Developing the Methods and Instruments
A community ‘report card’ is often the key element of a community monitoring process. The ‘report card’ allows for the community experience of service delivery (or its absence) to be summarized and placed before a varied audience of stakeholders to decide on ways to address the shortcomings and gaps. It is key instrument which converts experience to evidence, and hence needs to be carefully thought of and prepared. As in all report cards, including school report cards, the report card has some categories which will be assessed, which in this case are also called domains/issues for enquiry. In a school report card these can be Language, Math, Geography, History, and Science and in a report card around maternal health services these could be Coverage of Prenatal Services, Range of Services received, Cost of receiving services, Ease of Access, Quality of services and so on. For immunization services for young children these could be Coverage and dropout, Range of vaccines including revaccination, Outreach education, Parents knowledge about vaccines, Quality of care and so on.
Once the domains or key issues for the report card have been finalized one needs to further break up the issue into sub-issues and questions and then identify the appropriate sources of information. The graphic below provides an illustration of how key issues for enquiry may be further developed into sub-issues and questions using one example from the report card as illustration.
Once the domains or key issues for the report card have been finalized one needs to further break up the issue into sub-issues and questions and then identify the appropriate sources of information. The graphic below provides an illustration of how key issues for enquiry may be further developed into sub-issues and questions using one example from the report card as illustration.
In a similar manner questions can be developed from all the other issues or key domains in a report card.
Once the questions have been clearly articulated the source of information for each question needs to be identified, e.g. for the question “Is there educational material available at the centre” Information may be available from both the health centre and from the parents of small children (the intended users). In order to be certain/accurate about the information from the centre/facility the organizations may also choose to make an application under the Freedom of Information Act investigating whether educational material was available during the last year, how many were distributed and how, whether educational workshops were conducted and when? On the other hand parents and family members of small children (those whose children are the intended users of immunization services can be asked whether they were provided educational material around immunization, which vaccines, did they attend or were invited to attend educational workshops, did the patronage nurse visit their homes, how many times and so on. The information from the community may be taken through a group discussion or through individual interviews with parents of small children. A group discussion allows the enquiry process to get a sense of what the general situation in the community, but the information may not be very specific. Individual interviews allows for obtaining more specific information related to behaviours. The information about the outreach education and visits by patronage nurse can also obtained by interviewing the patronage nurse. The educational material may also be physically reviewed and someone from the enquiry team can also observe an educational workshop if one is scheduled during the period of enquiry. It thus emerges that information about the Outreach Education may be obtained from various sources viz.
Once the questions have been clearly articulated the source of information for each question needs to be identified, e.g. for the question “Is there educational material available at the centre” Information may be available from both the health centre and from the parents of small children (the intended users). In order to be certain/accurate about the information from the centre/facility the organizations may also choose to make an application under the Freedom of Information Act investigating whether educational material was available during the last year, how many were distributed and how, whether educational workshops were conducted and when? On the other hand parents and family members of small children (those whose children are the intended users of immunization services can be asked whether they were provided educational material around immunization, which vaccines, did they attend or were invited to attend educational workshops, did the patronage nurse visit their homes, how many times and so on. The information from the community may be taken through a group discussion or through individual interviews with parents of small children. A group discussion allows the enquiry process to get a sense of what the general situation in the community, but the information may not be very specific. Individual interviews allows for obtaining more specific information related to behaviours. The information about the outreach education and visits by patronage nurse can also obtained by interviewing the patronage nurse. The educational material may also be physically reviewed and someone from the enquiry team can also observe an educational workshop if one is scheduled during the period of enquiry. It thus emerges that information about the Outreach Education may be obtained from various sources viz.
- From the administrative records through FOI application
- From Community of parents of small children through a group discussion
- From parents of small children through individual interview
- From reviewing the material
- From observing a workshop
However for practical purposes it may not be possible to include all methods and respondents for gathering the information. A reasonable judgment has to be made keeping in mind issues of sampling, inclusion and triangulation (see later) as well time and resources available.
Using the same principle explained above a list of sub-issues and questions relating to each domain or key issue of the report card has to be drawn up and the possible sources of information listed for each question. This is illustrated in the table below:
Using the same principle explained above a list of sub-issues and questions relating to each domain or key issue of the report card has to be drawn up and the possible sources of information listed for each question. This is illustrated in the table below:
Issue/ Sub IssuesOutreach Education
Sub Issues - Outreach educational material Educational Workshops Visit by Patronage/Outreach nurse Language/simplicity of communication Parents Knowledge of Vaccines
Sub-Issues Names of the different vaccines/diseases prevented Vaccination calendar/ schedule Side effects and their management Quality of Services
Sub-Issues Physical Examination prior to vaccination Waiting time Privacy Interpersonal Commnication Satisfaction |
Questions (Sources of information)
Is there educational material available in the Centre? What kinds – posters/ pamphlets? (FOI, PGD, P Int)
Has it been distributed? How? (FOI,PGD) Is the language and visual of the material easy to understand? (PGD, MR) Were educational workshops organized? When? Where? Who attended? Who facilitated? (PGD, Pint, WO, PNInt) How many patronage nurses are attached to the centre? ( FoI, PGD) Do the conduct regular field visits? (PGD, P Int, PNInt) Are they able to cover the population according to norms? (PGD, PNInt) Are they able to communicate effectively with the community? (PGD, P Int, PNInt) What diseases are prevented by vaccines and which are not? Clarity (PGD, PInt, PNInt)
Which vaccines are to be given and what age of children? Timeliness (PGD, PInt, PNInt) Conditions when children should not be vaccinated? Contraindications (PInt, PNInt) Which are the vaccines that need to be given more than one and why? Regularity (PGD, PInt, PNInt) Any myths and misconceptions? (PGD, Pint, PNInt) What are common side-effects and how can they be minimized or managed. (PInt, PNInt) Q1 (PInt, DocInt, FO)
Q2 (PInt, DocInt, PGD) Q3(PInt, DocInt, PGD, FO) Q4 (PInt, PGD, FO) Q5 (PInt, DocInt, PGD) |
SourcesFOI Application (FOI)
Parents Interview (P Int) Parents Group Discussion (PGD) Patronage Nurse Interview (PNInt) Workshop Observation (WO) Material Review (MR) Interview with Doctor (DocInt)
Facility Observation ( FO) |
Once this list or table of questions with sources of information is drawn up, the tools of enquiry automatically emerge by putting together the questions for one particular source together. For example, all of the questions that plan to use a parents’ individual interview (P Int) as a source of information can be collected to form the questionnaire for the parents individual interview. Therefore, questions for the Parents Individual Interview guide will include;
Similarly instruments can be drawn up for Interview with patronage nurse, Focus Group Discussion with Community/Parents and even for issues that should be included in a FoI application.
Sampling – Addressing Issues of Inclusion, Representation and Validity
The evidence generated from the community monitoring process is geared towards evidence-based advocacy and one of the first tests that such evidence has to pass is the test of adequately representing the collective community reality, and not being anecdotal or a one-off experience. In formal research this issue is dealt in adopting an appropriate sampling methodology which allows some degree of representativeness in the findings. In community monitoring the issue of validity and representativeness is approached keeping the following issues in mind:
a. Differences in community experiences: For example, if there is reason to believe that within one community group there are two different experiences – i.e. the experience of those who are seasonal migrants may have a different experience of service delivery from those who are settled residents then the enquiry process will try to include both experiences by including at least one group discussion with each group and include both kinds of service users in the individual patient interview process.
4. When gathering information from more than one point of view, there may be differences of opinion. In such cases it is useful to keep in mind one question while deciding how much importance to give to a particular piece of information and this is– does this information come from ‘direct’ community experience. The source which relates to direct community experience should be give greater value. The process is explained later. There are also certain issues which are simply ‘unacceptable’ from a human rights perspective – for example a death of a child following immunisation, or refusal to provide ambulance to a woman in labor. These incidents are important in themselves and such experiences do not require any statistical justification and should be recorded in detail as case studies and reported.
In community monitoring adequate variety of sources of information and sufficient information about the experience of the socially excluded/marginalised provides the key justification about sample variety and size. The table below tries to provide a example how this may be done in actual practice
- Have you seen or been given educational material? What kinds – posters/ pamphlets?
- Have you participated in educational workshops? When? Where? Who attended? Who facilitated?
- Has the Patronage Nurses visited you?
- Communication of Patron nurses
- Do you know which diseases are prevented by vaccines and which are not?
- Do you know which vaccines are to be given and what age of children?
- Conditions, if any when children should not be vaccinated?
- Which are the vaccines that need to be given more than once and why?
- Sources of information? ( role of patronage nurse, materials and workshop)
- Any myths and misconceptions around vaccines?
- What are common side-effects and how can they be minimized or managed?
- Quality of Services Q1
- Quality of Services Q2
- Quality of Services Q3
- Quality of Services Q4
- Quality of Services Q5
Similarly instruments can be drawn up for Interview with patronage nurse, Focus Group Discussion with Community/Parents and even for issues that should be included in a FoI application.
Sampling – Addressing Issues of Inclusion, Representation and Validity
The evidence generated from the community monitoring process is geared towards evidence-based advocacy and one of the first tests that such evidence has to pass is the test of adequately representing the collective community reality, and not being anecdotal or a one-off experience. In formal research this issue is dealt in adopting an appropriate sampling methodology which allows some degree of representativeness in the findings. In community monitoring the issue of validity and representativeness is approached keeping the following issues in mind:
- Community monitoring does not aim at universal representativeness and it is a methodology for representing the experience of the marginalized or socially excluded communities. It is useful to mention and keep in mind this ‘specific’ interest from the beginning, and be careful to identify and include the experience of the socially excluded within the overall exercise.
- Community monitoring is a multi-method exercise and thus it aims to represent the community reality by including different points of view. In such an exercise the validity of the findings is not necessarily maintained through a representative sample but through a process of triangulation which allows different actors/stakeholders to contribute to developing the picture. Triangulation means asking the same question from multiple sources, to verify the answer and determine any differences in perspectives. For example, an idea of quality of care of services can be obtained from the service provider, the service recipient as well as observing the service delivery environment (without compromising confidentiality and privacy associated with service delivery).
- Another issue that needs to be kept in mind is that Community Monitoring tries to collect a variety of experiences and even though it presents the final report card as a single numeric or traffic light rating, this indicator is backed by a detailed narrative explanation which provides stories or community experiences. The narrative portion justifies the rating and explains any variations in experience and perspective that were obtained during the enquiry process. Examples of more complicated justifications include:
a. Differences in community experiences: For example, if there is reason to believe that within one community group there are two different experiences – i.e. the experience of those who are seasonal migrants may have a different experience of service delivery from those who are settled residents then the enquiry process will try to include both experiences by including at least one group discussion with each group and include both kinds of service users in the individual patient interview process.
4. When gathering information from more than one point of view, there may be differences of opinion. In such cases it is useful to keep in mind one question while deciding how much importance to give to a particular piece of information and this is– does this information come from ‘direct’ community experience. The source which relates to direct community experience should be give greater value. The process is explained later. There are also certain issues which are simply ‘unacceptable’ from a human rights perspective – for example a death of a child following immunisation, or refusal to provide ambulance to a woman in labor. These incidents are important in themselves and such experiences do not require any statistical justification and should be recorded in detail as case studies and reported.
In community monitoring adequate variety of sources of information and sufficient information about the experience of the socially excluded/marginalised provides the key justification about sample variety and size. The table below tries to provide a example how this may be done in actual practice
This table shows how each issue is being viewed from multiple stakeholder perspectives. In addition the number of enquiries (interviews, group discussion etc.) from each respondent source can also be mentioned alongside and then the table would look like the one below.
The number will depend upon our understanding of the community and the subject of the community monitoring. This process includes basic principles of sampling and requires some numerical understanding. Thus if we are looking at children who are under the age of 5 years we need to have some understanding of both the total population and the population of young children. This can be done through the mapping process which has been explained earlier.
