By Akie Yanagi, PhD, Evaluator
When you walk into a restaurant and see a couple frowning at each other, with the eyebrows brought together and corners of the mouth fallen downward on their faces, what do you suspect is going on between these two people? Are they talking about something pleasurable and fun, or something negative? Most people in our culture are likely to suspect the latter.
You made an observation of this couple unknowingly. People make unsystematic observations every day, but when this is done systematically, observations can provide a meaningful set of data or information. While part of what makes us human is the use of spoken languages, many communication experts argue that a big part of our lives consists of nonverbal communication – facial expressions, body language, and gestures. Observing nonverbal communication can provide us with a wealth of information.
An evaluation can be performed using different data collection tools such as surveys, individual in-depth interviews, focus group interviews, and secondary content analysis, but the observational method holds a very special place in the evaluation process. An observation allows the evaluator to view the situation in which a program or event is implemented, and they do not have to rely on the participant’s ability to provide information (e.g., surveys relying on other people’s response to provided questions).
Most importantly, an observation allows the evaluator to watch what people actually do, rather than relying on what they say they intend to do. People are complex and they often do not do what they say they would, so in such a case, incorporating observation into one’s evaluation can be extremely valuable as it is likely to capture the information that other tools cannot.
Simple Steps to Effective Observation Evaluations
An effective observation begins with identifying the focus of your observation, or considering what evaluation questions you are trying to answer through an observation.
Once the questions are identified, you must develop a system to collect your observational data. The evaluator must also ensure that the indicators of behaviors or events are clearly operationalized, measurable, and observable to the evaluator’s eye. It is important to prepare instructions to explain how the observational information should be collected.
An effective observation should also be systematic.
One way to ensure systematic data collection is to prepare a checklist, containing a set of questions that can help the evaluator measure events or behaviors of their interest. A checklist allows the evaluator to view the indicators they are looking for during the observation without having to rely on their memory, enhancing accuracy of their data collection. It also provides them with consistent data that can later be quantified. Preparing a checklist prior to the observation leads to a more focused, consistent, and reliable data collection. A checklist is also useful if there are multiple observers because they can check the inter-rater reliability between the observers during practice.
Observations can also be made more systematic if they are conducted more than once within the same evaluation. For instance, you may choose to observe the same event with different participants, at different sites, or using multiple observers. This allows for a comparison of the evaluator’s observational data, enhancing the reliability of their findings. In another case, the evaluator may conduct a casual, preliminary observation prior to a primary, formal observation to inform themselves on how the data should be collected, providing more specific directions for the latter.
Additional Considerations
The observational method comes with a few drawbacks:
- Observations can be time-consuming or costly as the evaluator must be on-site to conduct an observation and actually participate in a relevant event (and sometimes more than once).
- Conducting observations can also be physically demanding; sticking to your checklist and concentrating on the indicators for a long period of time can be tiring.
- There is always a risk of the observer’s personal bias due to their cultural backgrounds or due to human nature that people naturally pay attention to more conspicuous events than ordinary ones.
- If the evaluator’s identity and role are known, the observed people may not act naturally. Or in an opposite situation, if the evaluator decides to conduct an observation covertly, it may be difficult to explain what they are doing at the site when asked.
- The observational method does not necessarily provide the information about “why” certain things occurred during an observation.
Despite these drawbacks, combined with other data collection tools such as surveys and interviews, the observational method can compensate for what is missing from other tools, make the evaluation more comprehensive or whole, and thus can lead to more meaningful findings.
CCNY has extensive experience in all types of data collection, including the observational method. Contact us today and let’s talk about how we can help with your unique data collection challenges: info@ccnyinc.org.