by Sandy Sheppard and Lisa Kulka
Authentic and rigorous evaluation of programs and services is an ethical and oftentimes legal requirement for the work of non-profit organizations. This requirement has spawned a proliferation of external agencies offering evaluation services. Below are some guidelines that you can use to choose the right evaluator for your project.
First, you must also ask yourself¹ a few questions:
- What do you want to get out of an evaluation?
- What kind of evaluator do you need to get the information you want?
- Do you want a process evaluation that will document fidelity of implementation?
- Do you want a formative evaluation that will obtain information that can be used to develop or refine current projects?
- Do you want an outcome evaluation that will document the impact of your work?
Having a concrete understanding of what you want the evaluation to be able to demonstrate, relay, and/or provide will be crucial to identifying the “right” kind of evaluator for your project, as that person ultimately must have the skill set necessary to meet the objectives of your evaluation.
Beyond the mechanics² of forming an evaluation committee, defining the evaluation, developing a task description, and soliciting candidates, interviewing and selecting the evaluator, and writing and negotiating the contract, thoughtful questions³, and follow-up questions⁴,⁵ can be asked to determine goodness of fit between an evaluator and what you want to learn about your project:
- Does the candidate understand your project?
- Is the candidate aware of the range of possible sensitivities, practical considerations and contextual influences that are likely to arise?
- What would the candidate’s general approach be to your project?
- Does the candidate have the ability to choose the right approach?
- Does the candidate have the ability to select the right evaluation tools?
- Does the candidate believe the evaluation can be conducted for the available funds?
- What is the candidate’s prior evaluation experience?
- Has the candidate demonstrated the ability to make sense of complex data and detailed information, then disseminate it in a way that is relevant?
- How useful are the candidate’s previous evaluation reports?
- Does the candidate know the difference between research and evaluation?
- Does the candidate endorse the view evaluation and program development should work hand in hand?
- Does the candidate have good references?
- Will the candidate’s existing professional commitments interfere with the planned evaluation?
By considering these questions relative to the scope of your evaluation project, you will be well-positioned to identify candidates whose strengths and skill sets align most closely with your project objectives. While different candidates with varying backgrounds and experiences may be capable of producing a high-quality evaluation end product, you must be cognizant of what your evaluation must ultimately achieve, and whether or not your evaluator understands and shares that same vision. One’s approach to evaluation is just as important, if not sometimes more important, than their technical abilities.
“A good evaluation will benefit an organization in ways that will not be expected or predicted, and you may find yourself asking how you ever operated programs in the past without an evaluation. The key to a successful evaluation is having the right plan and having the right person(s) to implement an evaluation.“ -James Bell Associates⁶
References Cited
- https://www.jbassoc.com/wp-content/uploads/2018/03/How-Select-Evaluator.pdf
- Five of eight guidelines from: https://www.nationalservice.gov/sites/default/files/resource/Hiring_the_Right_Evaluator.pdf
- Ibid.
- http://nsfconsulting.com.au/how-to-choose-a-program-evaluation-consultant
- https://safesupportivelearning.ed.gov/sites/default/files/hec/product/prog-eval.pdf
- https://www.jbassoc.com/wp-content/uploads/2018/03/How-Select-Evaluator.pdf
For more information about CCNY’s Program Evaluation Toolkit, click here.