Question Everything

I have never struggled to come up with a question.  About anything.  I’m a naturally curious person.  This is part of why I think evaluation is the right field for me.  In evaluation, you have to question everything.  If you’re not questioning everything, you’re doing a disservice to yourself and your clients.  And you’re probably missing something.

Key evaluation questions are the foundation of any good evaluation.  How do you know what you are supposed to be evaluating if you do not have some sort of guiding question(s)?  Evaluation questions help to frame the evaluation and provide a scope or boundaries.  There are potentially endless questions about a program or policy that someone could ask.  But what do stakeholders really need to know as a result of the evaluation? What is important and what is not?  Without agreed-upon questions, evaluators won’t know where to focus.

learn-from-yesterday-live-for-today-hope-for-tomorrow-the-important-thing-is-not-to-stop-questioning

Here are some tips for developing your evaluation questions.

Co-create your evaluation questions with your client (and other relevant stakeholders): Getting stakeholder buy-in in the evaluation starts at the beginning.  By co-creating the evaluation questions, clients and stakeholders will be able to ensure that their voices are heard and that the evaluation will ultimately meet their needs.

Choose the right evaluation questions for the program and the evaluation: Not all evaluation questions are created equally. You have to make sure that the evaluation questions suit the type of evaluation you are doing and are appropriate for the program that is being evaluated.  Here are some helpful guides:

Formative Evaluation To what extent is the program feasible?

How appropriate is the program or policy for the intended population?

Process Evaluation To what extent was the program implemented as planned? What modifications were made and why?

What components of the program are participants satisfied with and why?

What are the facilitators and barriers to implementation of the program?

Outcome Evaluation To what extent did the program affect change in [expected outcome]?

Which participants experienced the expected change?

What unintended outcomes were produced?

To what extent can these changes be attributed to the program?

Economic Evaluation In what ways is the program cost-effective?

What is the social return on investment for the program?

Another good way to look at developing evaluation questions is to ask if the program or policy is appropriate, effective, and/or efficient.

Also, keep in mind the length and budget of the evaluation.  It is probably not appropriate to ask a question about long-term impact when the evaluation only lasts a year.  With a small budget, make sure that you are asking questions that you will actually be able to answer with the available resources.

Make sure your questions can be answered: This seems obvious.  But I have often been faced with questions that cannot be answered given the nature of the program or the scope of the evaluation.  No one wants to read an evaluation report where to answer to the questions is “well, maybe…”  That won’t look good for anyone.  Make sure that your questions are realistic and that you can answer them with the available resources and data.

Avoid “yes or no” questions: Asking yes/no questions is not nearly informative enough to help organizations make decisions about programming and policies.  To avoid this, try starting questions with “To what extent did…” or “How…” or “In what ways…”

Limit the number of evaluation questions: This goes hand-in-hand with ensuring that your evaluation question fit the type, scope, length, and budget of the evaluation.  Don’t set yourself up for failure by identifying 20 evaluation questions.  No matter the scope of the evaluation, people are not going to want to sit through a report, presentation, or dissemination with 20 questions. That’s way too many and people will lose interest.  Ideally, there should be between 4-7 evaluation questions, depending on the context.

If you are having trouble limiting the number of evaluation questions, facilitate a conversation with your client about priorities.  Write down all the possible evaluation questions that you and your client think should be answered.  Once you have those all written out, answer these questions:

  1. Do all of these questions fit the type of the evaluation? Scope of the evaluation? Timeline of the evaluation?  If the answer to any of these questions is “no,” get rid of the question.
  2. Are any of these questions asking essentially the same thing? Or is there any overlap between questions? If the answer is yes, get rid of one of them.
  3. If any of these questions were not answered, would the expectations for the evaluation not have been met? If the answer is yes, make sure you keep those questions. But if the answer is no, get rid of it.

I have found that a lot of times, clients just throw all their questions out there because they think the more questions, the better the evaluation.  But if you really make them think about it, there are only a couple of questions for them that are key.

Which brings us to…

Prioritize your evaluation questions: Even after you have limited your evaluation questions to 4-7 questions, it is still important to prioritize those questions.  Which questions are of most importance?  While your evaluation will answer all of the questions, prioritizing will help to guide you in your evaluation efforts.

Ensure accountability: I will typically include the co-created evaluation questions in the scope of work or contract.  With this, it protects me from a client adding an unintended question at the end of project or saying that the evaluation didn’t answer their questions.  But it also ensures that I am accountable to these questions and the needs of the client. Because the questions are outlined in the scope of work or contract, I have to answer those questions or else I am not meeting the expectations of the evaluation.  

Of course, this doesn’t mean that evaluation questions can’t be changed, added, or removed. But it also ensures that any change to the evaluation questions is agreed upon by the evaluator and the client.

 

Your evaluation is only has good as your planning. So make sure that you are taking the time at the beginning of your evaluation you clearly lay out the key evaluation questions.  How else are you going to know what to do?

 

Resources

Better Evaluation: Specify the Key Evaluation Questions

CDC: Developing Evaluation Questions

Better Evaluation: Framing the Evaluation: The Importance of Asking the Right Questions

Western Michigan University: Evaluation Questions Checklist for Program Evaluators

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s