Essay Grade

Evaluation (Including Outcomes
Evaluation)
© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.
This document provides guidance toward planning and implementing an evaluation
process for for-profit or nonprofit programs — there are many kinds of evaluations that
can be applied to programs, for example, goals-based, process-based and outcomesbased. Nonprofit organizations are increasingly interested in outcomes-based
evaluation. If you are interested in learning more about outcomes-based evaluation,
then see the sections Outcomes-Evaluation and Outcomes-Based Evaluations in
Nonprofit Organizations.
Sections of This Topic Include
Program Evaluation: carefully getting information to make decisions about programs
Where Program Evaluation is Helpful
Basic Ingredients (you need an organization and program(s))
Planning Program Evaluation (what do you want to learn about, what info is needed)
Major Types of Program Evaluation (evaluating program processes, goals, outcomes,
etc.)
Overview of Methods to Collect Information (questionnaires, interviews, focus groups,
etc.)
Selecting Which Methods to Use (which methods work best to get needed info from
audiences)
Analyzing and Interpreting Information
Reporting Evaluation Results
Who Should Carry Out the Evaluation?
Contents of an Evaluation Plan
Pitfalls to Avoid
Online Guides, etc.
Outcomes-Evaluation
General Resources
Also consider
Evaluations (many kinds)
Related Library Topics
Learn More in the Library’s Blogs Related to Program
Evaluations
In addition to the articles on this current page, see the following blogs which have posts
related to Program Evaluations. Scan down the blog’s page to see various posts. Also
see the section “Recent Blog Posts” in the sidebar of the blog or click on “next” near the
bottom of a post in the blog.
Library’s Business Planning Blog
Library’s Building a Business Blog
Library’s Strategic Planning Blog
A Brief Introduction …
Note that the concept of program evaluation can include a wide variety of methods to
evaluate many aspects of programs in nonprofit or for-profit organizations. There are
numerous books and other materials that provide in-depth analysis of evaluations, their
designs, methods, combination of methods and techniques of analysis. However,
personnel do not have to be experts in these topics to carry out a useful program
evaluation. The “20-80” rule applies here, that 20% of effort generates 80% of the
needed results. It’s better to do what might turn out to be an average effort at
evaluation than to do no evaluation at all. (Besides, if you resort to bringing in an
evaluation consultant, you should be a smart consumer. Far too many program
evaluations generate information that is either impractical or irrelevant — if the
information is understood at all.) This document orients personnel to the nature of
program evaluation and how it can be carried out in a realistic and practical fashion.
Note that much of the information in this section was gleaned from various works of
Michael Quinn Patton.
Program Evaluation
Some Myths About Program Evaluation
1. Many people believe evaluation is a useless activity that generates lots of boring data
with useless conclusions. This was a problem with evaluations in the past when
program evaluation methods were chosen largely on the basis of achieving complete
scientific accuracy, reliability and validity. This approach often generated extensive data
from which very carefully chosen conclusions were drawn. Generalizations and
recommendations were avoided. As a result, evaluation reports tended to reiterate the
obvious and left program administrators disappointed and skeptical about the value of
evaluation in general. More recently (especially as a result of Michael Patton’s
development of utilization-focused evaluation), evaluation has focused on utility,
relevance and practicality at least as much as scientific validity.
2. Many people believe that evaluation is about proving the success or failure of a
program. This myth assumes that success is implementing the perfect program and
never having to hear from employees, customers or clients again — the program will
now run itself perfectly. This doesn’t happen in real life. Success is remaining open to
continuing feedback and adjusting the program accordingly. Evaluation gives you this
continuing feedback.
3. Many believe that evaluation is a highly unique and complex process that occurs at a
certain time in a certain way, and almost always includes the use of outside experts.
Many people believe they must completely understand terms such as validity and
reliability. They don’t have to. They do have to consider what information they need in
order to make current decisions about program issues or needs. And they have to be
willing to commit to understanding what is really going on. Note that many people
regularly undertake some nature of program evaluation — they just don’t do it in a
formal fashion so they don’t get the most out of their efforts or they make conclusions
that are inaccurate (some evaluators would disagree that this is program evaluation if
not done methodically). Consequently, they miss precious opportunities to make more
of difference for their customer and clients, or to get a bigger bang for their buck.
So What is Program Evaluation?
First, we’ll consider “what is a program?” Typically, organizations work from their
mission to identify several overall goals which must be reached to accomplish their
mission. In nonprofits, each of these goals often becomes a program. Nonprofit
programs are organized methods to provide certain related services to constituents,
e.g., clients, customers, patients, etc. Programs must be evaluated to decide if the
programs are indeed useful to constituents. In a for-profit, a program is often a onetime effort to produce a new product or line of products.
So, still, what is program evaluation? Program evaluation is carefully collecting
information about a program or some aspect of a program in order to make necessary
decisions about the program. Program evaluation can include any or a variety of at
least 35 different types of evaluation, such as for needs assessments, accreditation,
cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based,
process, outcomes, etc. The type of evaluation you undertake to improve your
programs depends on what you want to learn about the program. Don’t worry about
what type of evaluation you need or are doing — worry about what you need to know to
make the program decisions you need to make, and worry about how you can
accurately collect and understand that information.
Where Program Evaluation is Helpful
Frequent Reasons:
Program evaluation can:
1. Understand, verify or increase the impact of products or services on customers or
clients – These “outcomes” evaluations are increasingly required by nonprofit funders as
verification that the nonprofits are indeed helping their constituents. Too often, service
providers (for-profit or nonprofit) rely on their own instincts and passions to conclude
what their customers or clients really need and whether the products or services are
providing what is needed. Over time, these organizations find themselves in a lot of
guessing about what would be a good product or service, and trial and error about how
new products or services could be delivered.
2. Improve delivery mechanisms to be more efficient and less costly – Over time,
product or service delivery ends up to be an inefficient collection of activities that are
less efficient and more costly than need be. Evaluations can identify program strengths
and weaknesses to improve the program.
3. Verify that you’re doing what you think you’re doing – Typically, plans about how to
deliver services, end up changing substantially as those plans are put into place.
Evaluations can verify if the program is really running as originally planned.
Other Reasons:
Program evaluation can:
4. Facilitate management’s really thinking about what their program is all about,
including its goals, how it meets it goals and how it will know if it has met its goals or
not.
5. Produce data or verify results that can be used for public relations and promoting
services in the community.
6. Produce valid comparisons between programs to decide which should be retained,
e.g., in the face of pending budget cuts.
7. Fully examine and describe effective programs for duplication elsewhere.
Basic Ingredients: Organization and
Program(s)
You Need An Organization:
This may seem too obvious to discuss, but before an organization embarks on
evaluating a program, it should have well established means to conduct itself as an
organization, e.g., (in the case of a nonprofit) the board should be in good working
order, the organization should be staffed and organized to conduct activities to work
toward the mission of the organization, and there should be no current crisis that is
clearly more important to address than evaluating programs.
You Need Program(s):
To effectively conduct program evaluation, you should first have programs. That is, you
need a strong impression of what your customers or clients actually need. (You may
have used a needs assessment to determine these needs — itself a form of evaluation,
but usually the first step in a good marketing plan). Next, you need some effective
methods to meet each of those goals. These methods are usually in the form of
programs.
It often helps to think of your programs in terms of inputs, process, outputs and
outcomes. Inputs are the various resources needed to run the program, e.g., money,
facilities, customers, clients, program staff, etc. The process is how the program is
carried out, e.g., customers are served, clients are counseled, children are cared for,
art is created, association members are supported, etc. The outputs are the units of
service, e.g., number of customers serviced, number of clients counseled, children
cared for, artistic pieces produced, or members in the association. Outcomes are the
impacts on the customers or on clients receiving services, e.g., increased mental
health, safe and secure development, richer artistic appreciation and perspectives in
life, increased effectiveness among members, etc.
Planning Your Program Evaluation
Depends on What Information You Need to Make Your
Decisions and On Your Resources.
Often, management wants to know everything about their products, services or
programs. However, limited resources usually force managers to prioritize what they
need to know to make current decisions.
Your program evaluation plans depend on what information you need to collect in order
to make major decisions. Usually, management is faced with having to make major
decisions due to decreased funding, ongoing complaints, unmet needs among
customers and clients, the need to polish service delivery, etc. For example, do you
want to know more about what is actually going on in your programs, whether your
programs are meeting their goals, the impact of your programs on customers, etc? You
may want other information or a combination of these. Ultimately, it’s up to you.
But the more focused you are about what you want to examine by the evaluation, the
more efficient you can be in your evaluation, the shorter the time it will take you and
ultimately the less it will cost you (whether in your own time, the time of your
employees and/or the time of a consultant).
There are trade offs, too, in the breadth and depth of information you get. The more
breadth you want, usually the less depth you get (unless you have a great deal of
resources to carry out the evaluation). On the other hand, if you want to examine a
certain aspect of a program in great detail, you will likely not get as much information
about other aspects of the program.
For those starting out in program evaluation or who have very limited resources, they
can use various methods to get a good mix of breadth and depth of information. They
can both understand more about certain areas of their programs and not go bankrupt
doing so.
Key Considerations:
Consider the following key questions when designing a program evaluation.
1. For what purposes is the evaluation being done, i.e., what do you want to be able to
decide as a result of the evaluation?
2. Who are the audiences for the information from the evaluation, e.g., customers,
bankers, funders, board, management, staff, customers, clients, etc.
3. What kinds of information are needed to make the decision you need to make and/or
enlighten your intended audiences, e.g., information to really understand the process of
the product or program (its inputs, activities and outputs), the customers or clients who
experience the product or program, strengths and weaknesses of the product or
program, benefits to customers or clients (outcomes), how the product or program
failed and why, etc.
4. From what sources should the information be collected, e.g., employees, customers,
clients, groups of customers or clients and employees together, program
documentation, etc.
5. How can that information be collected in a reasonable fashion, e.g., questionnaires,
interviews, examining documentation, observing customers or employees, conducting
focus groups among customers or employees, etc.
6. When is the information needed (so, by when must it be collected)?
7. What resources are available to collect the information?
Some Major Types of Program Evaluation
When designing your evaluation approach, it may be helpful to review the following
three types of evaluations, which are rather common in organizations. Note that you
should not design your evaluation approach simply by choosing which of the following
three types you will use — you should design your evaluation approach by carefully
addressing the above key considerations.
Goals-Based Evaluation
Often programs are established to meet one or more specific goals. These goals are
often described in the original program plans.
Goal-based evaluations are evaluating the extent to which programs are meeting
predetermined goals or objectives. Questions to ask yourself when designing an
evaluation to see if you reached your goals, are:
1. How were the program goals (and objectives, is applicable) established? Was the
process effective?
2. What is the status of the program’s progress toward achieving the goals?
3. Will the goals be achieved according to the timelines specified in the program
implementation or operations plan? If not, then why?
4. Do personnel have adequate resources (money, equipment, facilities, training, etc.)
to achieve the goals?
5. How should priorities be changed to put more focus on achieving the goals?
(Depending on the context, this question might be viewed as a program management
decision, more than an evaluation question.)
6. How should timelines be changed (be careful about making these changes – know
why efforts are behind schedule before timelines are changed)?
7. How should goals be changed (be careful about making these changes – know why
efforts are not achieving the goals before changing the goals)? Should any goals be
added or removed? Why?
8. How should goals be established in the future?
Process-Based Evaluations
Process-based evaluations are geared to fully understanding how a program works -how does it produce that results that it does. These evaluations are useful if programs
are long-standing and have changed over the years, employees or customers report a
large number of complaints about the program, there appear to be large inefficiencies
in delivering program services and they are also useful for accurately portraying to
outside parties how a program truly operates (e.g., for replication elsewhere).
There are numerous questions that might be addressed in a process evaluation. These
questions can be selected by carefully considering what is important to know about the
program. Examples of questions to ask yourself when designing an evaluation to
understand and/or closely examine the processes in your programs, are:
1. On what basis do employees and/or the customers decide that products or services
are needed?
2. What is required of employees in order to deliver the product or services?
3. How are employees trained about how to deliver the product or services?
4. How do customers or clients come into the program?
5. What is required of customers or client?
6. How do employees select which products or services will be provided to the customer
or client?
7. What is the general process that customers or clients go through with the product or
program?
8. What do customers or clients consider to be strengths of the program?
9. What do staff consider to be strengths of the product or program?
10. What typical complaints are heard from employees and/or customers?
11. What do employees and/or customers recommend to improve the product or
program?
12. On what basis do employees and/or the customer decide that the product or
services are no longer needed?
Outcomes-Based Evaluation
Program evaluation with an outcomes focus is increasingly important for nonprofits and
asked for by funders.An outcomes-based evaluation facilitates your asking if your
organization is really doing the right program activities to bring about the outcomes you
believe (or better yet, you’ve verified) to be needed by your clients (rather than just
engaging in busy activities which seem reasonable to do at the time). Outcomes are
benefits to clients from participation in the program. Outcomes are usually in terms of
enhanced learning (knowledge, perceptions/attitudes or skills) or conditions, e.g.,
increased literacy, self-reliance, etc. Outcomes are often confused with program
outputs or units of services, e.g., the number of clients who went through a program.
The United Way of America(http://www.unitedway.org/outcomes/) provides an
excellent overview of outcomes-based evaluation, including introduction to outcomes
measurement, a program outcome model, why to measure outcomes, use of program
outcome findings by agencies, eight steps to success for measuring outcomes,
examples of outcomes and outcome indicators for various programs and the resources
needed for measuring outcomes. The following information is a top-level summary of
information from this site.
To accomplish an outcomes-based evaluation, you should first pilot, or test, this
evaluation approach on one or two programs at most (before doing all programs).
The general steps to accomplish an outcomes-based evaluation include to:
1. Identify the major outcomes that you want to examine or verify for the program
under evaluation. You might reflect on your mission (the overall purpose of your
organization) and ask yourself what impacts you will have on your clients as you work
towards your mission. For example, if your overall mission is to provide shelter and
resources to abused women, then ask yourself what benefits this will have on those
women if you effectively provide them shelter and other services or resources. As a last
resort, you might ask yourself, “What major activities are we doing now?” and then for
each activity, ask “Why are we doing that?” The answer to this “Why?” question is
usually an outcome. This “last resort” approach, though, may just end up justifying
ineffective activities you are doing now, rather than examining what you should be
doing in the first place.
2. Choose the outcomes that you want to examine, prioritize the outcomes and, if your
time and resources are limited, pick the top two to four most important outcomes to
examine for now.
3. For each outcome, specify what observable measures, or indicators, will suggest that
you’re achieving that key outcome with your clients. This is often the most important
and enlightening step in outcomes-based evaluation. However, it is often the most
challenging and even confusing step, too, because you’re suddenly going from a rather
intangible concept, e.g., increased self-reliance, to specific activities, e.g., supporting
clients to get themselves to and from work, staying off drugs and alcohol, etc. It helps
to have a “devil’s advocate” during this phase of identifying indicators, i.e., someone
who can question why you can assume that an outcome was reached because certain
associated indicators were present.
4. Specify a “target” goal of clients, i.e., what number or percent of clients you commit
to achieving specific outcomes with, e.g., “increased self-reliance (an outcome) for 70%
of adult, African American women living in the inner city of Minneapolis as evidenced by
the following measures (indicators) …”
5. Identify what information is needed to show these indicators, e.g., you’ll need to
know how many clients in the target group went through the program, how many of
them reliably undertook their own transportation to work and stayed off drugs, etc. If
your program is new, you may need to evaluate the process in the program to verify
that the program is indeed carried out according to your original plans. (Michael Patton,
prominent researcher, writer and consultant in evaluation, suggests that the most
important type of evaluation to carry out may be this implementation evaluation to
verify that your program ended up to be implemented as you originally planned.)
6. Decide how can that information be efficiently and realistically gathered
(see Selecting Which Methods to Use below). Consider program documentation,
observation of program personnel and clients in the program, questionnaires and
interviews about clients perceived benefits from the program, case studies of program
failures and successes, etc. You may not need all of the above. (see Overview
of Methods to Collect Informationbelow).
7. Analyze and report the findings (see Analyzing and Interpreting Information below).
Overview of Methods to Collect Information
The following table provides an overview of the major methods used for collecting data
during evaluations.
Method
Overall Purpose
Advantages
-can complete
anonymously
-inexpensive to
administer
when need to quickly
-easy to compare and
questionnaires, and/or easily get lots
analyze
surveys,
of information from
-administer to many
checklists
people in a non
people
threatening way
-can get lots of data
-many sample
questionnaires already
exist
when want to fully
-get full range and
understand someone’s
depth of information
impressions or
-develops relationship
interviews
experiences, or learn
with client
more about their
-can be flexible with
answers to
client
questionnaires
-get comprehensive and
when want impression
historical information
of how program
-doesn’t interrupt
operates without
program or client’s
documentation interrupting the
routine in program
review
program; is from
-information already
review of applications,
exists
finances, memos,
-few biases about
minutes, etc.
information
to gather accurate
-view operations of a
observation
information about how program as they are
Challenges
-might not get careful
feedback
-wording can bias
client’s responses
-are impersonal
-in surveys, may need
sampling expert
– doesn’t get full story
-can take much time
-can be hard to
analyze and compare
-can be costly
-interviewer can bias
client’s responses
-often takes much time
-info may be
incomplete
-need to be quite clear
about what looking for
-not flexible means to
get data; data
restricted to what
already exists
-can be difficult to
interpret seen
focus groups
case studies
a program actually
actually occurring
behaviors
operates, particularly -can adapt to events as -can be complex to
about processes
they occur
categorize
observations
-can influence
behaviors of program
participants
-can be expensive
explore a topic in
depth through group -quickly and reliably get
discussion, e.g., about common impressions
-can be hard to
reactions to an
-can be efficient way to
analyze responses
experience or
get much range and
-need good facilitator
suggestion,
depth of information in
for safety and closure
understanding
short time
-difficult to schedule 6common complaints, – can convey key
8 people together
etc.; useful in
information about
evaluation and
programs
marketing
to fully understand or
-fully depicts client’s
depict client’s
-usually quite time
experience in program
experiences in a
consuming to collect,
input, process and
program, and conduct
organize and describe
results
comprehensive
-represents depth of
-powerful means to
examination through
information, rather
portray program to
cross comparison of
than breadth
outsiders
cases
Also consider
Appreciative Inquiry
Survey Design
Ethics: Informed Consent from Program Participants
Note that if you plan to include in your evaluation, the focus and reporting on personal
information about customers or clients participating in the evaluation, then you should
first gain their consent to do so. They should understand what you’re doing with them
in the evaluation and how any information associated with them will be reported. You
should clearly convey terms of confidentiality regarding access to evaluation results.
They should have the right to participate or not. Have participants review and sign an
informed consent form. See the sample informed-consent form.
How to Apply Certain Methods
Purposes and Formats of Questions Developing Questionnaires
Conducting Interviews
Conducting Focus Groups
Developing Case Studies
Selecting Which Methods to Use
Overall Goal in Selecting Methods:
The overall goal in selecting evaluation method(s) is to get the most useful information
to key decision makers in the most cost-effective and realistic fashion. Consider the
following questions:
1. What information is needed to make current decisions about a product or program?
2. Of this information, how much can be collected and analyzed in a low-cost and
practical manner, e.g., using questionnaires, surveys and checklists?
3. How accurate will the information be (reference the above table for disadvantages of
methods)?
4. Will the methods get all of the needed information?
5. What additional methods should and could be used if additional information is
needed?
6. Will the information appear as credible to decision makers, e.g., to funders or top
management?
7. Will the nature of the audience conform to the methods, e.g., will they fill out
questionnaires carefully, engage in interviews or focus groups, let you examine their
documentations, etc.?
8. Who can administer the methods now or is training required?
9. How can the information be analyzed?
Note that, ideally, the evaluator uses a combination of methods, for example, a
questionnaire to quickly collect a great deal of information from a lot of people, and
then interviews to get more in-depth information from certain respondents to the
questionnaires. Perhaps case studies could then be used for more in-depth analysis of
unique and notable cases, e.g., those who benefited or not from the program, those
who quit the program, etc.
Four Levels of Evaluation:
There are four levels of evaluation information that can be gathered from clients,
including getting their:
1. reactions and feelings (feelings are often poor indicators that your service made
lasting impact)
2. learning (enhanced attitudes, perceptions or knowledge)
3. changes in skills (applied the learning to enhance behaviors)
4. effectiveness (improved performance because of enhanced behaviors)
Usually, the farther your evaluation information gets down the list, the more useful is
your evaluation. Unfortunately, it is quite difficult to reliably get information about
effectiveness. Still, information about learning and skills is quite useful.
Analyzing and Interpreting Information
Analyzing quantitative and qualitative data is often the topic of advanced research and
evaluation methods. There are certain basics which can help to make sense of reams of
data.
Always start with your evaluation goals:
When analyzing data (whether from questionnaires, interviews, focus groups, or
whatever), always start from review of your evaluation goals, i.e., the reason you
undertook the evaluation in the first place. This will help you organize your data and
focus your analysis. For example, if you wanted to improve your program by identifying
its strengths and weaknesses, you can organize data into program strengths,
weaknesses and suggestions to improve the program. If you wanted to fully understand
how your program works, you could organize data in the chronological order in which
clients go through your program. If you are conducting an outcomes-based evaluation,
you can categorize data according to the indicators for each outcome.
Basic analysis of “quantitative” information(for information other than
commentary, e.g., ratings, rankings, yes’s, no’s, etc.):
1. Make copies of your data and store the master copy away. Use the copy for making
edits, cutting and pasting, etc.
2. Tabulate the information, i.e., add up the number of ratings, rankings, yes’s, no’s for
each question.
3. For ratings and rankings, consider computing a mean, or average, for each question.
For example, “For question #1, the average ranking was 2.4”. This is more meaningful
than indicating, e.g., how many respondents ranked 1, 2, or 3.
4. Consider conveying the range of answers, e.g., 20 people ranked “1”, 30 ranked “2”,
and 20 people ranked “3”.
Basic analysis of “qualitative” information(respondents’ verbal answers in
interviews, focus groups, or written commentary on questionnaires):
1. Read through all the data.
2. Organize comments into similar categories, e.g., concerns, suggestions, strengths,
weaknesses, similar experiences, program inputs, recommendations, outputs, outcome
indicators, etc.
3. Label the categories or themes, e.g., concerns, suggestions, etc.
4. Attempt to identify patterns, or associations and causal relationships in the themes,
e.g., all people who attended programs in the evening had similar concerns, most
people came from the same geographic area, most people were in the same salary
range, what processes or events respondents experience during the program, etc.
4. Keep all commentary for several years after completion in case needed for future
reference.
Interpreting Information:
1. Attempt to put the information in perspective, e.g., compare results to what you
expected, promised results; management or program staff; any common standards for
your services; original program goals (especially if you’re conducting a program
evaluation); indications of accomplishing outcomes (especially if you’re conducting an
outcomes evaluation); description of the program’s experiences, strengths,
weaknesses, etc. (especially if you’re conducting a process evaluation).
2. Consider recommendations to help program staff improve the program, conclusions
about program operations or meeting goals, etc.
3. Record conclusions and recommendations in a report document, and associate
interpretations to justify your conclusions or recommendations.
Reporting Evaluation Results
1.The level and scope of content depends on to whom the report is intended, e.g., to
bankers, funders, employees, customers, clients, the public, etc.
2. Be sure employees have a chance to carefully review and discuss the report.
Translate recommendations to action plans, including who is going to do what about the
program and by when.
3. Bankers or funders will likely require a report that includes an executive summary
(this is a summary of conclusions and recommendations, not a listing of what sections
of information are in the report — that’s a table of contents); description of the
organization and the program under evaluation; explanation of the evaluation goals,
methods, and analysis procedures; listing of conclusions and recommendations; and
any relevant attachments, e.g., inclusion of evaluation questionnaires, interview guides,
etc. The banker or funder may want the report to be delivered as a presentation,
accompanied by an overview of the report. Or, the banker or funder may want to
review the report alone.
4. Be sure to record the evaluation plans and activities in an evaluation plan which can
be referenced when a similar program evaluation is needed in the future.
Contents of an Evaluation Report — Example
An example of evaluation report contents is included later on below in this document.
Click Contents of an Evaluation Plan but, don’t forget to look at the next section “Who
Should Carry Out the Evaluation”.
Who Should Carry Out the Evaluation?
Ideally, management decides what the evaluation goals should be. Then an evaluation
expert helps the organization to determine what the evaluation methods should be, and
how the re