Evaluation

Top Tips 

Effective evaluations help organizations learn (Organizational Learning Tip Sheet) what works and why so they can improve the quality of their service delivery and policy change initiatives and achieve the desired impact.  Effective evaluations are based on solid planning, including a theory of change identifying the underlying assumptions about how and why the anticipated outcomes will occur and a logic model presenting the relationship between resources, activities and results.  Evaluation also serves an accountability function, demonstrating results to funders and stakeholders. Organizational leaders, program staff, funders, government agencies, other nonprofits and residents involved with your project will be interested in learning from your work and about your results.

Planning and conducting an evaluation helps:

1. stakeholders determine the value and quality of program activities and outcomes;

2. organizations learn how to improve the process of program delivery so that resources are directed in the most effective and cost-efficient way; and

3. plan for sustainability by determining which efforts should continue.  

• Will the evaluation be conducted by an internal (using program staff or volunteers) or external (using consultants from outside the organization) group?
• What are the budget and the timeline for completion?
• Who will the evaluation be presented to and what questions might they want the evaluation to answer?
• What will be included in the final report and who will receive it?

Spend some time learning about different evaluation methods and styles. Basic methods of data collection include:

• Quantitative – taking measurements of specific events or behaviors and applying statistics to analyze the data.
• Qualitative – capturing descriptive data (e.g. interviews, case studies) for review and analysis.
• Mixed Method evaluation combine quantitative and qualitative techniques and are generally considered to produce a more comprehensive understanding about a project’s accomplishments.

Two styles of evaluation include:

• Formative – evaluating the work plan’s processes (e.g. forming an advisory committee, delivering a training session, etc.) during a specific period of time.

• Summative – evaluating the outcome of work plan activities (e.g. number of residents who increased level of physical activity, number or types of public policies that were adopted by governing body, etc.) during a specific period of time.

The steps in conducting an evaluation generally include:

• Developing evaluation questions;
• Matching questions with appropriate information gathering techniques;
• Collecting data;
• Analyzing the data; and
• Providing information to interested audiences.

In selecting consultants (Find an Evaluator) or identifying an internal group, it’s important for them to: 

• Provide samples of evaluation reports they have produced for programs similar to yours.
• Demonstrate cultural competence and knowledge of your community environment (ethnic, political, geographic).
• Produce a work plan detailing how the evaluation will be conducted, what methods will be used, who will be involved, what the final report will consist of and timelines that will be followed. Ideally, there will be multiple methods proposed to collect data so that any significant finding can be confirmed by more than one source.
• Describe what protective actions will be put in place when human subjects are included in the evaluation.  

Online Tools, How-To Guides and Practical Resources

Innovation Network – Point K Learning Center
www.innonet.org/pointk
The Point K Learning Center features free, practical tools and resources for non-profit planning, evaluation and action.  The interrelated Evaluation Plan Builder™ and Logic Model Builder™ are designed to walk you through developing an evaluation plan or logic model for your initiative and to provide helpful information, examples and resources along the way.  An Evaluation Survey Builder, due out in 2007, will walk you through designing an evaluation survey and allow you to find field-tested surveys and questions tailored to your service area and data collection needs.  The site also includes an extensive listing of resources, including workbooks, tip sheets, articles and a collection of recommended planning and evaluation links and publications.

W.K. Kellogg Foundation Evaluation Toolkit and Handbook
www.wkkf.org/Default.aspx?tabid=90&CID=281&ItemID=2810002&NID=2820002&LanguageID=0

www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

Both the Toolkit and Handbook are targeted at Kellogg Foundation grantees, but are designed to be useful for anyone seeking to develop an evaluation.   The condensed, online Toolkit covers getting started, different evaluation approaches, developing evaluation questions, elements of the evaluation plan, budgeting, working with outside evaluators and additional resources.  The 116-page Handbook is the expanded version of the Toolkit and focuses on how to design and conduct a project evaluation.  It provides a framework for thinking about evaluation as a relevant and useful program tool.

The Evaluation Center – Checklists

www.wmich.edu/evalctr

This is a comprehensive evaluation site for Western Michigan University ’s Evaluation Center .  It includes refereed checklists for designing and managing an evaluation.   All checklists are easily downloadable in Adobe PDF format.  www.wmich.edu/evalcty/checlists

User-Friendly Handbook for Mixed Method Evaluations
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153
The Handbook includes information on how to conduct a mixed method evaluation that combines quantitative and qualitative techniques.  It provides detailed information on specific data collection methods and analysis for each type of measure. It was produced by the Division of Research, Evaluation and Communication of the National Science Foundation.

Learn More About Evaluation

American Evaluation Association 
The American Evaluation Association is a professional association of US and international evaluators.  They sponsor an annual conference and their website includes information on evaluation training and upcoming events focused on evaluation.  See the AEA website at www.eval.org

Building Evaluation Capacity

Preskill, H. and Russ-EFT, D., 2004 (Sage Publications, Inc.)

This book provides an overview of the practice of evaluation and provides activities for learning how to design and conduct evaluation studies.  To order visit Sage Publications www.sagepub.com or any of the online bookstores (amazon.com, barnesandnoble.com, etc.)

The Evaluation Exchange

www.gse.harvard.edu/hfrp/eval.html
The Evaluation Exchange is a free on-line evaluation periodical published by the Harvard Family Research Project.  It addresses current issues facing program evaluators of all levels, with articles written by the most prominent evaluators in the field. The publication highlights innovative methods and approaches to evaluation, emerging trends in evaluation practice and practical applications of evaluation theory.  The site includes an archive of past issues available in Adobe PDF or HTML format
www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive). Past issues covered Democratic Evaluation (Fall 2005) (www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/democratic-evaluation), Evaluation Methodology (Summer 2005) www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluation-methodology) and Community-Based Initiatives (Fall 2003) (www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluating-community-based-initiatives).  

The Manager's Guide to Program Evaluation: Planning, Contracting, and Managing for Useful Results. Mattessich, Paul, 2003. ( Amherst H. Wilder Foundation). The Manager's Guide to Program Evaluation describes the types of information to collect, spells out the four phases of evaluation and the steps in each phase, and offers advice on hiring and working with a professional evaluator.  To order online visit www.fieldstonealliance.org/productdetails.cfm?PC=28.
 

Multi-Cultural Health Evaluation Publications
The California Endowment and Social Policy Research Associates
This is a series of publications to help grantees, evaluators and funders better understand the field of multicultural health evaluation.  They are designed to help organizations explore their health evaluation projects within a multicultural and culturally competent context.  Available online at www.calendow.org/Collection_Publications.aspx?coll_id=74&ItemID=416

Evaluation of the Five-Year Expansion Program of California Healthy Cities and Communities

The California Endowment and Center for Civic Partnerships

Kegler, Michelle, Norton, B., and Aronson, R, 2003.  The complete cross-site evaluation report focusing on 20 California Healthy Cities and Communities that participated in a five-year expansion project, funded by The California Endowment, to enhance the capacity of recognized and emerging leaders in geographically and socially isolated communities. Available online at www.civicpartnerships.org/docs/publications/TCEFinalReport9-2003.pdf  

The Evaluation Forum

The Evaluation Forum has several useful publications on creating and managing outcome-based evaluation.  Outcomes For Success! 2000 Edition is an introduction to outcome-based evaluation which walks readers through the steps necessary to develop measurable outcomes and indicators for their program.   How to Manage and Analyze Data for Outcome-Based Evaluation covers a range of topics to help organizations collect and manage outcome data, using common software packages (e.g., Microsoft Excel and Microsoft Access). It contains step-by-step instructions and a disk with a practice data set.  To order online visit www.evaluationforum.com/publications

Improving Stakeholder Collaboration: A Special Report on the Evaluation of Community-Based Health Efforts

The California Wellness Foundation and Group Health Community Foundation

This report summarizes the learning from three national working conferences of experts in community-based health improvement to explore evaluation challenges, dilemmas, opportunities and innovations.  Participants in the conferences included 25 initiatives funded by The California Wellness Foundation, The California Endowment, Annie E. Casey Foundation, The Robert Wood Johnson Foundation and W.K. Kellogg Foundation, among others.  Available online at www.cche.org/pubs/ghcf-publication-stakeholder-collaboration.pdf  

Program Evaluation: Alternative Approaches and Practical Guidelines (3rd Edition).  Fitzpatrick, J., Worthen, Blaine, and Sanders, J., 2003. (Pearson Pub Co). Provides an overview of a wide variety of approaches to evaluation and contains practical guidelines and checklists for conducting successful evaluations. To order visit any of the online bookstores (amazon.com, powells.com, barnesandnoble.com).

Find an Evaluator

American Evaluation Association  
www.eval.org/find_an_evaluator/evaluator_search.asp
This site provides a searchable database by name/organization or area of expertise of American Evaluation Association members who provide evaluation consulting and technical assistance. 

The Evaluation Center

www.ec.wmich.edu/evaldir

The site provides a Directory of Evaluators with basic contact information from and about individuals and organizations who provide evaluation consulting services.  The directory can be searched by name/organization, area of specialty or geographic location.   

Specialty Topic - Advocacy Evaluation

The Evaluation Exchange – Advocacy and Policy Change

http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/advocacy-and-policy-change

The Spring 2007 issue (Volume XIII, No.1) of The Evaluation Exchange is the first of the Harvard Family Research Project’s “Hard to Measure” Evaluation Exchange series.  It describes how advocacy and policy change evaluations differ from other evaluations and gives examples of those differences in real life evaluation practices.  Also includes examples of new tools for advocacy and policy change evaluations. 

Innovation Network
www.innonet.org/advocacy
The Innovation Network is conducting research to identify a practical planning and evaluation framework for policy advocacy.  Currently the website includes a list of resources and articles on evaluating policy change and advocacy efforts, especially those addressing economic and legal issues, social justice and human rights.  See “8 Steps to Develop an Evaluation Plan” for tips on advocacy-focused evaluation planning.  

The Challenge of Assessing Policy and Advocacy Activities:  Strategies for a Prospective Evaluation Approach

The Challenge of Assessing Policy and Advocacy Activities Part II:  Moving from Theory to Practice 

The California Endowment and Blueprint Research & Design

These two publications examine the issues and challenges related to evaluating policy change efforts and make specific recommendations for conducting these types of evaluations.  The first publication (2005) presents guiding principles for policy change evaluation and makes recommendations for a prospective evaluation approach.  The second report (2006) provides recommendations and a framework for approaching policy and advocacy evaluation.  Both publications are available online at www.calendow.org.

Back to top