edpsy570 edpsy570jeld eval evaluacion evaluation health impact outcomes performance program program_evaluation programevaluation programme public

My tags:

From Wikipedia, the free encyclopedia

This article includes a list of references or external links, but its sources remain unclear because it lacks inline citations. Please improve this article by introducing more precise citations where appropriate. (February 2008)

This article's tone or style may not be appropriate for Wikipedia. Specific concerns may be found on the talk page. See Wikipedia's guide to writing better articles for suggestions. (December 2007)

Program evaluation is a systematic method for collecting, analyzing, and using information to answer basic questions about projects, policies and programs^[1]. Program evaluation is used in the public and private sector and is taught in numerous universities. Evaluation became particularly relevant in the U.S. in the 1960s during the period of the Great Society social programs associated with the Kennedy and Johnson administrations^[2][3]. Extraordinary sums were invested in social programs, but the impacts of these investments were largely unknown.

Program evaluations can involve quantitative methods of social research or qualitative methods or both. People who do program evaluation come from many different backgrounds: sociology, psychology, economics, social work. Some graduate schools also have specific training programs for program evaluation.

Contents 1 Dimensions of Program Evaluation 2 Determining Causation 3 Types of Program Evaluation 4 CDC framework 5 See also 6 External links 7 References

[edit] Dimensions of Program Evaluation

Program evaluators may assess programs on several dimensions to determine whether the program works. Rossi et al. (2004) divide these dimensions into 5 main categories: needs assessment, program theory, process analysis, impact analysis, and cost-benefit & cost-effectiveness analysis.^[4]

A needs assessment examines the nature of the problem that the program is meant to address. This includes evaluating who is affected by the problem, how wide-spread the problem is, and what effects stem from the problem. For example, for a housing program aimed at mitigating homelessness, a program evaluator may want to find out how many people are homeless in a given geographic area and what their demographics are.

The program theory is the formal description of the program's concept and design. This is also called a logic model or impact pathways^[5]. The program theory breaks down the components of the program and shows anticipated short- and long-term effects. An analysis of the program theory examines how the program is organized and how that organization will lead to desired outcomes. It will also reveal unintended or unforeseen consequences of a program, both positive and negative. The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders (see Participatory Impact Pathways Analysis).

Process analysis looks beyond the theory of what the program is supposed to do and instead evaluates how the program is being implemented. The evaluation determines whether target populations are being reached, people are receiving the intended services, staff are adequately qualified, etc.

The impact evaluation determines the causal effects of the program. More information about impact evaluation is found under the heading 'Determining Causation'.

Finally, cost-benefit or cost-effectiveness analysis assesses the efficiency of a program. Evaluators outline the benefits and cost of the program for comparison. An efficient program has a lower cost-benefit ratio.

[edit] Determining Causation

Perhaps the most difficult part of evaluation is determining whether the program itself is causing observed impacts. Events or processes outside of the program may be the real cause of the observed outcome (or the real prevention of the anticipated outcome).

Causation is difficult to determine. One main reason for this is self selection bias^[6]. People select themselves to participate in a program. For example, in a job training program, some people decide to participate and others do not. Those who do participate may differ from those who do not in important ways. They may be more determined to find a job or have better support resources. These characteristics may actually be causing the observed outcome of increased employment, not the job training program.

If programs could use random assignment, then they could find a strong correlation or association. Causation is not something that can be proved through correlation. A program could randomly assign people to participate or to not participate in the program, eliminating self-selection bias. Thus, the group of people who participate would be the same as the group who did not participate, and this would be helpful in

However, since most programs cannot use random assignment, causation cannot be determined. Impact analysis can still provide useful information. For example, the outcomes of the program can be described. Thus the evaluation can describe that people who participated in the program were more likely to experience a given outcome than people who did not participate.

If the program is fairly large, and there is enough data, statistical analysis can be used to make a reasonable case for the program by showing, for example, that other causes are unlikely.

[edit] Types of Program Evaluation

Program evaluation is often divided into types of evaluation^[7].

Evaluation can be performed at any time in the program. The results are used to decide how the program is delivered, what form the program will take or to examine outcomes. For example, an exercise program for elderly adults would seek to learn what activities are motivating and interesting to this group. These activities would then be included in the program.

Process Evaluation (Formative Evaluation) is concerned with how the program is delivered. It deals with things such as when the program activities occur, where they occur, and who delivers them. In other words, it asks the question: Is the program being delivered as intended? An effective program may not yield desired results if it is not delivered properly.

Outcome Evaluation (Summative Evaluation) addresses the question of what are the results. It is common to speak of short-term outcomes and long-term outcomes. For example, in an exercise program, a short-term outcome could be a change knowledge about the health effects of exercise, or it could be a change in exercise behavior. A long-term outcome could be less likelihood of dying from heart disease.

[edit] CDC framework

In 1999, the Centers for Disease Control and Prevention (CDC) published a six-step framework for conducting evaluation of public health programs. The publication of the framework is a result of the increased emphasis on program evaluation of government programs in the US. The six steps are:

1. Engage stakeholders
2. Describe the program.
3. Focus the evaluation.
4. Gather credible evidence.
5. Justify conclusions.
6. Ensure use and share lessons learned.

[edit] See also

• Impact assessment
• Impact evaluation
• Important publications in program evaluation
• Participatory Impact Pathways Analysis
• Policy analysis
• Policy studies

[edit] External links

• Administration for Children and Families The Program Manager's Guide to Evaluation. Discussion of evaluation, includes chapters on Why evaluate, What is evaluation.
• American Evaluation AssociationIncludes a link to Evaluation resources such as organizations, links sites, email discussion lists, consultants and more
• Canadian Evaluation Society Includes a link to Evaluation information such as services, professional development, resources, organizations, regional chapters
• CDC six-step framework. Also available here Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11). Includes a description of logic models in the Steps section.
• Handbook of Research Design & Social Measurement. Delbert Charles Miller, Neil J. Salkind (2002) Edition: 6, revised. Published by SAGE.
• The EvaluationWiki - The mission of EvaluationWiki is to make freely available a compendium of up-to-date information and resources to everyone involved in the science and practice of evaluation. The EvaluationWiki is presented by the non-profit Evaluation Resource Institute.
• Free Resources for Program Evaluation and Social Research Methods This is a gateway to resources on program evaluation, how to, on line guides, manuals, books on methods of evaluation and free software related to evaluation.
• International Organization for Cooperation in Evaluation
• Innovation Network Innovation Network is a nonprofit organization working to share planning and evaluation tools and know-how. The organization provides online tools, consulting, and training, for nonprofits and funders.
• Links to Assessment and Evaluation Resources List of links to resources on several topics, including: Centers; Community building; Education and training in evaluation; Foundations; Indiana government & organizations; Links collected by...; Logic models; Performance assessment & electronic portfolios; Political & private groups or companies; Professional assns, orgs & pubs; Purdue University; United States Government; Web searches for publications by author & topic; and Vivisimo topical meta searches.
• Maine Legislature's Office of Program Evaluation & Government Accountability An excellent example of a governmental Program Evaluation office with links to several detailed reports which include methodology, evaluations results, recommendations and action plans.
• National Legislative Program Evaluation SocietyIncludes links to state offices of program evaluation and/or performance auditing in the USA
• W.E. Upjohn Institute for Employment Research
• Employment Research Institute

[edit] References

1. ^ Administration for Children and Families (2006) The Program Manager's Guide to Evaluation. Chapter 2: What is program evaluation?.
2. ^ US Department of Labor, History of the DOL (no date) Chapter 6: Eras of the New Frontier and the Great Society, 1961-1969. http://www.dol.gov/oasam/programs/history/dolchp06.htm.
3. ^ National Archives, Records of the Office of Management and Budget (1995) 51.8.8 Records of the Office of Program Evaluation. http://www.archives.gov/research/guide-fed-records/groups/051.html.
4. ^ Rossi, P.H., Lipsey, M.W. & Freeman, H.E. (2004) Evaluation: A Systematic Approach.
5. ^ Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).
6. ^ Delbert Charles Miller, Neil J. Salkind (2002) Handbook of Research Design & Social Measurement. Edition: 6, revised. Published by SAGE,.
7. ^ University of Texas, Austin, Division of Instructional Innovation and Assessment (2007) Types of program evaluation http://www.utexas.edu/academic/diia/assessment/iar/programs/plan/types/.