Program evaluation

From Wikipedia, the free encyclopedia

Jump to: navigation, search

Program evaluation is a systematic method for collecting, analyzing, and using information to answer basic questions about projects, policies and programs[1]. Program evaluation is used in the public and private sector and is taught in numerous universities. Evaluation became particularly relevant in the U.S. in the 1960s during the period of the Great Society social programs associated with the Kennedy and Johnson administrations[2][3]. Extraordinary sums were invested in social programs, but the impacts of these investments were largely unknown.

Program evaluations can involve quantitative methods of social research or qualitative methods or both. People who do program evaluation come from many different backgrounds: sociology, psychology, economics, social work. Some graduate schools also have specific training programs for program evaluation.


[edit] Dimensions of Program Evaluation

Program evaluators may assess programs on several dimensions to determine whether the program works. Rossi et al. (2004) divide these dimensions into 5 main categories: needs assessment, program theory, process analysis, impact analysis, and cost-benefit & cost-effectiveness analysis.[4]

A needs assessment examines the nature of the problem that the program is meant to address. This includes evaluating who is affected by the problem, how wide-spread the problem is, and what effects stem from the problem. For example, for a housing program aimed at mitigating homelessness, a program evaluator may want to find out how many people are homeless in a given geographic area and what their demographics are.

The program theory is the formal description of the program's concept and design. This is also called a logic model or impact pathways[5]. The program theory breaks down the components of the program and shows anticipated short- and long-term effects. An analysis of the program theory examines how the program is organized and how that organization will lead to desired outcomes. It will also reveal unintended or unforeseen consequences of a program, both positive and negative. The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders (see Participatory Impact Pathways Analysis).

Process analysis looks beyond the theory of what the program is supposed to do and instead evaluates how the program is being implemented. The evaluation determines whether target populations are being reached, people are receiving the intended services, staff are adequately qualified, etc.

The impact evaluation determines the causal effects of the program. More information about impact evaluation is found under the heading 'Determining Causation'.

Finally, cost-benefit or cost-effectiveness analysis assesses the efficiency of a program. Evaluators outline the benefits and cost of the program for comparison. An efficient program has a lower cost-benefit ratio.

[edit] Determining Causation

Perhaps the most difficult part of evaluation is determining whether the program itself is causing observed impacts. Events or processes outside of the program may be the real cause of the observed outcome (or the real prevention of the anticipated outcome).

Causation is difficult to determine. One main reason for this is self selection bias[6]. People select themselves to participate in a program. For example, in a job training program, some people decide to participate and others do not. Those who do participate may differ from those who do not in important ways. They may be more determined to find a job or have better support resources. These characteristics may actually be causing the observed outcome of increased employment, not the job training program.

If programs could use random assignment, then they could find a strong correlation or association. Causation is not something that can be proved through correlation. A program could randomly assign people to participate or to not participate in the program, eliminating self-selection bias. Thus, the group of people who participate would be the same as the group who did not participate, and this would be helpful in

However, since most programs cannot use random assignment, causation cannot be determined. Impact analysis can still provide useful information. For example, the outcomes of the program can be described. Thus the evaluation can describe that people who participated in the program were more likely to experience a given outcome than people who did not participate.

If the program is fairly large, and there is enough data, statistical analysis can be used to make a reasonable case for the program by showing, for example, that other causes are unlikely.

[edit] Types of Program Evaluation

Program evaluation is often divided into types of evaluation[7].

Evaluation can be performed at any time in the program. The results are used to decide how the program is delivered, what form the program will take or to examine outcomes. For example, an exercise program for elderly adults would seek to learn what activities are motivating and interesting to this group. These activities would then be included in the program.

Process Evaluation (Formative Evaluation) is concerned with how the program is delivered. It deals with things such as when the program activities occur, where they occur, and who delivers them. In other words, it asks the question: Is the program being delivered as intended? An effective program may not yield desired results if it is not delivered properly.

Outcome Evaluation (Summative Evaluation) addresses the question of what are the results. It is common to speak of short-term outcomes and long-term outcomes. For example, in an exercise program, a short-term outcome could be a change knowledge about the health effects of exercise, or it could be a change in exercise behavior. A long-term outcome could be less likelihood of dying from heart disease.

[edit] CDC framework

In 1999, the Centers for Disease Control and Prevention (CDC) published a six-step framework for conducting evaluation of public health programs. The publication of the framework is a result of the increased emphasis on program evaluation of government programs in the US. The six steps are:

  1. Engage stakeholders
  2. Describe the program.
  3. Focus the evaluation.
  4. Gather credible evidence.
  5. Justify conclusions.
  6. Ensure use and share lessons learned.

[edit] See also

[edit] External links

[edit] References

  1. ^ Administration for Children and Families (2006) The Program Manager's Guide to Evaluation. Chapter 2: What is program evaluation?.
  2. ^ US Department of Labor, History of the DOL (no date) Chapter 6: Eras of the New Frontier and the Great Society, 1961-1969.
  3. ^ National Archives, Records of the Office of Management and Budget (1995) 51.8.8 Records of the Office of Program Evaluation.
  4. ^ Rossi, P.H., Lipsey, M.W. & Freeman, H.E. (2004) Evaluation: A Systematic Approach.
  5. ^ Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).
  6. ^ Delbert Charles Miller, Neil J. Salkind (2002) Handbook of Research Design & Social Measurement. Edition: 6, revised. Published by SAGE,.
  7. ^ University of Texas, Austin, Division of Instructional Innovation and Assessment (2007) Types of program evaluation
Personal tools