Demystifying Program Evaluation in Criminal Justice: A Guide for Practitioners

What is Program Evaluation?

Program evaluation that is rooted in science is critical for criminal justice.^[1] Criminal justice programs should engage in evaluation to provide proof effectiveness and legitimacy and justify taxpayer support.^[2] According to Weisburd (2003), criminal justice researchers have an ethical and professional “obligation to provide valid answers to questions about the effectiveness of treatments, practices, and programs.”^[3] In recent years, policy-makers and funders have focused on investing in evidence-based, criminal justice programs.^[4] In addition, a movement toward evidence-based policymaking has created an increased demand for quality research evaluation.^[5] An evaluation can allow a program to make improvements, better secure funding, and potentially expand.

Research is often conducted by academics at universities, think tanks, government agencies, and nonprofits. However, external research can be time consuming and costly. In a review of the top five academic criminal justice journals over five years, only 5.2 percent of articles were evaluation research.^[6]

Program Evaluation and Funding

Criminal justice programs are often costly and practitioners often compete to secure funds from county boards, city councils, federal and state granting agencies, and private foundations. Rigorously evaluated programs can distinguish themselves from the pack by showing evidence of program effectiveness.

When Should I Start my Program Evaluation?

Evaluation planning ideally starts during the initial stages of program development.^[7] This encourages program staff to clearly define goals and objectives and gather data to document program implementation and outcomes.

Practitioners should hold off on conducting outcome evaluations until their programs have operated long enough to adequately address initial implementation issues and produced enough data to rigorously analyze outcomes.^[8] Some researchers advocate for an “evaluability” assessment, which provides information on whether a program has reached the appropriate stage for systematic evaluation. Evaluability assessments also can prevent a premature undertaking of an impact evaluation.^[9]

It is worth noting that as a program embarks on an evaluation, staff and stakeholders may be nervous of negative findings about the program. Researchers conducting the evaluation may encounter their own challenges, such as isolating the effects of the program, competing interests of stakeholders, attrition, and resource constraints. However, these challenges should not impede evaluations which should be viewed as opportunities for programs to learn, improve, and grow.^[10]

Where Should I Start in Conducting a Program Evaluation?

Develop a logic model

Logic models provide a graphical depiction of a program that outlines and links relationships among program activities, outputs, and outcomes.^[11] Figure 1 depicts a logic model and explains each component. A logic model can be a first step in the development and management of a program and can guide evaluation activities.

Figure 1

Logic Model Example^[12]

Source: National Institute of Corrections

Develop research questions

Research questions will guide program evaluation and help outline goals of the evaluation. Research questions should align with the program’s logic model and be measurable.^[13] The questions also guide the methods employed in the collection of data, which may include surveys, qualitative interviews, field observations, review of data collected by the program, and analysis of administrative records. Finally, questions should be tailored to how the resulting information will be used (e.g. internal practitioners looking to incrementally improve the program, or external funders determining program continuation).^[14]

What Kind of Evaluations Can be Done?

An evaluation can focus on certain aspects of a program and to what extent they impact outcomes. The two main evaluation types— process and outcome evaluations—focus on specific program areas and are equally important.^[15] When considering the kind of evaluation needed, program administrators and evaluators should take an inventory of available resources, including staff time, to conduct or assist in an evaluation.

Sources: Miller, J. M., & Miller, H. V. (2015). Rethinking program fidelity for criminal justice. Criminology & Public Policy, 14(2), 339-349; Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach. (7th ed.). Thousand Oaks, CA: Sage Publications, Inc.

Research design options for outcome evaluations

The value of an outcome evaluation is directly related to what can and cannot be concluded so the most rigorous evaluation option should be employed.^[16] In research, outcome evaluations that incorporate randomized control trials, where participants are randomly assigned to an experimental group or a control group, are the gold standard.^[17] The groups are differentiated by the program or practice (outcome variable) being studied; the control group may receive alternative programming or no programming. This method allows evaluators to isolate the impact of the program on participants.

An alternative is quasi-experimental design which compares outcomes of program participants (the treatment group) with a similar group of individuals with which to compare (comparison group). The individuals are not randomly assigned, but the design allows for valid causal inferences, despite some drawbacks.^[18] The lack of random assignment causes non-equivalent test groups, which can limit the generalizability of the results, reduce internal validity, and restrict definitive conclusions about causality.^[19] Biased evaluation findings from individual studies may under- or overrepresent the effects of the program.^[20]

Ethical Considerations in Evaluation

In addition to considering sound methods, careful examination of how criminal justice research is conducted based on research ethics and standards is important. The evaluator has a responsibility to maintain integrity, balance, and fairness in study design and data collection methods and a duty to respect the security and dignity of all parties involved in the evaluation project. If the evaluation findings on an agency’s work are to be published and the agency receives federal funding, the research must follow the code of federal regulations for the protection of human subjects. The regulations outline procedures including approval of the research study from an institutional review board.^[21]

Putting It All Together: Using What is Learned from a Program Evaluation

As Escamilla and colleagues (2018) noted,

To seasoned practitioners with years of field experience, a formal evaluation guided by researchers may seem irrelevant. After all, most programs make adjustments over the course of development and changes on the ground are easily recognized. However, even the sharpest practitioners may not recognize that complex external factors unrelated to programmatic decisions may be driving observable changes.^[22]

An evaluation can provide insight and offer recommendations to improve programs. Rigorous evaluation research can assess not only whether the target problem was reduced, but determine if the program caused the problem to be reduced.^[23] Evaluation findings can be used in many ways, including:

• Demonstrating the effectiveness of your program.
• Identifying ways to improve your program.
• Modifying program implementation and operations.
• Demonstrating accountability to program stakeholders and funders.
• Justifying funding and potentially aid in securing additional funding.^[24]

Findings should be disseminated in documents or reports that can help other jurisdictions and programs learn from the evaluation. To maximize utilization of evaluation findings, the evaluator may tailor the report style to the intended audience (e.g. academic journal article, technical report, executive summary).^[25] The evaluation also may include information for the user about how the findings can be best employed and disseminated to additional constituencies.^[26]

Figure 2 depicts the steps of an evaluation, from logic model creation through sharing findings and making programmatic adjustments.

Figure 2

Steps in Program Evaluation^[27]

Source: Centers for Disease Control and Prevention, 1999

Building Evidence of Program Effectiveness

Evaluation findings of similar programs can build a body of knowledge, leading to the model being deemed evidence-based. An evidence-based program has strong evidence that it is effective based on reliable and replicated research, while an evidence-informed practice has less evidence of its efficacy (Figure 3).^[28] There are numerous program models that have proliferated at a rate far beyond that supported by accompanying evaluation research.^[29] Syntheses of multiple evaluations of similar program concepts can be employed to guide program and policy planning efforts.^[30] However, research evaluation is a continuous process that will always be needed. Many programs adapt evidence-based programs to meet the needs of their unique communities, participant populations, program policies, and organizational climates. Many programs may produce an impact in the short-term, however relatively few evaluations examine outcomes beyond a year following a program’s initiation. Therefore, continued research is needed on long-term outcomes to identify the sustained impact of programs and policies.

Figure 3

Building Evidence of Effectiveness^[31]

Source: Corporation for National and Community Service

Conclusion

The field of criminal justice has fallen behind other fields, such as medicine, marketing, and business, in its use of evaluation research to inform and improve programming.^[32] As a result, perception, anecdotal evidence, and “business as usual,” rather than rigorous, empirical testing, become influential in program development.^[33] Rigorous evaluation is necessary to make informed decisions on program improvement and bring accountability to the utilization of limited resources.

Evaluation research can provide important information to influence the policymaking in the criminal justice field. Conflicting stakeholder interests on policy or ideology should be superseded by rigorous research.^[34]

Evaluation also contributes to improved treatment and service provision given to individuals. Without rigorous evaluation, it is unknown whether a program is providing any benefit to its participants. Ineffective programs may even cause unintended harm to those who participate. There is increasing agreement that criminal justice programming and practices should be grounded in scientific research to the greatest extent possible.

Resources

• American Society of Evidence-Based Policing (ASEBP)
• Betagov
• Crimesolutions.gov
• Campbell Collaboration
• Centers for Disease Control and Prevention, Program Performance and Evaluation Office
• Coalition for Evidence-based Policy
• Cochrane Collaboration
• Council of State Governments: What Works in Reentry Clearinghouse
• An Introduction to Evidence-Based Practices (JRSA)
• OJJDP Model Programs Guide
• SAMHSA’s National Registry of Evidence-based Programs and Practices
• UC-Boulder’s Center for the Study of Prevention of Violence: Blueprints
• Washington State Institute for Public Policy