2022 Safe From the Start Process Evaluation

Introduction

Childhood exposure to violence (CEV) is a widespread public health concern. A national survey of over 4,000 children and youth revealed three in five children were exposed to at least one type of violence within the previous year.^[1] Additionally, CEV has been associated with both immediate and long-term developmental concerns. For example, CEV places children at increased risks for mental health problems and interpersonal relationship issues.^[2] While some evidence-based practices (EBPs) for victims of abuse have been effective at reducing the impact of CEV, process evaluations of community-based programs for young children are scarce.^[3] Process evaluations inform program development by documenting how program outcomes are reached and identifying how resources can be redirected to address programmatic needs and challenges and improve client outcomes.^[4]

Safe From the Start (SFS) is a program designed to develop, implement, and evaluate community-based models of intervention for young children exposed to violence and their families. Nine SFS sites operated in Illinois at the time of the study. The sites have shown promise in reducing the impact of violence exposure on children and families. For instance, 26% of children with social-emotional difficulties and 24% of caregivers with borderline- to clinical-levels of stress at intake significantly improved on the Ages & Stages Questionnaire: Social-Emotional (ASQ:SE) and Parenting Stress Index (PSI), respectively, after receiving SFS services.^[5] However, little is known about how program sites currently operate, their strategies for overcoming obstacles, and how the program can be improved.

We conducted the first process evaluation of the SFS program since Illinois Criminal Justice Information Authority (ICJIA) assumed administrative program oversite^[6] in 2013. We examined the program’s operation, with a focus on its program model comprising three core pillars: building coalitions, providing direct services, and raising public awareness about the issue and impact of CEV.

• Coalition building refers to providers’ partnerships with community-based agencies and with other local and state organizations serving families exposed to violence. It can help community-based intervention programs address community issues by collaborating with different organizations across multiple sectors. Coalitions commonly commit to a shared set of goals or targets and can mobilize talents and resources to develop widespread support for a community’s unmet needs, minimize duplicative efforts, and enhance visibility of one’s organization and services.^[7][8]

• Direct services provided by CEV prevention and intervention programs treat or support victims exposed to violence. Examples of direct services include home visits for high-risk families, trauma and symptom screening, referrals so families can access resources (e.g., mental health counseling), and education (e.g., positive parenting) for caregivers after exposure to violence.^[9]

• Public awareness involves providers engaging with community members to promote and disseminate information. It can be an effective approach for increasing community members’ knowledge of CEV and services available to young victims of violence. Also, it can provide communities with the resources needed to build their social networks vital for fostering positive youth functioning and healthy parent-child relationships.^[10]

Overall, the purpose of this process evaluation was to:

• Understand how sites operate.
• Assess sites’ collaboration with other agencies.
• Assess how clients are referred for SFS services.
• Identify client needs and challenges.
• Understand how clinicians deliver services and use best practices.
• Assess sites’ data collection processes.
• Document how sites conduct community outreach and increase awareness.

These evaluation findings are provided to inform stakeholders about both strong and weak program components. Recommendations are made on the efficiency and effectiveness of the SFS program, including suggestions for how stakeholders can better support sites, providers, and families exposed to community and/or domestic violence.

Method

Procedure

We conducted a process evaluation of nine SFS sites from December 2021 to April 2022. As part of the evaluation, we administered an online survey, facilitated focus groups, and observed virtual site visits conducted by ICJIA Federal and State Grants Unit staff. We recruited SFS service providers to participate in the survey and focus group through email. ICJIA grants staff invited us to attend the virtual site visits.

Data Sources

We analyzed data collected via an online survey, four focus groups, and nine virtual site visits. Participants included SFS site clinicians and administrative staff (e.g., directors, supervisors, managers). Some participants may have been a part of two or more data collection processes.

Participant Characteristics

At least 33 providers from nine sites participated in the process evaluation. Due to the anonymous nature of the survey, we were unable to calculate an unduplicated count of participants in our study. Program sites were located in various Illinois counties, including Cook, McLean, Peoria, and Rock Island. More clinicians (39.4%) participated in the evaluation than program supervisors (33.3%) or program directors (27.3%). Most participants (64.3%) had limited experience with the SFS program, having worked from only a few months to three years for the program. About a quarter of providers (28.6%) had worked between four and six years for the SFS program and one provider had been with the program for over six years.

Data Analysis

We created a predefined coding schema that was applied across data sources. Broad thematic codes were organized based on the study’s research questions and the three SFS pillars. We used NVivo, a qualitative research software, to review and identify patterns in participants’ responses to focus group questions. We conducted descriptive statistics of survey and site visit data.

Limitations

We encountered some research method limitations. The study may have been skewed by self-report bias due to the presence of clinical supervisors and directors at focus groups and grant monitors at site visits. Specifically, providers may have given more socially desirable responses rather than sharing their genuine views. In addition, the methods we used to collect the data limited our ability to conduct a more comprehensive analysis. For instance, while additional administrative data that could be used to support or expand on providers’ responses were available, we did not have IRB approval to analyze them for this study. Finally, we were unable to calculate an unduplicated sample size due to the anonymity of our survey; thus we cannot accurately describe whether the sample is representative of all sites.

Findings

Coalition Building

Providers reported three key objectives for building and maintaining a coalition:

• Build a robust network to better streamline the referral process for families.
• Collaborate with partners on public awareness endeavors to reach at-risk and/or underserved populations.
• Provide education and training to partners to increase clinicians’ knowledge of CEV and related topics.

Providers achieved these coalition objectives by exchanging referrals and sharing information and resources with their partners. Eight sites held at least one coalition meeting quarterly and six sites conduct at least one professional development training annually to ensure better collaboration among service agencies.

SFS partnerships were formed if:

• The agency had a similar service population as the SFS program.
• The agency was geographically close to the site.
• SFS meets an agency’s program need or the agency meets an SFS program need.
• The agency attended an SFS training session or presentation.

Providers indicated police departments were their most common partners. They reported capacity, staff retention, and the transition to virtual meetings were barriers to successful coalition member collaboration. However, the sites were able to successfully collaborate with coalition members. To encourage agency attendance and participation at coalition meetings, they provided professional development trainings and presentations.

Direct Services

The providers’ direct service activities involved making and receiving referrals, developing service plans, assessing staff capacity to provide treatment, providing therapeutic treatment, and monitoring client improvement. Providers commonly received referrals from the Department of Children and Family Services (DCFS), law enforcement, intra-agency programs, and caregivers. To develop a service plan, providers collaborated with case managers, crisis intervention managers, intake coordinators, and the courts. They also coordinated with other agencies to help meet families’ immediate needs (e.g., medical services, transportation, emergency housing) and secondary needs (e.g., childcare, educational, financial, and language needs). The providers asserted that it was important to address those needs so that families could remain in treatment. Also, they heavily relied on community partners to meet clients’ needs because they did not provide or did not have the capacity to provide all needed services. When referrals were received, sites assessed their capacity to serve referred families with consideration for clinical staffing levels, caseloads, clinician schedules, and case characteristics.

Sites provide therapy to children ages zero to five who have been exposed to violence. They also offer services to siblings under 18 years old and caregivers. Sites used a total of 17 different therapeutic treatment modalities. Table 1 summarizes treatment modalities used by multiple sites or those rated using the California Evidence-Based Clearinghouse’s Scientific Rating Scale (CEBC).^[11] The Scientific Rating Scale rates the strength of the research evidence supporting a practice or program using a scale of 1 to 5, with the following values:

• 1 = “Well-Supported by Research Evidence.”
• 2 = “Supported by Research Evidence.”
• 3 = “Promising Research Evidence.”
• 4 = “Evidence Fails to Demonstrate Effect.”
• 5 = “Concerning Practice.”
• NR = “Not Able to Be Rated” due to lack of available research evidence.

A value of 1 represents a practice with the strongest research evidence and 5 represents a concerning practice that appears to pose substantial risk to children and families. All sites used one of two treatment types, play therapy^[12] or Theraplay.^[13]

Table 1

Therapeutic Treatment Modalities Used by Sites

Note. Only modalities used by multiple sites or rated by the CEBC are listed here. Eight modalities, which were each used by a single site and not evaluated by the CEBC, were omitted. “-” indicates the modality was not included on the CEBC list.

As part of an evaluation protocol, sites gathered client information and administered assessments at specified intervals to track client progress while in service. While providers noted that the evaluation protocol had various benefits, the amount of time needed to administer the assessments and enter data into the SFS database as outlined in the protocol was a top challenge (Figure 1).

Figure 1

Highest Ranked Program and Protocol Challenges

Note. Sample was 13 providers. Nine providers did not respond to this survey question.

Additionally, we identified several needs and barriers to direct service provision. These included capacity issues, training gaps, a lack of caregiver engagement, and challenges associated providing services remotely. Nonetheless, sites demonstrated various strengths in providing direct services to families, such as striving to accommodate families’ busy schedules, using trauma-informed practices, offering psychoeducation to caregivers, and having internal supports in place (e.g., mentoring supervisors) to help alleviate work-related stress.

Public Awareness

Sites engaged in public awareness activities to promote and disseminate information about SFS services to community members and organizations serving similar age groups. SFS program and CEV awareness was shared through presentations to different agencies and at community events. Various professionals reached out to providers to request presentations, workshops, or training on such topics as trauma, healthy parent-child relationships and attachment, and the impact of domestic violence on young children.

Providers identified the following groups as underserved: victims of color, individuals without documents, families that only spoke Spanish, and LGBTQ+ individuals. They employed various strategies to reach underserved victims, including hiring bilingual staff and supplying outreach materials in multiple languages to mitigate language barriers and increase access to services. However, multiple providers did not have the staff capacity or financial resources to engage in outreach activities that targeted underserved groups.

The COVID-19 pandemic impacted the types of strategies providers used to conduct community outreach activities. Presentation topics shifted from CEV to pandemic-related areas. Also, providers turned focus to rebuilding connections that were lost early in the pandemic and attending community events. We found that the main barriers to public awareness and community outreach were related to staff capacity, low engagement or buy-in, and client accessibility to online platforms. While the pandemic exacerbated existing barriers to meeting public awareness goals, the sites’ use of virtual platforms created new opportunities to increase SFS program awareness and educate communities on CEV.

Providers’ Goals for Program Development

Providers recommended various SFS program goals for each pillar of work (Table 2).

Table 2

Providers’ Goals for the Program

Discussion

While providers noted many strengths, they encountered challenges particularly related to capacity and technology. We identified areas for programmatic improvement and offer recommendations.

Recommendations

Advocate for More Program Funding to Increase Capacity

Limited funding and resulting staff shortages created capacity challenges that affected coalition building, public awareness, and direct service work. Providers reported that staff left the program due to a lack of competitive salaries, benefits, and affordable childcare, which is consistent with recent research. Results also revealed that sites received an increased number of family referrals without additional financial resources. Furthermore, high staff turnover limited direct service capacity. With additional funding, sites would be able to offer more competitive salaries and benefits, helping to recruit new hires and retain current staff.

Advance Staff Knowledge and Skills through Training

Findings point to an increased need for training to build staff knowledge and skills, including on evidence-based practices (EBPs) and treatments and SFS programmatic processes. Providers stated that training opportunities were important for their professional development. Furthermore, research suggests career development offerings can improve workplace retention among health professionals.^[14] In addition, research on EBPs and treatments supports their effectiveness. According to Peña & Behrens,^[15] funders are more likely to invest in programs that apply research-based EBPs with their target populations as effective interventions often reduce taxpayer expenditures. Furthermore, trainings on SFS programmatic processes would help standardize practices among staff, which is particularly important for newer staff.

Focus on Expanding and Sustaining Coalition Partnerships

Most providers faced similar coalition challenges, including weakened agency connections, low or inconsistent meeting attendance from members, and few new partnerships. Findings suggested that providers need better guidance and tools for networking with other community-based service agencies and potential partners (e.g., early learning centers) similarly invested in preventing CEV. According to the U.S. Department of Health & Human Services,^[16] developing and strengthening partnerships helps build a more comprehensive and coordinated system for providing direct service and enables providers to better meet the needs of children and their families. Additionally, collaboration among organizations working with families exposed to violence can maximize resources, reduce siloing of services, and minimize duplicated efforts.^[17][18] Therefore, we recommend that providers focus on expanding and sustaining their coalition partnerships through various networking and team building strategies.

Explore Data Collection Methods to Further Assess Coalition Activities and Outcomes

The COVID-19 pandemic exacerbated coalition building challenges and depleted resources needed for interagency collaboration. Providers can better allocate their limited time and resources to gaps in their coalition activities with ongoing data collection. Collecting outcome data enables agencies to demonstrate to partners their progress toward programmatic goals and to obtain buy-in from new organizations.^[16:1] Additionally, logic models are useful for facilitating conversation about short and long-term outcomes.^[7:1][16:2] Another tool is the Coalition Effectiveness Inventory, useful for assessing if the coalition is informing policies and contributing to community change.

Enhance Engagement of Caregivers and Children in Services

Providers noted that some caregivers were reluctant to participate or allow their children to participate in services. Some sites successfully used strategies to keep families engaged in services longer, such as offering psychoeducation, parenting classes, and adult support groups to caregivers. Other sites should consider applying these strategies to increase caregiver engagement. Also, virtual service options made it possible for families who lived too far from site locations to receive services. However, providers reported that young children had difficulty staying engaged during these virtual sessions. Some sites provided families with therapy toolkits that allowed for interactive play and helped maintain children’s attention for longer periods of time during virtual sessions.

Enhance Cross-Site Collaboration and Coordination of Public Awareness Efforts

Most sites had difficulty providing both therapeutic services and engaging in public awareness activities due to limited staff capacity. Additionally, providers reported obtaining community buy-in through virtual outreach activities had been challenging. Research findings suggested that providers should enhance cross-site collaboration and improve the coordination of public awareness efforts to alleviate staff workload and increase community members’ awareness of the SFS program and the impacts of CEV. Tsao and Davis^[19] emphasized that efforts to address violence require consistent collaboration and stakeholder coordination. We recommend that SFS providers standardize their annual presentations and training offerings to better coordinate public education. Providers should use their bimonthly SFS all-sites conference calls to discuss potential opportunities for public awareness collaborations and to share their challenges or successes with community outreach.

Increase Knowledge and Accessibility of Services for Underserved Populations

While findings indicated providers had difficulty reaching underserved populations because of limited resources, SFS providers asserted that increasing underserved group awareness of the SFS program was an important programmatic goal. Underserved groups face multiple barriers to services, such as lack of transportation and services that feel welcoming, and limited service options for those living in rural areas and for people with disabilities.^[20] Additionally, when services are not sensitive to families’ identities and cultures, they are less likely to participate in services.^[21] Therefore, researchers recommended that providers build supportive relationships with community organizations to increase access to social supports^[22] and focus on collaborating with other local agencies to disseminate information on service options.^[10:1]

Future Directions for Research

The current process evaluation points to avenues for future research. First, researchers should consider evaluating individual sites. While we aggregated data across sites, each site was uniquely structured and served demographically distinct families. A closer examination of select sites would help identify their unique challenges and resource needs. Second, more research is needed to better understand strategies implemented by sites to retain SFS program staff and their effectiveness. Lastly, future researchers should conduct an outcome evaluation that incorporates client perspectives. While process evaluations provide invaluable information on a program’s operations, outcome evaluations are needed to better understand the program’s impact.^[23] Future research should assess program impacts on clients and provide opportunities for client study participation.

Conclusion

The SFS program is a community-based program established to help families exposed to violence, particularly young children under six years old. For more than two decades, SFS providers worked to expand their network of partnerships, provided direct services to families, and increased communities’ awareness of CEV and related topics. To paint a clear picture of program processes and outputs, process evaluations should be conducted at regular intervals. These evaluation findings would inform recommendations that could improve program operations and client outcomes.