|
Peer Support Program Evaluation Goals
| To measure who and how many youth seek and receive peer support services from Foundry To demonstrate the impact of peer support on youth, workers, and health system outcomes to support the further scaling up of peer support programs and initiatives To establish consistency across peer support programs including the roles of peer support workers | To enable quality improvement and to track user recovery | To understand the baseline effectiveness and to promote wider organizational uptake of peer support To provide ongoing feedback to understand and ensure that the needs of the program were being met To provide a process of quality improvement To inform peer support training for other programs in the organization | To provide quality improvement that addresses the needs of the organization using the peer support service | To inform quality improvement through ongoing feedback | To inform quality improvement and understand user and worker’s relationship with peer support | To ensure the program is adhering to a recovery model as outlined by the MHCC |
|
Peer Support Program Evaluation Guiding Principles and Practices
| Evaluation is developed through a process of co-design with youth Evaluation is informed by practice-based evidence | Evaluation is based on a recovery model | Evaluation is ongoing and developed through a process of co-design with youth Evaluation is informed by practice-based evidence | Evaluation is developed with the involvement of the business/organization that has requested the services of the peer support program | Evaluation is ongoing and based on a recovery model Evaluation is developed through a process of co-design involving youth | Evaluation is based on a recovery model and is developed through a process of co-design with youth | Evaluation is ongoing and is based on a recovery model of care |
|
Participants Involved in the Design and/or Conduct of Peer Support Evaluations
| Youth Advisory Councils are involved in all aspects of evaluation design and conduct (including analysis) Peer support users, workers, and clinicians participate in conducting the evaluation (on a voluntary basis) | Peer support users and workers participate in conducting the evaluation (on a voluntary basis) | A youth engagement committee is involved in the design of the evaluation Peer support users and workers participate in conducting the evaluation (on a voluntary basis) | Peer support workers and clinicians participate in conducting the evaluation (on a voluntary basis) | A youth and young adult advisory council is involved in the design of the evaluation Peer support users and workers participate in conducting the evaluation (on a voluntary basis) | Evaluation is done through a co-design process with youth users and workers Peer support users and workers participate in conducting the evaluation (on a voluntary basis) | Peer support users, workers, and clinicians participate in conducting the evaluation (on a voluntary basis) |
|
Peer Support Program Evaluation Data Collection and Analysis
|
Evaluation In-development:
Mixed-method longitudinal evaluation with a co-design approach
Guided by a splash and ripple logic model
Prior evaluations: Data on program outputs was collected using a youth experience and satisfaction survey and a service worker experience survey
| Data are collected using surveys Data are collected at baseline (intake) up to 9 months after program engagement C-PROM is used for recovery-based assessments | Data are collected using online surveys Quantitative and qualitative data are extracted from surveys | Data are typically collected using surveys | Evaluation is done using a pre-post analysis and results are aggregated across different programs that include peer support Data are collected using feedback forms | Evaluations take place after youth participation in peer support programs have concluded and results are typically analyzed on a quarterly basis Data are collected using the PSSIQI survey tool and analyzed by an external party and information is consolidated and shared back with the organization for interpretation PSSIQI can be used in conjunction with C-PROM | Evaluation is guided by a process-oriented logic model Data are collected using surveys CMHA offers yearly assessments of completed evaluations |
|
Peer Support Program Evaluation Outcomes
|
Prior Evaluations
Individual outcomes: user and worker satisfaction
Program outcomes: program outputs (e.g., attendance records, referrals to program)
Evaluation In-development
Individual outcomes: effectiveness of peer support over time through a repeated health measure based on an outcome rating scale with composite domains focusing on relationships to recovery (called Life Tracker)
System-level outcomes: connection to other external or internal services through peer support to understand impact to system-level services (e.g., emergency departments)
| Recovery-oriented outcomes: user’s “path to recovery” including feelings and attitudes of user’s personal recovery Individual outcomes: employment related outcomes Program outcomes: attendance, user information, duration of program enrolment System-level outcomes: hospitalizations | Individual outcomes: user and worker experiences | Program outcomes: peer support worker and clinician perceptions of the peer support program | Recovery-oriented outcomes: overall recovery, emotional regulation, empowerment, and social connections Individual Outcomes: education and employment | Recovery-oriented outcomes: change in emotions and feelings before and after peer support Individual outcomes: overall experience of the program and how strongly users agree with the set of 17 action statements to assess how users felt supported Program outcomes: assessment of changes that can be implemented into the program Demographic data are also collected | Recovery-oriented outcomes: community integration, overall fulfillment Individual outcomes: user and worker experiences Program outcomes: number of program users |
|
Equity considerations in evaluation
| JEDI work is being implemented throughout the organization and all aspects of program delivery, including peer support and evaluation Youth advisory councils include Indigenous, racialized, gender-diverse and youth from other marginalized and oppressed who are involved in co-designing evaluations Evaluation of their peer support training program is being done through an Indigenous lens The virtual care program was designed to increase access to programming and evaluation by breaking down barriers to participants for rural and remote There is an organization commitment to collecting Indigenous-data and stories during evaluation in a culturally responsible manner | Evaluation has highlighted a general gap in the uptake of populations using peer support services and efforts are being made to reach a broader population that is more reflective of the communities where peer support services are offered | There is an emphasis on having diverse perspectives from individuals with different lived experience informing the evaluation to ensure adequate community representation Participants involved in the evaluation are compensated for their involvement | Equity may not be on the client’s radar, which often means it is not considered Equity considerations should be incorporated into evaluations as evaluations become more common and consistent | There is an effort to shift away from using clinical language within the evaluation to minimize stigmatization | Multiple methods of data collection are used to help minimize barriers to youth providing feedback | Evaluations use a “client-led” approach which includes user perspectives to help inform future efforts that may impact barriers that prevent an equitable approach to program evaluation |
