Evaluation of the Strategy for Patient-Oriented Research (SPOR)
Final Evaluation Report

June 2023

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation – survey respondents, key informant interview participants, and Delphi panel participants. Also, thank you to those who supported the evaluation: Kate Benner and Sarah Boorman (Ference and Co.), Eva Maxwell and Kelly Wiens (Gelder, Gingras & Associates), the CIHR SPOR program team, CIHR Funding Analytics, members of the SPOR Working Group and Evaluation Advisory Committee.

The SPOR Evaluation Team

Carmen Constantinescu, Ellie Radke, Bruce Baskerville, Alison Croke, Jean-Christian Maillet, Rachelle Desrochers, Michael Goodyer, and Sarah Connor-Gorber.

For more information and to obtain copies, please contact: evaluation@cihr-irsc.gc.ca.

Table of Contents

• List of Tables
• List of Figures
• List of Acronyms
• Executive Summary
  • Program Overview
  • Evaluation Objective, Scope and Methodology
  • Key Findings
  • Recommendations
• Overview of SPOR Program
  • Program Description
• About the Evaluation
  • Purpose and Scope
  • Evaluation Context
  • Evaluation Methodology
• Evaluation Findings
  • Relevance
  • Design and Delivery
  • Performance
• Conclusions and Recommendations
• Appendix A: Tables
• Appendix B: Figures
• Appendix C: Detailed Descriptions of Core Elements
• Appendix D: Methodology – Additional Details
  • References
• Endnotes

List of Tables

• Table 1: CIHR Annual G&A Expenditures on SPOR by Core Element and Unspent Funds, 2010-11 to 2020-2021
• Table 2: CIHR Planned (based on TB submissions) and Actual Operating Costs on SPOR, 2010-11 to 2020-21
• Table 3: SPOR Partners Commitments for Funded Projects

List of Figures

• Figure 1: SPOR Logic Model
• Figure 2: SPOR Evolution by Core Elements
• Figure 3: Annual Allocations from TB and Annual SPOR G&A Expenditures by Core Element
• Figure 4: Needs Not Addressed by SPOR Reported by Researchers
• Figure 5: Number of PE Training or Mentoring Activities Offered by SUPPORT Units, Networks, and SEA, 2016-17 to 2019-20
• Figure 6: Number of Individuals Receiving Training or Mentoring in PE by SUPPORT Units, Networks, and SEA, 2016-17 to 2019-20
• Figure 7: Number and Type of Trainees Reported by Recipients
• Figure 8: Patient Involvement Reported by Patients vs. Recipients

List of Acronyms

Executive Summary

Program Overview

The Strategy for Patient-Oriented Research (SPOR) was launched in 2011 by the Canadian Institutes of Health Research (CIHR) as a response to a recognized need for greater uptake of research-based evidence to improve the health of Canadians while improving the cost-effectiveness of the health care system. Patient-oriented research (POR) is intended to focus on priorities that are important to patients and produce information that is taken up and used to improve health care practice, therapies, and policies. The goal of POR is to better ensure the translation of innovative diagnostic and therapeutic approaches to the point of care to ensure greater quality, accountability, and access of care. SPOR offers ongoing funding to support POR in Canada. Through SPOR, CIHR supports POR in Canada together with provincial and territorial ministries of health and other funding partners. SPOR consists of six core elements — Support for People and Patient-Oriented Research and Trials (SUPPORT) Units, SPOR Networks, Clinical Trials, Patient Engagement, Capacity Development, and Enabling Functions – that work to frame, facilitate and fund POR.

Evaluation Objective, Scope and Methodology

The objective of the evaluation is to provide CIHR senior management with valid, insightful, and actionable findings regarding the following:

• Needs addressed by SPOR and the program's alignment with CIHR and Government of Canada priorities;
• Effectiveness of the design and delivery of the program in supporting the achievement of intended outputs and outcomes; and
• Achievement of the program's expected outputs, and immediate, intermediate and ultimate outcomes.

The evaluation covers the period from 2016-17 to 2020-21. This is the second evaluation of the program since it commenced operations in 2011, with the first evaluation completed in 2016. Building on the first evaluation, this evaluation focused on the achievement of intermediate outcomes for elements of SPOR where sufficient time has elapsed and on the achievement of outputs and immediate outcomes for more recently implemented elements. The evaluation was committed to as part of CIHR's 2018-19 Evaluation Plan and designed to meet CIHR's requirements to the Treasury Board of Canada Secretariat (TBS) under the Policy on Results and the Financial Administration Act.

Key Findings

Relevance

There is a continued need to prioritize and foster patient-oriented evidence-informed health care in Canada, with substantive evidence of the relevance and benefits of patient engagement on the research process. POR is key to addressing the need for evidence-informed healthcare in Canada, with partners, patients and knowledge users highlighting its important contributions. There is a need to further increase awareness of both POR and SPOR among members of the health research community, patients, and decision-makers including a shared understanding of the benefits, challenges, and strategies for effective POR.

SPOR is aligned with the roles and responsibilities of the Government of Canada and CIHR's mandate to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and for the translation of research into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system" (S.C. 2000, c6). SPOR's objectives are well aligned with priorities in both CIHR's current and previous Strategic Plans. CIHR is well positioned to continue to play a leadership role in SPOR, particularly as a research funder and as a coordinating body or convener.

Design and Delivery

SPOR has largely been implemented as planned, with the implementation of SPOR elements evolving with a focus on strategic planning, the development and delivery of new programs and services, and phase II planning for SPOR SUPPORT Units and Networks. The implementation of SPOR has encountered challenges including resourcing limitations; inadequate guidance from CIHR on patient-engagement, including lack of harmonized patient compensation guidelines; uncertainty regarding the grant renewal process; and, challenging internal CIHR partnership processes. The SPOR program has responded to several unexpected shifts in the broader landscape, such as the COVID-19 pandemic, through the core elements demonstrating agility in adapting to the changing needs of patients, researchers, and the broader community. Monitoring of SPOR's implementation continues to be challenged by gaps in financial monitoring of grants and awards (G&A) expenditures, specifically the absence of unique coding for core elements, and operational spending, specifically a lack of information regarding direct salary.

As of 2021-22, the SPOR program had fully implemented actions for three out of six of the recommendations from the first evaluation, completed in 2016, with some actions for three recommendations partially implemented. Actions that remained partially implemented include: strengthening approaches to enable coordination, cross-learning and governance; supporting effective management and administrative functions within and across SUPPORT Units and Networks; and, revising the existing SPOR performance measurement strategy.

In general, the design features of SPOR support the achievement of intended outcomes; however, communication within and across the core elements was identified as inadequate, resulting in duplicative efforts rather than a cohesive approach. SPOR's current approach to patient engagement does not adequately support recruitment of diverse patient partners, with some patient partner groups disproportionately underrepresented in SPOR research. Although core elements demonstrate evidence of engagement with Indigenous community members, Indigenous communities remain underrepresented in SPOR-funded research.

SPOR's governance structure is not meeting its current objectives and lacks adequate patient representation. The National Steering Committee (NSC) has not met in recent years and generally provided advice rather than steering the SPOR program.

While collaboration between CIHR and partners was generally reported to be satisfactory, challenges remain, including: a lack of harmonized patient compensation standards, the need for a safe and supportive sharing environment for patient partners, and opportunities for increased awareness of ongoing SPOR activities.

A comparative review of international POR organizations suggests that using SPOR to inform an organization-wide patient engagement research funding model, in which patient and public engagement in all research programs is either encouraged or mandated, could optimize CIHR's investments in SPOR.

Challenges exist with the current management of performance measurement data including lack of clarity regarding performance indicators, inconsistent or missing indicators, double-counting, introduction of new indicators at the end of the reporting period, burden, and lack of alignment of Network and SUPPORT Unit work plans with the reporting requirements developed by CIHR. There is also limited evidence indicating that performance data are being used to inform decision-making regarding CIHR's implementation and optimization of SPOR.

Performance

SPOR's core elements are contributing to the achievement of immediate outcomes, including the generation of new knowledge, infrastructure, capacity development and engagement of patients and stakeholders. SPOR is generating and disseminating new knowledge as evidenced by the number of Knowledge Translation (KT) products^{Footnote 1} produced by the core elements based on the most recent annual reports in scope of the evaluation (2019-20) and trends over the evaluation period. Research platforms and other types of research infrastructure are established by the SUPPORT Units, SPOR Evidence Alliance (SEA) and Canadian Data Platform (CDP) and have responded to the needs of stakeholders by addressing identified barriers to data access and providing necessary evidence to knowledge users to drive decision making. Capacity in POR is developed as evidenced by the 2,221 training activities reaching 37,429 individuals across the SUPPORT Units, Networks and SEA. While there is evidence of engagement of patient partners in all aspects of research, there are opportunities to improve the level of patient engagement in research to avoid the perception of tokenism.

SPOR met or exceeded the 1:1 matching requirement by leveraging $1.16 in planned partner dollars for every CIHR dollar. However, it was not possible to determine if actual applicant partner investments met the matching requirement as applicant partner investments are not captured by CIHR's data systems nor were they systematically compiled from grant reports during the period covered by this evaluation.

SPOR's core elements are contributing to the achievement of intermediate outcomes, however, there are opportunities to strengthen contributions. Research evidence is being applied, as illustrated by guidelines, clinical practice, managerial decision-making, and policy documents citing SPOR-funded research. For example, findings from Primary and Integrated Health Care Innovations (PIHCI) research projects are supporting knowledge users with policy redesign in areas such as centralized waiting lists for primary care and reimagining health care delivery to reduce health care costs. SPOR's infrastructure and support services are aligned with and responding to the needs of stakeholders. Available evidence suggests that progress has been made in improving the clinical trials environment in Canada including the development of infrastructure for clinical trials which is supporting data access and addressing cost, capacity, and efficiency barriers; funding trialists to develop new methods that are low-cost, to generate relevant evidence and catalyze new partnerships and projects; and, supporting patient engagement in clinical trials.

Canadian capacity in POR is being strengthened and maintained, however, there are opportunities to strengthen the capacity for engaging with representative, equitable, and diverse patient populations, for example by re-establishing a governance structure with representation from patients, partners, and funders. Further, patient and stakeholder engagement is contributing to the achievement of intermediate outcomes, with some evidence of Indigenous Communities being active partners in both research and implementation of evidence-based improvements.

SPOR's core elements are contributing to the achievement of a cultural shift towards POR – a key expected ultimate outcome that should be maintained. At this point in time there is little evidence to demonstrate that SPOR has contributed to the expected ultimate outcomes to improve patient health care experiences, health outcomes or health system outcomes.

As expected, the COVID-19 pandemic has had a negative impact overall on recipients' ability to conduct research including reduced laboratory access and opportunities for collaboration.

Recommendations

The evaluation makes six recommendations aimed at improving the performance of SPOR to achieve its expected results.

Recommendation 1:

CIHR should use SPOR to inform an organization-wide approach to patient engagement in research to continue its leadership role, further investment and sustain progress on the outcome of a cultural shift toward POR.

Recommendation 2:

CIHR needs to do the following to improve the program design and delivery of SPOR:

• Increase awareness of the benefits of POR among members of the health research community, patients, and decision-makers.
• Enhance communications among and across SPOR core elements and CIHR institutes to avoid duplicative efforts, promote cohesion, and enhance partnerships.
• Improve overall program monitoring to ensure that research is delivering on intended objectives, such as the engagement of communities and patients in research and provide feedback.
• Establish consistent priorities, mandates and readiness across SPOR core elements to support linkages, alignment and coordination of initiatives.

Recommendation 3:

CIHR should re-establish an external and internal governance structure for SPOR with defined roles and responsibilities, including better representation from patients, partners, and funders, to improve CIHR's decision-making on SPOR.

Recommendation 4:

CIHR needs to improve patient and community engagement both in SPOR and in research in the following manner:

• Embed equity, diversity and inclusion considerations into the recruitment of patient partners to address the underrepresentation of important patient partner groups in research.
• Harmonize patient compensation standards across SPOR.
• Enhance accountability for meaningful patient engagement.
• Ensure consistency in engagement of Indigenous community members across SPOR core elements.

Recommendation 5:

CIHR should improve the management and reporting of SPOR performance measurement data to better inform decision-making by establishing a clear set of measures to track progress expected outcomes related to patient health care experiences, health, and health system.

Recommendation 6:

CIHR needs to further improve the following aspects of its financial monitoring and coding for SPOR:

• Grants and awards expenditures, especially coding of core elements and tracking of partner contributions.
• Operating and maintenance expenditures, specifically direct salary costs.

Overview of SPOR Program

Program Description

The Strategy for Patient-Oriented Research (SPOR) was launched in 2011 by the Canadian Institutes of Health Research (CIHR) as a response to a recognized need for greater uptake of research-based evidence to improve the health of Canadians while improving the cost-effectiveness of the health care system. Patient-oriented research (POR) is intended to focus on priorities that are important to patients and produce information that is taken up and used to improve health care practice, therapies, and policies. The goal of POR is to better ensure the translation of innovative diagnostic and therapeutic approaches to the point of care to ensure greater quality, accountability, and access of care. SPOR offers ongoing funding to support POR in Canada. Through SPOR, CIHR supports POR in Canada together with provincial and territorial ministries of health and other funding partners.

Concretely, SPOR aims to achieve the following^{Footnote 2} (see Figure 1: SPOR Logic Model):

• For patients, it means having a say in which health topics are researched;
• For researchers, it means benefiting from the perspectives and experiences of patients; and
• For the health care system, it means having access to the research evidence that decision-makers and health care providers need to improve care.

SPOR adheres to the following principles:

• Patients are involved in all aspects of research;
• Decision-makers and clinicians are involved throughout the entire research process to ensure integration into policy and practice;
• CIHR funding for SPOR initiatives is matched 1:1 with non-federal funding partners;
• Effective POR requires a multi-disciplinary approach; and
• Performance measurement and evaluations are integral components.

SPOR consists of six core elements — Support for People and Patient-Oriented Research and Trials (SUPPORT) Units, SPOR Networks, Clinical Trials, Patient Engagement, Capacity Development, and Enabling Functions – that work to frame, facilitate and fund POR. The first two years following the launch of the Strategy were focused on implementation design, including establishing a National Steering Committee (NSC), determining priorities, and creating funding opportunities for some of the core elements. The implementation of the different SPOR core elements began in 2013-14 when funding was initiated for four of the SUPPORT Units, two of the Networks, and the Canadian Clinical Trials Coordinating Centre (CCTCC). In the following years, more components of the core elements were implemented including: the Innovative Clinical Trials (iCT) Initiative, five additional SUPPORT Units, five more networks in Chronic Diseases, Patients Engagement Collaborations Grants, two enabling functions (i.e., the SEA and the CDP) and two capacity building components (i.e., the Patient-Oriented Research Awards and the National Training Entity). Additional details on SPOR's core elements are provided in Appendix C: Detailed Descriptions of Core Elements.

About the Evaluation

Purpose and Scope

The purpose of this evaluation is to provide CIHR senior management with valid, insightful, and actionable findings regarding the following:

• Needs addressed by SPOR and the program's alignment with CIHR and Government of Canada priorities;
• Effectiveness of the design and delivery of the program in supporting the achievement of intended outputs and outcomes; and
• Achievement of the program's expected outputs, and immediate, intermediate and ultimate outcomes.

By addressing these issues, the evaluation will help inform CIHR senior management decision-making and planning regarding the SPOR program, and meet the evaluation requirements outlined in the Policy on Results and subsection 42.1 of the Financial Administration Act.

The evaluation of the SPOR program was conducted by the CIHR Evaluation Unit and covers the period from 2016-17 to 2020-21 (with the extent of coverage of various elements dependent on when the elements were initiated; see Figure 2: SPOR Evolution by Core Elements). The extent to which the program has achieved its expected intermediate outcomes was measured by examining SPOR elements where sufficient time has elapsed. The extent to which outputs and immediate outcomes have been achieved was measured through more recently implemented elements. The evaluation design used a comprehensive approach with numerous lines of evidence to maximize depth of coverage of evaluation questions and rigour, and to triangulate data.

Evaluation Context

Previous Evaluation

This is the second evaluation of the SPOR and builds upon the first SPOR evaluation completed in 2016 which covered the period from inception in 2010-11 to 2015-16.

The findings from the first evaluation supported the continued need for the SPOR program, and its alignment with roles and responsibilities of the federal government and mandates of CIHR. The first evaluation made the following recommendations, agreed to by CIHR senior management in a management response to the evaluation:

• CIHR should increase efforts to strengthen SPOR's role in a common agenda for change to POR.
• CHIR should provide strategic guidance regarding how SPOR elements are to work together toward achieving SPOR's intermediate and long-term outcomes.
• CIHR should communicate plans for moving beyond the initial five-year funding period to manage sustainability expectations for CIHR investments in SPOR.
• CIHR should strengthen approaches to enable cross-learning, sharing of best practices, and collaboration; this should occur within and across SPOR elements and between CIHR and Canadian and International organizations.
• CIHR should continue to support effective management and administrative functions within funded SPOR SUPPORT Units and Networks and across these elements.
• CIHR should revise the existing SPOR performance measurement strategy to balance administrative/operational outputs with outcomes/impacts.

The current evaluation builds on the first evaluation to understand the effects of the actions taken in response to these recommendations, assess the achievement of immediate outcomes and, given that the program has been implemented for a decade, examine progress towards the achievement of intermediate and ultimate outcomes.

Evaluation Methodology

Evaluation Questions

The evaluation addresses the following specific questions.

Relevance

1. To what extent is there a continued need to prioritize and foster patient-oriented evidence-informed health care in Canada?
   1. To what extent is POR relevant to addressing this need?
2. To what extent has the role of the federal and provincial/territorial governments in SPOR been aligned with their respective roles and responsibilities in health care?
   1. To what extent is SPOR aligned with CIHR's mandate?

Design and Delivery

3. To what extent has SPOR been implemented as planned?
   1. How has SPOR been responsive to shifts in the broader landscape or needs identified by SPOR partners?
   2. How have recommendations from the 2016 formative evaluation of SPOR been addressed (e.g., sustainability, collaboration between core elements, performance measurement)?
4. How do the design features of SPOR (six core elements, principles) support the achievement of SPOR's intended objectives?
   1. How and to what extent are interconnections among the core elements fostered?
5. How effectively has the governance structure for SPOR (i.e., SPOR NSC, SPOR WG) guided the implementation of the strategy?
   1. To what extent has collaboration between CIHR and partners/stakeholders been satisfactory and effective?
   2. To what extent have communications around SPOR been satisfactory and effective?
6. What alternative models or approaches could optimize CIHR's investment in SPOR?

Performance

7. How and to what extent are the six core elements of SPOR contributing to the achievement of immediate outcomes?
   1. New knowledge in POR is generated and disseminated.
   2. Research networks, platforms and other types of research infrastructure are established.
   3. Capacity in POR is developed.
   4. Patients and other stakeholders are engaged in the generation of research knowledge and implementation of evidence-based improvements.
8. To what extent are the six core elements of SPOR contributing to the achievement of intended intermediate outcomes?
   1. Research evidence is applied.
   2. Infrastructure and support services respond to stakeholder needs.
   3. Clinical trials environment in Canada is improved.
   4. Canadian capacity in POR is strengthened and maintained.
   5. Patients and other stakeholders are active partners in both research and implementation of evidence-based improvements.
9. To what extent are the six core elements of SPOR contributing to achieving intended ultimate outcomes?
   1. Patient health care experiences and health outcomes are improved.
   2. Cultural shift toward POR is achieved.
   3. Improved health system outcomes through evidence-based practices.
10. To what extent has the COVID-19 pandemic impacted the delivery and performance of SPOR?
    1. Negative and positive consequences at the individual, research activity and strategy levels (e.g., lessons learned from pivoting to new ways of operating and trying to develop new solutions for primary health care and health research);
    2. Future anticipated changes to SPOR and SPOR-related research activity (e.g., innovations in virtual care will look post-pandemic era); and
    3. The effect that COVID-19 had on scientific productivity by gender, age, and other factors.

Evaluation Approach

The evaluation employed both quantitative and qualitative data collection methods and analyses. Consistent with best practices in program evaluation^{Footnote 3} as well as the Policy on Results, multiple lines of evidence were used to triangulate evaluation findings. This included a document review; administrative data review; an environmental scan; a bibliometric analysis; and surveys of recipients (n = 89), applicants (n = 49), and stakeholders of the program (n = 155), including co-applicants (n = 50), patient partners (n = 39)^{Footnote 4}, trainees (n = 39), other partners (n = 16)^{Footnote 5}, knowledge users (n = 11). There were also key informant interviews (n = 38) conducted with SPOR program management (n = 13), patient partners (n = 9), other partners (n = 9), knowledge users (n = 4), POR experts (n = 3); case studies (n = 5); and a Knowledge Readiness Levels analysis.

Gender-based Analysis Plus (GBA+) and equity, diversity and inclusion (EDI) considerations were built into the evaluation framework via specific evaluation questions and indicators.

Note that the reported denominator will change as it reflects the number of individuals who were posed the question. Given the large number of lines of evidence with varying sample sizes, the following qualifiers have been used to indicate the frequency of responses for some lines of evidence conducted, for consistency (i.e., surveys and key informant interviews). It is important to note that these qualifiers have been used in order to summarize statements about qualitative data; they should not necessarily serve as a measure of the importance of the respective finding.

Additional details about the methodology are provided in Appendix D: Methodology – Additional Details.

Limitations of this Evaluation

The evaluation leveraged a variety of data sources. The value of this evidence-based strategy lies in the efficiency of utilizing currently available data and synthesizing it through a single evaluative lens. However, as with all evaluations, this evaluation encountered some limitations (discussed in more detail in Appendix D: Evaluation Limitations and Mitigation Strategies). The main limitations associated with this evaluation are:

• Limited ability in attributing changes at the intermediate and ultimate outcome level to SPOR due to the complexity of the SPOR initiative and the health research funding landscape;
• Staggered timelines of implementation of the SPOR elements;
• Availability of data (i.e., limited ability to develop complete listings of researchers, trainees, patients, partners, and other stakeholders of SPOR);
• Analyzing performance and other secondary data sources (e.g., self-report data, possible double-counting, possible incompleteness); and
• Limited counterfactual (i.e., given there is no similar Canadian program, the only population for a counterfactual approach was researchers who applied, but did not receive, SPOR funding).

Evaluation Findings

Relevance

Key Findings:

There is a continued need to prioritize and foster patient-oriented evidence-informed health care in Canada.

It is evident that there is a continued need to prioritize and foster patient-oriented evidence-informed health care in Canada, with substantive evidence of the benefits of patient engagement on the research process. The need for POR is supported by considerable literature on patient partnership and community engagement in the production of evidence, the identified benefits engagement or patient involvement brings to the research process, and the work on principles and best practices that has been done to address the challenges in conducting POR. Evidence-based medicine has identified patient partnership in the production of evidence as one of the key ways of developing more trustworthy evidence and it has been described as a moral imperative that is associated with a number of benefits and challenges (Gill & Cartwright, 2021). A review of recent literature revealed key benefits of POR on the research process, including more relevant research topics and priorities, more relevant research outcomes, and uptake of evidence by health policy decision-makers. Benefits that were specific to patients included empowerment, prioritization of research relevant to the community, enhanced knowledge and skills, increased transparency and accountability, and more useful evidence for the purpose of knowledge translation (KT) (Vat et al., 2020).

POR is key to addressing the need for evidence-informed healthcare in Canada.

Survey findings and key informant interviews indicate that POR, through SPOR, is key to addressing the need for evidence-informed healthcare in Canada, with partners, patients and knowledge users highlighting its important contributions. On a 5-point scale from Not at All to a Very Great Extent, SPOR researchers and stakeholders surveyed felt that SPOR is addressing the need for POR in Canada to a moderate-to-great extent (Recipients: M = 3.8 out of 5, SD = 1.1, n = 84; Stakeholders: M = 4.1 out of 5, SD = 1.1, n = 165). Almost two-thirds of SPOR researchers indicated that their research project would not have proceeded had they not received SPOR funding (Recipients: 62%, n = 54), and indeed, almost two-thirds of unsuccessful applicants indicated that their POR project either did not proceed or had to be modified as a result of not receiving SPOR funding (66%, n = 31), emphasizing both the need for and importance of SPOR funding in supporting POR in Canada. Almost all key informants (25/30) indicated the continued need for POR in supporting evidence-informed health care, with partners, patients and knowledge users highlighting its important contributions, including capacity building, partnership and collaborations, engagement in decision-making and improvements to healthcare. Some key informants (9/26) reported that SPOR does not duplicate, but rather complements other patient engagement research activities across Canada.

Further awareness of both patient-oriented research and SPOR is needed, including a shared understanding of the benefits, challenges, and strategies for effective patient-oriented research among members of the health research community, patients, and decision-makers. Documents reviewed, key informants and SPOR researchers surveyed cited challenges or needs not generally being met by existing POR supports. A review of recent literature on the relevance of POR identified a number of potential challenges associated with POR, including challenges in the selection of patient partners; tokenism; logistical and practical barriers; patient exclusion from research stage; insufficient knowledge; absence/impact of patient compensation; traditional research culture; and challenges in participant recruitment (Martineau et al., 2020). Lack of awareness of SPOR was mentioned by some key informants (8/29) and two survey respondents, including comments that the SPOR program needs to continue to be communicated to patients, researchers, and decision-makers, with a few proposing that POR be integrated across all CIHR's granting activities.

SPOR is aligned with the roles and responsibilities of the federal government as well as CIHR strategic priorities.

There is evidence of an identified role for the federal government in supporting evidence-informed health care, with government mandate released during the evaluation reporting period emphasizing the importance of responding rapidly to ongoing changes in the healthcare system. The 2017 Mandate Letter from the Minister of Health communicated a need to keep up with advances in health technology that are rapidly changing health care across Canada and need for the federal government to continuously be a part of improving outcomes and quality of care for Canadians (Government of Canada, 2017). More recently, the 2021 Speech from the Throne outlined the need to strengthen the healthcare system for all Canadians, particularly seniors, veterans, persons with disabilities, vulnerable members of our communities, and those who have faced discrimination (Government of Canada, 2021b).

In terms of its alignment with CIHR, SPOR's objectives are very well aligned with the CIHR Act, CIHR's mandate, and CIHR's Strategic Plan priorities. SPOR's objective to improve care by integrating research evidence into the health care system is aligned with the CIHR Act (S.C. 2000, c6) as it acknowledges the importance of supporting initiatives that will lead to the improved health of Canadians. This objective is also aligned with CIHR's mandate for translation of research into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.

SPOR aligns with CIHR's Strategic Plan 2014-15 to 2018-19, Health Research Roadmap II: Capturing Innovation to Produce Better Health and Health Care for Canadians, as well as CIHR's Strategic Plan 2021-2031, a Vision for a Healthier Future. SPOR's principle to involve patients in all aspects of research aligns with Strategic Direction 2 of CIHR's Strategic Plan for 2014-15 to 2018-19, to mobilize health research for transformation and impact through its intent to build, shape and mobilize research capacity to address critical health issues that are important to patients and Canadians (CIHR, 2015). Additionally, SPOR is specifically mentioned as a means of incorporating POR into policy and practice within Research Priority A: enhancing patient experiences and outcomes through evidence-informed health innovations. Further, the objectives of SPOR are closely aligned with all five priorities of CIHR's new Strategic Plan (CIHR, 2021). Specifically, Priority E to integrate evidence in health decisions cites the creation of SPOR as helping to shape the field of knowledge mobilization in Canada and moving evidence into Canadian health systems.

CIHR is well-positioned to play a leadership role in SPOR.

Almost all key informants (30/35) expressed that CIHR is well-positioned to play a leadership role in SPOR. In addition to being the major health research funder in Canada, areas where CIHR is best positioned to extend its leadership role include that of a national convener that brings stakeholders together to ensure that patient engagement is integrated across the research cycle. In addition, key informants described CIHR as a catalyst for patient engagement in research that includes such things as supporting KT, developing methodologies and protocols and best practices, creating patient engagement tools for researchers, an honest broker for the provincial and territorial jurisdictions, building capacity in POR, and setting the standards for guidelines and policies for such matters as patient compensation.

Design and Delivery

Key Findings:

SPOR has largely been implemented as planned.

A review of documentation and key informant interviews revealed that SPOR has been largely implemented as planned, with the implementation of SPOR elements evolving with a focus on strategic planning, the development and delivery of new programs and services, and phase II planning for SPOR SUPPORT Units and Networks. Administrative data indicates that SPOR evolved from initial planning stages and the solidifying of partnerships in 2011 to the funding of SPOR core elements starting in 2013-14. The first two years following the launch of the Strategy were focused on implementation design, including establishing a National Steering Committee (NSC), determining priorities, and creating funding opportunities for the first core elements. The implementation of the different SPOR core elements began in 2013-14, when funding was initiated for four of the SUPPORT Units (Alberta, Manitoba, Maritimes, and Ontario), two of the Networks (Adolescent Connections to Community-driven Early Strengths-based Stigma-free Services [ACCESS] Open Minds and the Primary and Integrated Health Care Innovations [PIHCI] Network), and the Canadian Clinical Trials Coordinating Centre (CCTCC). In the following years, more components of the core elements were implemented including: the innovative Clinical Trials (iCT) Initiative, five additional SUPPORT Units, five more Networks (Chronic Disease), Patient Engagement Collaborations Grants, two enabling functions (the SPOR Evidence Alliance [SEA] and the Canadian Data Platform [CDP]) and two capacity building components (the Patient Oriented Research Awards and the National Training Entity [NTE]). Further, phase II funding has begun for some Networks and SUPPORT Units, such as the Alberta SUPPORT Unit and the CDNs. Figure 2: SPOR Evolution by Core Elements depicts the implementation of components of the SPOR core elements at different points in time.

SPOR expenditures increased as new elements were added over time, starting at $14.4M in 2013-14 with a peak at $67.6M in 2017-18, and remaining stable at approximately $60M per year for the period 2018-2021. Following significant underspending on G&A in the period of the first evaluation while implementing the strategy, CIHR has consistently spent 89% or more of its allocation from Treasury Board (TB) within the fiscal year, including overspending in years 2016-17 and 2017-18, resulting in $6M (10%) in additional G&A spending beyond what was allocated to CIHR by TB. SUPPORT Units have received the highest levels of investment, totaling $228M, followed by Networks $88M, Clinical Trials $40M, Enabling Functions $13M, Capacity Development $10M and Patient Engagement $2.8M (Figure 3: Annual Allocations from TB and Annual SPOR G&A Expenditures by Core Element). The total cumulative expenditures for the SPOR program for the period of 2010-2021 was $390,981,571.

The first evaluation of SPOR reported a consistent decrease in Foundational Investments^{Footnote 6} as SPOR focused its efforts on the core elements. Whether this trend continued could not be assessed since the Foundational Investments made using funding outside the SPOR Ring-Fenced funding were not included in financial reporting on the SPOR program past the period of the first SPOR evaluation (Table 1: CIHR Annual G&A Expenditures on SPOR by Core Element and Unspent Funds, 2010-11 to 2020-21). The evaluation team was unable to find documentation to confirm the nature of the investments (programs and grants), and if those investments ended in fiscal year 2015-16 or continued thereafter.

Several challenges to implementation were identified within program documentation and key informant interviews. Inhibiting factors identified in program documentation and supported by key informant interviews included uncertainty of funding (2/14) and staffing challenges (i.e., under-resourcing leading to stress completing annual operations such as annual reporting, daily operations, and grant renewals at the core element level, and inability to provide timely feedback on annual reports at the program management level (2/14). The most frequently reported inhibiting factor from the document review was the COVID-19 pandemic, as well as patient partner turnover, a health system transformation, delays with data access, and the implementation of a new strategic plan for the data platform and services component requiring a greater demand of resources. CIHR staff, SPOR entity leads, and knowledge users interviewed also identified CIHR not providing enough guidance on patient engagement policies or standards (4/14) or holding grant recipients to a patient engagement standard (2/14), and that the internal CIHR partnership processes among SPOR entities could be improved (4/14).

The review of administrative data indicate that, while the SPOR program overspent its planned operating expenditures considerably in early years of the program, annual operational spending has remained within 92% and 108% of the allocations from TB since 2016-17 (Table 2: CIHR Planned (based on TB submissions) and Actual Operating Costs on SPOR, 2010-11 to 2020-2021).

The administrative costs for SPOR are derived from a combination of actual expenditures and estimates of direct salary costs. In the case of FTEs and salary costs, the reported total FTE estimate was consistently lower (ranging from 20.05 to 23.1) than the 27.75 planned FTEs for SPOR for the period under review (2016-17 to 2020-21). However, the lower number of reported FTEs did not result in reduced operational spending due to the salary costs of the number of senior professional positions reported. CIHR does not have a robust method to track staff time associated with SPOR activities across the entire organization. Therefore, for the purpose of this evaluation, Financial Planning and Advisory Services asked the SPOR staff to review the list of positions used to derive the planned 27.75 FTEs in the first evaluation and provide an estimated FTE for those positions for each year from 2016-17 to 2020-21. It should be noted that due to time constraints these estimates were not validated by the implicated CIHR business units, and the mid-range salary for each position was used to estimate direct salary costs.

The estimated total cost of administering the SPOR Program as a percentage of Total Program Expenditures varied between 6.2% and 7.5% over the period of the evaluation. This is high relative to CIHR overall, which has an average of 5.4%, but is lower than SPOR's planned administrative costs percentage of 7.7%. It is important to note that the validity of this estimate is affected by the method used to estimate FTEs contributing to SPOR.

SPOR has been responsive to shifts in the broader landscape.

According to documents reviewed and key informant interviews, the SPOR program has been responsive to several unexpected shifts in the broader landscape, such as the COVID-19 pandemic, through the core elements demonstrating agility in adapting to the changing needs of patients, researchers, and the broader community. Many core elements stated in program documentation that the pandemic resulted in temporary closures to a few of their partner institutions (e.g., Memorial University), unplanned and unforeseen staffing departures and leaves of absence, as well as some projects and events being placed on hold (e.g., the Canadian Association for Health Services and Policy Research conference). Many SPOR core elements, including SUPPORT Units, Networks and the CDP reported shifting priorities to address the urgent needs of health policy and practice due to COVID-19, while simultaneously maintaining important non-COVID-19 research. Since the start of the pandemic, documentation from core elements revealed how agile they were in meeting unexpected requests for support from within each province while maintaining regular functions.

Funding uncertainty was another frequently cited reason for changes to the SPOR program among documentation reviewed. Sustainability and succession planning have been challenging for some of the SPOR SUPPORT Units and Networks given funding uncertainty for health research and POR from both federal and provincial sources. Due to the grant funding coming towards the end of its first cycle, one SUPPORT Unit reported losing several staff to local industries where they could be offered more job security, which led to the loss of some institutional memory. The core elements through program documentation indicated that they have been quick in replacing some of these staff and ensuring that there is cross training of staff to cover vacancies. With the second bridge period from CIHR in place, some of the SUPPORT Units felt they were better able to advocate for continued support for several of its partnered POR supports.

With the publication of the new SPOR II guidelines and associated funding envelopes it became clear that some SUPPORT Units would need to become a leaner team, this resulted in several individuals being given their notice. Staff changes always cause some degree of upset to workflow however the Units reported that remaining team members have continued to work efficiently, and priorities have been adjusted to ensure that all the required work is being completed in a timely manner. Additionally, retention of patient partners has required some flexibility on the part of the core elements. A number of patient partners were elderly and considering leaving their home province to be with family. For example, to adapt to the potential loss of patient partners, the Patient Advisory Council in Newfoundland was tasked with developing a patient partner recruitment and retention policy.

Some key informants (10/29) indicated that SPOR had sufficient flexibility to respond to changing environment or landscape and identified needs. One partner expressed how SPOR has taken projects from an idea to a competitive stage and one patient partner expressed that CIHR did a good job educating patients on what to expect from participating in research. Some key informants (13/35) also reported that SPOR adapted in response to patient partner needs. For example, elements of projects were modified or adapted according to the input and needs of patients and SPOR continued to adapt by responding to identified gaps, such as the need for capacity building and through the process of co-creation with patients.

Design features support the achievement of intended outcomes.

Overall, program documentation revealed that SPOR's core elements are supporting the achievement of the program's intended outcomes. Long-standing entities, including the SUPPORT Units, Networks, and iCT support the achievement of both SPOR's immediate and intermediate outcomes. SUPPORT Units provide decision-makers and health care providers with the means to connect research with patient needs so that evidence-based solutions can be applied to health care and then shared throughout the country. They also generate new knowledge in POR, exist as established infrastructure across Canada, develop the capacity for POR in Canada, and engage stakeholders in the generation of research and implementation of evidence-based improvements. Similarly, Networks support the achievement of intermediate outcomes, including playing a key role in capacity development aimed at fostering the next generation of patient-oriented researchers. They also contribute to SPOR's intermediate outcomes by focusing on specific health challenges identified as priorities in multiple provinces and territories and generating evidence and innovations designed to improve patient health and health care systems. This supports research evidence being applied and responding to stakeholder needs. The iCT contributes to immediate outcomes, including supporting trialists to develop new POR methods that are low-cost, expected to generate relevant evidence, and that catalyze new partnerships and projects going forward. It also supports progress towards SPOR's intermediate outcome to improve the clinical trials environment in Canada.

The SPOR Capacity Development and Patient Engagement entities, which have been funded more recently in 2019-20, best support achievement of more immediate outcomes given that limited time has passed since their implementation. The Capacity Development entity is intended to address gaps and areas of opportunity identified in POR capacity development in Canada, which facilitates the achievement of strengthening and maintaining the Canadian capacity in POR. The Patient Engagement entity plays a key role in ensuring patient engagement is achieved throughout all levels of SPOR. Together, SPOR's core elements are achieving SPOR's intended outcomes.

Many key informants (11/20) identified areas of improvement in SPOR design, including avoiding duplication and increasing communication. Instances of duplication exist primarily within the SPOR environment itself and include such areas as training, patient engagement frameworks, and evidence synthesis. Further, views among many key informants (11/20) on the connections or collaborations among core elements were mixed. Positive experiences (5/11) included connectedness among the Chronic Disease Networks and SUPPORT Units reaching out to iCT grant recipients. In contrast, negative experiences (6/11) included problems in communication and cohesion, recognition of too much siloed activity within and across SPOR elements, and that connections or collaborations among elements can be hit and miss.

Collaborations between CIHR and SPOR partners face some challenges.

While collaboration between CIHR and partners was generally reported to be satisfactory, challenges remain. The SPOR Summit, last held in the fall of 2018, was identified in program documentation as a key initiative that fostered collaborations between CIHR and SPOR stakeholders with the aim of exploring and sharing their experiences and knowledge promoting POR, highlighting early successes and lessons learned, and learning from experts on subjects such as patient engagement, SPOR capacity development, governance, and KT. Participants had the opportunity to participate in panel discussions, plenary and poster sessions, and network with others involved in POR. Though it aims to meet once every 18 months, the SPOR Summit has not met since 2018.

Many key informants (18/35) expressed satisfaction with the extent of collaboration with partners identifying the opportunities provided for connection between SUPPORT units and the platforms. Conversely, a few key informants (2/35) expressed some communication and partnership challenges, such as not having their work recognized or endorsed, or not understanding how best to work together.

From a patient partner perspective, many key informants (9/18) expressed satisfaction with SPOR patient engagement. At the same time, key informants offered several areas for improvement to collaboration, much of it centered on improvements to patient engagement. While considerations when paying patient partners in research are included within SPOR's Patient Engagement Framework, many key informants (9/18) repeatedly indicated a lack of harmonization or policy regarding patient compensation. These findings were consistent with evidence from the case studies that found that compensation practices appeared to vary across SPOR entities and projects based on difference guidelines and policies developed within the respective jurisdictions. In addition, a few patient partners (2/18) expressed the need for safe, supportive, and respectful environments to improve collaboration with recognition that researchers on occasion can be intimidating and unappreciative of patient involvement. A few key informants (3/18) also provided input on the need for CIHR improvements to coordination of operations in terms of consistency across the various elements of SPOR and CIHR.

Patient engagement does not adequately support recruitment of diverse patient partners.

SPOR's current approach to patient engagement does not adequately support recruitment of diverse patient partners, with some patient partner groups disproportionately underrepresented in SPOR research. Though there is evidence of incorporation of EDI and GBA+ considerations into the design of SPOR within program documentation, surveys and key informant interviews found a lack of diverse patient partner representation. Program documents revealed evidence of incorporation of EDI and GBA+ in the design of SPOR, research projects led by or involving Indigenous researchers and partners, integration of Indigenous methodologies into SPOR research, as well as research aimed at reducing gender disparities, and the involvement of Sex and Gender-based Analysis Plus (SGBA+) champions in core elements.

The Saskatchewan SUPPORT Unit reported engaging with Indigenous patients and family advocates throughout every phase of projects aimed at exploring health challenges faced by Indigenous patients. For example, the unit funded a project aimed at better understanding and advocating for Miyo-Mācihowin (good health and well-being) among Indigenous Peoples living with Inflammatory Bowel Disease. The Ontario SUPPORT Unit also reported engaging in several projects on Indigenous health, either led by or involving Indigenous partners. For example, two studies were conducted, one on the quality of end-of-life cancer care for First Nations people in Ontario and another on the health determinants and outcomes of Inuit living in Ottawa, Canada.

There were also examples of GBA+ incorporation in the design of SPOR core elements. Approximately half (five out of nine) of SPOR SUPPORT Units reported SGBA+ champions in the 2019-20 reporting year. The remaining SUPPORT Units intended to recruit a SGBA+ champion or partner to meet SGBA+ needs. Other SPOR core elements (i.e., Networks and Enabling Platforms) also have SGBA+ champions. For example, Can-SOLVE CKD, IMAGINE, CHILD-BRIGHT and DAC have established SGBA+ champions and the CPN has commenced addressing sex and gender in research with a SGBA+ champion who is a member of the Patient-Oriented Research Committee. These champions participate on working groups and work to integrate sex and gender considerations into research project design, analysis, and dissemination.

This was supported by SPOR stakeholders surveyed, in that they felt that the SPOR research project(s) they were involved in embedded EDI into all aspects of research to a moderate to great extent (M = 3.4, SD = 1.2, n = 154). Further, they generally did not report experiencing any EDI-related barriers (e.g., being a visible minority, gender, being an Indigenous person, or being a person with one or more disabilities) to participating in SPOR (Ms = 1.1-1.7, SDs = 0.6-1.0, ns = 6-175).

However, survey findings suggest a lack of diverse representation among patient partners participating in SPOR, and barriers to Indigenous partners participating in SPOR. Patient partner diversity, specifically the engagement of underrepresented patient partners, was the most frequently cited gap (n = 13) in SPOR identified by SPOR researchers (see Figure 4: Needs Not Addressed by SPOR Reported by Researchers). This was supported by the demographics of the patient partner sample obtained for the survey: patient partners who responded to the survey were Caucasian (87%, n = 34), women (62%, n = 24), and between the ages of 42 and 80 years old (M = 66.0, SD = 9.7, n = 36). Similar to the findings of this evaluation, Abelson and colleagues (2022) found that patient partners working across health system settings in Canada predominantly identified as female (77%), white (84%) and university educated (70%). The two patient partners surveyed who self-identified as Indigenous (100%, n = 2) reported experiencing barriers to participating in SPOR related to being Indigenous to a moderate extent.

Survey findings were supported by many key informants (15/32) who expressed concerns about the lack of diversity and inclusion, particularly the lack of engagement of racialized, Indigenous, disabled or other marginalized patient partners and that the majority of patient partners were older, retired, white women. Many key informants (15/32) also identified a number of GBA+ implementation challenges, mainly, the lack of diversity among patient partners (7/32) and that SPOR could do more to communicate GBA+ implementation guidance (7/32). For example, one patient partner described themselves as the only working age, male, disabled, and person of colour engaged in SPOR research and stated the SPOR is relying too much on the people who just happen to show up. Another patient partner noted that they were generally the only Indigenous partner engaged in the research. Some key informants (7/32) expressed that there was an opportunity to do more to improve the diversity and inclusion of patient partners by acknowledging that there is a problem and developing and communicating GBA+ implementation guidance.

SPOR's governance structure is not meeting its current objectives and lacks adequate patient partner representation.

SPOR's governance structure is not meeting its current objectives and lacks adequate patient representation. The NSC has not met since 2018 and a few key informants believed it generally provided advice rather than steering the SPOR program (3/22). Despite almost all key informants (19/22) attributing the NSC to providing advice to CIHR, having adequate representation, and the meetings being useful, the governance challenges identified overshadowed any of the past successes of the NSC.

Many key informants (13/22) described the SPOR governance not working, confusing or absent as being very problematic. For example, key informants (13/22) described how it was not entirely clear how decisions were made as to the distribution of SPOR funding and that the governance structure did not do much in terms of oversight of the SPOR entities with major issues being left to the SPOR management team at CIHR to resolve. Some partner key informants (3/8) also expressed confusion about SUPPORT Units and Networks governance in terms of not knowing if partnering should happen with a Network or a provincial SUPPORT Unit or both. A few key informants (3/22) noted that oversight rested principally with the SPOR management team at CIHR without patient or partner representation. In addition, a few key informants (2/22) spoke to the challenges of working with different federal, provincial, and territorial health systems and that consideration needs to be given as to how provinces and territories influence the direction that SPOR takes. Most importantly, some key informants (6/22) indicated that the lack of representation of patients and partners in SPOR governance was a critical issue, particularly for a program with a goal to include the active collaboration of patients, providers, researchers and decision-makers.

Key informant interviews with experts from the Patient-Centered Outcomes Research Initiative (PCORI) and Consumer and Community Involvement (CCI) in Australia provided insights on the engagement of patients and partners in governance. Both PCORI and CCI have citizen or patient and other stakeholder representation on governance bodies. PCORI has patient representation and people with lived experience on the Board of Governors, various advisory committees, merit review panels in terms of assessment of applications, and peer review panels for when draft final research reports are produced, as PCORI is required by legislation to publish and make public as lay summaries all the findings from every funded research project. In comparison, CCI has consumer advisory group involvement in governance at the state and national levels.

There is an opportunity to inform an organization-wide patient engagement research funding model.

A comparative review of international POR organizations suggests that using SPOR to inform an organization-wide patient engagement research funding model, in which patient and public engagement in all research programs is either encouraged or mandated, could optimize CIHR's investments in SPOR.

Currently, SPOR closely resembles PCORI in the United States, a specialized government research funding program for POR. PCORI key informant interviews revealed that PCORI's approach to public and patient involvement in research is at three levels – engagement with organizations, programs, and individual investigators, clinicians and patient partners or family members. In contrast, the CCI key informant described the approach in Western Australia to consumer and community involvement in research as primarily focused on capacity building and connecting people with lived experience with research opportunities. CCI has supported the grant review panel process to ensure that consumer and community involvement is an important criterion of the grant review scoring matrix for the National health and Medical Research Council and guidelines for public involvement in research have existed for a considerable period of time in Australia. Similarly, the Future Health Innovation Research Fund in Australia has recently mandated the involvement of the public and patients in research. These efforts have resulted in a rapid increase in CCI membership, from approximately 2,000 members 18 months ago to over 6,500 at the time of the interview (January 2023).

The comparative review revealed challenges among PCORI's specialized government research funding. With public and patient involvement being an unfunded mandate in PCORI's model, there are fewer resources to support and better understand public and patient involvement than if it were mandated. The lack of resources and mandate specifically for public and patient involvement means there is no way to ensure consistency and quality of involvement across PCORI; these challenges are similar to those faced by SPOR. Moving to an organization-wide model that mandates public involvement could address these challenges.

Further, an organization-wide model is more closely aligned with CIHR's new Strategic Plan than SPOR's current specialized government research funding model and would assist SPOR's progress towards achieving a cultural shift towards POR. CIHR's new Strategic Plan (2021-31) champions a more inclusive concept of research excellence that recognizes patients, the public, providers, decision-makers, and other users of research outputs as active collaborators throughout the entire research process. This strategy falls under Priority A of the new CIHR Strategic Plan to "Advance Research Excellence in All Its Diversity". Further, four key informants expressed views on alternative approaches to SPOR including the broadening of patient and public involvement in research beyond SPOR to all CIHR research funding opportunities. Almost all CIHR staff, SPOR entity leads and knowledge user key informants (14/17) interviewed shared thoughts on the evolution of SPOR that focused on the potential for informing and implementing POR across CIHR, the evolution of patient partners in decision-making, and the future sustainability of the SPOR Strategy. The CIHR Strategic Plan, evidence from the comparative review of organizations, and key informant interviews support an organization-wide patient engagement research funding model.

The previous evaluation's recommendations were not fully addressed.

The document review highlighted that according to the 2020-21 and 2021-22 annual updates to the SPOR Management Action Plan, only three out of the six recommendations made in the 2016 Evaluation of SPOR have been fully implemented. SPOR management state that they have fully implemented the evaluation's recommendations that CIHR should: 1) "increase efforts to strengthen SPOR's role in a common agenda for change to POR", 2) "provide strategic guidance regarding how SPOR elements are to work together toward achieving the Strategy's intermediate and long-term outcomes", and 3) "communicate plans for moving beyond the initial five-year funding period to manage sustainability expectations for CIHR investments in SPOR" and "provide clear communications regarding SPOR funding and options beyond the current five-year funding commitment to some elements". As of 2021-22, the remaining three recommendations from the 2016 Evaluation are partially implemented, with some steps taken towards full implementation.

There is limited evidence indicating that performance data are being used to inform decision-making.

There is limited evidence indicating that performance data are being used to inform decision-making regarding CIHR's implementation and optimization of SPOR. The first SPOR Evaluation, completed in 2016, recommended revising the initial SPOR performance measurement strategy to better measure impact in the second five-year cycle Evaluation of SPOR (CIHR, 2016). The strategy was revised with guidance from the SPOR WG and other SPOR stakeholders in April 2018. Given the partnered nature of the SPOR program, the SUPPORT Unit performance measurement framework and corresponding indicators were collectively built with the SUPPORT Unit performance measurement leads and approved by SPOR WG. The annual reporting template that is used to collect the data for these indicators is also revised with SUPPORT Unit performance measurement leads on an annual basis to reflect changing reporting needs. Based on the current document review findings, performance data are currently being collected annually from five out of the six SPOR core elements, including all SUPPORT Units, Networks, the iCT, the SEA, and the CDP, but there is limited evidence of how the performance data collected are being used for decision-making regarding CIHR's implementation and optimization of SPOR.

Challenges exist with the current management of performance measurement data.

Challenges exist with the current management of performance measurement data including lack of clarity regarding performance indicators, inconsistent or missing indicators, double-counting, introduction of new indicators at the end of the reporting period, burden, and lack of alignment of Network and SUPPORT Unit work plans with the reporting requirements developed by CIHR.

In general, SPOR elements recognize the importance of performance monitoring for ongoing management of operations, planning and decision-making. However, the key informant interviews and review of documents revealed challenges associated with annual reporting and performance measurement including: lack of clarity regarding some categories of measurement (e.g., supervision versus mentoring activities); inconsistent or missing reporting categories (e.g., research professionals); double-counting and lack of clear guidance on how to avoid double counting; introduction of new reporting categories at the end of the reporting period; and performance reporting burden despite the process being co-developed by CIHR and the core elements. A few SUPPORT Units also stated that CIHR's reporting requirements do not align with their Unit's work plans and therefore do not capture the full range of SUPPORT Unit activities or their impacts. For example, an independent evaluation report commissioned by the Quebec SUPPORT Unit described the performance reporting experience as cumbersome.

Some key informants (7/20) provided comments regarding performance measurement in the context of the annual reporting process. A few key informants found the annual reporting process to be too burdensome (2/20) with no clarity on its use, not necessarily aligned to provincial and/or SPOR entity priorities (1/20), and that increased emphasis needed to be placed on measuring SPOR impacts (3/20) rather than activities or immediate outputs. Finally, one key informant commented that there is little reliability on how patient engagement is measured.

Performance

Key Findings:

SPOR's core elements are contributing to the achievement of immediate outcomes.

SPOR's core elements are contributing to the achievement of immediate outcomes, including the generation of new knowledge, infrastructure, capacity development and engagement of patients and stakeholders.

SPOR is generating and disseminating new knowledge.

SPOR is generating and disseminating new knowledge as evidenced by the number of KT products (e.g., peer-reviewed journal articles, social media campaigns, conference presentations) produced by the core elements based on the most recent annual report in scope of the evaluation (2019-20) and trends over the evaluation period.

Overall, new knowledge is generated through peer-reviewed journal articles. For 2019-20, SPOR core elements produced a total of 1,531 peer-reviewed journal articles, with the SUPPORT Units directly producing approximately two-thirds of total peer-reviewed journal articles (n = 1,057). The bibliometric analysis revealed that publications funded by SPOR performed better than the global set of publications in POR, according to the citation indicators. SPOR publications received three times more citations than average (average relative citation of 3.08), and close to 26% of these publications were among the 10% most cited worldwide (2.6 times higher than expected), 16.2% were among the top 5% (3.2 times higher than expected) and 4.7% were among the top 1% (almost five times higher than expected).^{Footnote 7}

The findings from the modified Delphi panel of experts for the KRL assessment of a sample of SPOR knowledge products found that two-thirds of SPOR knowledge products are at the 'application' level of scientific maturity and that 20% and 14% are at the level of 'foundational' and 'real-world', respectively. These findings indicate that most SPOR knowledge products are mainly classified as providing applied knowledge with the potential to improve individual or public health.

Knowledge is also generated and disseminated by SPOR core elements through other KT products and events. In 2019-20, SPOR core elements directly produced a total of 999 conference presentations, 881 reports/technical reports, 647 social media campaigns, 494 KT-related workshops, meetings, and webinars, 413 plain language publications, 307 educational materials, 123 online KT tools, and 26 books/book chapters. The variety of KT products produced by SPOR core elements, particularly plain language publications and KT-related workshops, meetings, and webinars, demonstrates an emphasis on increasing accessibility of information to various audiences. For example, the IMAGINE Network developed a "Quality Indicators in Inflammatory Bowel Disease Care" initiative and clinical practice guidelines for patients with irritable bowel syndrome. A preliminary analysis of these publications using altmetric (metrics and qualitative data that are complementary to traditional, citation-based metrics) revealed that IMAGINE's KT outputs were tweeted by 2,269 users, received 269 publication citations, were mentioned by 269 bloggers, picked up by 61 news outlets, and referenced four times on Wikipedia, demonstrating the reach of these outputs.

Research platforms and other types of research infrastructure are established.

Program documentation from the period under review indicates that several different types of SPOR infrastructure supports (i.e., research platforms, resources) have been established or are currently under development.

Infrastructure has been established for advancements in data availability. In 2020-21, the SUPPORT Units received a total of 829 data access requests, a 25% increase from the previous year. There were several examples of infrastructure to support data access within program documentation. The Newfoundland SUPPORT Unit has established health information and data analytics platforms that have provided faster computation, increased data storage security and the ability to create customizable software solutions. Through a strategic partnership and investment with the Institute for Clinical Evaluative Sciences (IC/ES), the Ontario SUPPORT Unit has expanded its repository of linked datasets (from 40 to 60) to enhance accessibility of its data and analytics platform, establish new data sharing agreements and establish novel data and analytics partnerships (e.g., Vector Institute). Case studies revealed that the SPOR CDP and Data Platforms services offered through the SUPPORT Units are on track to advance data availability and support researchers in breaking ground in multi-jurisdictional research projects through developing partnerships with data centers, supporting shared learning, and providing navigational/coordination support to researchers.

SUPPORT Unit annual reports also indicate that progress has been made to establish infrastructure for KT. For example, the Alberta SUPPORT Unit's KT Platform has developed a "living lab concept" (the Advancing Implementation Science in Alberta Initiative) aimed at advancing and accelerating the application of POR results into health care practice for improved patient health and system performance.

Case studies indicate that SPOR infrastructure has helped to forge connections with priority stakeholder groups in POR, including knowledge users, through collaborative events offered by the SUPPORT Units and Networks (e.g., policy roundtables, Bridge Events, learning series) and by PIHCI and the SEA. Case studies also showed coordinated action across SPOR entities through advancing infrastructure such as the CDP and NTE. The launch of the NTE has facilitated new dialogue between the SPOR entities to develop a sustainable and common path for training and capacity building in POR going forward. This has supported SPOR partners to identify common priorities/needs (e.g., need for sustainable mentorship) and reduce duplication based on activities/resources advanced by each entity. In addition, the CDP Data Access Support Hub, NTE and SEA infrastructure and support services advanced by SPOR, had specific impacts for research trainees and early career investigators. The infrastructure offered by the CDP Data Access Support Hub is expected to support early career researchers and researchers new to multi-jurisdictional research. The SEA is supporting early career investigators to expand their network of collaborations, disseminate research, and collaborate with patient partners and other stakeholders on evidence synthesis requests. Finally, the NTE is expected to centralize and optimize training for trainees and early career investigators in POR and other competency areas.

Administrative data indicate that the SPOR program was successful in attracting partnership investments and for every dollar that CIHR committed to SPOR, the competition and applicant partners committed $1.16 (Table 3: SPOR Partners Commitments for Funded Projects), where competition partners are organizations that partnered with CIHR at the competition-level to contribute financially or in-kind to the Funding opportunities while applicant partners are individuals and organizations that partnered with grant recipients directly to contribute cash and/or in-kind resources to the funded research. Some elements were better at leveraging partner dollars than others: Enabling Functions attracted $1.13 for every CIHR dollar while Clinical Trials secured $1.34 and SPOR Networks secured $1.48. Across all competitions, SUPPORT Units and Patient Engagement leveraged $1.09 and $0.71, respectively, for every dollar committed by CIHR. When only the competitions with an applicant partner matching requirement were considered, SPOR program overall continued to exceed the matching ratio with $1.28 for every CIHR dollar, and the ratio of leveraged partner dollars increased or remained the same across element. The SPOR Networks and Clinical Trials elements were most successful in attracting funding beyond the mandatory 1:1 requirement, securing $1.53 and $1.76 dollars for every CIHR dollar, respectively. Only the Patient Engagement element did not meet the 1:1 matching requirement from applicant partners, reaching only $0.85 for every CIHR dollar.

To estimate leveraged partner funds across SPOR competitions and the matching ratios for SPOR competitions with an applicant partner requirement, the applicant partner investments were taken as the planned applicant partner contributions, cash and in-kind, at the time of application from the Matching Contribution Verification Tables. Despite applicant partnership being a central component of SPOR, the process for tracking planned applicant partner commitments is resource intensive, requiring the manual extraction of information on actual partner investments from individual grant reports. Consequently, it is prone to inconsistencies in how the data are compiled and summarized, making what data are available challenging to analyze quantitatively. CIHR has no system-enabled mechanism for tracking actual applicant partner investments over the course of the grant nor was a process in place during the specific timeframe of this evaluation to systematically compile applicant partner investments across all SPOR elements.

Capacity in POR is being developed.

Capacity in POR has been developed as evidenced by patient engagement training opportunities offered across SPOR core elements and by trainees involved in SPOR research. In terms of patient engagement training opportunities offered, SUPPORT Units, Networks and SEA offered a total of 10,532 training activities to 112,185 individuals between 2016-17 and 2019-20, with 2,221 activities to 37,429 individuals in 2019-20.^{Footnote 8} During the 2019-20 reporting period, the SUPPORT Units provided 1,388 training activities reaching 22,875 participants compared to 1,850 training activities reaching 10,703 people in 2016-17. Interestingly, the number of training activities offered by SUPPORT Units remained relatively stable while the number of individuals attending has consistently increased over the years. For Networks, the number of training activities offered consistently increased from 2016-17 to 2019-20, while the number of individuals receiving training varied across years (see Figure 5: Number of PE Training or Mentoring Activities Offered by SUPPORT Units, Networks, and SEA, 2016-17 to 2019-20 and Figure 6: Number of Individuals Receiving Training or Mentoring in PE by SUPPORT Units, Networks, and SEA, 2016-17 to 2019-20).

All SPOR recipients surveyed (100%, n = 92) reported including trainees in their SPOR-funded research. See Figure 7: Number and Type of Trainees Reported by Recipients for a breakdown of the number and type of trainees involved in SPOR research as reported by SPOR researchers. SPOR recipients who involved trainees indicated that trainees were involved in research and professional skill development to a great extent on average (M = 4.1, SD = 1.0, n = 71; M = 4.2, SD = 1.1, n = 71, respectively). Surveyed SPOR recipients indicated that trainees were involved in networking and/or collaboration activities to a moderate extent (M = 3.8, SD = 1.2, n = 71), and interdisciplinary research opportunities to a great extent (M = 4.2, SD = 1.0, n = 71). Overall, trainees surveyed were greatly satisfied with the training they received during their involvement in SPOR research (M = 4.2, SD = 0.9, n = 37). Similarly, trainees on average felt that the training they received benefitted them to a great extent to very great extent (M = 4.2, SD = 0.9, n = 37), and that they will very likely stay in POR (M = 3.9 out of 4, SD = 0.3, n = 39).

Many key informant patient partners (5/9) expressed satisfaction in the quality of training and mentoring offered, while some (3/9) indicated they had not received any formal training. Cited suggestions for improvement of training offered including the need for focused patient engagement workshops (1/9), ongoing mentorship, contact and support for patient partners with illness (2/9), training from an Indigenous perspective (1/9), and that on occasion training provided was too focused on SPOR rather than POR in general (1/9).

Patients are being engaged in the generation of research knowledge.

Patients are engaged in the generation of research knowledge and implementation of evidence-based improvements, as evidenced by the extent and nature of engagement of patients in research. While there is evidence of engagement of patient partners in all aspects of research, there are opportunities to improve the level of patient engagement in research to avoid the perception of tokenism.

Within program documentation, the SUPPORT Units reported having engaged patients and members of the public in meaningful ways throughout the research process. Areas of patient involvement include: governance; building capacity for POR through training and dissemination; connecting with patient and community groups; ensuring EDI for patient partners; supporting POR and patient engagement; and, patient partner leadership activities.

When surveyed directly, between 28% and 49% of SPOR patient partners that responded indicated that they were involved in various phases of the SPOR research project, including in the development of the research idea/question (46%, n = 18), development of the research proposal (49%, n = 19), data collection phase/project implementation (44%, n = 17), interpretation of results (49%, n = 19), KT activities (49%, n = 19), and training/supervising research staff and trainees (28%, n = 11). Notably, patient partners' reported involvement in these research activities was lower than researchers' reports of patient partner involvement in the same activities (see Figure 8: Patient Involvement Reported by Patients vs. Recipients). For example, 67% of researchers reported patient partner involvement in the development of the research idea/proposal compared to just 46% of patient partners reported involvement in this activity.

Despite almost all SUPPORT Units and Networks reporting that patients were engaged at the highest level ("Empower")^{Footnote 9}, less than 20% of patient partners reported being engaged at this level. Instead, patient partners on average reported being engaged at the lowest levels of "Inform" or "Consult" in the development of the research idea/question (M = 1.8, SD = 1.6, n = 24), development of the research proposal (M = 1.6, SD = 1.7, n = 24), data collection phase/project implementation (M = 1.5, SD = 1.8, n = 24), interpretation of results (M = 1.7, SD = 1.6, n = 24), and KT activities (M = 2.1, SD = 1.8, n = 24).

Despite reporting being engaged at the lowest levels (inform/consult), over half of stakeholders surveyed felt that their involvement in the SPOR research team(s) had an impact on the research to a moderate to great extent (M = 3.7, SD = 1.2, n = 148), with patient partners reporting the impact of their involvement to a similar extent (M = 3.4, SD = 1.0, n = 34). Some patient key informants (3/9) even reflected on how their experiences had a positive impact on their well-being, empowerment, and ability to self-advocate. They conveyed that patient partnership has had a significant impact not only on the knowledge gained about research but on their own sense of confidence and empowerment to ask questions and be able to advocate on behalf of patients.

At the same time, key informants highlighted challenges to patient engagement strategies. They expressed the need to support meaningful involvement of patient partners in all aspects of the research as opposed to tokenistic roles (7/27), the need to further train both researchers and patients on meaningful patient engagement in research (7/29), and the need to align researcher and patient partner needs (2/27). Other observations included a SPOR entity lead indicating the importance of evaluating patient engagement at the end of each funded grant and two patients commenting on how researchers or investigators have been enriched by the patient engagement experience. In addition, a knowledge user reflected that patient engagement could be brought to the next level by integrating it into the overall grant application process with guidance on reaching out to SPOR SUPPORT Units as needed. Some patients (3/9) highlighted challenges with patient compensation.

Stakeholders are being engaged.

Program documentation, surveys, and key informant interviews indicate that stakeholders are engaged in SPOR. Program documentation from the SPOR core elements revealed engagement with a variety of different key stakeholders, with more than half of the core elements reporting having engaged with the following types of stakeholders: health system/care practitioners, health system/care managers, health organizations, federal and provincial representatives (including policy-makers), community organizations (including policy-makers), charitable organizations, industry, the media, researchers and academics, and research funding organizations. Similarly, surveyed SPOR researchers indicated having involved a variety of different key stakeholders in their research projects. For example, SPOR researchers reported an average of ten academic partners (SD = 18.0, n = 63), four public sector partners (SD = 6.0, n = 44), three non-profit sector partners (SD = 4.3, n = 44), two government partners (SD = 3.5, n = 41), and one private sector partner (SD = 1.6, n = 44) on their research project(s). Further, stakeholders reported that the SPOR research project(s) they were involved in were interdisciplinary to a moderate to great extent (M = 3.7, SD = 1.1, n = 145), and some key informants (3/9) indicated that stakeholders were engaged in forums such as policy roundtables and/or research relevance committees.

SPOR's core elements are contributing to the achievement of intermediate outcomes.

SPOR's core elements are contributing to the achievement of intermediate outcomes, however, there are opportunities to strengthen contributions.

Research evidence is being applied.

Research evidence is being applied, as illustrated by guidelines, clinical practice, managerial decision-making, and policy documents citing SPOR-funded research. Several projects across the SUPPORT Units and Networks achieved impacts in primary care reform, cost avoidance, and re-design of services, whereas other projects were on the pathway to impact through expected scale and spread or development of new partnerships that were leveraged going forward. Furthermore, the case studies found strong evidence to suggest that the pan-Canadian infrastructure offered by the SEA is generating relevant and timely knowledge products, which are influencing practice, policy, and public health guidelines in a range of settings (e.g., international organizations, federal/provincial/territorial governments, health systems) and have potential to positively impact health outcomes. There is also evidence of SPOR research informing policy with SPOR publications published between 2011 and 2019 being cited within policy documents approximately 2.7 times more than the global average. The analysis revealed that SPOR publications had been cited by government reports (e.g., the Public Health Agency of Canada [PHAC]) and grey literature. Findings across these activities/projects point to promising impacts of research generated through SPOR, however, it was not always possible to assess the resulting impacts from practice and policy change, which may point to the need for more time to realize this impact.

There were several examples of SPOR research informing guidelines, clinical practice, and/or managerial decision-making also found within program documentation. The B.C. SUPPORT Unit supported the evaluation and implementation of a clinical toolkit for patients on clozapine at the Vancouver General Hospital. In the Maritimes, the New Brunswick Department of Health required help from the Maritimes SUPPORT Unit to inform their response to COVID-19. The Northwest Territories SUPPORT Unit was requested by the Government of Northwest Territories Department of Health and Social Services to develop culturally safe communications on the history of pandemics on Indigenous Peoples in the Northwest Territories for front line staff.

Furthermore, most SPOR researchers surveyed (70%, n = 79) reported participating in decision-making bodies (e.g., policy networks, boards, advisory groups) to at least some extent (M = 3.0, SD = 1.2, n = 76) and surveyed stakeholders reported that the SPOR research they were involved in has been used to inform guidelines, clinical practice, or decision-making to a moderate to great extent (M = 3.4, SD = 1.2, n = 141).

This finding is supported by several case study key informants across SPOR entities, who generally agreed that SPOR projects/initiatives are providing valuable evidence to decision makers in-line with their needs, however more time may be needed to assess tangible impacts. To illustrate, case study key informants with the SEA outlined several projects across a range of topic areas that supported knowledge users at provincial, federal, and international levels to address evidence needs. Furthermore, there was evidence from PIHCI Network projects that research findings are supporting knowledge users with policy redesign in areas such as centralized waiting lists for primary care and reimagining health care delivery to reduce health care costs. At this time, the outcomes of these policy designs are unknown, however projects such as these outline the potential for SPOR projects to improve patient care outcomes through health system change (see project highlight box).

Some examples from the SUPPORT Units (e.g., Newfoundland and Labrador SUPPORT Unit, Northwest Territories SUPPORT Unit, Maritimes SUPPORT Unit) were provided within the case studies where support was provided to local governments to re-imagine health service and program delivery in-line with population needs. As an example, the Northwest Territories Healthy Family Program Renewal Project led by Hotıì ts'eeda and supported the Northwest Territories Department of Health and Social Services will implement a new parenting and childhood program that is rooted in community-connected, Indigenous strengths-based, and culturally competent programming. Findings since the renewed implementation are unknown, though key informants indicated that the project recommendations were well-received (e.g., findings were relevant and logical) and supportive of the department's needs.

Infrastructure and support services are responding to the needs of stakeholders.

SPOR advanced an expansive range of infrastructure based on the priorities/mandates of the respective entities and needs of stakeholders. Findings across program documentation, surveys, and activities/services examined as part of the case studies indicate that SPOR infrastructure is generally aligned with stakeholder needs, particularly with respect to addressing barriers to data access (e.g., across and within jurisdictions), meeting routine and priority evidence needs to drive decision making, supporting coordination across partners (e.g., across jurisdictions, SPOR entities, stakeholder groups), and meeting stakeholder-specific needs.

Based on program documentation and survey findings, stakeholders' views were mostly positive regarding the extent to which their needs have been met by infrastructure and support services; however, there were some noted reservations. Specifically, SPOR stakeholders surveyed report that the SPOR research project(s) they were involved in are responding to their needs as a stakeholder to a moderate to great extent (M = 3.8, SD = 1.1, n = 155); the same was true for the needs of their organization (M = 3.8, SD = 1.1, n = 120), and patients (M = 3.8, SD = 1.2, n = 37). This aligns with findings from the document review, for example, in the Alberta SUPPORT Unit annual report, stakeholders agreed that infrastructure and support services respond to stakeholder needs, while others noted more time is needed to meet health system needs and that Alberta SUPPORT Unit serves clients on an as-needed basis rather than an overall health system focus. Several types of research infrastructure and support services (e.g., clinical trials, KT) have been established through SPOR, however, the growing demand for SPOR data may be exceeding the capacity of the existing infrastructure.

Case study findings related to infrastructure (e.g., SUPPORT Units, Networks, CDP, SEA, NTE) suggest that these services are streamlining/facilitating data access, supporting knowledge users to access high-quality evidence, and building new linkages between stakeholder groups, jurisdictions, and SPOR entities. While some infrastructure is newly developed/launched (e.g., NTE, CDP, SEA), findings to date suggest that these services are aligned with stakeholder needs and are expected to address known gaps (e.g., coordinated capacity for evidence synthesis, data access, and training and capacity building). Furthermore, SPOR developed and advanced several pan-Canadian initiatives to support collaboration and shared learning, reduce duplication, and streamline processes through the CDP, SEA, NTE, and PIHCI.

While case study key informants highlighted the ongoing complexities to accessing data from other jurisdictions, there is evidence that SPOR infrastructure is addressing gaps in data access through building capacity within SUPPORT Units and supporting collaboration across jurisdictions. Evidence from case studies show that SPOR has also developed supportive infrastructure to effectively meet the routine and priority evidence needs of knowledge users. For instance, several SUPPORT Units led priority research projects and/or responded to knowledge synthesis requests, which supported knowledge users to re-design or adjust government policies and social programs. In addition, the SEA research query service provided national-level support in knowledge synthesis, guideline development, and KT for both knowledge users and patient partners in-line with evidence needs. Overall, evidence from case study key informants across projects/activities indicate that services are generally aligned with the needs of knowledge users as they are generating timely and relevant evidence to support decision-making as well as integrating knowledge users throughout the research process.

With respect to delivering responsive and relevant infrastructure and support services, some gaps and/or unmet needs were identified across case studies. First, engaging and diversifying expertise outside of the 'traditional research model' will further support researchers to engage in POR. For example, a researcher involved with PIHCI spoke to the need for PIHCI to continue to move beyond the traditional model of research in order to realize the full vision of SPOR. Second, the broad range of research projects can limit relevance to knowledge users. Key informants in one case study noted that the differing priorities and needs between knowledge users, researchers, and other team members can lead to projects that are out of scope for knowledge users. Finally, differing priorities, mandates, and readiness across SPOR entities created some challenges for building linkages and aligning needs/priorities for coordinated initiatives. However, key informants emphasized that establishing mechanisms to support discussion and transparency (e.g., through the CDP, NTE) is important to improve coordinated and sustainable action. Findings from the CDP case study identify the unprecedented complexity of the data landscape and the limitations to the CDP as a result of legislative and procedural barriers as well as the differences in data infrastructure across Canada. While key informants indicated that progress had been made, navigating these barriers will be an important challenge going forward.

Progress is being made to improve the Canadian clinical trials environment.

Available evidence suggests that progress has been made in improving the clinical trials environment in Canada including the development of infrastructure for clinical trials. Services, infrastructure, and activities funded through the SUPPORT Units, Networks, CCTCC, and iCT funding opportunities examined in case studies were considered the key SPOR's investment into the clinical trials environment. Evidence from these initiatives suggest that SPOR has made some progress toward this outcome, though more data is needed to fully assess impact. To illustrate, SPOR has developed supportive infrastructure for clinical trials (e.g., research and data services offered by several SUPPORT Units, research networks and recruitment platforms developed across Networks, CTO research ethics harmonization), which are supporting data access and addressing cost, capacity, and efficiency barriers among projects/services explored as part of the case studies. For example, Clinical Trials Ontario (CTO), an Ontario SUPPORT Unit partner, has used the SUPPORT Unit investment to create a single point of access ethics system that continued to grow in reach and use, for example, supporting 235 new studies and 662 center applications in 2018-19. This investment has led to a streamlined single point of access for clinical trials ethics approval in Ontario. The Can-SOLVE CKD Network also launched a re-imagined Canadian Nephrology Trials Network that in 2019 reviewed 11 clinical research study protocols. This national hub for enhancing clinical research in nephrology has a refreshed vision, governance structure, and set of priorities, which placed increased emphasis on patient partners following a 2018 workshop coordinated by Can-SOLVE CKD.

Evidence from an iCT project suggests that SPOR funding is supporting trialists to develop new methods that are low-cost, expected to generate relevant evidence, and that catalyze new partnerships and projects going forward. In addition to this, several activities to support and advance patient engagement in clinical trials were noted (e.g., CTO Decision Aid project, Alberta SUPPORT Unit Pragmatic Clinical Trials Certificate, Can-SOLVE CKD Nephrology Trials Network).

Together, these activities point to some improvements to the clinical trials environment, however there was significant variation across SUPPORT Units with respect to capacity to support trialists. Infrastructure and capacity to support clinical trials varied across SUPPORT Units and over the course of Phase I, with capacity already existing within Ontario, Alberta, and Québec, and growing within Manitoba, Saskatchewan, Newfoundland and Labrador, and the Maritimes. There were gaps in evidence that has resulted in the inability to assess actual improvement in the Canadian clinical trial environment.

Canadian capacity in POR is being strengthened and maintained.

Canadian capacity in POR is being strengthened and maintained however, there are opportunities for SPOR to strengthen the capacity for engaging with representative, equitable, and diverse patient populations (e.g., by re-establishing a governance structure with representation from patients, partners, and funders). The MSSU's Associate Scientist program, which launched in 2018-19, continued to grow and establish itself within the MSSU community, with a 33% increase in members in 2019-20, and the NTE is expected to advance a more comprehensive approach to capacity development, including by offering learning, mentorship, and funding opportunities. The MSSU has also explored additional mentoring opportunities such as creating an inventory of existing mentorship programs in SPOR entities and partner organizations (e.g., Diabetes Action Canada, Beatrice Hunger Cancer Research Institute) and conducted an environmental scan of literature to identify barriers and enablers to mentorship programs that exist around POR.

Similarly, the Alberta SUPPORT Unit's PaCER Training Program (see project highlight box), launched in 2012, is a certificate based, three-course training program aimed at teaching patients how to conduct research projects by, for and with patients.

Case studies found that trainees who completed the PaCER program were engaged in projects that spanned a range of topic areas, including kidney health, youth and e-mental health, and intensive care. All trainees agreed that PaCER prepared them with the necessary skills and expertise to conduct qualitative patient-oriented research, including qualitative research techniques, data management, and ethics protocols. Furthermore, trainees in the case studies spoke to the value of being involved in research projects that aligned with their lived experience and/or interests, some of which led to publications or further opportunities to inform research within their interest area. For example, one trainee explained that their internship project, focused on transfer from ICUs to the hospital, was subsequently leveraged in a CIHR multi-year grant to further investigate transfer experiences. Similarly, a SPOR entity lead noted that some PaCER teams continued to collaborate with the research sponsor (e.g., Alberta Health Services) after completion of the program to further develop and disseminate their research, whereas other graduates were later engaged in governance and patient advisory roles within the health system (e.g., health quality councils, health safety organizations).

The case studies also found that Networks contributed to strengthening capacity building. For example, the Can-SOLVE CKD Network developed the Wabishki Bizhiko Skaanj Learning Pathway. This learning pathway is comprised of five training programs to support patient-oriented kidney research: a toolkit for helping teams do patient-oriented research; Indigenous cultural competency training; storytelling; knowledge mobilization principles; and a general overview of kidney health research in Canada. The programs were developed based on findings from a 2017 needs assessment survey and are rooted in the core values of Respect, Reciprocity, Relevance, Relationships, and Reflection.

Program development is driven by a Training and Mentorship Committee, with representation from researchers, Indigenous and non-Indigenous partners, Can-SOLVE CKD staff, and representatives from the Kidney Foundation of Canada. In addition to this, the Network also offered thesis supervision, mentorship and collaborated with kidney research fellowship trainees (i.e., KRESCENT) to support career development and capacity in conducting POR with attention to culturally safe research (see project highlight box).

Despite evidence from the case studies that the SUPPORT Units are developing resources and training for researchers and patient partners to collaborate in POR (e.g., through workshops, patient skill building), there are ongoing challenges with the level of engagement and readiness for true patient engagement in research. This impedes the strength and maintenance of POR capacity in Canada and indicates that there may be a need to expand the reach of SUPPORT Unit activities to new research communities. Furthermore, it was noted that increasing the diversity of the BC SUPPORT Unit team that is interfacing with the academic community (i.e., increasing patient partner representation, increasing racial diversity) is also an opportunity going forward.

Patient and stakeholder engagement is contributing to the achievement of intermediate outcomes.

Several examples of patient engagement contributing to the achievement of SPOR's intended intermediate outcomes were presented within program documentation and key informant interviews. Many SUPPORT Units and Networks cited meaningful participation of patients in projects. The CPN reported their patient partners have impact on federal and provincial policy by sitting on the federal government's Canadian Pain Task Force, and patient partners are involved in various engagement activities to inform reports to the Minister of Health. IMAGINE's patient partners engaged with healthcare policy advisors and health system administrators to explore the value of results from their projects on rapid learning health systems. The Ontario SUPPORT Unit contributed to a catalogue of organizations in Ontario that are placing patient partnership in research in their organizational structures and have placed patients as decision makers at policy roundtables. The Newfoundland and Labrador SUPPORT Unit has been involved in supporting a key provincial health information custodian in developing a patient engagement plan for their organization that includes basic patient engagement guidelines to their staff.

Some partner, patient and knowledge user key informants (5/22) reported that patient involvement in research is key and that patients had impacted the research through influence on research priorities, outcome measures and applicability of findings. In addition, key informants (5/22) acknowledged that progress has been made to increase the involvement of patients in the selection of research outcomes and the meaningfulness of research; however as noted above, in the context of CIHR the view that patients were not active in decision-making was expressed. Regarding the level of involvement of stakeholders, many partner key informants (4/9) indicated that partners were generally actively involved in research as well as helping to formulate SPOR policy and on occasion treated as funders rather than full partners and that there should be greater partnership accountability.

In order to assess SPOR's engagement with Indigenous communities, one case study focused specifically on the extent to which Indigenous stakeholders were active partners in both research and implementation of evidence-based improvements. This case study found that most SPOR entities involved and/or actively partnered with Indigenous peoples (e.g., researchers, patients, community members, Elders, Knowledge Keepers) in governance activities as well as in the design, delivery, and/or implementation of relevant projects, with a noted increase in capacity to engage Indigenous partners over time. Findings from key informants involved in some Indigenous-focused initiatives suggest that SPOR projects are supporting positive outcomes for individuals, families, and communities when initiatives are community-driven, flexible, and self-determined. To further support meaningful partnerships and generate relevant knowledge, key informants saw opportunities to improve ongoing and meaningful inclusion of Indigenous partners (e.g., reducing one-time engagement), reduce engagement burden, and support self-determination in Indigenous POR as priorities going forward.

SPOR is contributing to the achievement of a cultural shift towards POR.

SPOR's core elements are contributing to the achievement of a cultural shift towards POR – a key expected ultimate outcome that should be maintained. For example, case studies found patient partners and/or community members feeling empowered as a result of involvement in SPOR projects, which improves their ability to advocate in the health system and increase self-efficacy. Research is being informed by, and relevant to, patient priorities, which ultimately generates applicable evidence to improve patient care. Case study key informants were generally most equipped and able to assess progress toward this ultimate outcome and almost all agreed that the projects, initiatives, and/or services in question had contributed to a cultural shift within their research teams, organizations/networks, and/or in local institutions.

For example, key informants involved in the Maritimes SPOR SUPPORT Unit highlighted that the SUPPORT Unit activities supported diffusion and uptake of POR in local government and in other institutions (e.g., universities). Similarly, key informants in the Can-SOLVE CKD Network indicated that researchers trained in POR through the Network were modelling POR in other research networks/institutions (e.g., IC/ES, Canadian Nephrology Trials Network), which had led to uptake of patient engagement practices in research outside of SPOR.

Case study key informants (in particular, patient partners) in two SPOR entities suggested that there are opportunities to improve patient engagement practices and that not all researchers/leadership have wholly embraced POR. To illustrate, a patient partner in one SUPPORT Unit felt that patients were still minimally involved or excluded from some activities, and another patient partner noted that staff turnover between Phase I and II had resulted in lost progress and reversion to tokenistic engagement practices. However, patient partners generally agreed that patient engagement had improved over time.

Most key informants (27/35) provided examples of how SPOR was contributing to the achievement of the ultimate outcome of a cultural shift toward POR including patient involvement in the design of research and POR being adopted by researchers outside of SPOR. Given the success of patient involvement in research, some key informants (3/27) expressed that the pace was not fast enough for the goal of culture change and that in the end there should not be the need for SPOR if culture change has been reached. Other comments from key informants related to culture change included a SPOR entity lead reflecting on the growing interest in POR, two partners mentioning SPOR contributing towards building a learning health system, and a knowledge user identifying implementation science and KT barriers within the health system.

There is little evidence of improvement of patient health care experiences, health outcomes or health system outcomes.

At this point in time there is little evidence to demonstrate that SPOR has contributed to the expected ultimate outcomes to improve patient health care experiences, health outcomes or health system outcomes.

Many key informants (17/35) provided insights on the impact of SPOR on improving patient health care experiences and health outcomes. However, approximately half of key informants who responded (9/17) indicated that it was too early to expect overall improvements in patient health care experiences and health outcomes from SPOR funded research as it can take as long as 20 years for change to take place in the health system. At the same time, one key informant indicated that breakthroughs were on the cusp of coming to market, another identified quality of life improvements, one identified practice and health policy changes, one identified improvements to emergency practice, and another identified kidney screening improvements for Indigenous communities.

Evidence from case study key informants, which was supported by document review, suggests that several projects are on the pathway to achieving impact, but more time may be needed to assess achievements. To illustrate, an analysis of projects within the PIHCI Network suggests that some projects will see spread and scale (e.g., PriCare project, SPIDER project), increase capacity for data access and use (e.g., Children's Health Profile and Trajectory Initiative in New Brunswick and Prince Edward Island), or are expected to inform policy/practice changes due to engagement of relevant knowledge users (e.g., Evaluating Older Adult Care Continuums in Alberta and Manitoba, Identification of frailty using administrative and electronic medical record data); however, concrete evidence was not yet captured.

SPOR researchers surveyed reported that their research resulted in improved health system outcomes to a moderate extent (M = 2.7, SD = 1.1, n = 66), and SPOR stakeholders surveyed reported that the SPOR research they were involved in positively impacted health system outcomes to a slightly greater extent (M = 3.5, SD = 1.2, n = 137). Notably, patients surveyed reported that the SPOR research they were involved in had a more positive impact on health system outcomes (M = 3.5, SD = 1.2, n = 30) than SPOR researchers reported. However, just over half of SPOR researchers stated that it is too early in their research to accurately report on ultimate outcomes such as improved health system outcomes (n = 16). Of these recipients, some stated that the COVID-19 pandemic contributed to project outcomes not being at the stage they anticipated (n = 3). A few others stated that their research has successfully resulted in improvements to POR in general, such as new POR research guidelines (n = 2). A few surveyed grant recipients cited research contributing to improvements in patients' health outcomes (n = 3 such as informing cerebral venous thrombosis management guidelines (n = 1), improved awareness of signs of delirium for critically ill patients (n = 1), and the development of WHO guidelines on the "Clinical Characterization and Management of COVID-19" (n = 1).

For improved health system outcomes through evidence-based practice, many key informants (15/35) indicated that translating evidence into practice was challenged by implementation roadblocks (2/15), difficulties engaging policymakers and decision-makers (2/15), and difficulty in determining SPOR's contribution to ultimate outcomes (2/15). Regarding implementation roadblocks, patients expressed how the implementation of research findings is challenged by how overwhelmed front-line health care workers are. One SPOR entity lead and one partner summarized difficulties in engaging decision-makers in terms of willingness and capacity to engage. Key informants generally felt that SPOR has made progress, but that assessing ultimate outcomes was a challenge.

Given that only 14% of knowledge products were considered ready for real-world application based on the Delphi panel of experts for the KRL assessment, SPOR funded research needs to mature, advance and be taken up to achieve the expected ultimate outcomes to improve patient health care experiences, health outcomes or health system outcomes.

The COVID-19 pandemic has had a negative impact on recipients' ability to conduct research.

As expected, the COVID-19 pandemic has had a negative impact overall on recipients' ability to conduct research including reduced laboratory access and opportunities for collaboration. Case studies, surveys and key informant interviews provided evidence of the impact of COVID-19 on SPOR. Many of the SPOR entities (e.g., SUPPORT Units, SEA, Networks) were leveraged to meet urgent evidence needs to inform provincial policy, guidelines, and public health measures. To illustrate, the SEA was a key asset during the COVID-19 pandemic to support decision-makers with time-sensitive evidence needs, at times in as little as five days. In 2020-21, SEA delivered three COVID-19 rapid reviews for the Ontario Ministry of Health and Long-Term Care COVID-19 Evidence Synthesis Table and 12 rapid reviews for decision-makers in high-impact organizations such as the World Health Organization (WHO), Health Canada, PHAC, and others. For example, the SEA responded to the evidence needs of WHO through a rapid review of COVID-19 transmission in long-term care facilities. The evidence generated from the review informed the publication of a clinical practice guideline by WHO and was used by clinicians and organizations internationally. Similarly, some of the SUPPORT Units were engaged by local government, federal agencies, and organizations to provide timely evidence for decision-making. The Maritimes SPOR SUPPORT Unit produced rapid response reports for the provincial Department of Health, rapid reviews on COVID-19 related concerns for the Vitalité Health Network, Health Canada, the PHAC, and nine background summaries for the Nova Scotia COVID-19 Therapeutics and Prophylactics Advisory Group.

Many SPOR researchers surveyed (55%, n = 53) and most SPOR stakeholders surveyed (76%, n = 116) indicated that the COVID-19 pandemic had a negative impact on their research or their involvement with SPOR research. Many SPOR researchers (42%, n = 41) and stakeholders (n = 90, 58.8%) reported adapting or pivoting their research to COVID-19 related research activities. Examples of impacts of COVID-19 on researchers' SPOR projects were delays in research progress, experiencing staffing challenges, competing caregiver demands, and barriers to engaging with healthcare providers or conducting research in healthcare settings. The most frequently cited implication of pivoting to COVID-19 related activities were shifting activities from in-person to virtual (n = 11) and requiring more time to complete research projects than initially anticipated (n = 7). One-third of SPOR researchers (33%, n = 32) anticipate future impacts to their research as a consequence of COVID-19.

Almost all key informants (31/33) provided insights into both the positive and negative consequences of the COVID-19 pandemic. Negative consequences were identified more often than the positive and included: research projects having been cancelled or delayed and the pandemic induced inability to spend project funding (12/33); staff turnover and burn-out (8/33); impacts on the health of individuals, particularly certain population groups, and the health system overall (8/33); challenges with recruitment and maintaining partnerships (7/33); and the challenge of a digital divide with some groups not having equal access to online technologies (1/33). Positive consequences and observations included: how research teams were able to quickly pivot to COVID-19 (6/33); the ability to continue working virtually and online (10/33); for some the impression that access and participation in research improved (4/33); savings of time and money (1/33); and that the pandemic interrupted but did not end POR (2/33).

Many key informants (18/33) provided insights into future anticipated changes to SPOR as a consequence of the COVID-19 pandemic and a few key informants did not anticipate any future changes (3/33). For those key informants that did anticipate changes, increased access to patients due to virtual communications (3/33) and anticipated impacts to the health system (2/33) were identified. A few key informants also thought that patients at risk and living with health conditions (2/33) will be impacted for the future. Finally, key informants (2/33) thought about the future in the context of COVID-19 and the health system and reflected on the need for post COVID-related research such as post-COVID-19 condition and accessibility of needed health services.

Conclusions and Recommendations

Conclusions

Relevance

The evaluation concludes that there is a continued need to prioritize and foster patient-oriented evidence-informed health care in Canada, with evidence of the relevance and benefits of patient engagement on the research process.

SPOR is well aligned with the roles and responsibilities of both the Government of Canada and CIHR. SPOR's objectives are aligned with CIHR's mandate of supporting initiatives that will lead to the improved health of Canadians and a strengthened healthcare system, as well as several priorities outlined in CIHR's previous and current Strategic Plans. SPOR also aligns with government priorities to continuously strengthen the healthcare system outlined in a Mandate Letter from the Minister of Health and the 2021 Speech from the Throne.

The evaluation found that CIHR is well positioned to continue to play a leadership role in SPOR, particularly as a research funder and as a coordinating body or convener. To maximize CIHR's investments in SPOR, it would be beneficial to further increase awareness of both POR and SPOR among members of the health research community, patients, and decision-makers, including a shared understanding of the benefits, challenges, and strategies for effective POR.

Design and Delivery

The evaluation found that SPOR has largely been implemented as planned, with the implementation of SPOR elements evolving with a focus on strategic planning, the development and delivery of new programs and services, and phase II planning for SPOR SUPPORT Units and Networks. Interviewees noted several opportunities to enhance the implementation SPOR, such as increased guidance from CIHR on patient engagement and harmonized patient compensation guidelines.

Monitoring of SPOR's implementation continues to be challenged by gaps in financial monitoring of grants and awards (G&A) expenditures, specifically the absence of unique coding for core elements as well as a lack of documentation for the Foundational Investments reported in the first evaluation, and operational spending, specifically a lack of information regarding direct salary.

While the evaluation found that the design features of SPOR generally support the achievement of intended outcomes, several opportunities to improve the design of SPOR were noted. The current evaluation found that SPOR's governance structure is not meeting its current objectives and lacks adequate patient representation. The NSC has not met in recent years and generally provided advice rather than steering the SPOR program. There is an opportunity for CIHR to re-establish a governance structure to improve SPOR's decision-making on POR.

There is also an opportunity to improve the management and utility of performance measurement data regarding clarity and consistency of performance indicators, streamlining data collection, ensuring alignment of Network and SUPPORT Unit work plans with CIHR's reporting requirements. These improvements could better inform decision-making to ensure CIHR's optimization of the implementation of SPOR.

Further, a comparative review of international POR organizations suggests that using SPOR to inform an organization-wide patient engagement research funding model, in which patient and public engagement in all research programs is either encouraged or mandated, could optimize CIHR's investments in SPOR.

Performance

SPOR met or exceeded the 1:1 matching requirement by leveraging $1.16 in planned partner dollars for every CIHR dollar. However, it was not possible to determine if actual applicant partner investments met the matching requirement as applicant partner investments are not captured by CIHR's data systems nor were they systematically compiled from grant reports during the period covered by this evaluation.

SPOR is currently achieving its immediate outcomes, particularly in the generation of new knowledge, infrastructure, and capacity development. The evaluation found SPOR is generating and disseminating new knowledge as evidenced by the number of KT products produced by the core elements and found that SPOR core elements develop capacity for POR in Canada while being responsive to stakeholders to provide evidence necessary to drive decision making. While there is evidence of engagement of patient partners in all aspects of research, there are opportunities to improve the level of patient engagement in research to avoid the perception of tokenism.

The evaluation also found evidence that SPOR is making progress towards achieving its intermediate outcomes. There were examples of SPOR research informing guidelines, clinical practice, and/or managerial decision-making found within program documentation. Available evidence suggests that progress has been made in improving the clinical trials environment in Canada; however, more data regarding the outcomes of these engagement activities is needed to fully assess impact of participation in trials on patients.

SPOR's core elements are contributing to the achievement of a cultural shift towards POR. Almost all case study key informants agreed that the projects, initiatives, and/or services in question had contributed to a cultural shift within their research teams, organizations/networks, and/or in local institutions. However, more time is needed to assess how SPOR has contributed to the expected ultimate outcomes to improve patient health care experiences, health outcomes and health system outcomes.

Recommendations

The evaluation makes six recommendations aimed at improving the performance of SPOR to achieve its expected results.

Recommendation 1:

CIHR should use SPOR to inform an organization-wide approach to patient engagement in research to continue its leadership role, further investment and sustain progress on the outcome of a cultural shift toward POR.

Recommendation 2:

CIHR needs to do the following to improve the program design and delivery of SPOR:

• Increase awareness of the benefits of POR among members of the health research community, patients, and decision-makers.
• Enhance communications among and across SPOR core elements and CIHR institutes to avoid duplicative efforts, promote cohesion, and enhance partnerships.
• Improve overall program monitoring to ensure that research is delivering on intended objectives, such as the engagement of communities and patients in research and provide feedback.
• Establish consistent priorities, mandates and readiness across SPOR core elements to support linkages, alignment and coordination of initiatives.

Recommendation 3:

CIHR should re-establish an external and internal governance structure for SPOR with defined roles and responsibilities, including better representation from patients, partners, and funders, to improve CIHR's decision-making on SPOR.

Recommendation 4:

CIHR needs to improve patient and community engagement both in SPOR and in research in the following manner:

• Embed equity, diversity and inclusion considerations into the recruitment of patient partners to address the underrepresentation of important patient partner groups in research.
• Harmonize patient compensation standards across SPOR.
• Enhance accountability for meaningful patient engagement.
• Ensure consistency in engagement of Indigenous community members across SPOR core elements.

Recommendation 5:

CIHR should improve the management and reporting of SPOR performance measurement data to better inform decision-making by establishing a clear set of measures to track progress expected outcomes related to patient health care experiences, health, and health system.

Recommendation 6:

CIHR needs to further improve the following aspects of its financial monitoring and coding for SPOR:

• Grants and awards expenditures, especially coding of core elements and tracking of partner contributions.
• Operating and maintenance expenditures, specifically direct salary costs.

Appendix A: Tables

Table 1: CIHR Annual G&A Expenditures on SPOR by Core Element and Unspent Funds, 2010-11 to 2020-2021

Source: CIHR Electronic Information System Data

Table 2: CIHR Planned (based on TB submissions) and Actual Operating Costs on SPOR, 2010-11 to 2020-21

Source: Treasury Board Submissions and CIHR Electronic Information System Data

Table 3: SPOR Partners Commitments for Funded Projects

Source: CIHR Electronic Information System and Matching Contribution Verification Tables

Appendix B: Figures

Figure 1: SPOR Logic Model

Source: Program Documents

Figure 2: SPOR Evolution by Core Elements

Source: Program Documents

Figure 3: Annual Allocations from TB and Annual SPOR G&A Expenditures by Core Element

Source: Treasury Board Submissions and CIHR Electronic Information System Data

Figure 4: Needs Not Addressed by SPOR Reported by Researchers

Source: CIHR Evaluation Unit Researcher Survey

Figure 5: Number of PE Training or Mentoring Activities Offered by SUPPORT Units, Networks, and SEA, 2016-17 to 2019-20

Source: Program Documents

Figure 6: Number of Individuals Receiving Training or Mentoring in PE by SUPPORT Units, Networks, and SEA, 2016-17 to 2019-20

Source: Program Documents

Figure 7: Number and Type of Trainees Reported by SPOR Recipients

Source: CIHR Evaluation Unit Researcher Survey

Figure 8: Patient Involvement Reported by Patients vs. SPOR Recipients

Source: CIHR Evaluation Unit Researcher Survey

Appendix C: Detailed Descriptions of Core Elements

This section provides a detailed description of each of the SPOR core elements.

Support for People and Patient-Oriented Research and Trials (SUPPORT) Units

SUPPORT Units are specialized, multidisciplinary research service centers located in provinces and territories across Canada. SUPPORT Units were created to provide the necessary expertise to pursue POR and help lead reforms in response to locally-driven health care needs. Additionally, they are tasked with facilitating decision-making within the health services setting, fostering the implementation of best practices, and promoting collaboration among researchers engaged in POR. SUPPORT Units provide decision-makers and health care providers with the means to connect research with patient needs so that evidence-based solutions can be applied to health care and then shared throughout the country.

SUPPORT Units are a collaborative effort between CIHR and the provinces and territories, which have a significant role in directing the work carried out by the Units. The SUPPORT Units have dual priorities, in that they align with the provincial and territorial priorities, as well as support the research needs of the other SPOR-funded entities. The SUPPORT units are at varying stages of implementation:

• SUPPORT Units implemented:
  • 2013-14: Alberta, Manitoba, Maritimes, Ontario
  • 2014-15: Newfoundland & Labrador, Quebec
  • 2015-16: British Columbia, Saskatchewan
  • 2016-17: Northwest Territories
• SUPPORT Unit in review stage:
  • Yukon
• SUPPORT Unit in development:
  • Nunavut

Phase I SUPPORT Units are made up of six component areas in addition to the cross-cutting theme of patient engagement:

1. Data Platforms and Services providing access to administrative datasets, data analysts, a central platform for primary data collection, and data ambassadors (e.g., data sharing agreements);
2. Methods for Support and Development providing access to expertise such as biostatistics, epidemiology, and clinical trial design;
3. Applied Health Systems Research, Knowledge Translation, and Implementation Science via activities that put knowledge into action and enhance uptake;
4. Real-World Clinical Trials providing assistance in areas such as innovative design, data management, statistical analysis, ethics approvals and multi-site management;
5. Training and Capacity Development providing training, mentoring and career development for clinical, health services and systems research and methodological patient-oriented researchers; and
6. Consultation and Research Services supporting researchers in areas such as design, measurement, methods development, data analysis, economic evaluation, literature review and scientific writing.

Each SUPPORT Unit has a governance structure, which includes appropriate mechanisms for Patient Engagement. Each SUPPORT Unit must also undertake a demonstration project(s). These are intended to show the benefits of POR, and demonstrate the effectiveness of the model.

In 2013, the SPOR SUPPORT Unit Council (SSUC) was created with the mandate to provide an opportunity for information sharing and collaboration among SUPPORT Units. It does not have powers to compel actions. In addition, the following Working Groups (WGs) that report to the SSUC were established:

• Performance Measurement Working Group (2014);
• Knowledge Translation Working Group (2014);
• Patient Engagement Working Group (2014);
• Capacity Development Working Group (2015);
• Data Working Group (2015); and
• Communications Working Group (2017).

Networks

Networks are pan-Canadian research teams that represent a collaboration of patients, health service providers, policy/decision makers, and health researchers. Networks focus on specific health challenges identified as priorities in multiple provinces and territories, and are intended to pursue research and generate evidence and innovations designed to improve patient health and health care systems.

Each SPOR Network has a governance structure, which includes appropriate mechanisms for Patient Engagement. There are currently seven funded Networks: The ACCESS Open Minds Network, announced in 2014, is the result of a funding partnership between CIHR and the Graham Boeckh Foundation. AOM seeks to improve the care provided to young Canadians (aged 11 to 25 years) with mental illness, by assisting in connecting patients and youth with researchers, health care professionals, and decision-makers in order to foster the translation of research findings into practice and policy. ACCESS Open Minds aims to bring about transformational change in 14 service-delivery sites in communities across Canada, giving youth faster access to services and addressing adolescent and youth mental health and well-being.

The Primary Care Research Network (formerly the Primary and Integrated Health Care Innovations (PIHCI) Network) funded in 2014, is a key CIHR initiative under both SPOR and the CIHR Community-Based Primary Health Care Signature Initiative. As a Network of networks, PIHCI builds on regional and national assets in community-based primary and integrated health care; and is overseen at a pan-Canadian level by a Network Leadership Council (NLC) and supported by a Network Coordinating Office. It aims to foster a new alliance between research, policy and practice to create dynamic and responsive learning systems across the country that develop, evaluate and scale up new approaches to the delivery of horizontally and vertically integrated services within and across sectors of health care (e.g., public health, home and community care, primary, secondary, and tertiary care) as well as outside the health sector (e.g., education, social services, housing). In 2022, the Primary and Integrated Health Care Innovations Network adopted a new name, the Canadian Primary Care Research Network (CPCRN). The new SPOR Primary Care Research Network will integrate and build upon PICHI to expand patient-oriented primary care innovations to new sites, settings, and populations.

Five SPOR Chronic Disease Networks (CDNs) were funded in early 2016 with the objective of translating existing and new knowledge generated by basic biomedical, clinical, and population health research into testing of innovations that can improve clinical science and practice. These Networks also aim to foster policy changes, leading to transformative and measurable improvements in patient health outcomes and improvements in efficiency and effectiveness of healthcare delivery within five years. The CDNs consist of CHILD-BRIGHT, CPN, DAC, IMAGINE, and Can-SOLVE CKD.

The PIHCI member networks, each representing a unique jurisdiction in Canada (including ten provinces and one territory), were funded in three cohorts with start dates between January and September 2015. The first of these cohorts saw the end of their grant term on December 31, 2019.

CIHR offered funding extensions in December 2019 to all PIHCI member networks, as needed, enabling continued operations until September 30, 2021. It is anticipated that the CDNs will have access to funds, because of the slow ramp up and the extended Authority To Use Funds period, into 2022-23 and AOM has been approved for a no-cost extension until March 31, 2024.

The development of the Phase II Network Funding Opportunity was a major focus for CIHR in the second half of 2019-20, with expected launch of a funding opportunity s in early 2021 and funding release as early as fall 2021 (this is anticipatory, as the competition timeline will be developed in collaboration with the CIHR Program Design and Delivery team).

Capacity Development

Capacity Development aims to build capacity for researchers, patients, health care professionals, health system administrators/decision-makers and other stakeholders to work together to conduct POR and to apply the results. It is intended to bring about culture change, link partners together, and effectively support training and career development in POR.

The SPOR Capacity Development Framework was designed in 2015 to encourage a shared vision, key principles, and considerations for capacity development in SPOR. In alignment with this framework, training, mentoring, and career support is to be integrated into the SPOR Networks and SUPPORT Units, with each Network and SUPPORT Unit being required to articulate a training and capacity development strategy. The SEA and the iCT Initiative are also required to articulate a training and capacity development strategy.

To date, most capacity development activities have been carried out by SPOR-funded entities such as the SUPPORT Units and the Networks. Beginning in 2019, CIHR launched the SPOR Capacity Development Initiative and the first of the CDI funding streams to lead a coordinated approach to training and capacity development.

The SPOR Capacity Development Initiative is intended to address gaps and areas of opportunity identified in POR capacity development in Canada. The Initiative is expected to increase capacity in POR and support viable career paths in POR by:

1. Coalescing the community of POR leaders, mentors, coaches and trainees to share and innovate approaches to POR capacity building;
2. Positioning future leaders in the POR community;
3. Integrating POR principles, practices, and findings within Canadian health care contexts; and
4. Incorporating Indigenous, Western, and other paradigms of POR, training, mentoring, and leadership to enrich training opportunities and career development in a culturally safe manner.

The Initiative consists of:

1. A National Training Entity (NTE) (funding opportunity launched in 2019 and funding started in March 2021) to serve as a central body for systematic training and mentoring of POR leaders in Canada. The NTE aims to ensure a more comprehensive, national strategy to building capacity that supports inclusive approaches appropriate to the specific learning needs of various communities and populations in Canada. This work involves moving beyond common and/or siloed approaches to training by engaging stakeholders and/or community members from all relevant disciplines, sectors, geographies and cultural perspectives. Core functions will include:
   1. Serve as a central body for systematic POR training and mentoring in Canada;
   2. Cultivate a community in POR capacity development;
   3. Support mentors and trainees, including Patient-Oriented Research Award recipients and trainees from other SPOR-funded entities; and
   4. Develop collaborations and partnerships in capacity development.
2. A suite of Patient-Oriented Research Awards to support trainees transitioning into independent careers and to assist health systems organizations in integrating POR principles and findings into Canadian health care contexts by developing in-house 'implementation specialists' (i.e., individuals who bring a spectrum of influence in health policy or practices, for example, policy analysts or nurse managers). Recipients of this funding will have the opportunity to collaborate with the NTE and with existing SPOR-funded entities. These awards include:
   1. Transition to Leadership Stream (funding started April 2020): These two-phased, six-year awards aim to support the timely transition of individuals in a research fellowship into an independent research career. Phase 1 of the award provides a stipend and an allowance for professional development and research. Phase 2 of the award will provide recipients of Phase 1 with salary support and a research allowance to launch their POR career. The competition is complete and the 22 Award recipients are starting their Phase 1 Award during the period of April 1, 2020 to January 1, 2021, and transition to Phase 2 is underway.
   2. Health System Stream (funding opportunity Winter/Spring 2020): These two-year grants aim to build capacity for the uptake of POR in the health system by providing health systems or related organizations with an opportunity to embed POR principles, practices and findings within Canadian health care contexts. This will be accomplished by implementation specialists (individuals who bring a spectrum of influence in health policy or practices) employed by these health systems or related organizations, who will receive training, mentorship, and a practical opportunity to embed patient-oriented research within their organization. These grants will be flexible and tailorable to the specific needs of the host organization, implementation specialist, and relevant community and stakeholders. In addition, there are other capacity building funding opportunities including: the POR Catalyst Grants, the SPOR-sponsored ICS Planning and Dissemination Grants, and the POR Impact Assessment Grants, which were all funded toward the end of FY 2019-20.

Patient Engagement

Patient engagement supports efforts to engage patients in a meaningful way through active collaboration in governance, priority setting, and the conduct of research, as well as in summarizing, distributing, sharing, and applying its resulting knowledge. Patient engagement in research will ensure the relevance of the research and improve its translation into policy and practice, contribute to more effective health services and products, and ultimately, enhance the quality of life of Canadians. A key objective of SPOR is for patients, researchers, health care providers, and decision-makers to actively collaborate to build a sustainable, accessible and equitable health care system.

Patient engagement is a key component of all SPOR funded activities including patients in their governance structures, engaging patients in their range of activities, contributing to the literature on patient engagement methods and evaluation, and creating resources/training for patient engagement. A SPOR Patient Engagement Framework was designed in 2014 to establish key concepts, principles and areas for patient engagement to be adopted by all SPOR partners. The core areas for engagement outlined in the framework include:

1. Patient engagement in governance and decision-making (patient representation on all SPOR funding opportunity peer reviews, boards/committees such as the SPOR National Steering Committee (NSC), SUPPORT Units, and Networks);
2. Capacity building for patient engagement (via research funding mechanisms; training strategies; partnerships with other organizations/networks; shared-responsibility with patients); and
3. Tools and resources (pool of diverse patient participants; education, orientation, and training; best practices for engagement approaches and role definitions).

In alignment with the framework, CIHR funded some small-scale SPOR patient engagement grants, co-developed a POR curriculum with relevant stakeholder groups, and identified key considerations for compensating patient partners.

• Patient Engagement Grants: The Patient Engagement Collaboration Grants were launched in 2014 and 2015 with a focus on implementing patient engagement into research projects. As of March 31, 2016, 11 projects were funded with objectives to: identify and implement inclusive engagement mechanisms, processes and approaches that value patient perspectives, experiences and skills throughout the research process; and facilitate opportunities for researchers and knowledge users, including patients, to work together to identify problems and gaps, set priorities for research, and produce and implement solutions. This funding opportunity was subsequently retooled and offered twice more in 2017 and 2018 as the Patient-Oriented Research Collaboration Grants. This version increased the funding amount and the scope of the grant, which focused more on conducting POR studies or projects as opposed to simply implementing patient engagement into a study or project. To date, end of grant reports have been received for the Patient Engagement Collaboration Grants, and end of grant reports for the Patient-Oriented Research Collaboration Grants are still being received.
• POR Curriculum: CIHR developed a national-scope curriculum that could be delivered by SUPPORT Units and/or Networks to create a common, standardized set of materials and to bring alignment in understanding of patient engagement across different organizations. The Foundations in Patient-Oriented Research curriculum was designed to build mutually beneficial relationships for conducting POR by ensuring that relevant stakeholders (i.e., patients, researchers, health care professionals and health system decision-makers) have a common foundational understanding of POR, the research enterprise, and team dynamics. 3 Several SUPPORT Units and Networks continue to deliver and adapt the curriculum, including efforts to build and maintain capacity within their staff to deliver the workshops.
• Patient Compensation: CIHR led the development of a framework document on patient compensation, which provides general guiding principles that can be used when offering payment to patient partners engaged in research and research-related activities. This framework document promotes the view that, whenever possible, patients should be offered appropriate payment for their added value to the research activity to which they are contributing.

Clinical Trials

An important goal of SPOR is to strengthen organizational, regulatory, and financial support for, and enhance patient and clinician engagement in, clinical trials in Canada. SPOR has developed the following components to improve Canada's clinical trials environment:

Canadian Clinical Trials Coordinating Centre

The Canadian Clinical Trials Coordinating Centre (CCTCC), launched in 2014, was a body that brought stakeholders together to strengthen the Canadian clinical trials environment and promote Canada as a leading destination for clinical trials globally through policy development, advocacy, standardization, and evaluation. Funding for CCTCC was not renewed as of 2019-20. Some initiatives to date include:

• Canadian Clinical Trials Asset Map, a unique, robust, searchable web-based database designed to showcase Canada's clinical research strengths to all stakeholders, including clinical trial sponsors and position Canada as an attractive global destination for the conduct of clinical trials.
• Model Clinical Trial Template Agreement which provides a standard model contract for use by clinical trial sites and sponsors in negotiating phase II and phase III clinical trial agreements.
• Streamlining Research Ethics Review for Multi-Centre Trials, a SPOR initiative to gather information on streamlining initiatives in Canada and making recommendations for improving the process of ethics review for multi-centre patient-oriented research.

Innovative Clinical Trials

In response to the evolving Clinical Trials environment, the iCT Initiative was created to build capacity in innovative clinical research methodologies and increase the amount of clinical research undertaken in Canada. The iCT Initiative was designed to increase Canadian competitiveness in innovative clinical research and provides a stimulus for trialists to adopt new methodologies. The objectives of the SPOR iCT Initiative are to:

1. Build capacity in iCT by:
   1. Attracting Canadian clinical investigators with a focus on shifting their research programs to include innovative methodologies and adopt the principles of SPOR; and,
   2. Enhancing Canada's capacity and expertise in innovative and cost-effective trial methodologies.
2. Increase the intensity of iCT research nationally.

Phase I of the iCT Initiative includes three types of grants, with a focus on pragmatic real world studies of comparative effectiveness and implementation science:

1. Catalyst Grants to provide seed funding for novel research perspectives;
2. Mentorship Chairs to develop capacity in iCT through provision of salary and operating funds; and
3. Multi-Year Grants to support innovative clinical trials and allow researchers to build/develop/improve leadership and research planning.

Also funded as part of the iCT Initiative was the Rewarding Success Initiative a new funding model that will reward success as a means of incentivizing research teams and their healthcare partners to enhance value-based care, health system sustainability, and health outcomes. The research teams and their partners designed, implemented, and evaluated interventions in healthcare organization(s) that aim to produce healthcare cost savings and/or improved health system efficiency. Teams use iCT design to institute complex interventions that will allow them to fail fast and iterate to improve the likelihood of success. Use of an iCT design enables unambiguous attribution of the effect of the intervention(s) employed to improve value and efficiency in health care.

Enabling Functions

Enabling Functions support the development of national platforms, including data and systematic reviews and guidelines development, with the aim of providing national-level support for POR. To date, this element consists of two initiatives.

SPOR Evidence Alliance

The SEA (funded in 2017) was established to provide national-level support in knowledge synthesis, systematic reviews, clinical practice guidelines development, KT, and POR. This Canada-wide alliance of researchers, research trainees, patients, healthcare providers, policy makers and organizations aims to provide evidence-based answers through research to ensure high-quality information is available to inform decisions.

SEA is committed to building a Canadian health system that is increasingly informed and improved using best available evidence and innovations uncovered by the health research community. The goal of SEA is to respond to at least 100 questions over its five-year term of funding, conduct systematic reviews, develop guidelines, provide mentorship and training opportunities, and engage in knowledge translation to ensure health care providers have the evidence to provide patients with the right care at the right time. SEA areas of focus presented in the 2018-19 SEA Performance Report pertain to the following categories:

• Query Services
• Canadian Clinical Practice Guidelines developers Asset Map
• Stakeholder Engagement
• Advancing Knowledge
• Training and Capacity Development
• Governance
• Sex and Gender
• Relevance to Indigenous Peoples
• Financials

SPOR Canadian Data Platform

The objective of the SPOR CDP, funded in October 2018, is to develop a distributed network that facilitates and accelerates multi-jurisdictional research by connecting existing Canadian centers of excellence that already collect and work with health and health-related data across Canada. SPOR CDP aims to address major barriers and inefficiencies in accessing or using multi-jurisdictional or national data for POR.

The SPOR CDP is establishing a single stop for researchers to request access to rich health and social data from various sources across the country. Currently, the data access process differs across jurisdictions in terms of the forms required, the fees charged, the physical location of the data, and more. By reducing barriers to information access, the CDP will enable investigators to conduct multi-jurisdictional, people and patient-focused research more efficiently. Communications with key stakeholders including the SPOR SUPPORT Units and other SPOR funded entities are being established to clarify the objectives and role of the SPOR CDP and understand the needs of other stakeholders. The SPOR CDP has developed its Data Access Support Hub (DASH), which was launched in early 2020 and serves as the data access portal for the platform.

As these two initiatives have only recently been established, the evaluation will consider SEA and the SPOR CDP from an implementation evaluation perspective only.

Appendix D: Methodology – Additional Details

Additional details about the multiple lines of evidence and methodology used in the evaluation are presented in this appendix.

Administrative and Financial Data Analysis

The review of SPOR program records and administrative data included EIS data from the period 2010-2021, partnership data for planned contributions for the period 2010-2017 from SPOR program based on EIS/SSRE, InfoNet, ResearchNet, and online data, and partnership data for planned contributions for the period 2018-2021 from SPOR program based on InfoNet and ResearchNet. The administrative and financial data analysis provided information about the program's descriptive and performance (e.g., number of grants and awards funded, the financial investments, etc.), descriptive data for GBA+ analysis, data for the edge lists necessary for the SNA, and sampling frames for the surveys, case studies, and interviews with various stakeholders involved in the program.

Document and Literature Review

The document review component involved the analysis of relevant SPOR documents including SPOR performance reports (e.g., annual reports, summary reports, previous evaluations, newsletters), SPOR general documentation (e.g., strategies, TB submissions, SPOR committee and sub-committee materials), partner documentation, federal and provincial policy documents, and other relevant federal government documents. The specific documents reviewed (n = 82) were selected based on recency, relevance, maturity, and materiality of available documents provided to the Evaluation Unit from the SPOR program.

The literature review focused on academic and professional sources at the national and international level to assess the extent to which POR is addressing the ongoing need in Canada to prioritize evidence-informed health care. A comparative review examining the experience of other countries such as the U.S., U.K., and Australia in implementing POR strategies was also conducted. The review focused on recently published documents that are high level narrative or systematic reviews of POR. If reviews were not available, then the review searched recently published editorial or opinion pieces in highly cited journals such as The New England Journal of Medicine or Lancet, etc.

Bibliometric and Altmetric Analysis

Bibliometric indicators are recognized as valuable measures of scientific productivity and quality. The bibliometric and altmetric analysis was completed by Science-Metrix to evaluate the performance of POR publications in Canada and globally. The bibliometric database was mainly built on Scopus data and complemented by other sources such as Unpaywall, PlumX, Overton, PatentSight, ClinicalTrials.gov, and PubMed.

Surveys of Researchers and Stakeholders

The SPOR researcher survey targeted all researchers who have either received (i.e., recipients) or applied (i.e., applicants) for a SPOR funding opportunity associated with a research project between January 2015 and December 2020. The researcher sample frame was constructed from CIHR grant data. Recipients of SPOR funding or applicants to SPOR funding opportunities which were not associated with a defined research project (e.g., travel awards, funding for a SPOR core element, training grants and awards, fellowship grants and awards) were removed from the sample for the purposes of this survey. The final sample was comprised of 506 researchers, 245 recipients and 259 applicants, with a 36% response rate for recipients (n = 89) and 18% for applicants (n = 47). The inclusion of applicants allowed for counterfactual comparison, explored whether alternative funding was obtained and the extent to which patient engagement, capacity development and research outcomes were impacted by not obtaining SPOR funding. Forty-one percent of SPOR recipients surveyed received their funding from collaboration grants (n = 100), 30% from operating grants (n = 77), 29% from catalyst grants (n = 74), 11% from project grants (n = 27), and 5% from other iCT grants (n = 10).

The SPOR stakeholder survey targeted stakeholders who have been involved in SPOR in some capacity (e.g., co-applicants, trainees, patient partners, other partners, knowledge users). However, the current stakeholder lists from SPOR program and SPIR consultations did not include the entire population of stakeholders. Therefore, a respondent-driven sampling approach was taken, in which SPOR researchers received a link to a short survey to forward on to any individual involved in SPOR. This allowed the Evaluation Team to collect full names and email addresses of additional SPOR stakeholders. Data obtained from the respondent-driven sampling approach were combined with the lists obtained from the SPOR program and SPIR consultations. The final sample was comprised of 738 stakeholders, with a 24% response rate (n = 179). SPOR stakeholders surveyed consisted of 28% co-applicants (n = 50), 22% patient partners (n = 39), 22% trainees (n = 39), 13% other (employees of SPOR Networks or SUPPORT Units, lead of a training committee, institutional authority, institutional partners, research staff; n = 24), 9% other partners (n = 16), and 6% knowledge users (n = 11).

Key Informant Interviews

Key informant interviews provided insights concerning the relevance of the SPOR program, in terms of its alignment with the Canadian need for POR and assessed performance of the program by assessing the achievement of the program's immediate, intermediate, and ultimate outcomes. They also helped evaluators assess how the design and delivery of the program supports the achievement of these intended outcomes.

In total, 38 key informant Evaluation Limitations and Mitigation Strategies interviews were conducted from seven respondent groups:

• CIHR staff (VP LHS, SPOR WG Chairs; n = 5);
• POR experts (n = 3);
• SPOR funded entities (n = 6);
• Knowledge users (n = 4);
• NSC members (n = 2);
• Partners (n = 9);
• Patient partners (n = 9).

The interviews were approximately 45-60 minutes long, fully confidential and semi-structured. Respondents received an interview guide prior to the interview, to allow them to consider the questions in advance.

Knowledge Readiness Levels Analysis

The Knowledge Readiness Levels (KRLs) allow a classification of the research results' scope from basic bench and lab work to the establishment of policies or tools for prevention, screening, diagnosis, or treatment. Research results were identified as Knowledge Products (KPs), specifically peer-reviewed publications, for the purposes of the evaluation. There are three descriptive ranges of knowledge products: scientific foundations, applications, and real-world contexts. Each range consists of three knowledge readiness levels within that range.

The CIHR evaluation team leveraged existing databases from the College of Reviewers as the primary and key source of contacts for this exercise. The invitation was sent to 200 experts that had relevant clinical and/or academic expertise in POR, clinical trials, systematic reviews, epidemiology, or qualitative research, of whom 47 responded they were interested. Those experts who demonstrated interest were arranged into 4 groups of 11-12 each for the Delphi process. The Delphi process involved a facilitator who sought individual assessments from the pool of experts in several rounds until a consensus was reached to classify the KPs into their KRL.

Case Studies

The aim of the case studies was to collect evidence as to what extent SPOR is achieving its intermediate outcomes, provide a rich narrative of qualitative data from stakeholders of the program, and describe impact, results, challenges, lessons learned and contributions of the different aspects of SPOR studied in the case studies. The five case studies were selected around SPOR's intermediate outcomes and used combinations of SPOR core elements as units (e.g., SUPPORT Units, Networks) and/or sub-units (e.g., projects, researchers, trainees) of analysis for the larger cases. The case study for the intermediate outcome of patients and stakeholders are active partners in both research and implementation of evidence-based improvements focused on Indigenous. The selection of units and sub-units for each case were determined based on the duration of funding/project, the material investment, the innovation within collaboration or research, the topic area, the regional representation, and the extent of patient engagement within the example.

Evaluation Limitations and Mitigation Strategies

References

• Abelson, J., Canfield, C., Leslie, M., et al. (2022). Understanding patient partnership in health systems: lessons from the Canadian patient partner survey. BMJ Open, 12, 1-8. doi:10.1136/bmjopen-2022-061465
• CIHR. (2011). Canada's Strategy for Patient-Oriented Research.
• CIHR. (2015). Strategic plan 2014-15 – 2018-19: Health research roadmap II: Capturing innovation to produce better health and health care for Canadians.
• CIHR. (2016). Evaluation of the Strategy for Patient-Oriented Research.
• CIHR. (2021). Strategic plan 2021-2031: A vision for a healthier future.
• Gill, P.J. & Cartwright, E. (2021). Partnering with patients in the production of evidence. BMJ Evidence-based Medicine, 26(3), 73-76. Doi: 0.1136/bmjebm-2020-111339.
• Government of Canada. (2000). Canadian Institutes of Health Research Act.
• Government of Canada. (2017). Minister of Health Mandate Letter.
• Government of Canada. (2019). Speech from the Throne to open the First Session of the Forty-Third Parliament of Canada.
• Government of Canada (2020). Speech from the Throne to open the Second Session of the Forty-Third Parliament of Canada.
• Government of Canada. (2021A). Government of Canada creates national training platform for patient-oriented research.
• Government of Canada. (2021b). Speech from the Throne to open the First Session of the Forty-Fourth Parliament of Canada.
• Martineau, J.T., Minyaoui, A. and Boivin, A. (2020). Partnering with patients in healthcare research: a scoping review of ethical issues, challenges, and recommendations for practice. BMC Medical Ethics, 21(34). https://doi.org/10.1186/s12910-020-0460-0
• Saskatchewan Centre for Patient-Oriented Research (2018). SCPOR Patient-Oriented Research Level of Engagement Tool.
• Staley K. (2009). Exploring Impact: Public involvement in NHS, public health and social care research. INVOLVE, Eastleigh.
• Vat, L.E., Finlay, T., Schultmaker-Warnaar, T.J., Fahny, N., et al. (2020). Evaluating the "return on patient engagement initiatives" in medicines research and development: A literature review. Health Expectations, 23, 5–18. DOI: 10.1111/hex.12951

Date modified:

2023-11-10