Overview of the Reforms to CIHR’s Open Suite of Programs: Peer Review Expert Panel

November 2016

Table of contents

1. Objective
2. Origins for the design, including rationale for the changes (PREP Q1, 2)
3. Overview of the Design (Panel Q1)
4. Foundation and Project Grant Program Implementation Process (Panel Qs 2, 3, 4)
5. Description of evidence to date (Panel Qs1, 2)
   • 5.1. Comparisons of success rates between Open Operating, Foundation, and Project Grant Programs
   • 5.2. Funding results: 2015 Foundation Grant Competition
   • 5.3. Funding Results: Spring 2016 Project Grant Competition
6. Overview of areas for growth and ongoing monitoring (PREP Q4, 6)
   • College of Reviewers
   • Performance Measurement

1. Objective

The objective of this briefing paper is to synthesize existing CIHR evidence, analysis, and external links in a single document for the purposes of orienting the Peer Review Expert Panel to:

• the design of CIHR's reforms to its open suite of investigator-initiated research programs and peer review processes and how these reforms respond to the peer review challenges identified by CIHR;
• the extent to which these changes in program architecture and peer review relate to the challenges posed by the breadth of the CIHR mandate, the evolving nature of science, and the growth of interdisciplinary research;
• CIHR's mechanisms for ensuring peer review quality and impacts.

This document is not meant to be an evaluation of the reforms, nor is it intended to replace the interpretation of data and/or context that will be offered during the face-to-face Panel meetings on January 16-18, 2017. Rather it is intended to provide a description and evidence base to support a common understanding about the reforms and complement what the Panel will hear during their meetings.

2. Origins for the design, including rationale for the changes (PREP Q1, 2)

As stated in the CIHR Act, CIHR's mandate is to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system."^{Footnote 1}

The Canadian Institutes of Health Research (CIHR) is the major Canadian Government funder of research in the health sector and classifies its research across four "pillars" of health research: biomedical; clinical; health systems services; and population health. CIHR invests approximately $1 billion (CAN) each year to support health research.^{Footnote 2} This investment supports both investigator-initiated and priority-driven research.

CIHR classifies investigator-initiated research as research where individual researchers and their teams develop proposals for health related research on topics of their own choosing. Just over half of CIHR's $1 billion budget are used to support investigator-initiated research.^{Footnote 2} The balance is spent on priority-driven research, which refers to research in areas identified as strategically important by the Government of Canada; in this case themed calls for research proposals are made.

Until 2014, the Open Operating Grants Program (OOGP)^{Footnote 3} was CIHR's primary mechanism through which investigator-initiated research was supported. The specific objectives of the OOGP were to contribute to the creation, dissemination and use of health-related knowledge, and to help develop and maintain Canadian health research capacity, by supporting original, high quality projects or teams/programs of research proposed and conducted by individual researchers or groups of researchers, in all areas of health.

In 2009, CIHR's Health Research Roadmap was released and this five-year plan introduced a bold vision to reform the peer review and open funding programs. Beginning in 2010 CIHR started the process of reforming its investigator-initiated research programs, including the OOGP, and the related peer review processes. These reforms were informed by significant contributions and also influenced by three main lines of evidence:

1. Data from a poll of the scientific community: a 2010 IPSOS Reid poll conducted by CIHR found strong support from the research community to fix a peer review system that was perceived as 'lacking quality and consistency'.

   Figure 1. Stakeholder Satisfaction - Peer Review (Percent of respondents who provided an opinion), Results from 2010 IPSOS Reid Study

   

2. A recommendation of CIHR's second International Review Panel in 2011:

   'CIHR should consider awarding larger grants with longer terms for the leading investigators nationally. It should also consolidate grants committees to reduce their number and give them each a broader remit of scientific review, thereby limiting the load'.

3. Findings from the 2012 evaluation of CIHR's Open Operating Grant Program (OOGP) which recognized challenges in open funding across pillars of research and supported the need to reduce peer review and applicant burden.

CIHR conducted several rounds of consultations with its stakeholder communities prior to and throughout the reforms process.^{Footnote 4} The earliest consultations, which led to the proposed design, resulted in a number of challenges being identified with CIHR's existing funding architecture and peer review processes,^{Footnote 5} and the resulting re-design was intended to address those challenges. The challenges included:

• Funding program accessibility and complexity;
• Applicant burden/"churn";
• Application process / attributes do not capture the correct information;
• Insufficient support for new/early career investigators;
• Researchers and knowledge user collaborations not fully valued;
• Lack of expertise availability;
• Unreliability/inconsistency of reviews;
• Conservative nature of peer review;
• High peer reviewer workload.

CIHR moved to a new system of investigator–initiated grants, including a number of design elements intended to respond to these challenges. These changes were piloted and implemented over the 2010-2016 period and were designed to directly respond to the challenges identified through stakeholder input. Key points along this plan are outlined in Figure 2 below.

The new open suite of programs means that the majority of its investigator-initiated research funding is now being awarded through its Foundation Grant Program and Project Grant Program.^{Footnote 6} With this said, it is important to note that CIHR also funds awards and priority-driven grants through other funding mechanisms and peer review processes that are not discussed in this paper.

The objectives of the new open suite of programs are as follows:

• Foundation Grant Program (one competition per year) - designed to contribute to a sustainable foundation of health research leaders by providing long-term support for the pursuit of innovative, high-impact programs of research; and,
• Project Grant Program (two competitions per year) - designed to capture ideas with the greatest potential for important advances in health-related knowledge, the health care system, and/or health outcomes, by supporting projects with a specific purpose and defined endpoint. The reforms have been met with mixed reaction by Canada's health research community.
• The vision of the College of Reviewers (College) is to establish an internationally recognized, centrally-managed resource that engenders a shared commitment across the Canadian health research enterprise to support excellence in peer review of the diverse and emerging health research and knowledge translation activities that span the spectrum of health research. The College is intended to be a national resource that over time will serve the peer review needs of CIHR and its partners. The implementation of the College will ensure:
  • A more stable base of experienced reviewers from Canada and abroad;
  • Collaborative approaches to peer review across health research funding organizations;
  • Inclusion of quality assurance mechanisms;
  • Systematic recruitment processes;
  • Ongoing education resources and mentorship programs; and,
  • A valued recognition program.

Figure 2. Timeline of CIHR Reforms process

3. Overview of the Design (Panel Q1)

The purpose of this section is to provide an overview of the reports and processes that informed the new funding program design and how the design of the Foundation and Project Grant Programs relate to the former CIHR Open Operating Grants Program. The resulting application process to the Foundation Grant Program and Project Grant Program are illustrated in Figures 3 and 4 below.

The Foundation and Project Grant Programs were designed to respond to the challenges outlined earlier. The specific design elements associated with these programs and how they respond to one or more of these challenges are below in Figure 5 and discussed more fully in the following seminal documents:

• Design Discussion Document - Proposed Changes to CIHR's Open Suite of Programs and Enhancements to the Peer Review Process. This document, released in March 2012, provides a description of CIHR's thinking on the newly proposed design. It defines a set of funding mechanisms for the new Open Suite of Programs and a set of design parameters that were tabled for implementation.
• What CIHR Heard: Analysis of Feedback on the Design Discussion Document. This document, released in August 2012, provides an account of the feedback received from CIHR's research community regarding the proposed changes to the Open Suite of Programs and peer review process. This feedback was obtained through cross-country consultations about the design.
• Designing for the Future: The New Open Suite of Programs and Peer Review Process. This document, released in December 2012, outlines the key elements of the design of the new Open Suite of Programs and peer review processes. It also signalled CIHR's commitment to a clear transition plan to the new design and CIHR's intent to an iterative design and implementation process, including pilot studies.

Figure 3. Schema of Foundation Grant Program Competition Cycle

Figure 4. Schema of Project Grant Program Competition Cycle

Figure 5. Design elements of the new programs and how they were intended to address identified challenges

As can be seen from Figure 5 (above) the design elements deliberately respond to multiple challenges. A description of how these design elements compare to the former Open Operating Grants Program is provided in Table 1. Table 1 reflects the state for the current competitions for the Foundation and Project Grant programs.

Table 1. Design element differences across Open Operating (OOGP), Foundation, and Project Grant Programs

4. Foundation and Project Grant Program Implementation Process (Panel Qs 2, 3, 4)

The purpose of this section is to provide an overview of the documents and processes that informed ongoing implementation of the design and related peer review processes.

Since 2013, CIHR has piloted specific design elements associated with the new programs, including structured applications, remote review, rating scale, and streamlined CV. These pilots were conducted in a 'live' manner (i.e., piloted during the course of routine program delivery across several programs). Pilots were conducted so that CIHR could monitor outcomes in an evidence-informed manner. The pilots were closely scrutinized to safeguard the reliability, consistency, fairness and efficiency of the competition and peer review processes. The results from pilots completed to date have been analyzed and consolidated into the following reports:

• Fall 2013 Knowledge Synthesis Pilot Report
• Spring 2013 Fellowships Competition Report

In 2015, CIHR commissioned various reviews of different aspects of the reforms implementation project. These were initiated by CIHR management to assess CIHR's internal systems and processes for the implementation of the reforms, and to allow for adjustments in a timely manner on projects of this complexity. The reports included:

• The Internal Audit Consulting Engagement Report looked at lessons learned from the Reforms Implementation Project. It was completed by the CIHR Internal Audit Group. The report focused on governance and administrative practices linked to the project management and internal reorganization in order to deliver the new Foundation and Project Grant programs. The report found that the reforms implementation project benefited from well-developed planning tools and that the pilots were rolled out on time. The report also found opportunities for improvement in the areas of information-sharing, communications, reporting, project planning, and stakeholder engagement, all of which are being addressed by CIHR.
• The Reforms CRM Project Independent Third Party Review by Interis Consulting was specifically designed to assess the implementation of the business systems required to support the new program delivery processes. CIHR sought expert advice to provide recommendations concerning implementing complex, transformative business systems; opportunities for improvements included clarifying roles and responsibilities as well as project schedules and scope. CIHR is currently taking steps to implement the report's recommendations such as establishing new governance committees to monitor scope and timelines for the projects.
• The CIHR Management Response, released in May 2016, addresses the recommendations in both reports. These two reviews allowed management to receive feedback as the new managerial and business systems were being implemented which allowed for course corrections. CIHR continues to implement the reports' recommendations and is undertaking the necessary steps to address project management challenges.

These reviews identified a number of internal strengths, most notably that:

• key planning activities were properly undertaken;
• document and planning tools were well developed;
• the pilots occurred during the specified timeframes, thanks to a dedicated project team; and,
• employee engagement in the pilots resulted in greater dedication and commitment during times of change.

The reviews also identified a number of opportunities for improvement, involving five areas:

• Governance;
• Communications and stakeholder engagement;
• Information for senior level committee decision-making;
• Project management and planning; and,
• Internal stakeholder engagement and impact on personnel.

CIHR accepted the recommendations of the reviews and, in response, is taking the actions listed below in Table 2.

Table 2. Areas for action and corresponding CIHR response following CIHR Internal Reviews, 2015

In 2016, CIHR outlined how the 2014 "live pilot" of the Foundation Grant Program would inform subsequent competitions. Table 3, taken from the executive summary of this report, outlines a number of enhancements to the 2015 Foundation Grant "live pilot" competition informed by the 2014 "live pilot" and related survey data from reviewers, applicants, and Competition Chairs.

Table 3. Enhancements implemented to the 2015 Foundation Grant "live pilot" following 2014 pilot.

On July 13, 2016, at the request of the Minister of Health, CIHR hosted a Working Meeting with members of the health research community to review and jointly address concerns raised regarding CIHR's peer review processes, particularly associated with the Project Grant Program. Table 4 outlines the key outcomes of that meeting and complements the full report of the meeting.

Table 4. Action areas and requested changes to the 2016 Project Grant competition (Fall launch) following the July 13th, 2016 Working Meeting of CIHR and its research community

The Peer Review Working Group was established as an outcome of the July 13th, 2016 Working Meeting between CIHR and representatives from its research community. Under the leadership of Dr. Paul Kubes, the Peer Review Working Group discussed each of the July 13th Working Meeting outcomes and made recommendations for action, outlined in Table 5. The CIHR response to these recommendations is outlined in Table 6. It should be noted that Dr. Kubes has an ongoing role with CIHR as the College of Reviewers Executive Chair.

Table 5. Recommendations from the Peer Review Working Group for the 2016 Project Grant competition

Table 6. CIHR implementation of recommendations from the Peer Review Working Group for the CIHR Project Grant competition (Fall 2016 )

5. Description of evidence to date (Panel Qs1, 2)

The purpose of this section is to provide a high-level summary of data related to the overall CIHR funding context.

Comparisons of success rates between Open Operating, Foundation, and Project Grant Programs

Figure 6 presents data on the number of applications received and approved and the success rates for its Open Operating Grants Program (OOGP), the Foundation Grant Program and the Project Grant Program. It can be seen that the success rates for these programs has declined over time and current success rates for the Investigator-Initiated programs sits at ~13%.

Figure 6. Distribution of application pressure and success rates in the CIHR Investigator-Initiated Grant programs, 2006-07 to 2015-16

The success rate varies across several important areas for monitoring:

• CIHR's four pillars of health research (Figure 7)
• Sex of the Nominated Principal Investigator (Figure 8)
• Career stage of the Nominated Principal Investigator (Figure 9)

Figure 7 provides the overall decline in success rates which are evident in the previous graph, but also depicts the variability by pillar. Biomedical research has had historically higher success rates than all other pillars; however it seems that the Foundation Grant Program is shifting this pattern slightly, at least in the first two competitions held for the Program. Figure 8 depicts a similar pattern in success rates by sex, however depicts sex differences in Foundation Grant program success rates.

Figure 7. Distribution of success rates of Investigator-Initiated Grant programs by primary pillar, 2006-07 to 2015-16

Figure 8. Distribution of success rates of Investigator-Initiated Grant programs by sex of NPI, 2006-07 to 2015-16

Figure 9 shows a persistent difference in the success rates for new/early career investigators when compared to established investigators (mid career and senior) in the OOGP. Within the new programs, which both have mechanisms for ensuring appropriate review of and protected funding for new/early career investigators, this gap seems to be closing.

Figure 9. Distribution of success rates of Investigator-Initiated Grant programs by career stage of NPI, 2006-07 to 2015-16

Funding results: 2015 Foundation Grant Competition

The 2015 Foundation Grant Competition will support 120 research programs for a total of $292M over 7 years.

New/Early Career Investigators Cohort – 33 funded
Average grant size: $1,140,364
Median grant size: $1,067,334
Grant duration: 5 years

Mid-career/senior investigators – 87 funded
Average grant size: $3,282,939
Median grant size: $2,838,143
Grant duration: 7 years

The figures below provide a further breakdown of the 2015 Foundation Grant Competition results by applicant cohort (new/early career investigator cohort, Figure 10 and mid/senior career investigator cohort, Figure 11). Specifically these figures present data on the proportion of applications funded relative to the proportion submitted on a range of applicant profile variables. For the Foundation Grant Program, these cohorts are reviewed and ranked separately by design.

Figure 10. Breakdown of new/early career investigator cohort by proportion of applications submitted and funded, 2015 Foundation Grant Competition

Figure 11. Breakdown of mid/senior career investigator cohort by proportion of applications submitted and funded, 2015 Foundation Grant Competition

The 2015 Foundation Grant competition was supported by a three-stage competition and review process.

Stage 1

• A total of 432 individual reviewers participated in the peer review process which was monitored by 30 virtual chairs.
• The peer-review process resulted in total of 4,418 individual reviews.
  • 14 applications (2%) had three reviewers assigned to them
  • 104 applications (11%) had 4 reviewers assigned to them
  • 792 applications (87%) had 5 reviewers assigned to them
• 265 applications were successful at stage 1 and were invited to submit a stage 2 application (260 stage 2 applications were submitted - 77 early/career investigators, 183 established investigators).

Stage 2

• A total of 171 individual reviewers participated in the peer review process which was monitored by 16 Virtual Chairs.
• The peer-review process resulted in total of 1,298 individual reviews.
  • Two applications (1%) had 4 reviewers assigned to them
  • 258 applications (99%) had 5 reviewers assigned to them
• 160 applications were successful at stage 2.
• Based on stage 2 results, CIHR identified the highest ranked applications to be considered for funding. These applications constituted the "green zone" and did not require further evaluation.

Final assessment stage

• The final assessment stage committee was comprised of 16 virtual chairs from stage 2, a Chair and a Scientific Officer.
• 96 applications with high consolidated rankings and significant variation in those rankings were discussed.

Funding Results: Spring 2016 Project Grant Competition

The 2016 Project Grant competition will support 491 research grants, plus an additional 127 bridge grants, for a total investment of $375,579,347 over five years. The 491 grants approved were awarded to 468 individual nominated principal investigators (23 NPIs were awarded two grants). Of the 491 grants, 98 were awarded to new/early career investigators. The average grant size/duration is approximately $791,000 over four years. The median grant size/duration is approximately $703,000 over five years.

Figure 12 presents the proportion of applications funded relative to the proportion submitted on a range of applicant profile variables.

Figure 12. Breakdown of proportion of applications submitted and funded, Spring 2016 Project Grant Competition

Stage 1

Each application was reviewed by up to five reviewers. That said, there was a small subset of applications for which this was not possible. Applications with only 1 or 2 reviewers were sent directly to the final assessment stage for a full review.

• 1,664 individual reviewers participated in the peer review process.
• Each reviewer received approximately 10 applications.
• The peer-review process resulted in total of 15,405 individual reviews.
  • 4 applications (less than 1%) had 1 reviewer assigned to them
  • 48 applications (1%) had 2 reviewers assigned to them
  • 550 applications (14%) had 3 reviewers assigned to them
  • 2,390 (63%) had 4 reviewers assigned to them
  • 821 (22%) had 5 reviewers assigned to them

Final assessment stage

• The single final assessment stage committee was comprised of 31 virtual chairs from stage 1, including a Chair and a Scientific Officer.
• 100 applications were considered by this committee. They included the highest ranked applications with significant review variances, as well as the under-assigned applications from stage 1.
• After a face-to-face meeting and discussion, the committee recommended an additional 20 applications for full funding.

6. Overview of areas for growth and ongoing monitoring (PREP Q4, 6)

The purpose of this section is to describe a number of areas for growth and ongoing monitoring related to the implementation of the Foundation and Project Grant Programs and related peer review support mechanisms. In particular, this section describes the role and status of the College of Reviewers and related objectives and the draft Performance Measurement Framework.

College of Reviewers

In June 2016, CIHR announced the inaugural slate of College Chair appointments. The role of the College Chair is to ensure that the peer review system supports the selection of the most innovative and cutting-edge proposals for research and knowledge translation, while continuing to be fair, well-managed and transparent. College Chairs are appointed by the Chief Scientific Officer (CSO) and interact with CIHR through the CSO office and the College Branch. The Chairs will play a vital role in helping to build a College that is recognized as a valued resource throughout Canada and around the world.

Sixteen candidates were appointed as College Chairs on July 1, 2016. Working together, College Chairs will oversee strategies aimed at attracting and retaining outstanding peer reviewers. They will also ensure that the necessary frameworks are in place to provide peer reviewers with the relevant learning, mentoring, and support required to deliver fair and quality reviews.

Dr. Paul Kubes (University of Calgary) was appointed as the Executive College Chair. Working in close collaboration with CIHR's Chief Scientific Officer and his Chair colleagues, he will be responsible for ensuring peer review excellence across all fields of health research and for adapting peer review to the evolving profile of science and convergence of research fields.

The first College Chairs meeting was held October 2016. Terms of Reference for the College Chairs are under development and expected to be finalized in early 2017. The next meeting of the College Chairs, scheduled for early January 2017, will focus on determining the requirements to launch and open the College as a peer reviewer resource dedicated to ensuring excellence in peer review.

Table 7. Status of core College functions overseen by College Chairs

Performance Measurement

CIHR conducts surveys of applicants, reviewers, and Chairs (both Virtual Chairs for Foundation and Competition Chairs for Project) following each stage of both the Foundation and Project Grant Programs as well as ongoing monitoring of application and competition statistics.

In addition, all of CIHR's major programs have a performance measurement framework, including a logic model. The logic model provides a graphic illustration of how its activities will contribute to the achievement of the expected outcomes. Statements within the model are intended to be high-level and concise; definitions and details are presented in an accompanying narrative. The draft logic model for the Investigator-Initiated Programs is presented in Figure 13. The depiction implies a linear, sequential nature of activities, outputs, and outcomes. In actuality logic model components are dynamic and often occur in an iterative nature. For instance, the completion of one activity can be dependent upon the outputs or outcomes from one or more other activities. To maintain the clarity of the model, this iterative process is not depicted with connectors between individual statements; only the general flow of logic from activities through to ultimate outcomes is shown.

Figure 13. Draft program logic model for Investigator-Initiated Research at CIHR

Table 8. Draft performance measurement framework for Investigator-Initiated Research at CIHR

Date modified:

2021-03-30