Designing for the Future: The New Open Suite of Programs and Peer Review Process

December 19, 2012

Table of contents

• Message from CIHR's Science Council
• Executive Summary

1. Purpose and Introduction
2. A New Open Suite of Programs
3. Project Scheme
   • Overview of the Architecture
   • Overview of the Mechanics
   • Proposed Structured Review (Adjudication) Criteria
4. Foundation Scheme
   • Overview of the Architecture
   • Overview of the Mechanics
   • Proposed Structured Review (Adjudication) Criteria
5. College of Reviewers
6. Transitioning to the New Open Suite of Programs and Peer Review Process
   • Piloting of Key Peer Review Design Elements
   • Phasing-in the New Open Suite of Programs
   • Transitioning from Existing Programs and Processes
   • Implementing the College of Reviewers
   • Supporting Transition: Orientation for Applicants and Reviewers
7. Conclusion

• Annex A: Project Scheme Early Draft Adjudication Worksheet
• Annex B: Foundation Scheme Early Draft Adjudication Worksheet
• Annex C: Financial Modelling of the New Open Suite of Programs
• Annex D: Overview of the Research Plan

Message from CIHR's Science Council

As the major federal funder of health research in the country, CIHR must take appropriate action to ensure the long-term sustainability of its contributions to the health research enterprise. This includes having an Open Suite of Programs capable of developing and supporting a well-trained base of investigators with the skills and expertise needed to design and conduct innovative and diverse research and knowledge translation activities aimed at improving health. It also includes an expert peer review system that is well-managed, fair, and transparent in the selection and support of the most innovative and cutting-edge health-related proposals.

Over the years, the community has made it clear that CIHR's current research funding programs need to be streamlined to reduce program complexity, and to ensure that researchers spend less time writing and applying for multiple grants to support their research. We have also heard that CIHR's peer review system does not always adequately accommodate the diverse health-related applications it receives, and that the most appropriate experts are not always engaged in reviewing individual applications.

The Design Discussion Document (released in February 2012) introduced our proposal to address these challenges with its Open Suite of Programs and peer review processes. From the beginning, there has been an appetite from the community to be more involved in the design process. Members of the community shared their thoughts at over 80 face-to-face meetings across the country, through written correspondence, and through Web-based questionnaires. As summarized in What CIHR Heard: Analysis of Feedback on the Design Discussion Document (released in August 2012), there are diverse views across different health fields and career stages on the specific details of the proposed new Open funding schemes, and on how peer review should be conducted.

In addition to the feedback received on the Design Discussion Document, we engaged with institutions, University Delegates, Institute Advisory Boards, partners, associations, and a number of advisors from the research community on aspects of the proposed design. All comments and suggestions were considered in light of CIHR's mandate, its vision for the new Open Suite of Programs, and the multiple challenges we set out to address. This document outlines the revised, high-level design of the two new Open funding schemes and peer review process.

CIHR's Governing Council has endorsed this design, and has directed us to continue to plan for a thoughtful transition from the current Open programming. We are confident that this shared vision for the new Open Suite of Programs and peer review process will enhance CIHR's ability to sustainably deliver on its investigator-driven strategy. The health research landscape is changing, and now is the opportunity for CIHR and its community to design for the future, and position Canadian health researchers for success in this new world.

CIHR Science Council

Executive Summary

Overview of the new Open Suite of Programs and Peer Review Process

The design of the new Open Suite of Programs sets clear objectives for its investigator-driven grants program. This includes integrating CIHR's current suite of Open funding mechanisms into a simpler system that is flexible enough to accommodate today's changing landscape of health research. As a result, the existing suite of Open funding mechanisms was reorganized into two new schemes:

• Project Scheme
• Foundation Scheme

The Project Scheme is designed to capture ideas with the greatest potential for important advances in health-related knowledge, the health care system, and/or health outcomes, by supporting projects with a specific purpose and defined endpoint. There will be two Project competitions per year. As a result of the feedback we received, we re-designed the competition process for the Project Scheme to have a two-stage review process. The assessment criteria for this scheme will be based on the quality of the idea, and the project's feasibility. In Stage 1, all applications will be reviewed remotely by five reviewers. In Stage 2, face-to-face meetings will be held to integrate the results of the independent reviews, and to discuss "grey zone" applications, with particular emphasis on applications with large variances in independent reviewer rankings.

The Foundation Scheme is designed to contribute to a sustainable foundation of health research leaders by providing long-term support for pursuit of innovative, high-impact programs of research. There will be one Foundation competition per year. The Foundation Scheme will have a three-stage review process. The first two stages will involve remote review by five reviewers. The assessment criteria at Stage 1 will focus on the caliber of an applicant. Stage 2 will integrate elements of the caliber of the applicant, while focusing on the quality of the proposed program of research. As in the Project Scheme, the final stage of the review will involve a face-to-face meeting. Again, the objective of the final stage is to integrate the results of the independent expert reviews, and to discuss "grey zone" applications, with particular emphasis on applications with large variances in independent reviewer rankings.

Each scheme has been designed to accommodate the different ways research is conducted and applied in the various areas of health research and knowledge translation. CIHR anticipates that these two funding mechanisms will deliver on the goals of CIHR's investigator-driven grants program, and significantly advance health-related knowledge, the health care system, and health outcomes.

Table i. Summary of key features in the new Open Funding Schemes.

The College of Reviewers

To support the implementation of the new funding schemes, CIHR will also establish a College of Reviewers. The College is designed to serve as a framework for organizing and managing groups of reviewers, with a vision to:

"Establish an internationally recognized, centrally-managed resource that engenders a shared commitment across the Canadian health research enterprise to support excellent peer review of the diverse and emerging health research and knowledge translation activities that span the spectrum of health research."

The College of Reviewers is intended to enhance CIHR's current peer review system by:

• Supporting systematic recruitment to identify and mobilize the appropriate expertise for all funding applications submitted to CIHR;
• Developing customized orientation and development programs to provide reviewers with the knowledge and resources necessary to conduct consistent and fair reviews; and
• Establishing reviewer incentives and recognition approaches to attract, and retain, the breadth and depth of expertise required to review the breadth of applications received.

Transitioning to the New Open Suite of Programs and Peer Review Process

The transition to the new Open Suite of Programs will be a multi-year process. Starting with pilots of key design elements in 2013, changes to CIHR's current Open Suite of Programs and peer review processes will be implemented over a number of years until CIHR reaches its projected intake of grantees. Once the transition is complete, CIHR will support approximately 750 Foundation grants and 2,200 Project grants, for a total of 2,950 grants, through its new Open funding mechanisms. These numbers will increase should CIHR receive increases to its base budget.

Piloting Peer Review Design Elements

CIHR will conduct various pilot studies and tests for the new peer review process between early 2013 and mid-2016. These pilots will assess the proposed design elements, new processes, and systems to support the processes. If significant changes are required to the design as a result of the pilots, timelines for the overall transition will be adjusted.

We are committed to continuous assessment of the outcomes of the new peer review process. We will conduct research projects that will involve rigorous analytics to gain a greater understanding of the complexities of peer review, and of how best to measure the ongoing performance of the new Open Suite of Programs. It is our intent to share our findings with the research community, and contribute to the body of literature on peer review.

Phasing-in the New Open Suite of Programs

The phase-in of the new Open funding schemes will occur over the course of several competition cycles. The Foundation Scheme launched through two "live pilot" competitions (as described on p. 31) with application deadlines scheduled for fall 2014 and fall 2015. The first regular Foundation competition application deadline is scheduled for fall 2016. The first Project competition application deadline is scheduled for spring 2016.

We are committed to ensuring that adequate support is available to applicants, reviewers, and academic institutions throughout the transition process to help the research community navigate through this transition period with minimal disruption. This support begins immediately with the College of Reviewers, which has an anticipated launch date of spring 2013.

Transitioning from Existing Programs and Processes

We recognize the importance of carefully managing the implications for grantees who are currently supported by CIHR's existing Open Suite of Programs. CIHR will hold three more Open Operating Grant Program competitions (spring 2013, fall 2013, and spring 2014). There will also be a transitional Open Operating Grant Program competition in spring 2015, which will be held in parallel to the first "live pilot" of the Foundation Scheme. Other existing open programs will be phased-out after the launch of the first Project Scheme competition.

Given the scope and scale of the proposed changes, and the intention to learn from the results of the pilots, course corrections and adjustments to timelines may be required along the way. It is our intention to keep the research community informed of any changes to its transition plan as the Open Suite of Programs and peer review enhancements are implemented.

1. Purpose and Introduction

In February 2012, CIHR released a Design Discussion Document that outlined the initial thinking on changes to CIHR's Open Suite of Programs and peer review processes. Feedback was solicited and received from the research community on the proposed new design. A summary of this feedback was documented in the "What CIHR Heard" report, which was released in August 2012. As a result of this feedback, we modified design elements and made changes that include suggestions from, and address concerns raised by, the research community. The purpose of this document is to describe the revised high-level design of the new Open Suite of Programs and peer review processes. It summarizes the architecture and the mechanics of the new Open Suite of Programs, provides an overview of the College of Reviewers, and outlines the transition plan.

We thank the research community, our partners, and other CIHR stakeholders for taking the time to consider, and provide feedback on, the initial design of the new Open Suite of Programs and peer review processes.

Although this document outlines the key elements of the design of the new Open Suite of Programs and peer review processes, it is not meant to signal the end of the design process. There are still details to be confirmed before we implement the design, and the transition plan identifies a number of important pilots. After each pilot, time will be taken to adjust the design, if required.

Keeping the research community and other stakeholders involved and informed as we move forward with the implementation of the proposed changes remains an important priority.

2. A New Open Suite of Programs

The design of the new Open Suite of Programs sets clear objectives for CIHR's investigator-driven grants program. This includes integrating CIHR's current suite of Open funding mechanisms into a simpler system that is flexible enough to accommodate today's changing landscape of health research. As a result, the existing suite of Open funding mechanisms was reorganized into two new schemes:

• The Foundation Scheme is designed to provide long-term support to research leaders at any career stage with demonstrated track records of success, including new/early career investigators with excellent training and early-career productivity, to pursue innovative, high-impact programs of health research (including knowledge translation).
• The Project Scheme is designed to provide support for important, innovative, and impactful health-related projects brought forward by researchers and knowledge users¹ for a specific purpose and period of time.

The objective of the new Open Suite of Programs is to accommodate a diverse mix of applicants at any career stage from across the spectrum of health research and knowledge translation (Figure 1).

Figure 1. Overview of the objectives of the new Open Suite of Programs (Foundation Scheme and Project Scheme)

Long description: Figure 1

This strategy was designed to accommodate the different ways that research is conducted and applied in the various areas of health. CIHR anticipates that these two funding mechanisms will deliver on the goals of CIHR's investigator-driven grants program, and significantly advance health-related knowledge, the health care system, and health outcomes.

Other CIHR funding streams (e.g., strategic initiatives, training awards) are not directly affected by the re-design, and will continue to be a part of CIHR's strategy to develop research priority areas and support a sustainable pipeline of talented new health researchers and health-related professionals for the Canadian health research enterprise.

3. Project Scheme

Overview of the Architecture

The Project Scheme is designed to capture ideas with the greatest potential for important advances in health-related knowledge, the health care system, and/or health outcomes, by supporting projects with a specific purpose and defined endpoint. The best ideas may stem from new, incremental, innovative, and/or high-risk lines of inquiry or knowledge translation approaches. It is CIHR's intent to capture all of these ideas in the Project Scheme.

The Project Scheme is expected to:

• Support a diverse portfolio of health-related research and knowledge translation projects at any stage, from discovery to application, including commercialization;
• Promote relevant collaborations across disciplines, professions, and sectors;
• Contribute to the creation and use of health-related knowledge.

Grant Value and Duration

Project grant values and durations will be commensurate with the requirements of the project proposed, and will vary depending on the field, proposed approach, and scope of activities. Applicants will be eligible to apply to the Project Scheme for grants within the range of approximately $25,000 to $750,000 per year, with grant durations ranging between 1 to 5 years, reflecting the wide variety of projects presently funded in various Open programs.

Concerns have been raised about whether the proposed range of grant values for the Project Scheme will be sufficient to support certain types of research, in particular, clinical research such as randomized controlled trials. Historically, CIHR randomized controlled trial grant applications have requested an average of $282,000 per year, over 3.6 years. This is within the range of grant values for the Project Scheme. An alternative approach to funding large Project grants through CIHR's strategic funding envelope is currently under development.

Current modeling suggests that, based on the proportion of CIHR's Open funding grants budget allocated to this Scheme (approximately 55%), CIHR will be able to support an annual intake of approximately 939 grants of varying sizes and duration at steady state, once the phase-in and ramp-up period is over. It should be noted that modeling estimates have been based on a budget envelope of approximately $500 million, which is a conservative estimate of the combined current budgets of the Open Operating Grant Program and other existing Open funding mechanisms.

Eligibility

Project Leader(s) (Nominated Principal Applicant; Principal Applicants)

In the Project Scheme, CIHR anticipates supporting a diversity of projects from all areas of health research; some may be completely new, and some may build on the outputs of earlier projects.

Eligible applicants will include one or more independent researchers and/or knowledge users affiliated with an eligible institution.

The Project Scheme is accessible to eligible applicants working independently or in teams with one or more Project Leader. In cases where Principal Applicants are applying as a collective (e.g., as equivalent co-leaders), the Nominated Principal Applicant must be affiliated with an eligible institution.

A Foundation Grant Program Leader (e.g., Nominated Principal Investigator or Principal Investigator from the Foundation Scheme) will not be eligible to apply to the Project Scheme as a Project Leader (e.g., Nominated Principal Investigator or Principal Investigator). However, as CIHR supports collaborative approaches to health research and knowledge translation, there is no restriction on Foundation Grant Program Leaders participating in Projects as collaborators.

Collaborators and Partners

For some types of projects, partner collaboration or knowledge user involvement may be necessary, or even critical, to deliver on the objectives of the project and to achieve maximum impact. Each stage in the research process presents an opportunity for collaboration with knowledge users and/or non-academic partners (e.g., development or refinement of the research questions, selection of the methodology, data collection and tools development, selection of outcome measures, interpretation of the findings, crafting of the message, and dissemination of the results).

Applicants will be assessed on whether they have been able to secure the appropriate expertise needed to complete the project, and will need to demonstrate a sufficient level of engagement from these collaborators and/or partners in their application as appropriate.

Institutional Commitment

Overview of the Mechanics

In the February 2012 Design Discussion Document,a three-stage competition and application-focused review process was proposed for the Project Scheme. Members of the research community have raised concerns that it would be difficult to evaluate the quality of an idea/concept without also assessing the feasibility. As a result of this feedback, the remote review stages (Stage 1 and 2) were merged into a single stage for the Project Scheme. The two-stage Project Scheme competition and review process is illustrated below (Figure 2).

There will be no restriction on the number of Project grants that may be held by a single investigator during the transition period. This may be revised after transition.

Figure 2. Two-stage competition process for the Project Scheme. This figure represents the outcome of a single competition

Long description: Figure 2

Although application requirements are still being discussed, current thinking suggests that a Stage 1 application would be approximately 5-7 pages with very specific limits on attachments. Preliminary data will not be required. At Stage 1, applicants will submit a full proposal describing the idea/concept, its importance, the approach, the complement of expertise that will deliver on the objectives and the intended impacts of the project, the suitability of the environment, and the project budget. Although budgets will not factor into an application's overall ranking, peer reviewers will be advising CIHR on what constitutes an appropriate level of support for the project.

Stage 1 review will be conducted remotely by expert reviewers, and supported by an Internet-assisted platform that will enable communication among reviewers in a virtual space.

• Step 1: Each application will be assigned to five reviewers, based on optimal matching between the application content and reviewer expertise. Each reviewer will assess up to 20 applications.
• Step 2: Each reviewer will conduct a structured assessment of each assigned application against the established review criteria. The reviewer will assign a rating to each criterion according to a defined adjudication scale (Annex A). Each reviewer will then generate an initial ranked list of his/her assigned applications. This process will be assisted by an online support tool.
• Step 3: For each application, initial reviews will be shared with the other four reviewers assigned to the application. Reviewers will have an opportunity to initiate or participate in electronic discussions.
• Step 4: Once the discussions are complete, reviewers will confirm the final reviews and relative rankings of their set of applications, and submit the results electronically.
• Step 5: CIHR will consolidate all individual reviewer rankings into an average ranking for each application.
• Step 6: A combined ranking list will be computed for the competition. All applicants will receive structured feedback from each assigned reviewer.

Stage 2, the final assessment stage, will involve a face-to-face discussion by a number of inter-disciplinary committees (approximately 6-10). These committees will be responsible for integrating the results of the Stage 1 reviews. The discussion will focus on "grey zone" applications, with particular emphasis on applications with large variances in independent reviewer rankings. These committees will make recommendations on which "grey zone" applications to fund.

CIHR invests significantly in peer review, and many meritorious applications go unfunded. In the spirit of advancing health research and contributing to a sustainable health research enterprise, CIHR is exploring opportunities to share its pool of unfunded meritorious applications at the end of each competition with other health research funding organizations (e.g., partners, academic institutions, etc.). Discussions with health research funding organizations are underway.

The importance of reviewer matching

Matching applications to appropriate reviewers is essential to implementing application-focused review. We are currently assessing various IT-based tools to assist with the matching processes. These tools are intended to support human judgment and experience in validating matching assignments. We envision that external experts will help support this process. A number of pilots are being designed to ensure these IT-based tools are effective and are appropriately integrated with expert guidance.

Reporting and Re-Application Process

As per our current processes, end-of-grant reporting will be expected within 18 months of the Project grant term-end date.

Project grants are funded to achieve a specific purpose with a defined beginning, middle, and end. It is anticipated that new concepts, questions, or hypotheses will emerge from the outcomes of previous work, which may form the basis of subsequent project proposals. Some of these subsequent questions and hypotheses may be incremental in nature, following directly from results obtained from previous Project Grants. Project grantees interested in building on the results of previous projects can submit a new application, emphasizing sequential lines of inquiry, to the Project Scheme competition.

Proposed Structured Review (Adjudication) Criteria

CIHR continues to work with a variety of stakeholders including institutions, University Delegates, Institute Advisory Boards, and advisors from the research community, to develop the proposed structured review criteria. The adjudication criteria shown below outline the current thinking on what should be peer reviewed in the Project Scheme (Figure 3).

Figure 3. Proposed adjudication criteria for the Project Scheme at each stage of the multi-stage competition and review process

Long description: Figure 3

Peer reviewer guidelines will emphasize the importance of assessing each criterion in the context of research field, career stage, and institution setting. For more information on CIHR's current thinking on the proposed Project Scheme adjudication criteria, see Annex A.

4. Foundation Scheme

Overview of the Architecture

The Foundation Scheme is designed to contribute to a sustainable foundation of health research leaders, by providing long-term support for pursuit of innovative, high-impact programs of research.

The Foundation Scheme is expected to:

• Support a broad base of researcher leaders across career stages, areas, and disciplines relevant to health;
• Develop and maintain Canadian capacity in research and other health-related fields;
• Provide research leaders with the flexibility to pursue new, innovative lines of inquiry;
• Contribute to the creation and use of health-related knowledge through a wide range of research and/or knowledge translation activities, including any relevant collaboration.

As described in the February 2012 Design Discussion Document, a separate stream will be established for new/early career investigators, to capture emerging research leaders.

Grant Value and Duration

Foundation grant values will be commensurate with the requirements of the research proposed, and will vary depending on the research field, research approach, and scope of program activities. Investigators will be able to apply to the Foundation Scheme for grants within the range of approximately $50,000 to $1.5 million per year. Established investigators will be awarded 7-year grants, and new/early career investigators will be awarded 5-year grants. It is expected that an applicant's funding history will be one factor in determining future levels of funding.

Current modeling suggests that, based on the proportion of CIHR's Open funding grants budget allocated to this Scheme (45%), CIHR will support an annual intake of approximately 114 Foundation grants of varying sizes at steady state, once the phase-in and ramp-up period is over. It should be noted that modeling estimates have been based on a budget envelope of approximately $500 million, which is a conservative estimate of the combined current budgets of the Open Operating Grant Program and other existing open funding mechanisms.

The ability to transition current holders of CIHR grants (whether single or multiple) into the Foundation Scheme has been a subject of much discussion in the research community. Current thinking suggests that successful applicants to the Foundation Scheme with single or multiple sources of Open funding would have their existing, ongoing funds rolled-up into the budget of their new Foundation grant. This process would apply to current grant holders with multiple sources of Open funding (e.g., Open Operating Grant Program; Open Knowledge Translation funding programs), as well as future Project grantees interested in transitioning or consolidating their individual Projects into a cohesive program of research.

Eligibility

Program Leader(s) (Nominated Principal Applicant; Principal Applicants)

Foundation grants are about supporting research leaders from any career stage to build and conduct programs of health research across CIHR's mandate. Eligible applicants will include new and established independent researchers with a demonstrable track record of excellence and impact in their field of study. These researchers must be affiliated with an eligible institution.

In some areas, programs of research are co-managed by two or three researchers of equal standing. The eligibility of multiple Program Leaders on a single Foundation grant has been widely discussed in the research community. As a result of these discussions, single applications from multiple Program Leaders will be eligible in the Foundation Scheme. Such joint applicants will need to convincingly demonstrate synergy and a history of co-managing programs of research. It is expected that while the broader research team may evolve over time, Program Leaders (whether single or joint) will remain unchanged over the course of the grant.

Collaborators and Partners

Collaborations and partnerships are likely within the Foundation Scheme, and may change over time as the program evolves. Applicants will be assessed on whether they have the appropriate expertise and level of engagement from collaborators at the onset of the program, and a plan to engage future expertise. The level of engagement and/or contribution from collaborators and partners with Program Leaders in the Foundation Scheme will depend on the extent to which contributions (cash and/or in-kind) are essential for delivering on the program objectives. There is no intention to implement a requirement for matching funds for Foundation grants.

New/Early Career Investigators and Mid-Career Investigators

In the August 2012 What CIHR Heard: Analysis of Feedback on the Design Discussion Document report, we summarized respondent opinions that including a separate stream for new/early career investigators was a strength, but that CIHR should revise the definition of a new/early career investigator to be more reflective of actual early career trajectories. As a result, the definition of a new/early career investigator has been revised to:

"Any applicant who, at the time of application, has assumed his/her first independent academic position (e.g., faculty appointment) no more than 5 years (60 months) ago."

The success of new/early career investigators in the Foundation Scheme will be actively monitored. Peer reviewers will be trained to assess new/early career investigators to take into consideration their career stage, research field and institution setting. The Stage 3 assessment committee will review this pool of "grey zone" new/early career applicants separately.

Some members of the research community also expressed concern that there will not be adequate support for mid-career investigators in the Foundation Scheme. Mid-career investigators play an important role in the sustainable pipeline of research excellence. As with the process for new/early career investigator applications, peer reviewers will be oriented on how to appropriately apply adjudication criteria to mid-career investigators. We remain committed to monitoring the outcomes of competitions to ensure that they are accessible to researchers from all health fields at any career stage. A specific strategy for mid-career investigators will be considered, if required, based on multi-year analysis of competitions.

Institutional Commitment

As a condition of funding, Foundation grant applicants will be required to secure a formal commitment from their host institution to provide significant support beyond what is described in the Memorandum of Understanding (MOU) on the Roles and Responsibilities in the Management of Federal Grants and Awards.

The requirement for enhanced institutional support in the Foundation Scheme is intended to strengthen the partnership between CIHR and institutions, in order to jointly:

• Ensure that health research leaders succeed in their programs of research; and
• Contribute to a sustainable Canadian health research enterprise.

The longer duration, larger value, and greater autonomy associated with the proposed Foundation grants provide researchers with the flexibility needed to pursue innovative, high impact programs of research. We considered the recommendation to enforce the same requirement for enhanced institutional support in both the Foundation and Project Schemes, but decided to maintain the requirement for enhanced institution support for Foundation grants only. This requirement will help ensure that successful applicants have the institutional support required to optimize the investment made in the Foundation Scheme, and that the potential outcomes of CIHR-funded health research are maximized.

Overview of the Mechanics

The Foundation Scheme will be supported by a three-stage competition and application-focused review process that will focus reviewer attention on specific structured review criteria at each stage (Figure 4).

Note that Foundation Grant Program Leaders can only hold one active Foundation grant at a time.

Figure 4. Three-stage competition process for the Foundation Scheme

Long description: Figure 4

The focus of the review will be different at each stage. Stage 1 of the review will assess the caliber of an applicant (based on their vision for the proposed program of research), key achievements, and significant contributions. Although application requirements are still being discussed, current thinking suggests that the Stage 1 application would be approximately 3-4 pages, with specific attachments related to the Program Leader's CV(s). Applicants deemed to be competitive will be eligible to submit an application to Stage 2, where the review will assess the quality of the proposed program of research, research capacity, and supporting environment for the proposed program. Applicants will be asked to describe their proposed research concept and approach; the complement of expertise needed to deliver on the objectives and intended impacts of the proposed research; their plan to train and mentor outstanding individuals in health research and/or other health-related non-academics fields; the suitability of the research environment (including institutional support); and the program budget. Current thinking suggests that the Stage 2 application would be approximately 10-12 pages. Although budgets will not factor into an application's overall ranking, peer reviewers will be advising CIHR on what constitutes an appropriate level of support for the program of research.

Stages 1 and 2 of the Foundation Scheme will be conducted remotely by expert reviewers, and supported by an Internet-assisted platform that will enable communication among reviewers in a virtual space. All applicants will receive structured feedback from each assigned reviewer at each stage.

• Step 1: Each application will be assigned to five reviewers, based on optimal matching between the application content and reviewer expertise, with each reviewer assessing up to 20 applications.
• Step 2: Each reviewer will conduct a structured assessment of each assigned application against the established review criteria. The reviewer will assign a rating to each criterion according to a defined adjudication scale (Annex B). Each reviewer will then generate an initial ranked list of their assigned applications. This process will be assisted by an online support tool.
• Step 3: For each application, initial reviews will be shared with the other four reviewers assigned to the application. Reviewers will have an opportunity to initiate or participate in electronic discussions.
• Step 4: Once the discussions are complete, reviewers will confirm the final reviews and relative rankings of their set of applications, and submit the results electronically.
• Step 5: CIHR will consolidate all individual reviewer rankings into an average ranking for each application.
• Step 6: A combined ranking list will be computed for Stage 1 of the competition. Approximately 350 applicants will be invited to submit an application to Stage 2.

Stage 2 applications will follow a similar review process. At the end of the process, all individual rankings will be consolidated, with a ranked list created for Stage 2 of the competition.

Stage 3, the final assessment stage, will involve a face-to-face discussion by an inter-disciplinary committee. This committee will be responsible for integrating the results of the remote Stage 1 and 2 reviews. The discussion will focus on "grey zone" applications, with particular emphasis on applications with large variances in independent reviewer rankings. This committee will make recommendations on which "grey zone" applications to fund. Unsuccessful applicants with very low competition rankings will be ineligible to apply to the next Foundation Scheme competition, but will be eligible to apply the following year (i.e., two years from the initial Stage 1 Application deadline).

Reporting and Re-application Process

To meet federal reporting and accountability requirements, Foundation grant holders will be expected to report periodically on their productivity and achievements. This reporting process is still under development, and will be simple and easy to complete.

Foundation grantees interested in continuing their program of research beyond the duration of the Foundation grant may submit a new application in Year 6 to the regular Foundation Scheme competition. In exceptional circumstances, Foundation grantees may choose to submit a new application in an earlier year with the goal of obtaining a higher level of grant support. In all instances, current policies on early re-application would apply. If a subsequent Foundation grant is not secured, end-of-grant reporting will be expected within 18 months of the term-end date of the Foundation grant.

Proposed Structured Review (Adjudication) Criteria

As with the Project Scheme's adjudication criteria, CIHR worked with a variety of stakeholders to develop the proposed Foundation Scheme adjudication criteria. The adjudication criteria shown below outline the current thinking on what should be peer reviewed in the Foundation Scheme at each stage of the multi-stage competition and review process (Figure 5).

Figure 5. Proposed adjudication criteria for the Foundation Scheme at each stage of the multi-stage competition and review process

Long description: Figure 5

Peer reviewer guidelines will emphasize the importance of assessing each criterion in the context of research field, career stage, and institution setting. For more information on CIHR's current thinking for the proposed Foundation Scheme adjudication criteria, see Annex B.

5. College of Reviewers

The Vision and Objectives of the College of Reviewers

The College of Reviewers is intended to establish an internationally recognized, centrally-managed resource that engenders a shared commitment (across the Canadian health research enterprise) to support excellence in peer review across the diverse and emerging health research and knowledge translation activities that span the spectrum of health research.

The success of the College is underpinned by:

• The shared commitment of its members and health research funding organizations, manifested through various roles and responsibilities;
• The diversity of its members'expertise, experience, perspectives across disciplines, professions, and sectors related to health;
• A formal structure that provides support and orientation for, and maintains the quality and credibility of, the College;
• A base of qualified experts who are supported to uphold the excellence of the peer review system (Figure 6).

Figure 6. Vision and components of the College of Reviewers

Long description: Figure 6

The success factors outlined above have informed the following College of Reviewers objectives and key design elements (Table 3):

To deliver on the vision and objectives, the College of Reviewers will be built as a centrally-managed, national resource comprising a number of components that focus on the recruitment, orientation, performance management, and recognition of College members.

College members will be organized according to "faculties of expertise." These faculties will have dedicated Faculty Chairs responsible for monitoring and building an up-to-date faculty of reviewers. Various analyses are underway to document the profile of the current reviewer base, and to inform future recruitment strategies and targets.

CIHR plans to work early and closely with health research funding partners to build a College of Reviewers that is truly a collaborative national resource. Partners will be able to access the base of expertise and other features within the College, and work jointly with CIHR on expanding the base of expertise, on orienting reviewers, and on recognizing outstanding reviewers.

Roles and Responsibilities within the College of Reviewers

We envision different roles within the College of Reviewers with different levels of responsibility. Roles will include not only those related directly to the peer review of applications, but also those that support the quality of the peer review system as a whole (e.g., recruitment, mentoring and orientation, matching of reviewers to applications).

Examples of some of the other proposed roles and responsibilities are outlined below (Figure 7):

College of Reviewers Recruitment Program

The College of Reviewers recruitment strategy will ensure that the College has an appropriate base of qualified experts to uphold an excellent peer review system for the diverse and emerging health research and knowledge translation activities that span the spectrum of health research.

There will be a number of processes to support recruitment. These processes, which are described in Figure 8, include:

We will work with a number of key stakeholders, including academic institutions and partners, to identify and nominate potential candidates to the College of Reviewers.

College of Reviewers Orientation and Development Program

The objective of the orientation and development program is to ensure that College members efficiently acquire the appropriate knowledge and support to conduct high-quality peer review, in accordance with CIHR policies and review criteria. The orientation and development program is being designed to ensure that all members are informed, and supported in their various activities within the College, through specialized orientation modules for specific roles. A learning management system will be set up to enable College members to access online material.

In our discussions with the research community, we heard that it was important to provide opportunities for new/early career investigators to learn about peer review from more experienced peers. As part of the design, it is expected that more experienced members in the College will be trained to serve as mentors to new peer reviewers.

College of Reviewers Recognition and Performance Management Program

The objective of the recognition and performance management program is to establish policies and mechanisms that will monitor and promote quality in peer review, and recognize members'contributions and performance.

We will work with researchers and academic institutions to design and implement meaningful rewards and incentives. We are also committed to working with institutions to increase recognition of the value of peer review service.

We will monitor key milestones and report on activities for all members of the College of Reviewers. This data will be used to inform potential recognition activities, and will provide information on workload distribution and levels. This will be used to develop appropriate approaches for minimizing the overall burden on individual reviewers.

6. Transitioning to the New Open Suite of Programs and Peer Review Process

The implementation of the design for the Open Suite of Programs and peer review processes represents a significant change not only for the research community, but also for CIHR. Much of what we have heard to date from the community has stressed the importance of taking a measured approach to implementing these changes.

The transition to the new Open Suite of Programs will be a multi-year process, with changes to both CIHR's Open Suite of Programs and peer review processes implemented over a number of years until the projected intake of grantees is accomplished.

Once the transition is complete, the new schemes will support approximately 750 Foundation grants and 2,200 Project grants for a total of 2,950 grants based on current budget envelopes (Figure 9).

A number of scenarios were considered to determine a sustainable intake of grantees over the longer term. These scenarios looked at the average grant values and durations, competition cycles, and available budget for future competitions.

In developing this transition strategy, CIHR considered the following elements:

• Ensuring fairness and transparency;
• Ensuring access to the new funding schemes for a diverse pool of researchers and knowledge users;
• Minimizing the external impacts of the proposed timelines on institutions and the research community as a whole;
• Ensuring that the IT-systems and solutions developed to support application-focused review are extensively tested for user-acceptance and functionality.
• Ensuring a sustainable process that will minimize gaps in Open funding opportunities, manage future financial impacts, and ensure the uptake of new researchers.

It is CIHR's intent to keep the research community informed of any changes to its transition plan as the Open Suite of Programs and peer review enhancements are implemented.

The transition strategy includes three phases:

1. Piloting key peer review design elements;
2. Phase-in of the new funding schemes;
3. Gradual phase-out of existing Open funding programs.

The figure below (Figure 10) illustrates the three phases of CIHR's draft transition timelines. These phases are further defined and expanded on in the sections that follow.

Figure 10. Draft transition timelines for the piloting of peer review design elements, the phasing-in of the new Open Suite of Programs and peer review processes, and the phasing-out of existing Open programs. CIHR recognizes there will be impacts on funded and unfunded researchers. CIHR is analyzing the options to address these impacts. As a result, competition start dates and decision dates may shift forwards or backwards.

Long description: Figure 10

Piloting of Key Peer Review Design Elements

The success of transition to the new Open Suite of Programs and peer review processes rests on our ability to efficiently and effectively pilot and test the functionality of peer review design elements.

A major driver for the piloting stage of the transition strategy is the need to select and test the most appropriate IT solutions for the proposed peer review enhancements. As shown in Figure 10 above, we have allotted time during the transition period to analyze the outcomes of pilot studies, and to engage in comprehensive user testing.

Piloting peer review design elements will allow CIHR to adjust and refine processes and systems in order to best support applicants and reviewers. Two existing CIHR programs have been used to pilot some elements of the new design. The New Investigator Salary Award Program competition, held in June 2011, tested elements of a multi-stage competition and review process. This competition included an independent, individual review at Stage 1, followed by a face-to-face interdisciplinary committee review of select applications at Stage 2. A recent Community-Based Primary Health Care Team Grant competition also tested elements of the multi-stage competition and review, application-focused review, and structured review processes. Lessons learned from these competitions have helped refine the design of the new Open funding schemes and peer review processes.

A number of additional pilots will occur between early-2013 and mid-2016 (Table 4). Each will involve assessing one or more of the following design elements: multi-stage competition process; remote (virtual) screening process; application-focused review; structured review; a simplified rating scale; the number of reviewers per application; and the functionality of online discussion mechanisms.

These pilots will be conducted using competitions of smaller size so that we can properly train applicants, orient reviewers, and monitor outcomes in a managed fashion. Participants will be asked to provide feedback on the design elements through surveys and focused discussions.

In addition to these pilots, we will assess the longer-term outcomes of the new peer review process, as described in the overview of its research plan (Annex D).

Research projects will involve rigorous analytics to inform not only adjustments to the new design, but also a greater understanding of the complexities of peer review. It is our intent to share our findings with the research community, and contribute to the body of literature on peer review.

Phasing-in the New Open Suite of Programs

The phase-in of the new Open funding schemes will occur over the course of several competition cycles, as funds currently directed to ongoing commitments from the existing Open Suite of Programs become available. The Foundation Scheme will be launched first (as a "live pilot"), with the Project Scheme following thereafter (Table 5).

The phase-in of the Foundation Scheme would start with two "live pilot" competitions. "Live pilot" competitions will be regular competitions where CIHR will implement specific application intake strategies to manage application pressure.

Successful applicants from the first "live pilot" competition will be the first Foundation grantees. Applicants and reviewers will be asked to provide feedback on the competition process, which will inform adjustments to the second Foundation Scheme "live pilot." Similar intake strategies will be applied to the second "live pilot" to manage application pressure.

A transitional Open Operating Grant Program competition (spring 2015) will be held in parallel to the "live pilot" competition to allow those who are unsuccessful after Stage 1 of the Foundation Scheme competition an opportunity to apply to the Open Operating Grant Program competition.

The first unrestricted Foundation Scheme competition is planned for summer 2016, with an application deadline of fall 2016. It is expected that the first three Foundation Scheme competitions ("live pilots" and first regular competition) will have a greater than average annual intake as part of ramp-up efforts to achieve steady state in 2018.

The first Project Scheme competition will open to all eligible applicants in winter 2016, with an application deadline of spring 2016. Two Project Scheme competitions will be held regularly each year, with application deadlines planned for spring and fall.

Transitioning from Existing Programs and Processes

We recognize the importance of carefully managing the implications for researchers who are supported by CIHR's existing Open Suite of Programs, and are committed to minimizing disruptions. This section describes the phase-out of the Open Operating Grant Program and the Open knowledge translation programs. Our ability to afford the new Foundation and Project competitions relies on the phase-out of the current and future planned commitments to the existing open funding mechanisms.

The Open Operating Grant Program

Three more Open Operating Grant Program competitions (spring 2013, fall 2013 and spring 2014) will be held. There will also be a transitional Open Operating Grant Program competition held in parallel to the first "live pilot" of the Foundation Scheme (spring 2015). This will allow those who are not entering the Foundation Scheme "live pilot" an opportunity to apply for funding, and those who are not successful after Stage 1 of the Foundation Scheme competition an opportunity to apply to the Open Operating Grant Program competition.

Applicants will be able to register as early as October 2014 for the spring 2015 transitional Open Operating Grant Program competition. It is anticipated that CIHR will be able to support approximately 600 grants in the transitional Open Operating Grant Program competition.

Due to cash-flow and operational constraints, CIHR cannot support two Open Operating Grant Program competitions during the phase-in period of the transition. As such, there will be no competition with a fall 2014 application deadline. We recognize that this may impact currently funded researchers whose Open Operating Grant term-end dates fall within that period, as well as unfunded researchers seeking support. We are in the process of analysing a number of options to mitigate this effect. Discussions are underway to determine the most feasible and appropriate option for CIHR and its research community.

The Open Knowledge Translation (KT) Programs

The new Project Scheme was designed to reduce barriers and support a diverse suite of projects, including those which have been funded through Open knowledge translation programs. We recognize the importance of sustaining the Open knowledge translation programs until the Project Scheme is fully implemented. As a result, the Open knowledge translation funding mechanisms will have a delayed phase-out planned for the fall of 2016.

Implementing the College of Reviewers

The implementation of the College of Reviewers has been integrated into the implementation of the new funding schemes. The transition period for the College of Reviewers is anticipated for spring 2013 to 2016. Spring 2013 will mark the official launch of the College of Reviewers, with the first two waves informing ongoing recruitment strategies. The figure below illustrates and describes CIHR's recruitment strategy for the College of Reviewers (Figure 11).

Figure 11. College of Reviewers recruitment waves during the transition period. Adjustments to timelines may be required along the way.

The various waves of recruitment will be supported by the policies, procedures, and technology requirements specific to each wave.

Supporting Transition: Orientation for Applicants and Reviewers

We recognize that transitioning to the new Open Suite of Programs and peer review processes represents a significant change for applicants, reviewers, institutions, and partners. Ensuring that the research community is able to navigate through this transition period with minimal disruption is a priority for us all. To achieve this, CIHR is developing an orientation program that will:

• Provide orientation to all applicants, reviewers and research administrators about the new schemes (objectives, application requirements, review criteria, etc.);
• Provide guidance on roles and responsibilities for College members and institutions;
• Support the mentoring of new/early career reviewers.

Due to the large geographical distribution of researchers and reviewers, the primary orientation delivery method will be self-directed Web-based learning. Online, interactive orientation materials will be divided into short, modular segments of no longer than five to ten minutes. Instructor-led webinars will also be used to provide an interactive environment (using voice and/or chat) to facilitate discussion with participants. We are also discussing approaches that involve institutional administrative staff who currently support researchers through the application process.

As part of its orientation activities, CIHR will provide the research community with information, systems, and tools that will enable them to experience program requirements and competition processes on a trial basis. These features will also be made available to institutions who wish to provide specific training to their students and researchers on how to use CIHR systems.

7. Conclusion

Sustaining and supporting a strong health research enterprise is an essential part of Canada's ongoing ability to innovate and compete in a modern, knowledge-based, global economy. There is a paradigm shift underway in the way health research is being conducted around the world. Multidisciplinarity, networked collaborations, and timely research in emerging areas are quickly becoming the norm. Over the past several years, we have become increasingly aware of the need to modernize existing frameworks and systems to better capitalize on Canada's health research strengths, and to better capture the evolution in the health research landscape.

CIHR's investigator-driven strategy is central to its mandate. Our goal in designing the new Open Suite of Programs and peer review processes is to develop a high quality, flexible and sustainable system capable of identifying and supporting excellence in research and knowledge translation in all areas of health. Such changes will support and accelerate the transformational shift in the health research landscape that is already underway.

The February 2012 Design Discussion Document outlined our vision and plan to address the current and substantial challenges with our funding mechanisms and peer review processes, while the August 2012 What we Heard: Analysis of Feedback on the Design Discussion Document documented the research community's varied perspectives on the proposed changes. Since then, we have been working closely with a variety of stakeholders to further develop and change and refine the design.

Through the re-design of the new Open Suite of Programs, CIHR and its research community have created a future-oriented system that is poised to accelerate the advancement of health knowledge and improvements to health research, the health care system, and health outcomes. Work is now underway to refine implementation details and approaches for the transition period.

Given the scope of the proposed changes, and the intention to learn from the results of the pilots, course corrections and adjustments to implementation timelines may be required along the way. As the design phase for the new Open Suite of Programs and peer review process comes to an end, we invite the research community to join us in implementing this sustainable funding framework, which is capable of identifying and supporting excellence across the entirety of CIHR's mandate.

Annex A: Project Scheme Early Draft Adjudication Worksheet

Title:

Applicant:

Instructions:

• Step 1. Assess each criterion as per the adjudication scale and qualify each with 3-5 strengths and/or weaknesses in point form in the space provided.
• Step 2. Provide an overall assessment statement that takes into consideration all criteria.

Adjudication Scale

• A - For this sub-criterion, the application excels in most or all relevant aspects
• B - For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others
• C - For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others
• D - For this sub-criterion, the application has reasonably addressed all relevant aspects, but does not excel in any
• E - For this sub-criterion, the application fails to provide convincing information and/or has major flaws or gaps

Criterion 1: Concept

To assess the concept, the following sub-criteria apply:

Quality of the Idea (A, B, C, D or E)

• Are the overall goal and the objectives of the project well-developed and clear?
• Is the idea founded on a logical integration of concepts?
• Are the potential results/outputs clearly described?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Importance of the Idea (A, B, C, D or E)

• Are the proposed results/outputs likely to advance knowledge and/or make contributions to health research, the health care system and/or to health?
  • Will they address a significant need or gap?
  • Will there be a significant and/or sustained positive influence?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Criterion 2: Feasibility

To assess the likelihood of successfully implementing the project and achieving significant and sustainable impact, the following sub-criteria apply:

Approach (A, B, C, D or E)

• Are the objectives and milestones of the project realistic and achievable?
• Are the key approaches and methods appropriate to maximize the likelihood of achieving:
  • the objectives of the project?
  • the potential benefits/impacts of the project?
• Does the approach identify potential challenges and appropriate mitigation strategies?
• Does the approach include appropriate research and/or knowledge translation strategies and/or methods?
• Will there be sufficient financial resources to complete the project?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Expertise (A, B, C, D or E)

• Does the project have strong leadership and demonstrate the appropriate complement of expertise and experience needed to maximize the likelihood of achieving:
  • the objectives of the project, and
  • the potential benefits/impacts proposed?
• Where relevant, are the roles and responsibilities of any project team member clearly defined and appropriate?
• Where relevant, is there an appropriate and demonstrated level of engagement and/or financial (or other commitment) from key research and/or knowledge user collaborators?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Quality of environment (A, B, C, D or E)

• Is there appropriate access to the required personnel, facilities, infrastructure and other key resources to carry out the project?
• Is the environment (institutional or other) suitable for the proposed project?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Budget Recommendation

Reviewers are asked to consider the proposed budget of the research program, and provide a justified recommendation as to whether the program budget should be:

• Accepted, as described by the applicant; or
• Reduced.

Note that the budget recommendation is not integrated in the overall rating of the application, but rather is intended to inform funding levels and the applicant.

Budget Recommendation

• Are the requested resources sufficient to support the project (e.g., realistic and well-justified); taking into account other sources of funding, as appropriate?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Recommendation:

___ I recommend the proposed budget, as described by the applicant(s).
___ I recommend a reduction in the proposed budget to the level of $________ per annum.

Assessment Overview

Provide an overall assessment summary of the application given what you understand to be the purpose of the project.

Annex B: Foundation Scheme Early Draft Adjudication Worksheet

Stage 1

Applicant:

Title:

Instructions:

• Step 1. Assess each criterion as per the adjudication scale and qualify each with 3-5 strengths and/or weaknesses in point form in the space provided.
• Step 2. Provide an overall assessment statement that takes into consideration all criteria.

Adjudication Scale

• A - For this sub-criterion, the application excels in most or all relevant aspects
• B - For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others
• C - For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others
• D - For this sub-criterion, the application has reasonably addressed all relevant aspects, but does not excel in any
• E - For this sub-criterion, the application fails to provide convincing information and/or has major flaws or gaps

Criterion 1: Vision/Program Direction

Vision/Program Direction (A, B, C, D or E)

• Are the vision and overall objectives of the proposed program of research well-articulated?
• Is the vision forward looking, creative, and appropriately ambitious?
• Does the vision aim to significantly advance knowledge, health research, the health care system and/or health outcomes?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Criterion 2: Caliber of the Applicant

Research Leadership (A, B, C, D or E)

• Do the applicant(s) (Program Leader(s)) demonstrate significant and effective leadership or co-leadership with respect to:
  • Establishing, resourcing, and directing a program of research?
  • Managing mentoring/training activities, knowledge translation activities and collaborations?
  • Recognition and influence in their research field, including advancing direction in the field and building networks/communities?

  Note: In the case of multiple Program Leaders, also consider the above in the context of joint prior work of the named Program Leaders.

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Productivity (A, B, C, D or E)

• Have the applicant(s) (Program Leader(s)) demonstrated an excellent level of research outputs, based on prior work?
• Has the applicant'(s) previous work generated high quality research outputs?

  Note: In the case of multiple Program Leaders, also consider the above in the context of joint prior work of the named Program Leaders.

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Significance of Contributions (A, B, C, D or E)

• Have the applicant(s) significantly advanced knowledge, health research, the health care system and/or health outcomes?
• Have the applicant(s) (Program Leader(s)) engaged, trained, and launched the careers paths of outstanding individuals in research and/or other health-related non-academic fields?

  Note: In the case of multiple Program Leaders, the assessment of significant contributions should also consider the prior joint work of the named Program Leaders.

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Assessment Overview

Provide an overall assessment summary of the application given what you understand the program of research to encompass.

Stage 2

Applicant:

Title:

Instructions:

• Step 1. Assess each criterion as per the adjudication scale and qualify each with 3-5 strengths and/or weaknesses in point form in the space provided.
• Step 2. Provide an overall assessment statement that takes into consideration all criteria.

Adjudication Scale

• A - For this sub-criterion, the application excels in most or all relevant aspects
• B - For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others
• C - For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others
• D - For this sub-criterion, the application has reasonably addressed all relevant aspects, but does not excel in any
• E - For this sub-criterion, the application fails to provide convincing information and/or has major flaws or gaps

Criterion 1: Quality of the Program of Research

Research Concept (A, B, C, D or E)

• Are the goal(s) and objectives of the proposed program well-articulated?
• Is there conceptual coherence within the program of research?
• Are the potential program outputs significant? Are they likely to significantly advance knowledge, health research, the health care system and/or health outcomes?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Research Approach (A, B, C, D or E)

• Are the approach and key activities appropriate to deliver on the proposed program objectives?
• Does the approach include appropriate research and knowledge translation strategies and/or methods?
• Does the approach account for flexibility in direction as the program evolves?
• Does the approach identify potential challenges and appropriate mitigation strategies?
• Does the approach include an appropriate plan or framework to measure progress and success?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Criterion 2: Research Capacity

Expertise (A, B, C, D or E)

• Does the applicant(s) (Program Leader(s)) bring the appropriate expertise and relevant experiences to lead and deliver on the objectives of the research program and to achieve the potential benefits?
• Is there an appropriate complement and level of engagement and/or commitment (financial or in-kind) from relevant research and/or knowledge user collaborators to deliver on the objectives of the research program?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Mentorship (A, B, C, D or E)

• Does the research program incorporate a comprehensive plan for engaging, training and mentoring trainees and new/early investigators for research careers and/or other career paths in health-related non-academic fields?
• Are the proposed activities, project(s) and mentor(s) suitable?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Criterion 3: Quality of the Support Environment

Quality of the Environment (A, B, C, D or E)

• Has the applicant(s)' (Program Leader(s)') host institution(s) demonstrated a sufficient commitment to enable the long-term success of the various components of the program of research?
• Is the institution (or other environment) suitable for conducting the proposed research, training, knowledge translation and other relevant activities?
• Is there appropriate access to the required personnel, facilities and infrastructure to support the proposed research program?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Budget Recommendation

Reviewers are asked to consider the proposed budget of the research program, and provide a justified recommendation as to whether the program budget should be:

• Accepted, as described by the applicant; or
• Reduced.

Note that the budget recommendation is not integrated in the overall rating of the application, but rather is intended to inform funding levels and the applicant.

Budget Recommendation

• Are the requested resources sufficient to support the project (e.g., realistic and well-justified); taking into account other sources of funding if any?
• Has the annual budget taken into account the potential for program evolution?

Strengths: [Enter free-form or bulleted text here]

Weaknesses: [Enter free-form or bulleted text here]

Recommendation:

___ I recommend the proposed program budget as described by the applicant(s).
___ I recommend a reduction to the proposed program budget to the level of $______ per annum.

Assessment Overview

Provide an overall assessment summary of the application given what you understand the program of research to encompass.

Annex C: Financial Modelling of the New Open Suite of Programs

CIHR currently funds approximately 3,000 Nominated Principal Investigators through the existing Open Suite of Programs. The intention is to fund a similar number of Nominated Principal Investigators as the changes are implemented. Given an annual budget of about $500 million, and considering the number of grants to award in each scheme and the average grant size and duration, CIHR anticipates allocating 45% of its Open Funding budget to the Foundation scheme and 55% to the Project scheme (Table C1).

Foundation Scheme

Foundation scheme grants would be for 7 years (e.g., established investigators) and 5 years (e.g., new/early career investigators) with a weighted average value of approximately $300,000 per year. With a projected annual Foundation Scheme budget of $225 million and a funding ratio of one 5-year grant to five 7-year grants (average duration of 6.7 years), the annual intake for the Foundation scheme would average 114 grants, for a steady state total of approximately 750 Program grants (Table C2).

Project Scheme

Most Project scheme grants would be 3 or 5 years in duration, with a weighted average value of approximately $125,000 per year. It is anticipated that one third of Project grants would be for 1 or 2 years. With a projected annual Project Scheme budget of $275 million and these funding ratios of grants of varying duration, the annual intake for the Project scheme would average 939 grants, for a steady state total of approximately 2,200 grants (Table C3).

Annex D: Overview of the Research Plan

During the re-design of the Open Suite of Programs and peer review processes, CIHR noted the scarcity of literature on peer review, and encountered some gaps in its own application, peer review, and funding data. During this process, CIHR's research community has also communicated the importance of validating the proposed changes through data analysis, pilot studies, and the testing of new systems, so that decisions related to the proposed changes are evidence-informed.

In response, CIHR developed a Research Plan with the goal of informing a peer review process that will support the best funding decisions in the context of the objectives of CIHR's funding mechanisms. The Research Plan was designed with the following objectives:

• To increase the understanding of the complexities of peer review;
• To demonstrate and validate solutions to identified challenges in peer review;
• To better measure, monitor, and evaluate the ongoing performance of the peer review process and funding programs;
• To inform CIHR's community of research plan activities and findings.

In addition to the short-term pilot studies described in the Design Document, CIHR has begun to develop longer-term research projects to assess features of the new peer review processes, such as its reliability, consistency, and fairness (Table D1).

CIHR remains committed to piloting and rigorously evaluating various aspects of the reforms, and to investigating opportunities to study the feasibility and functionality of the proposed design elements. It is CIHR's intent to take a measured approach to evaluating its success, and identifying opportunities to adjust and enhance the system, as required.

1. CIHR defines a knowledge user as an individual who is likely to be able to use the knowledge generated through research to make informed decisions about health policies, program, practices or other knowledge end users such as those from the private sector.

Date modified:

2012-12-20