Review guidelines – Priority-driven initiatives

Table of contents

• 1. Introduction
• 2. Review process
  • 2.1 Principles of review
  • 2.2 Types of review processes
  • 2.3 Adjudication models
  • 2.4 Evaluation criteria and factors
  • 2.5 Ranking and rating scale
• 3. Review committees and meetings
  • 3.1 Selection of CIHR review committee members
  • 3.2 Committee members
  • 3.3 Committee members roles and responsibilities (in person or teleconference)
  • 3.4 Sequence of steps at a committee meeting

1. Introduction

CIHR provides funding to researchers for innovative and targeted research that responds to the changing health needs and priorities of Canadians across all aspects of health, including biomedical research, clinical research, health system research, health services, the health of populations, societal and cultural dimensions of health and environmental influences on health, and other research as required.

The goal of CIHR’s priority-driven initiatives is to advance health knowledge and its application in specific areas of research identified by CIHR in consultation with other government departments, partners and stakeholders, in order to improve health systems and/or improve health outcomes in these priority areas.

The purpose of these guidelines is to provide information regarding the review process for priority-driven initiatives.

For detailed regulations concerning all aspects of CIHR funding programs, please refer to the CIHR Funding policies.

2. Review process

The integrity of the review process for strategic programs relies on well-established principles and policies that:

• ensure fair and effective evaluation; and,
• support CIHR’s objectives and strategic funding targets.

2.1 Principles of review

CIHR’s principles of review are:

• confidentiality;
• absence of conflict of interest;
• fairness; and,
• transparency.

Peer Review refers to the process used by CIHR to review applications submitted for funding. CIHR’s peer review process involves the evaluation of applications by a group of reviewers, who have (individually or collectively) the required experience and/or expertise, as determined by CIHR, to assess the quality and/or potential impact of the proposed research and/or research related activities, within the context of the funding opportunity objectives. CIHR’s peer review process is supported by academic reviewers, as well as non-academic reviewers, including industry, government and community-based representatives.

CIHR relies on peer review to ensure its mandate to excel, according to internationally accepted standards, in the creation of new knowledge and its translation into:

• improved health for Canadians;
• more effective health services and products; and,
• a strengthened Canadian health-care system.

In addition, the CIHR review process is supported by a number of policies and guidelines which help to ensure the quality of the review system and the quality of the research that is funded through its research programs.

All committee members (Chair, Scientific Officer, reviewers) must agree to:

• Abide by CIHR’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations.
• Declare their conflicts of interest with applications.
• Treat applications as protected information. Committee members must not divulge their contents to others, or use any information herein for any other purpose than peer review.
• Ensure that all review materials personally used are handled safely and disposed of according to the Guide on Handling Documents used in Peer Review.

2.1.1 Confidentiality

Confidentiality is information about a person that will not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. The use of this information for any other purpose is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.

CIHR instructs committee members to not approach or communicate in any way with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the applicant’s chances of success or failure. In turn, applicants are not to contact committee members, including the Chair and Scientific Officer, regarding the status of their applications (ratings, rank, etc.).

By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of CIHR's funding decision for a competition.

The identity of the reviewers will not be revealed to the applicants. However, a list of review committee members is typically published on the CIHR website 60 days after the approval of funding of a competition.

2.1.2 Conflict of interest

CIHR makes every effort to ensure that its decisions are fair and objective by identifying and addressing any conflict of interest between an applicant and a reviewer. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a conflict of interest means a conflict between a participant's (e.g., reviewers and observers) duties and responsibilities with regard to the review process, and a participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the participant:

• would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
• has a professional or personal relationship with an applicant or the applicant's institution; or,
• has a direct or indirect financial interest in a funding opportunity or application being reviewed.

A conflict of interest may be deemed to exist or perceived as such when committee members, reviewers or observers:

• are a relative or close friend, or have a personal relationship with the applicants;
• are in a position to gain or lose financially/materially from the funding of the application;
• have had long-standing scientific or personal differences with the applicants;
• are currently affiliated with the applicants' institutions, organizations or companies—including research hospitals and research institutes;
• are closely professionally affiliated with the applicants, as a result of having in the last six years:
  • frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
  • been a supervisor or a trainee of the applicants;
  • collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
  • been employed by the institution, when an institution is the applicant; and/or
• feel for any reason unable to provide an impartial review of the application.

All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff is responsible for resolving areas of uncertainty. All committee members must read and agree to abide by the COIC policy prior to viewing any application information.

2.1.3 Fairness

Success of the review system is critically dependent upon the willingness and ability of all reviewers to:

• exercise rigorous judgement;
• be impartial and reasonable; and,
• understand and take into account in a balanced way the particular context of each application.

A constructive, quality review helps applicants by pointing out strengths and weaknesses that could be improved upon.

In circumstances where the Chair or CIHR staff deems that the review expresses comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way or contains factually incorrect information regarding CIHR’s policies, CIHR reserves the right to remove such information from the review.

2.1.4 Transparency

CIHR ensures transparency in the review process through a number of different mechanisms. All applications submitted to CIHR are evaluated by reviewers who provide an overall assessment of the application. This assessment is performed using the application requirements and review criteria established for each individual competition, as seen in the relevant funding opportunity. In addition, the Scientific Officer (SO) notes capture the discussion during the peer review deliberations. All reviews and SO notes, as applicable, are shared with applicants.

On its website, CIHR publishes the Peer Review Committee Membership Lists when appropriate, and lists all CIHR successful applications by competition through its Funding Decisions Database.

2.2 Types of review processes

CIHR invites experts with various perspectives from the health research community (e.g., health researchers, health related professionals, policy makers, community leaders, patients and citizens, etc.) to become members of review committees to:

• evaluate applications submitted for a particular funding opportunity;
• rate them on their merit using a defined set of evaluation criteria so they can be ranked by CIHR in order of priority for funding; and,
• make recommendations on the budget needed to support the application.

Review committees make recommendations for funding to CIHR and partners, who in turn make the final funding decisions. Review for priority-driven initiatives is typically conducted using two fundamental types of review processes: committee meeting (in person or teleconference) and virtual review. Additional design elements may be used based on the needs of the competition.

2.2.1 Committee meeting

A committee meeting generally has reviewers meeting in person, or via teleconference, to discuss and rate the applications for a given competition. Reviewers assign an initial rating, which starts the discussion amongst committee members. After discussing the strengths and weaknesses of the application, a consensus score is reached by the assigned reviewers and the committee members, with the exception of the Chair and Scientific Officer. Reviewing members are then asked to score within +/-0.5 of the consensus score. The average of the committee members' score, or combined ratings for the application, is converted into a final ranking for the application. The rank order number generated for each application within a particular competition is used by CIHR to allocate funding.

2.2.2 Virtual review

Virtual review includes an electronic, at-home, review process, whereby each application is assigned to two or more reviewers who will provide a review and a recommendation using a rating scale.

For virtual review, reviewers do not meet to discuss the applications.

During virtual review, the role of the reviewer is the same as previously described in the context of committee meetings. Funding recommendations are made solely based on scores submitted by reviewers, along with their written review.

If there are sufficient discrepancies in the reviewers’ scores, reviewers may be given an opportunity to discuss the reviews and confirm their scores.

2.3 Adjudication models

2.3.1 Relevance review

The relevance review process is used by strategic leads and/or partners to assess the alignment of an application with a specific research theme described in the funding opportunity (FO). As the name implies, the process is used when it is important for applications to be relevant to, or in alignment with, targeted research components of the FO. The relevance review process typically takes place prior to the peer review process. Applications deemed relevant will proceed to the next step.

2.3.2 Scientific review

Scientific review is the standard review mechanism for assessing the scientific excellence of proposals submitted to a competition. It uses a clear set of evaluation criteria to measure key aspects of the proposals in relation to the main scope and objectives of the funding opportunity. Peer reviewers rate each criterion individually and provide a single overall score for the purpose of comparison, ranking and making funding recommendations. Weighting of criterion may be applied to indicate differing degrees of importance; however, criteria are typically treated equally. Only those applications that exceed the fundable threshold based on a consolidated overall score are considered for funding.

2.3.3 Merit review

Merit review is a type of review that uses separate scores, or ratings, for potential impact and scientific merit. This evaluation approach may be used to assess research projects that engage knowledge-users throughout the research process to inform the research plan, carry out the project, and apply the research findings.

In general, the potential impact score of an application reflects the importance of the project to the knowledge-users and the likelihood that it will have a substantive and sustainable impact on health outcomes, practice, programs and/or policy in the study context. Only those applications that exceed the fundable threshold on both potential impact and scientific merit will be considered for funding. The mean of the two scores will be calculated for the applications with a fundable score on both criteria to determine the final rating and establish a ranking list.

2.4 Evaluation criteria and factors

CIHR grant proposals typically use the following five criteria:

• Research approach
• Originality of the proposal
• Applicant(s)
• Environment of the research
• Impact of the research

These criteria may be elaborated and defined by specific factors which are considered by reviewers during the evaluation process. It should be noted, however, that some of these factors may not be relevant to every grant application. Also, each criterion and corresponding factors are not listed in order of importance, and the relative weighting of each criterion may vary depending on the objectives of each specific funding opportunity as defined by CIHR.

For more information, please refer to the Revised Grants Evaluation Criteria – Interpretation Guidelines. You should also consult the particular funding opportunity for complete program and review details, as many funding opportunities have their own set of evaluation criteria based on their unique research and funding objectives.

San Francisco Declaration on Research Assessment

CIHR is a signatory of the San Francisco Declaration on Research Assessment (DORA), which “recognizes the need to improve the ways in which the outputs of scholarly research are evaluated”. DORA recognizes that scholarly outputs include a broad range of contributions and impacts and are not limited to published journal articles.

Reviewers are to assess productivity and progress broadly by taking into consideration:

• a range of contributions (e.g., research publications, reports, books, guidelines, datasets, code, tools, training and mentorship, community engagement, standards, software, and commercialized products) and impacts (e.g., influence on policy and practice, health outcomes, societal outcomes, and distinctions-based, meaningful and culturally safe health research), and
• the context of the applicant (e.g., leave history, career stage, area(s) of research, experiential knowledge, diverse career paths, family responsibilities, pandemic impact) and how it may have affected their progress.

Metrics such as number of publications and citations, and size/number of research grants should not be used in isolation to assess productivity. Journal-based metrics (e.g., Journal Impact Factors) should not be used as surrogate measures of the quality and impact of individual research publications. As stated in DORA, the “scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published”.

2.5 Ranking and rating scale

Typically, during the review process, an application is provided a rating by the review committee. This rating is then used to generate a rank ordered list of applications which is used by CIHR to make funding decisions. Applications are funded until all available funding is exhausted or there are no more additional fundable applications.

Please note that in some competitions there may be a dedicated source of funds for a specific research area. These applications would still be funded in rank order but they are considered a separate funding stream/pool.

CIHR’s priority-driven peer review mechanisms commonly use a scale of 0.0 to 4.9 to rate applications, with 4.9 being the highest possible rating. Ratings are assigned qualitative descriptors (see Tables below) to help guide reviewers in their rating of applications. In order to be considered for funding, an application must receive a score of 3.50 or higher. Other rating scales may be used to support the objectives of a specific competition.

2.5.1 CIHR's rating scale for most programs

2.5.2 CIHR's rating scale for the Merit Review peer review process

Merit Review employs separate scores for Potential Impact and Scientific Merit.

* Only those applications that exceed the threshold rating of 3.50 on both potential impact and scientific merit will be considered for funding. The mean of the two scores will be calculated for the applications with a rating of 3.50 and above on both criteria to determine the final rating and establish a ranking list.

2.5.3 Binning process

The binning process can be used to focus discussions on meritorious applications at the committee meeting (in person or teleconference).

The binning process, in addition to a rating scale, is used to help differentiate between applications. As part of this process, applications are placed into groups to distinguish between applications that should and should not be discussed.

2.5.4 Streamlining of applications

Committee members have the option to not discuss an application if the average of the initial scores is clearly above or below what is considered a fundable score. This is referred to as high or low streamlining of an application. In this case, the initial scores will be averaged together to make the final score. Note that Scientific Officer notes will not be taken, and therefore, will not be provided to the applicant.

Because there is a limited amount of funds available for every competition/program, it is important that committee members focus their discussions on the most competitive applications to ensure that an accurate rank-order list is generated.

An application may be streamlined if:

• the binning process, if used, shows it is not competitive;
• it is clearly above the funding cut-off projection;
• it is clearly below the funding cut-off projection;
• the average of the reviewers' initial ratings is < 3.50; and/or,
• there is no objection from the other committee members that the application not be discussed.

3. Review committees and meetings

3.1 Selection of CIHR review committee members

While CIHR program design and delivery personnel and management are responsible for making the final decisions on committee membership, they extensively consult with relevant groups to determine the appropriate complement of members. Committee members are selected following the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC).

3.2 Committee members

A CIHR review committee typically consists of reviewers and sometimes a Chair, a Scientific Officer and a reader, depending on the needs of the adjudication model. Individual committee members are selected for their knowledge, expertise and/or experience.

Committee membership as a whole considers one or more of the following aspects:

• the need to cover the full range of research areas, relevant methodologies and experience for which the committee is responsible;
• the necessity for reviewing capability in both English and French so that applications in either official language can be evaluated by the committee; and,
• the need for regional representation and representation by gender proportionate to membership in the Canadian health research community.

3.3 Committee members roles and responsibilities (in person or teleconference)

3.3.1 Committee Chair

Persons serving the role of Chair in a committee meeting are expected to follow a clear set of general instructions as defined by CIHR.

Before the committee meeting, the Chair may:

1. Suggest names of potential committee members to CIHR during both the formation of the review committee and the assignment of applications, as required. 
2. Work with CIHR staff to establish a review sequence or agenda, and to determine when streamlining will be employed, if applicable.
3. Work with CIHR staff to manage conflicts of interest relative to review committee members.

At the committee meeting and in the role of moderator, it is the Chair’s responsibility to ensure that the review committee functions smoothly, effectively and objectively, and that a positive, constructive, fair-minded environment in which research proposals are evaluated is established and maintained. To do so, the Chair, may:

1. Provide opening remarks to the committee.
2. Make the proper introductions and explain the meeting process to members, as appropriate.
3. Ensure that all committee members who are in conflict with an application leave the meeting room before discussions between members about that particular application begin.
4. Appoint a delegate as Chair or Scientific Officer (SO) when either individual leaves the meeting room due to a conflict of interest with an application or any other reason. Whenever possible, the same individual should not occupy both roles in the context of the same committee meeting.
5. Fulfill an oversight role – does not rate applications or vote during the committee meeting.
6. Ensure the involvement of the entire committee in evaluating each application.
7. Ensure peer review is conducted in accordance with the San Francisco Declaration on Research Assessment (DORA).
8. Work with the Scientific Officer, as required, to summarize the discussion around each application, before the consensus rating is reached.
9. Ensure that a consensus rating is reached.
10. Ensure that both the budget and term of the grant are discussed amongst committee members and that comments are added to the SO notes, as appropriate.
11. Ensure that specific ethical concerns and other CIHR requirements are addressed, and that any related discussion is captured in the SO notes.

At the end of the committee meeting, the Chair may:

1. Propose that any application that has been identified by committee members as being inconsistently reviewed in the review process is discussed again.
2. Allow time to receive feedback from the committee members regarding the effectiveness and functioning of the committee.

3.3.2 Scientific officer (SO)

Individuals serving the role of Scientific Officer (SO) at a committee meeting are responsible for supporting the Chair in his/her role during the review committee meeting and are expected to follow a clear set of general instructions as defined by CIHR.

Before the committee meeting, the Scientific Officer may:

1. Suggest names of potential committee members to CIHR during the formation of the review committee, as well as during the assignment of applications, as required. This could include identifying potential external reviewers to fill specific expertise gaps.

At the committee meeting, the Scientific Officer:

1. Takes official notes of the committee discussions for each application (SO notes) and ensures that comments are relevant to and in alignment with the San Francisco Declaration on Research Assessment (DORA). These will be sent to applicants along with the reviewer reports, if applicable.
2. Reads back the SO notes to receive validation from committee members and to receive additional input before a consensus on the application is reached by the committee and member votes are cast.
3. Reads back the summary of changes to the budget and term of the grant on the application, and ensure that special considerations related to ethics and/or other issues have been flagged for the attention of CIHR and that these are recorded within the evaluation, if applicable.

At the end of the committee meeting, the Scientific Officer:

1. Finalizes and posts the SO notes directly to ResearchNet or sends them to the program coordinator no later than 5 working days following the meeting.

Notes:

• Scientific Officers do not rate applications or vote during the committee meeting.
• The SO notes should provide the applicant with insight into the committee discussion of his/her application. Their purpose is two-fold:
  • to identify the determining factors in the evaluation of the application by the committee; and,
  • to describe how the budget and term was determined by the committee (if applicable).
• SO notes should be clear and concise, and give objective and constructive feedback to the applicant. They should:
  • include the strengths and weaknesses of the application, in alignment with DORA;
  • address the issues that had the greatest impact on the evaluation, as they relate to the evaluation criteria in the funding opportunity;
  • address aspects of the committee discussion that were not captured in the reviewer written reports;
  • describe how reviewer disagreements, as seen in the individual reviewer reports, were reconciled by identifying which view was favored by the committee; and,
• CIHR will not edit or modify the SO notes without the Scientific Officer’s approval.

3.3.3 Reviewers

Reviewers are committee members assigned to review one or more applications. Applications are assigned to a minimum of two reviewers for assessment, who participate in the rating of applications and submit an in depth written report that is provided to the applicant. If applicable, they present their review at the committee meeting where they lead the discussion, and vote during the meeting. They also participate in the discussion and rating of all other applications before the committee for which they are not in conflict. Note that reviewers may also be assigned other applications as readers.

Before the committee meeting, reviewers must:

1. Evaluate the applications assigned to them by providing a critical assessment of the application, as well as constructive feedback based on the objectives and evaluation criteria described in the funding opportunity:
   • consider all factors and the strengths or weaknesses of the applications in relation to each criteria and in alignment with DORA;
   • emphasis may be placed on specific criteria in the funding opportunity details, in order to meet funding program objectives;
   • focus their comments on the factors most relevant to the rating;
   • provide comments on the budget requested and a formal recommendation, including clear and detailed reasons for any recommended budget or term cuts, if applicable; and,
   • provide comments on issues that they feel should be flagged, as required. These concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the application.
2. Provide an initial rating (this rating can be changed).

Note: The written report and the initial rating should be submitted on ResearchNet (usually one week before the committee meeting and no later than the day before the committee meeting).

Once a review is submitted as final, reviewers can view reviews submitted by other reviewers. However, they can no longer modify their reviews prior to the meeting. Following the meeting, reviewers have one week to directly upload changes to ResearchNet.

3.3.4 Readers

Reviewers may also be assigned applications in the role of reader. Readers are responsible only for reading an application and are not required to submit a written review. Rather, they serve as a formal discussant in the committee and aid in reaching a consensus rating.

They may be asked to provide an initial score if the reviewers are not in close agreement. This information will not be recorded and is only used for discussion at the committee meeting.

While no written review is required from the reader, it is recommended that he/she prepares speaking notes and be prepared to discuss the application with the other reviewers at the committee meeting.

3.4 Sequence of steps at a committee meeting

The following steps are typically used during the meeting after all members in conflict have left the room/teleconference. Other models may be used to support the design of the competition.

1. The reviewers announce their initial ratings and, if applicable, their overall assessment of quality, to determine if the application should be streamlined.
2. The first reviewer gives a brief synopsis of the proposal, followed by the assessment of the strengths and weaknesses of the proposal according to the evaluation criteria for the funding opportunity and comments on any issues to be flagged.
3. The other reviewers follow, concentrating on points of agreement or disagreement, and elaborating on points not addressed by the first reviewer.
4. Following any other points from the reader and the discussion of the application by the entire committee, the Scientific Officer reads the SO notes and seeks clarification/validation
5. The Chair seeks a "consensus rating" from the reviewers who may revise their initial ratings as they see fit. If a consensus cannot be reached, the Chair declares the consensus score, usually by using the mean value of the revised respective scores after discussion (i.e., round up, if necessary, to obtain a single decimal point).
6. All committee members, with the exception of the Chair and Scientific Officer, must then cast individual confidential votes within ± 0.5 of the consensus rating. The internal reviewers are not bound to the consensus rating. The final rating assigned to the proposal is the average of these confidential votes.

Date modified:

2024-01-16