Open Knowledge Translation Pilot Competitions (2013-2016) – Peer Review Manual

Contents

Purpose of this manual

The purpose of this manual is to provide reviewers with a step-by-step walk through of the adjudication process for the Open Knowledge Translation pilot competitions. This manual is divided into two sections which correspond to the two stages of the adjudication process as illustrated in Figure 1 below.

Section 1 outlines the adjudication process for Stage 1 of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: 1) preliminary review of the applications, 2) asynchronous online discussion, and 3) submission of the final rank list to CIHR.

Section 2 outlines the adjudication process for the final assessment stage of the competition. It describes the responsibilities of reviewers participating in this stage, which includes the following: 1) pre-meeting activities, and 2) a face-to-face committee meeting.

The Final Assessment Stage face-to-face committee will be composed of reviewers who have not participated in the previous review stage.

In addition to reviewing the material in this manual, it is essential that peer reviewers read and become familiar with the Funding Opportunity. All CIHR peer reviewers are also expected to:

For detailed regulations concerning all aspects of CIHR funding programs, please refer to the CIHR Funding Policies.

Figure 1. Overview of the Adjudication Process of the Open Knowledge Translation Pilot Competitions

Figure 1: Long description

Section 1 – Adjudication Process for Stage 1

This section outlines the responsibilities of reviewers participating in the Stage 1 adjudication process. The adjudication process includes the following: 1) preliminary review of the applications, 2) asynchronous online discussion, and 3) submission of the final rank list to CIHR.

1.1 Preliminary Reviews of Applications

Stage 1 focuses on the Concept and Feasibility.

Each application will be assigned to five reviewers, based on optimal matching between the application content and reviewer expertise. Each reviewer will receive between 10 and 15 applications. Reviewers will access their assigned applications through ResearchNet and complete their reviews remotely. It is expected that it will take approximately 1.5-2 hours to complete each review.

The application will be presented in a structured format to align with the adjudication criteria. Reviewers will also be asked to review the applicant(s) CV(s). Through their CV(s), applicants will highlight their recognitions, funding history, activities and contributions that best demonstrate their leadership, significant contributions and productivity in the context of their research field(s).

Reviewers will be asked to provide a rating and to justify the rating by briefly stating the strengths and weaknesses for each sub-criterion as outlined below. It is important for reviewers to clearly articulate the strengths and weaknesses as they will be used for three purposes: 1) to provide applicants with feedback, 2) to provide the other reviewers assigned to the application with a justification for the ratings and rankings given to the application, and 3) to provide reviewers in the final stage with a justification for the ratings and rankings given to the application.

The specific adjudication criteria are outlined in the Open Knowledge Translation Pilot Competitions (2013-2016) – Interpretation Guidelines Adjudication Criteria.

Criterion 1. Concept (50%)

Quality of the Idea (25%)

Descriptors

  1. Are the overall goal and objectives of the project well-defined and clear; with distinct outputs that support advances in health-related knowledge, health research, health care, health systems, and/or health outcomes?
  2. Is the rationale of the project idea sound, logical and valid?
Importance of the Idea (25%)

Descriptors

  1. Is the proposed contribution(s) of the project well-defined, clear, and significant with respect to advancing health-related knowledge, health research, health care, health systems, and/or health outcomes?
  2. To what extent does the project respond to the objective(s) of the Funding Opportunity?*

* Reviewers will evaluate this in the sub-criterion entitled "Importance of the idea". However, applicants can address this question in any of the 4 sub-criterion text boxes.

Criterion 2. Feasibility (50%)

Approach (25%)

Descriptors

  1. Are the approaches and methods appropriate to deliver the proposed output(s) and achieve the proposed contribution(s)?
  2. Are the timelines and related deliverables of the project realistic?
  3. Does the proposal identify potential challenges and appropriate mitigation strategies?
Expertise, Experience, and Resources (25%)

Descriptors

  1. Does the applicant(s) bring the appropriate expertise and experience to lead and deliver the proposed output(s), and to achieve the proposed contribution(s)?
  2. Is there an appropriate level of engagement and/or commitment from the applicant(s)?
  3. Is the environment (academic institution and/or other organization) appropriate to enable the conduct and success of the project?

Reviewers should take into consideration the career stage, research field and institution setting of all applicants when assessing each criterion. New/early career investigators may show evidence of leadership, contributions and productivity in different ways than established researchers. The evidence should be notable compared to peers in similar fields and career stages.

More guidance is provided for reviewers in the Open Knowledge Translation Pilot Competitions (2013-2016) – Interpretation Guidelines Adjudication Criteria. These guidelines will be assessed after each pilot and may be refined if required.

To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale acknowledging that many of the ratings are likely to fall into this range. Reviewers are encouraged to use the full breadth of the scale and should use the increased granularity within the top descriptors to express differences within these categories. To facilitate this, the following scale including descriptors and definitions is provided:

Table 1. Rating Scale
Descriptor Abbreviation Definition

Outstanding

O++

For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.

O+

O

Excellent

E++

For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.

E+

E

Good

G

For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.

Fair

F

For this sub-criterion, the application broadly addresses all relevant aspects. Major revisions are required.

Poor

P

For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

As each review is completed, ResearchNet will automatically show a ranked list of the individual reviewer’s set of assigned applications based on the ratings (highest to lowest). Reviewers may make adjustments to their ratings until they are comfortable with the rankings, however, there will be no requirement to break ties at this stage of the adjudication process. Ultimately it is the application’s ranking that will determine its standing within the competition and will be used to make funding decisions.

Reviewers will also be asked to flag applications in ResearchNet that involve any of the issues outlined below. These issues are not to be considered as criteria for adjudication unless they impact the scientific quality of the application. For detailed regulations concerning these issues, please refer to the CIHR Grants and Awards Guide.

  1. Ethical and other policy requirements: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may flag specific issues, such as the use of human participants, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  2. Human pluripotent stem cell research: Applications involving the use of human stem cells which will be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”, but it is essential that this be reviewed by reviewers.
  3. Section 56 of the Federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Reviewers should flag such applications to CIHR staff who will follow up before funds are released, if the application is funded.

1.2 Budget Assessment

Reviewers will be required to review the requested budget and justification. Applicants will be asked to submit their budget using a modular template with budget increments. Reviewers will be required to determine if the requested budget is appropriate to support the proposed program of research and if it is realistic and well-justified.

Please note that the budget assessment must not be factored into the scientific assessment. Moreover, detailed scrutiny of each item will not be expected. Reviewers may recommend that the budget remain as requested or recommend an adjusted amount. If a reviewer adjusts the budget, he/she will be required to provide comments to justify their recommendation.

During the asynchronous online discussion reviewers will be required to discuss any discrepancies in their budget recommendations. In cases where a discrepancy still exists, the budget will be reconciled at the end of Stage 1 by CIHR staff and the virtual chair. The virtual chair may communicate with Stage 1 reviewers if there is a need for clarification.

Specific instructions are outlined in the Open Knowledge Translation Pilot Competitions (2013-2016) – Interpretation Guidelines Adjudication Criteria.

Budget Recommendation

The budget recommendation is intended to inform CIHR funding levels. While budget recommendations are not factored into the overall rating of an application, CIHR values the experience and perspectives of reviewers in estimating overall budget requirements of the proposed project.

The reviewers will then be asked to submit their preliminary reviews to CIHR in order to proceed to the next step.

1.3 Asynchronous Online Discussion

Once the preliminary reviews are submitted, the other reviewer’s reviews will be made available. Reviewers will be able to discuss each of their assigned applications with the four other reviewers who were assigned the same application. An asynchronous online discussion tool has been developed to allow reviewers to communicate with each other to share expert perspectives and to discuss any discrepancies.

The objective of the discussion will be to discuss and understand discrepancies in reviews and budget recommendations. There will be no requirement to reach consensus. Each reviewer’s name and their preliminary review (ratings, written comments, budget recommendation and ranking) will be visible for each application. At this point, if any reviewer has suggested a change in the budget, this should also be discussed. If there are limited discrepancies in reviews, it is possible that a discussion may not be required.

The discussion period will last 5 days and reviewers will be made aware of the planned discussion period for the competition well in advance of the deadlines. Reviewers may log in at any time during the discussion period and will not be expected to participate in online discussions simultaneously.

All five reviewers assigned to an application will be able to participate in the discussion as soon as all of the reviews for that application have been submitted. The discussion for all applications will open on the established deadline and it is therefore important to respect the established deadlines. Reviewers who have not submitted their preliminary reviews will not be able to participate in the discussion and this will be flagged in their online status.

A virtual Chair will also be assigned to the application to moderate the discussion. The virtual Chair will be tasked with ensuring that discussions take place if warranted, and may prompt reviewers to discuss. CIHR staff may also initiate a discussion, or provide input where necessary, to ensure that CIHR policies and procedures are followed.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include changing their ratings, rankings and/or editing their comments.

It should be noted:

  1. Every reviewer will have the ability to initiate a discussion thread. Every comment will be shared with every reviewer assigned to that application. Once comments are posted, reviewers will not be able to delete or edit them.
  2. Reviewers will be able to post comments for the attention of CIHR staff; however they will be visible by the other reviewers.
  3. A notification email will be sent daily to advise reviewers of new posts.
  4. As applications are discussed, reviewers may feel the need to update their preliminary ratings and reviews. This can happen at any time both during and after the discussion period. These modifications will not be accessible to other reviewers.
  5. A transcript of the online discussions will not be given to the applicants.

1.4 Submit the Final Rank List to CIHR

Once the asynchronous discussions are complete, reviewers will be asked to finalize their reviews and their individual ranked list. At this point, reviewers must break any ties in their individual ranked list. Once ties are eliminated, reviewers may modify the rank order of their assigned applications by moving the applications up or down the rank list. The original rank order will remain visible as a reference point. Once satisfied with the rank order of applications that they have been assigned, the reviewers will be asked to submit the final rankings to CIHR. Note that reviewers will not see the final rank lists of other reviewers. At this point, the adjudication process for Stage 1 is complete.

Once all rankings have been received, CIHR will calculate a consolidated ranking for each application and compile a list of all applications in the competition ranked from highest to lowest. Each application will have five reviews, a ranking from each reviewer, a consolidated ranking and a standard deviation for Stage 1.

CIHR will determine which applications will be discussed at the Final Assessment Stage based on the number of applications, consolidated ranking, standard deviation and funds available for the competition.

Please note CIHR is still finalizing the methodology that will be used to determine the number of applications moving to the Final Assessment Stage.

An overview of the Stage 1 adjudication process is outlined in Figure 2 below.

Figure 2. Stage 1 Overview

Preliminary Reviews of Applications

Reviewers will be asked to provide a rating for each adjudication criterion, and then to justify the rating by providing comments on the strengths and weaknesses. To ensure consistency, reviewers must adhere to a common adjudication scale. Note that granularity has been built into the top descriptors of the scale to allow reviewers to indicate finer differences between high quality applications.

The adjudication criteria are:

Criterion 1: Concept

  • Quality of the Idea (25%)
  • Importance of the Idea (25%)

Criterion 2: Feasibility

  • Approach (25%)
  • Expertise, Experience and Resources (25%)

Budget Assessment

Reviewers will be required to review the requested budget and justification. Applicants will be asked to submit their budget using a modular template with budget increments. Reviewers will be required to determine if the requested budget is appropriate to support the proposed program of research and if it is realistic and well-justified.

Please note that the budget assessment must not be factored into the scientific assessment. Reviewers may recommend that the budget remain as requested or recommend an adjusted amount. If a reviewer adjusts the budget, he/she will be required to provide comments to justify their recommendation.

Asynchronous Online Discussion

Once the preliminary reviews have been submitted, the other reviewer’s preliminary reviews (ratings and comments) will be made available to reviewers who have been assigned the same application. Reviewers will be able to read the preliminary reviews and discuss each of their assigned applications with the four other reviewers who were assigned the same application. This is the reviewers’ opportunity to share their expert perspectives, and to discuss any rating discrepancies in a virtual environment.

Reviewers will be required to discuss any discrepancies in their budget recommendations. In cases where a discrepancy still exists, the budget will be reconciled at the end of Stage 1 by CIHR staff and the virtual moderator/chair. The virtual chair/moderator may communicate with Stage 1 reviewers if there is a need for clarification.

Once the discussion for each application is complete, reviewers will be given the opportunity to make adjustments to their reviews as required. This may include changing their ratings, rankings and/or editing their comments.

Submit the Final Ranked List to CIHR

A reviewer’s applications will be ranked in order from highest-rated to lowest-rated based on the ratings given to the application by the reviewer.

At this stage, some applications may be tied (i.e., have the same rank). The reviewer will be required to break these ties by moving one application ahead of the other.

Once the reviewer is satisfied with the ranked list of applications, he/she will submit the ranked list to CIHR. Once all ranked applications have been received, CIHR will compile a list of all applications in the competition ranked from highest to lowest.

The number of applications, consolidated ranking, and the standard deviation of the applications, as well as the funds available for the competition will be used to identify which application will be discussed at the Final Assessment Stage.

Section 2 – Adjudication Process for Final Assessment Stage

This section outlines the responsibilities of reviewers participating in the Final Assessment Stage. The adjudication process includes the following: 1) pre-meeting activities, and 2) a face-to-face committee meeting.

This stage will be completed by one interdisciplinary committee. It is anticipated that the committee will comprise 13 to 15 reviewers covering the full spectrum of health research. The committee will include a Chair and Scientific Officer (SO).This committee will be responsible for integrating the results of the Stage 1 reviews and making recommendations on which applications should be funded.

2.1 Pre-Meeting Activities

Prior to the face-to-face meeting, each committee member will be assigned a subset of between 10 and 15 applications and each application will be assigned to five committee members. It is expected that it will take approximately 1-2 hours to complete each review. For each application, the committee member will have access to information from Stage 1, including the reviews, the consolidated rankings, standard deviations and the full applications.

In order to help committee members differentiate between applications, a binning system will be used. Reviewers will be asked to assign applications to a “Yes” bin (to be considered for funding) or a “No” bin (not to be considered for funding). Each reviewer will be allocated a minimum number of applications that may be placed in the “Yes” and “No” bins. This number will depend on the funds available for a given competition. Committee members will be asked to submit their recommendations to CIHR prior to the meeting.

This information will be consolidated and the Chair of the committee will have an opportunity to review the recommendations from the committee members prior to the meeting. CIHR staff will prepare a schedule for the face-to-face committee meeting in consultation with the Chair and SO. An illustration of the process is provided in Figure 3.

Figure 3. Final Assessment Stage – Pre-Committee Meeting Overview

Reviewer assigned to 10 – 15 applications.

Reviewer reads the Stage 1 reviews of their assigned applications.

Based on the reviews, the Final Assessment Stage reviewer must choose which applications should be considered for funding and which ones should not. The reviewer will receive a maximum number of applications that they may place in the “Yes” Bin (to be considered for funding), and the rest of the applications will be placed in the “No” Bin (not to be considered for funding).

Based on the yes/no binning recommendations reviewers made as part of the pre-meeting activities, CIHR will rank all the Final Assessment Stage applications in order from the highest ranked to lowest ranked. The primary sort criteria will be the number of “Yes” votes received, and the secondary sort criteria will be based on the Stage 1 results.

2.2 Face-to-Face Committee Meeting

The interdisciplinary committee will be asked to make recommendations on which applications should be considered for funding. This meeting will be supported by electronic voting technology.

Prior to the meeting, CIHR will have compiled the results from the pre-meeting activities where committee members binned their assigned applications. At the meeting, committee members will see the grant applications ordered from highest-rated to lowest-rated according to a combination of the pre-meeting yes/no binning recommendations and Stage 1 results. Based on these results, CIHR will place the applications into one of three groups:

  1. Group A (applications that the majority of the assigned committee members recommended should be funded )
  2. Group B (applications for discussion at the meeting)
  3. Group C (applications that the majority of the assigned committee members recommended should not be funded)

These groups are illustrated in Figure 4 below.

Figure 4. Pre-meeting Assessment Placed into Groupings

Figure 4: Long description

The meeting will start with the committee Chair asking the committee members to validate the groupings of applications. To validate, the committee Chair will ask if any committee member would like to move any of the proposed Group A or Group C applications into Group B so that they may be discussed by the committee. Once the committee is satisfied with the groupings, the Group A and Group C applications will be locked in and will not be discussed any further at the meeting.

Please note that an estimated funding cut-off line will be displayed within the list of Group B applications. This line is calculated using both the available budget for the competition and the budget requested in the grant applications. The estimated funding cut-off line is meant to be a tool to facilitate discussion and may not be entirely accurate as budget reductions may occur.

The committee should consider the estimated funding cut-off line when adding applications to Group B for discussion. For example, if only five grants will be able to be funded from the Group B pool, then it may not be effective or efficient to move 20 grants from Group C into Group B. However, the grant application budget should not be a factor in making recommendations for funding (i.e., should not be used to vote a high-budget grant out of the funding range in favour of more, but smaller-budget, grants). CIHR is committed to funding excellent applications.

The Chair will moderate the discussion of all Group B applications, and the Scientific Officer (SO) will prepare the notes that summarize the key points discussed for each application.

For each application that is discussed, the following process will be used:

  1. Any committee member in conflict with the application will be asked to leave the room.
  2. The Chair will ask the five reviewers who reviewed the application to indicate whether they voted “Yes” or “No” during the pre-meeting activities.
  3. The Chair will ask the assigned member(s) who voted the application “Yes” to start the discussion and then the member(s) who voted “No” to continue the discussion. The discussion will focus on the key points that led to their yes/no assessments.
  4. Committee members not assigned to the application will be asked to contribute to the conversation by asking questions and adding their opinions/advice/expertise to clarify points of disagreement.
  5. The SO will read the SO notes, which will capture the key elements of the discussion, to the committee for approval.

Once all of the applications in Group B have been discussed, members will be asked to vote on the Group B applications. Members will be provided with a maximum number of “Yes” votes (to be considered for funding). This maximum will be based on both the number of applications in Group B and the estimated funding cut-off line.

The committee Chair will go through the Group B applications one at a time, calling for a vote on each application. Using the electronic voting tool, committee members will vote either “Yes” or “No” for any application with which they are not in conflict. The Chair and SO will not be permitted to vote.

Committee members will be expected to adhere to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations. While committee members will not be in the room during the discussion of the applications with which they are in conflict, they will not be asked to leave during the voting process. They will not be permitted to vote on any in-conflict applications, however, they will see the final rank order of all applications.

The committee will conclude its work after all the members have agreed on the applications they recommend for funding. These recommendations will be summarized and submitted to CIHR for the final review of budgets and the funding decision. Applicants will be informed of the results of the competition once CIHR’s Scientific Council has approved the grants to be funded. All applicants will be sent a Notice of Decision.

An important component of any peer review committee meeting is the final review of the committee’s effectiveness and functioning, as well as a discussion of policy issues that may have arisen in the course of its deliberations. This discussion is important as it provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the piloted adjudication process.

Applicants, research administrators and reviewers will be asked to complete surveys after each stage to provide their feedback. Feedback received through the surveys will help refine the design of the new Open funding schemes and peer review processes.

Date modified: