CIHR Reviewers’ Guide for Doctoral Research Awards

Table of Contents

Introduction

On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide instructions on the peer review process specific to the Canada Graduate Scholarships Doctoral Award (CGS D) and the Doctoral Foreign Study Award (DFSA) programs.

Note: As part of reviewer’s engagement in peer review at CIHR, we ask reviewers to take a moment to complete or update the Reviewer Profile on ResearchNet. The Reviewer Profile has been developed as a tool to build and support reviewer expertise management.

Peer Review at CIHR

Information on CIHR’s objectives, governance and policies,an outline of the roles and responsibilities of peer review committee members, and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting the reviews.

Summary of the Peer Review Process

As part of your engagement in peer review at CIHR, we ask that you take a moment to complete or update your Reviewer Profile on ResearchNet. The Reviewer Profile has been developed as a tool to build and support reviewer expertise management.

At this time, we ask that you please login to ResearchNet and select “My Reviewer Profile” from the left-hand navigation bar to create or update your Profile. When completing your Reviewer Profile, in the Reviewer Type section under the Researcher group please select “Other” and enter “Post Doctoral Fellow”. Please only complete the sections that are most applicable to you based on your experience as well as your ability and willingness to review. For additional information on how to proceed, please view a quick learning module entitled Completing your Reviewer Profile. 

An individual structured review process via the online ResearchNet platform is used. The review process is completed in one stage: an individual review and rating of an assigned set of applications (i.e. there is no committee meeting). All eligible applications received will be assigned to two (2) reviewers.

There are two (2) peer review committees. Each application received will be assigned to the committee with the scientific area that has been chosen by the applicant, which most closely aligns with the applicant’s proposed research activities. For more information regarding the committee mandates refer to: Peer Review Committees and Mandates – Doctoral Research Award.

Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:

  • Step 1: Read the pertinent documentation
  • Step 2: Identify conflicts of interests
  • Step 3: Conduct in-depth review of assigned applications
  • Step 4: Submit reviews and ratings on ResearchNet
  • Step 5: Be prepared for a re-review

Step-by-Step Instructions

Step 1: Read the pertinent documentation

The peer review process is described in detail in this document. It is essential to read the document and be familiar with it. It is also important to read the following:

Step 2: Identify conflicts of interests

To identify conflicts of interests, reviewers must:

  • Log into ResearchNet.
  • On the home page, click on the link of their assigned committee to open the main task list.
  • Complete the task Review Confidentiality and Conflict of Interest Guidelines (once completed, it will “open” the other tasks).
  • Open the Manage Conflicts task.
  • For each of the assigned applications, use the information provided to indicate if you are able to review or if there is a conflict. If there is a conflict, CIHR will reassign the application to another reviewer.
  • Unlike grants, CIHR assigns the Doctoral Research Award applications to the various peer review committees based on a review of the scientific area identified by the applicant and not by the specific expertise of reviewers, which explains why you may be assigned applications that are not necessarily within your area of expertise. From a non-specialist's perspective, we ask that you assess the intellectual challenge of the research in which the candidate will be involved. Although you may not be familiar with the field, we ask that each reviewer complete their reviews from the point of view of a generalist.
    • Note: As other reviewers within your committee will also declare conflicts, there is a likelihood that you will receive additional application assignments. The calibration of workloads will be maintained to ensure a fair peer review process.

Step 3: Conduct in-depth review of assigned applications

Once conflicts have been identified, the full content of the remaining assigned applications will be available under the task “Conduct Reviews”. Reviewers should then follow the steps below.

3.1 Review the evaluation criteria

Reviewers should first become familiar with the evaluation criteria found at the end of this document in Appendix A. This appendix provides an interpretation of each criterion and identifies which elements of the application to review for that criterion.

CIHR is a signatory of the San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs include a broad range of contributions and are not limited to published journal articles. When assessing applications, reviewers should consider:

  • A range of contributions (e.g., research publications, reports, books, guidelines, datasets, code, tools, training and mentorship, volunteerism, community engagement, standards, software and commercialized products) and impacts (e.g., influence on policy and practice, influence on direction of thought, use of research results by stakeholders, health outcomes, societal outcomes, and distinctions-based, meaningful and culturally safe health research); and
  • The context of the applicant (e.g., health problems, family responsibilities, disabilities, trauma and/or loss, pandemic impact, career stage, area(s) of research, experiential knowledge and diverse career paths) and how it may have impacted the applicant's performance (see range of contributions). Please see specific notes about research area and career stage:
    • Reviewers should consider the research area/discipline of the applicant when reviewing an application, for example, publication productivity can vary when comparing a biomedical researcher, a clinician or a social scientist.
    • Reviewers should consider the career stage of the applicants to better assess and calibrate their set of applications, for example, direct entry to fellowship from PhD vs. entry after several years outside academia. Please note that, while CIHR has removed the eligibility requirement that all applicants be within 3 years of their PhD, this is not intended to encourage additional time spent in postdoctoral positions. Reviewers are asked to think critically about whether the training position for which an individual is applying for funding will have the desired career benefits and impact compared to other applicants.

Reviewers should guard against placing too much value on the number of contributions – the focus must be on the impact and significance of the contributions. Reviewers should not use journal-based metrics such as the Journal Impact Factor as surrogate measures of quality and/or excellence as they introduce bias into the review process. Citation rates vary between disciplines and contexts; members must be mindful of this when considering them as part of their evaluation.

As stated in DORA, the “scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published”. Applicants are instructed to elaborate and comment on their role, positive impact and importance of their research contributions.

3.2 Read the assigned applications

Reviewers should read all of their assigned applications in detail before rating any of them, and take notes to capture their impressions of the applications.

It is important to note that many candidates will likely be conducting research outside of the reviewer’s research specialty. Therefore, they should review the application with a generalist’s perspective and assess the overall quality of the research proposed by the candidate, using the appropriate evaluation criteria.

It is important that reviewers are aware of the importance of the integration of sex and gender in research proposals and assess their appropriateness in their assigned set of application. To learn more about Sex and Gender considerations in Research, reviewers are encouraged to complete these training modules:

When evaluating applications that involve First Nations, Inuit and Métis Peoples, reviewers are strongly encouraged to review the following training module and literature:

To ensure that all applications are evaluated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research on the candidate or the proposed institution (e.g. publications via PubMed, etc.).

3.3 Learning for participants in peer review

CIHR offers a number of learning modules to help you gain in-depth knowledge about our programs, processes and tools. These modules are intended to ensure that all participants in the peer review process have the same base knowledge of the processes and policies in order to conduct effective and fair peer review.

Addressing bias

As reviewers read the applications, they should be aware of and take actions to mitigate against bias related to gender, ethnicity, disability, Indigenous identity and Indigenous ways of knowing, institution size, region, age, language and interdisciplinary approaches to research. For example, it is important to be aware that:

  • Career interruptions for childbearing and raising can affect opportunities for knowledge production, publications and related variables;
  • Different disciplines and environments offer different opportunities for research contributions (e.g., publications, influence on policy and practice, patents, knowledge mobilization activities, etc.);
  • The reputation of institutions should not affect the reviewer’s view of applicants or their research training environment;
  • It is important to take steps to mitigate bias in reviewers’ thought process about difference in culture (e.g. Indigenous Peoples);
  • Recognize Indigenous identity in a reconciliation, nation-to-nation framework; and,
  • A point should be made of respecting Canada’s linguistic duality by recognizing the value of research in French and the value of research in Francophone minority communities.

To learn more about bias in peer review, reviewers must complete the related learning module.

Sex and Gender in Health Research

Reviewers must complete one of the two Sex and Gender in Health Research online modules self-selected by each reviewer to align with their area of expertise:

3.4 Rate the assigned applications

Reviewers are then asked to rate their assigned applications against each of the evaluation criteria described in Appendix A, using CIHR’s traditional rating scale (below). It is particularly important that the full scale be used.

Descriptor Range Outcome
Outstanding 4.5 – 4.9 May be Funded
Excellent 4.0 – 4.4
Very Good 3.5 – 3.9
Good 3.0 – 3.4 Not Fundable
Average 2.0 – 2.9
Below Average 1.0 – 1.9
Not Acceptable 0.0 – 0.9

Applications with an overall weighted score below 3.5 are not eligible for CIHR funding, including those from partnerships/priority announcement programs.

3.5 Provide a written assessment for each assigned application

Reviewers are asked to provide a short written assessment for each assigned application that supports their ratings. The written reviews will provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training.

Comments should highlight the strengths and weaknesses of each evaluation criterion:

  • Keep it simple;
  • Use familiar descriptors, such as those from the CIHR rating scale that align with your rating;
  • Include justification, context and an explanation of your comments, if applicable, for each topic introduced;
  • Respect DORA principles by considering a range of contributions and impacts and avoiding the use of journal-based metrics such as Journal Impact Factors as surrogate measures of quality. Do not use metrics such as number of publications and citations, and size and number of grants or awards in isolation to assess the applicant – instead, balance quantitative metrics with qualitative parts of the proposal;
  • Be clear and concise;
  • While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
  • Use objective and non-inflammatory language – avoid any biased language or approach;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.
  • When referring to the applicant use gender neutral pronouns or phrases. For example, use “they” or “the applicant,” rather than “he” or “she”.

The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the comments and should not identify themselves in order to ensure the confidentiality of the review process. For additional guidance on conducting high quality reviews please refer to CIHR’s Review Quality Expectations, as well as the Conducting Quality Reviews learning module.

3.6 Flag issues for CIHR’s attention

Any concerns regarding the application, eligibility, research with human participants, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments unless they impact the scientific quality of the application. For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards.

Similarly, if you believe there may be a misrepresentation of information in an application or a breach of Agency policy, please inform us immediately so we can determine whether the issue needs to be addressed through the responsible conduct of research (RCR) process. For a list of potential breaches, please refer to the Tri-Agency Framework: Responsible Conduct of Research.

Step 4: Submit reviews and ratings on ResearchNet

As the reviewers perform their evaluation, the reviews can be saved as drafts by selecting “Save draft copy” on ResearchNet. This allows them to make changes at a later time. However, in order to submit the reviews and ratings to CIHR, they must select “Submit Final Review”. Afterwards, they will no longer be able to modify them.

It is important for reviewers to respect the deadline provided by submitting the reviews and scores in ResearchNet by the date specified via correspondence with CIHR staff. Delays in the peer review process will jeopardize CIHR’s ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that they may not be able to submit their reviews on or before the deadline, they must contact CIHR staff as soon as possible.

Step 5: Be prepared for a re-review

Once all the overall weighted scores are submitted, CIHR will perform a discrepancy review by calculating the final rating for each application. CIHR will then identify applications that are at risk of an unfair decision because of a wide gap between the two reviewers’ overall scores. In such cases, CIHR will ask both reviewers to reconsider their initial assessment and resubmit scores. To do so, they will be asked to get in contact with each other to discuss the application. This second review usually reduces the gap between overall scores to an acceptable size. If the discrepancy persists, CIHR will obtain a third review (unless the persisting discrepancy is found to have no impact on the overall top-ranked applications).

For this purpose, it is recommended that reviewers keep their working notes on file up until a few weeks after the competition results have been announced.

Ranking of applications

Upon completion of the peer review process, CIHR will generate a ranking list for the applications. This information will be used for funding decision-making.

Feedback

An important component of the peer review process is the review of the committee’s effectiveness and feedback on policy issues that may have arisen in the course of the process. This feedback may be used to support CIHR’s ongoing efforts to maintain an effective and high-quality peer review system.

Since there are no face-to-face or teleconference meetings, the reviewers’ feedback should be communicated to the committee coordinator by email.

Appendix A – Evaluation criteria

The following criteria are to be used for evaluating applications:

Description Information Source Notes/Advice for reviewers
Research Ability and Potential (weight in overall score: 50%)

Indicators of Research Ability and Potential:

  • Quality of research proposal
    • specific, focused, and feasible research question(s) and objective(s)
    • clear description and soundness of the proposed methodology
    • significance and expected contributions to research
  • Relevant training; such as academic training, lived experience and traditional teachings
  • Demonstration of ability and potential to carry out proposed research relative to expectations of someone with the candidate’s academic experience
  • Quality of contributions and extent to which they advance the field of research. Contributions may include: publications, patents, reports, posters, abstracts, monographs, presentations, creative outputs, knowledge translation outputs, community products
  • Demonstration of sound judgment and ability to think critically
  • Demonstration of responsible and ethical research conduct, including honest and thoughtful inquiry, rigorous analysis, commitment to safety and to the dissemination of research results, and adherence to the use of professional standards
  • Demonstration of research, originality, initiative, autonomy, relevant community involvement and outreach
  • The ability to communicate theoretical, technical and/or scientific concepts clearly and logically in written and oral formats
  • Training Expectations
  • Research Project Summary
  • Common CV
  • “Publication List” attachment, if provided
  • Sponsor’s Assessments (keeping in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors)
  • “Leaves of Absence and Impact on Research” section of the Common CV, if provided
  • Special Circumstances” document (if provided)
  • Most candidates will be conducting research outside your research specialty. From a non-specialist's perspective, assess the intellectual challenge and excitement of the research in which the candidate will be involved.
  • Review the candidate's training expectations and the proposed research project summary, as well as research experience and achievements.
  • Review the list of articles, presentations and other publications produced by the candidate and assess these activities relative to your expectations of someone with their academic experience.
  • Consider breadth of science covered and the frequency of publication. Bear in mind that publication activity patterns vary among health science disciplines.
  • In considering the candidate's contribution to the publications, take into account the number of co-authors for each paper and the prominence of the candidate's name on the list of authors. Do not consider publications listed with a "Submitted" status, unless available in a recognized scientific public archive.
  • Examine the sponsor assessments, recognizing that positive comments are common while negative ones are not. Provide a score based on your overall impression (e.g. extent to which the box scores and narratives are consistent, length of time sponsor has known the candidate, relationship of the sponsor to the candidate, etc).
Relevant experience and achievements obtained within and beyond academia (weight in overall score: 50%)

Indicators of relevant experience and achievements obtained within and beyond academia:

  • Scholarships, awards and distinctions (amount, duration and prestige)
  • Academic Record, such as:
    • Transcripts
    • Duration of previous studies
    • Program requirements and courses pursued
    • Course load
    • Relative standing in program (if available)
  • Professional, academic, and extracurricular activities and collaborations with supervisors, colleagues, peers, students and members of the community, such as:
    • teaching, mentoring, supervising and/or coaching
    • managing projects
    • participating in science and/or research promotion
    • community outreach, volunteer work and/or civic engagement
    • chairing committees and/or organizing conferences and meetings
    • participating in departmental or institutional organizations, associations, societies and/or clubs
  • Common CV
  • Academic transcripts
  • Training Expectations
  • Sponsor’s Assessments (keeping in mind that candidates have no opportunity within the application to provide a justification for their choices of sponsors)
  • “Leaves of Absence and Impact on Research” section of the Common CV, if provided
  • “Special Circumstances” document (if provided)
  • Review information on presentations, research prizes and other indicators of the candidate's research productivity.
  • Review undergraduate academic transcripts and graduate transcripts.
  • Assess other research activity of the candidate relative to your expectations of someone with their academic experience.
  • Assess professional, academic and extracurricular activities relative to your expectations of someone with their experience.
  • Consider breadth of science covered, size and importance of meetings involved, frequency of conference presentations and research honours or awards. Bear in mind that it may vary among health science disciplines.
Date modified: