Mock Review Toolkit: Pre-Simulation Training

CIHR expects that all research applicants will include sex and gender into their research designs, methods, analysis, and interpretation, and/or dissemination of findings within their research proposal, when appropriate. SGBA is an approach that systematically examines sex-based (biological) and gender-based (socio-cultural) differences between men, women, boys, girls and gender-diverse people. The purpose of SGBA is to promote rigorous and reproducible science that is sensitive to sex and gender and therefore has the potential to expand our understanding of health determinants for all people. The SGBA section of the CIHR website contains helpful resources for applicants and peer reviewers alike, providing CIHR’s definitions for sex, gender and SGBA, as well as information on applying SGBA to the development and assessment of research proposals.

CIHR will extend invitations to members of the health research community to join specific Project Grant Peer Review Committees (PRCs), based on their area(s) of expertise. Reviewers will be recruited based on a set of selection criteria and in consultation with Committee Chairs and Scientific Officers. The Chairs also have a role in the selection of Scientific Officers.

Standing peer review committees have been established for the Project Grant competition. Committee core membership will be recruited for a term of service (typically 3 years, or 6 competitions). To maintain stability in membership, while providing a mechanism for membership renewal, a rotational system will be established for one third of the membership on a yearly basis. The membership may also be supplemented by additional members as required for a specific competition, based on the applications received and expertise needed for their review.

These terms will also address the benefits of renewing the membership so that new perspectives are continually incorporated into the peer review process.

Applications are initially assigned to the applicant’s first choice committee. Based on information provided at registration, CIHR staff review the initial committee assignments; if the application pressure is too high in a particular committee, the committee will be split in two, in consultation with the Committee Chair and the two Scientific Officers.

For the purpose of the simulation, only a single Scientific Officer is required per Committee.

Chairs and Scientific Officers are then asked to review the assignment of applications to their committee based on the committee mandate. Applications may be reassigned if they are more appropriate (or more closely aligned) to the mandate of another committee and can be better assessed by that committee. The final authority for the assignment of applications to a peer review committee rests with CIHR.

After confirming the assignment of applications to PRCs, applications are assigned to reviewers who identify any conflicts of interest that they may have and declare their ability to review the applications, in accordance with the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations. The Committee Chair and Scientific Officers, along with CIHR staff, assign each application to three reviewers based on their declared level of expertise.

For the simulation, Reviewers must sign the Conflict of Interest and Confidentiality Agreement for Peer Reviewers and Peer Review Observers form included in the Toolkit prior to receiving their applications to review. Reviewers may also be asked to declare their Ability to Review as part of the application assignment process if warranted.

Significant Peer Review Experience

• Previous experience as a grant program Committee Chair or Scientific Officer; or significant previous experience as a peer review committee member for a grant program; and
• Past peer review performance met high standards (Chairs and Scientific Officers were engaged, followed appropriate policies, fulfilled their role well).
• Independent Investigator status at a University or Research Institution.
• Tri-council funding (or equivalent) has been held within the last 5 years

Research Experience

• Independent Investigator status at a University or Research Institution
• At least one recent federally funded (or equivalent) peer reviewed grant as a Principal Investigator

Review Experience

• At least two peer review roles at CIHR or other recognized organization
• Completion of a training module on bias in peer review
• Completion of a training module on review quality
• Completion of learning modules on sex- and gender-based analysis in health research

Knowledge, Expertise and Lived Experience

• Expertise within CIHR’s mandate

Knowledge Users will be recruited using a combination of the criteria above, as appropriate. Applications that are identified as having an integrated knowledge translation (iKT) component will be assessed by both researcher and knowledge user reviewers.

Peer Review committee membership lists for Project Grant competitions are posted online approximately 60 days after the competition funding decisions have been published on the CIHR website.^{Footnote 3}

Note, some mock applications provided through the Toolkit do not include a Summary of Progress as this section was recently introduced to the Project Grant competition.

Reviewers must also assess the Summary of Progress. This two-page document supports the research proposal by allowing applicants to:

• Contextualize any results from research activities that support the current application;
• Describe how the application fits within their overarching research program and why the requested funds are needed
  • This should include a clear outline how the current budget request is distinct from funds currently held (as applicable) or overlaps and/or differs from applications submitted to other funding agencies/organizations (pending grants).
• Outline the impact of specific factors (e.g. leave, the COVID-19 pandemic) on their research progress.

Please note that the Summary of Progress is a narrative and not a detailed accounting of progress and funding. Details on funding can be found in the applicant’s CV and the Summary of Progress is complemented by other components of the application.

Reviewers provide their initial rating for each application to one decimal place in advance of the peer review committee meeting. Note that reviewers are not bound by the initial rating and can change it during the peer review committee meeting.

Reviewers also highlight the strengths and weaknesses of the proposal based on the evaluation criteria. Reviewers are encouraged to provide strengths and weaknesses for each evaluation criterion; strengths and weaknesses that contributed to the application rating must be clearly articulated, as they will be used to:

• provide the other reviewers assigned to the application with a justification for the rating given to the application
• provide applicants with feedback

Reviewers comment on whether the integration of sex (as a biological variable) and/or gender (as a socio-cultural determinant of health) is a strength, a weakness or not applicable to the proposal. Reviewers will also be asked to provide recommendations to the applicants on how they might improve the strength of their applications with respect to the integration of sex and/or gender. Resources to support this assessment can be found on the CIHR website.^{Footnote 5}

The rating scale ranges from 0.0-4.9. The table below outlines the rating scale and definitions. Reviewers are encouraged to use the full range of the scale.

1. Is the project idea creative?
   • The project idea is among the best formulated ideas in its field, stemming from new, incremental, innovative, or high-risk lines of inquiry; new or adapted research in Practice science, or health care, or health systems or health outcomes. When applicable, knowledge translation/commercialization approaches/methodologies should be considered, as well as opportunities to apply research findings nationally and internationally.
2. Is the rationale of the project idea sound?
   • The project rationale is based on a logical integration of concepts.
3. Are the overall goals and objectives of the project well-defined?
   • The overall goal and objectives of the project are well-defined and clear.
   • The goal states the purpose of the project, and what the project is ultimately expected to achieve.
   • The objectives clearly define the proposed lines of inquiry and/or activities required to meet the goal.
   • The proposed project outputs (i.e., the anticipated results of the project) are clearly described and aligned to the objectives.
4. Are the anticipated project contributions likely to advance basic health-related knowledge, or health care, or health systems or health outcomes?
   • The context and needs (issues and/or gaps) of the project are clearly described.
   • The anticipated contribution(s) (e.g. publishing in peer-reviewed journals) are clearly described, and should be substantive and relevant in relation to the context of the issues or gaps.
   • The anticipated contribution(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related.

Considerations

This sub-criterion is not intended to assess feasibility of the project, expertise of the team or the potential of success. These will be assessed under Criterion #2: Feasibility

Research should focus on addressing an issue (e.g., hypothesis or question, problem, need or gap) in any area across the spectrum of health (basic biomedical, health-related knowledge, health care, health systems, and/or health outcomes).

Depending on the nature of the project, it may have a research and/or knowledge translation/commercialization focus. Also, depending on the nature of the project, the rationale may be well-supported by evidence (e.g., literature review, previous findings, environmental scan, market analysis, stakeholder or partner input). However, this level of justification is not required for all types of projects (e.g., high-risk lines of inquiry).

In cases where projects have a primary implementation, or knowledge translation / commercialization (application and uptake of research findings) focus, the importance of the research should be validated as being substantive and relevant by stakeholders and partners, i.e., by those who could directly benefit from, or make use of, the project outputs.

Indigenous Health Research (IHR) committee Considerations

The proposed research must be relevant to First Nations, Inuit and/or Métis priorities and have the potential to produce valued outcomes from the perspective of First Nations, Inuit and/or Métis participants and Indigenous peoples more broadly.

Global Health Research

Projects that have a global health research focus, or include international collaborations, are eligible for support through the Project Grant program. CIHR welcomes all research, from fundamental to applied, with the potential to advance health-related knowledge, and/or improve health outcomes for Canadians and the broader global community.

1. Are the approaches and methods appropriate to deliver the proposed output(s) and achieve the proposed contribution(s) to advancing health-related knowledge, health care, health systems, and/ or health outcomes?
   • The research and/or knowledge translation/commercialization approaches, methods and/or strategies are well-defined and justified in terms of being appropriate to accomplish the objectives of the project.
   • Is sex (as a biological variable) and/or gender (as a socio-cultural factor) taken into account in the research design, methods, analysis and interpretation, and/or dissemination of findings?
   • Opportunities to maximize project contributions to advance health-related knowledge, health care, health systems and/or health outcomes should be proactively sought and planned for, but may also arise unexpectedly.
2. Are the timelines and related deliverables of the project realistic?
   • Timelines for the project should be appropriate in relation to the proposed project activities. Key milestones and deliverables should be aligned with the objectives of the project, and be feasible given the duration of the project.
3. Does the proposal identify potential challenges and appropriate mitigation strategies?
   • Critical scientific, technical, or organizational challenges should be identified, and a realistic plan to tackle these potential risks should be described. An exhaustive list is not expected.

Sex and Gender Considerations (if applicable)

Evidence demonstrates that biological and social differences between women and men contribute to differences in health risks, health services use, health system interaction and health outcomes. Accounting for sex and gender in health research has the potential to make health research more rigorous, more reproducible, and more widely applicable. CIHR expects that all research applicants will integrate sex and gender into their research designs when appropriate, as indicated on the Sex, Gender and Health Research webpage. Resources to assist reviewers in their assessment of the integration of sex and gender in the research design are available on CIHR’s website.

Indigenous Health Research (IHR) committee Considerations

In addition to demonstrating scientific excellence (Western, Indigenous, or both), the proposed research approaches and methods must respect Indigenous values and ways of knowing and sharing, and abide by the Tri-Council Policy Statement Chapter 9: Research Involving the First Nations, Inuit and Métis Peoples of Canada, and/or Indigenous partnering community / organizational ethical guidelines, or clearly explain why other guidelines have been developed and agreed upon with the study governance body.

Other Considerations

• Applications submitted to the Project Grant competition may include an integrated knowledge translation approach or may have a knowledge translation focus, with at least one knowledge-user and one researcher. CIHR defines a knowledge user as an individual who is likely to be able to use the knowledge generated through research to make informed decisions about health policies, programs and/or practices. A knowledge user can be, but is not limited to, a practitioner, policy-maker, educator, decision-maker, health care administrator, community leader, or an individual in a health charity, patient group, private sector organization or media outlet.
• CIHR defines integrated knowledge translation as a way of doing research with researchers and knowledge users working together to shape the research process – starting with collaborations on setting the research questions, deciding the methodology, being involved in data collection and tools development, interpreting the findings and helping disseminate the research results.
• Designs, approaches, and methodologies will vary by project (e.g., field of research, target audience) and should include a knowledge translation approach, when applicable, that is appropriate to the nature of the project outputs.

1. Does the applicant(s) bring the appropriate expertise and experience to lead and deliver the proposed output(s), and to achieve the proposed contribution(s)?
   • The applicant(s) should demonstrate the combined expertise and experience needed to execute the project (i.e., deliver the proposed outputs as well as achieve the proposed contribution(s)).
   • The roles and responsibilities of each applicant should be clearly described and linked to the objectives of the project.
2. Is there an appropriate level of engagement and/or commitment from the applicant(s)?
   • The level of engagement (e.g., time and other commitments) of each applicant should be appropriate to the roles and responsibilities described.
3. Is the environment (academic institution and/or other organization) appropriate to enable the conduct and success of the project?
   • Project applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, and/or supplies to:
     • Carry out their respective roles, and;
     • As a collective, manage and deliver the proposed output(s), and achieve the proposed contribution(s).
4. Does the applicant adequately demonstrate productivity and progress of their research program?
   • In their Summary of Progress, the applicant should:
     • Outline the most relevant accomplishments
     • Demonstrate their productivity

Note, this element of the sub criterion is only applicable to applications that contain the Summary of Progress section. Mock applications provided through the Toolkit do not include a Summary of Progress as this section was recently introduced to the Project Grant competition.

Reviewers must assess productivity broadly (i.e., not just based on publications) and consider the applicant’s context (e.g., career stage, leave history). CIHR has signed San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs are not limited to published journal articles but can include a broader range of outputs. Reviewers are encouraged to include these in their assessments.

Indigenous Health Research (IHR) Committee Considerations

• Appropriateness of the team based on their overall scientific experience (Western, Indigenous, or both) and skills as well as their Indigenous community-based research experience, track record, relevance of past experience, including expertise related to Indigenous lived experience(s).

Other Considerations

• The required complement of expertise will vary by project. Applications with an integrated knowledge translation approach or knowledge translation focus must include knowledge users in defining/refining research questions, informing the research plan, conducting research, interpreting research findings, understanding the receptor community, leading dissemination activities, etc. Knowledge users may also be responsible, and accountable for the application/uptake of the project outputs. The nature, breadth and depth of the applicants’ experiences and contributions should be assessed in the context of the applicants’ career stages.
• Applicants that have taken leaves of absence in the past seven years (e.g., parental, bereavement, medical, or administrative leave) may include a PDF document (no page limits) to supplement the publication information for that equivalent period of time. Whatever length of time an applicant has taken off from research in the past seven years is the amount of time that they may include in the attachment. Note that leaves of absence should also have been included in the appropriate section of the CV. Reviewers should therefore review this document in order to ensure that they have an accurate profile of applicants’ research activities and achievements.
• Project environments should be assessed according to their ability to support the proposed project activities. Institutions often function as “networked” environments or interdisciplinary networks, which means there may be multiple satellite environments contributing to the support environment. Reviewers should consider that for smaller institutions, or affiliated research facilities where resources and/or services may be obtained through networks, or may be contracted out.

When assessing an application for the integration of sex and/or gender, Reviewers should:

• Complete one of the training modules provided on the integration of sex and/or gender:
  • Introduction to Sex and Gender Considerations in Basic Science (8 min.)
  • Introduction to Sex and Gender Considerations in Clinical and Epidemiological Research (11 min.)
• Access the following resource for additional information on the integration of sex and/or gender:
  • CIHR YouTube Video: Assessing Sex and Gender Integration in Peer Review
• Read the section entitled “Other Project Information” of the application to gain general insight into the applicants’ consideration of sex and/or gender. Applicants use this section to indicate whether they have taken sex and/or gender into account in the research design, methods, analysis and interpretation, and/or dissemination of their findings, and to provide a brief justification for their decision. Please note, this section only appears in more recent Project applications.
• Critically assess the full proposal to determine whether sex and/or gender was appropriately integrated throughout the application or if the exclusion of sex and/or gender was justified.
• Indicate whether the integration of sex and/or gender was a strength, a weakness or not applicable to the proposal, as well as provide recommendations to the applicants on how they might
  • improve their applications with respect to the integration of sex and/or gender.
• Incorporate your assessment into the application’s overall grant score (if applicable). While there is no separate score associated with the assessment of sex and/or gender, reviewers should take sex and/ or gender into consideration for the Approaches and Methods sub-criterion (if the reviewer deems sex and/or gender is applicable).

When in committee, reviewers should discuss the proposal’s integration of sex and/or gender prior to reaching a consensus score.

The SGBA section of the CIHR website provides helpful resources for applicants and peer reviewers alike, including CIHR’s definitions for sex, gender and SGBA, as well as information on applying SGBA to the

• How to integrate sex and gender into research
• Key considerations for the appropriate integration of sex and gender in research