Mock Review Toolkit: Running the Simulation
On this page
4.1 Assigning Applications and At-Home Reviews
After signing the Conflict of Interest and Confidentiality agreement and receiving all Pre-Simulation Training materials, Reviewers are assigned applications for review. Reviewers can either be assigned a set of pre-selected applications or applications that fit, as best as possible, the Themes or Areas of Science they indicated during the promotion period or Reviewers may be provided with application summaries and asked to complete an Ability to Review task.
(Optional) Ability to Review Task
This optional Ability to Review task allows Reviewers to indicate the extent to which their expertise aligns with provided applications. Facilitators may decide to conduct the Ability to Review task to both better replicate CIHR’s Peer Review process and/or to try and best match Reviewer expertise to an application.
It is important to note, even if the Ability to Review process is conducted, Reviewer expertise does not need to align perfectly as this is a training exercise. Remind participants that reviewers often underappreciate their Ability to Review when reading application summaries, and that they will likely have much more to contribute than they would initially believe.
Resources
Sample Email: Application Assignment
If the Ability to Review process is conducted, this email is sent to Reviewers and lists application summaries for Reviewers to indicate their expertise alignment
Ability to Review Template
Reviewers will complete the Ability to Review task using this template, which should be attached to the Application Assignment Email.
Assign Applications to Reviewers
We recommend each Reviewer in a Light Simulation be assigned 1-2 applications, while in the Full or Internal Simulation we recommend 2-4 applications per Reviewer.
The Facilitator will also have to balance the number of Reviewers assigned to each application. In a typical Project competition, each application is assigned three Reviewers: a primary and to two secondary. The primary Reviewer is the first to give their assessment at the Committee meeting, including a brief synopsis, while secondary Reviewers concentrate on points of agreement or disagreement.
While we recommend following the standard model (1 Primary and 2 Secondary Reviewers/application), we recognize that there will many instances where the size of the committee or the number of applications available necessitates that more than 3 Reviewers be assigned to an application. We encourage the Facilitator to adapt application assignments as necessary to fit their simulation – be it having more than three Reviewers per application, having multiple primary Reviewers, or even having Reviewers with specific tasks for a given application (e.g. providing a detailed perspective on the incorporation of Sex and Gender). For additional details on the roles played by primary and secondary Reviewers during Committee, please see the Roles, Responsibilities and Scripts section later in the Toolkit.
Use the Master Planner resource to help track and assign applications. Please contact the College of Reviewers for this resource.
Send Applications to Reviewers
Mock Review Templates must be provided to Reviewers along with their assigned applications. Reviewers will conduct their reviews at-home using the provided Mock Review Templates prior to the Committee meeting. The Mock Review Template allows Reviewers to; summarize the application, provide an overall score, note strengths and weaknesses, assess sex and gender considerations, and provide a budget recommendation.
Resources
Sample Email: At-Home Reviews
Details instructions on completing at-home reviews using the Mock Review Template
Mock Review Template
Reviewers will complete the Ability to Review task using this template, which should be attached to the Application Assignment Email.
4.2 Committee Meeting
Committee Meeting Agenda and Confirmation
An email should be sent out to participants in advance of the Committee meeting to confirm the meeting’s date/ time/location and provide a meeting agenda. Facilitators can adapt the sample email and agenda to fit their specific timelines. If the meeting is virtual, we recommend Facilitators include items listed under Preparation and Room Setup below as part of their communication to participants.
Resources
Sample Email: Confirmation of Committee Meeting
Meeting Agenda Template
Must be attached to the Confirmation of Committee Meeting email. If the Committee meeting is in-person, this must also be printed for all participants – see Preparation and Room Setup below
Run the Committee Meeting
Preparation and Room Setup
Note, these are suggestions for room setup; please adapt as necessary.
If the Committee meeting is in-person, these items should be printed ahead of the meeting. If the Committee meeting is virtual, these resources can be sent to participants prior to the virtual meeting as part of the Confirmation of Committee Meeting package above.
In-meeting Resources
Number | Document name | Description |
---|---|---|
1 per participant | Sequence of Steps for Committee Meeting | A step-by-step description of the Committee meeting. Useful for all participants. |
1 per participant | RQA Checklist | A practical tool to assist Reviewers in writing quality reviews. Useful for entire committee. |
1 per Facilitator | Scoring Sheet | A scoring sheet that allows for tracking of scores for Facilitators OR all participants. |
1 per application | SO Notes Template | A template for Scientific Officers to write their notes. |
1 per Chair and Scientific Officer | Executive Roles, Responsibilities and Scripts | Scripts to assist committee executives in their role. |
1 per Reviewer | Reviewer Roles, Responsibilities and Scripts | Scripts to assist Reviewers in their role. |
If the Committee meeting is to be in-person, we suggest the table be setup as depicted below to better facilitate discussion.
Roles, Responsibilities and Scripts
Facilitator
The Facilitator is responsible for organizing the Committee meeting(s) and ensuring that everyone has the appropriate documentation.
The Facilitator will:
- provide all necessary documentation
- discuss conflicts of interest
- clarify policies and/or administrative processes
- keep track of the time
- ensure Chair knows when it is time to take a break
Chair
It is the Chair's responsibility to ensure that the review committee functions smoothly, effectively and objectively, and that a positive, constructive, fair-minded environment in which research proposals are evaluated is established and maintained.
The Chair will:
- provide opening remarks to the committee, including an outline of the structure and agenda of the day.
- explain the meeting process to the committee, including the review of SGBA considerations in the applications.
- briefly discuss the budget and term components and remind Reviewers that they are not part of the Committee meeting.
- ensure that all committee members who are in conflict with an application leave the meeting room before the discussion of the application.
- appoint a delegate as Chair or Scientific Officer when either individual leaves the meeting room due to a conflict of interest with an application or for any other reasons.
- fulfill an oversight role – does not rate applications nor vote during the Committee meeting.
- ensure the involvement of the entire committee in evaluating each application.
- work with the Scientific Officers, as required, to summarize the discussion around each application, before the consensus rating is reached.
- ensure that a consensus rating is reached by the assigned Reviewers.
- ensure that specific ethical concerns and other CIHR requirements are addressed, and that any related discussion is captured in the Scientific Officer notes.
Scientific Officer
Reviewers can rotate the role of an SO to gain experience. If this is the case, ensure that all “SO Reviewers” receive a copy of the SO Note Template (available in Preparation and Room Setup above). You can also include a Mock SO Note (included with some, but not all Mock Applications) to provide Reviewers with an example.
The Scientific Officer (SO) assumes the role of note taker. The SO will:
- take official notes of the committee discussions for each application (SO Notes). The SO Notes should provide the applicants with insight into the committee discussion of their applications. They should be clear and concise and give objective and constructive feedback to the applicants. They should:
- include the strengths and weaknesses of the applications discussed by the committee.
- address the issues that had the greatest impact on the evaluation, as they relate to the program's evaluation criteria.
- address aspects of the committee discussions that were not captured in the Reviewers' reports.
- describe how Reviewer disagreements, as seen in the individual Reviewer reports, were reconciled by identifying which view was favored by the committee.
- read back the SO Notes to the committee for validation and for additional input before a consensus rating on each application is reached by the assigned Reviewers, and all the members' votes are cast.
- ensure that special considerations related to ethics and/or other issues are also recorded in the SO Notes, if applicable.
Reviewers
The Reviewer evaluates each of the applications assigned to them by providing a critical assessment of the applications, as well as constructive feedback based on the program's objectives and adjudication criteria described in the funding opportunity.
The Reviewer will:
- consider all factors and the strengths and weaknesses of the applications in relation to each adjudication criterion.
- focus their comments on the factors most relevant to their ratings.
- provide comments on the integration of sex (as a biological variable) and/or gender (as a socio-cultural determinant of health) in the applications, if applicable.
- provide comments on the budgets requested and a formal recommendation to CIHR in the "Budget" section, including clear and detailed reasons for any recommended budget or term cuts, if applicable (*Note: discussion of budget and term support is not currently within scope of this Mock Review Toolkit).
- provide comments on issues that they feel should be flagged, as required. These concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the applications.
- provide their initial ratings to one decimal place using the provided Mock Review Templates
- familiarize themselves in advance of the Committee meeting with the applications to be assessed by their committee as this will facilitate discussions at the face-to-face Committee meeting.
- present to the committee the review of their assigned applications.
- participate in the committee discussions.
- vote on all the applications discussed by the committee and for which they are not in conflict with.
Reviewer Scripts
Primary Reviewer(s) Script:
- Once Chair introduces Reviewers, present initial scores to committee.
- Present a brief overview of the application, premise of the research, study design etc.
- Discuss strengths and weaknesses.
- Listen to other reviews.
- Following Secondary Reviewer(s) discuss consensus score.
- Discuss further if necessary.
- Confirm content of notes from Scientific Officer at the end of the discussion.
Secondary Reviewer(s) Script:
- Once Chair introduces Reviewers, present initial scores to committee.
- Listen to Primary Reviewer(s) overview of the application and comments.
- Present strengths and weaknesses not mentioned by previous Reviewers.
- Once all discussion has taken place, Reviewers discuss consensus score.
- Discuss further if necessary.
- Confirm content of notes from Scientific officer at the end of the discussion.
- Date modified: