Final update from Dr. Paul Kubes, Chair of the CIHR Peer Review Working Group

September 14, 2016

The CIHR Peer Review Working Group has held three two-hour teleconferences since my last update, and as our work wraps up, I wanted to provide the research community with an overview of the outcomes of our discussions. But first, I would like to join CIHR in thanking the members of the Working Group for taking time out of their busy schedules – and in some cases, interrupting their vacations – to lend their ideas and expertise to our deliberations. Their efforts will benefit all those researchers who count on CIHR funding to undertake their important work.

Thank you as well to all those who took the time to share their feedback with me; I brought forward many of the issues you raised with me via email and it helped inform our decision-making.

The Working Group had candid and productive discussions, and we were able to cover a lot of ground in five short weeks. Our face-to-face meeting brought out engaging discussions, where members raised innovative ideas, worked through logical simulations, constructively challenged one another, and arrived at consensus for optimal peer review, for almost all recommendations. The majority of members have agreed to a key set of recommendations for improving the peer review process for the upcoming Project Grant competition. CIHR has committed to taking these recommendations under consideration and providing updates on their implementation. I invite you to consult this table for a full list of recommendations and the rationale for them.

There are a few recent updates that I would like to highlight, including:

  • Four reviewers per application at Stage 1, two at Stage 2: The Working Group recommends that each application be reviewed by four reviewers at Stage 1. And then at Stage 2, two of the original four reviewers – who will be chosen based on the expertise needed, not on how they ranked grants, to minimize bias – will go to Ottawa to participate in face-to-face discussions. We felt it was important that all reviewers be made aware when they are invited that they could be selected to attend the face-to-face meetings. Further, Stage 1 reviewers' names will be made available with their reviews at the face-to-face meetings. We saw this approach as a way to strengthen reviewer accountability throughout the two-stage process.
  • How applications move from Stage 1 to Stage 2: As mentioned previously, the Working Group recommends that approximately 40 per cent of Project Grant applications should move on to Stage 2 for face-to-face review. The Working Group’s recommendation is to have a process where the top-ranking ~30 per cent of applications across clusters plus those in the top ~30 per cent within their cluster will go forward to Stage 2. This approach is recommended until data collected during the next Project Grant can inform this ranking strategy and address issues related to applicants who are also reviewers. The remaining portion will include applications such as those that have large scoring discrepancies, were specifically flagged by Competition Chairs, or were highly ranked within their clusters. The final breakdown of the 40 per cent will depend on the actual applications submitted during the competition.
  • Lifting the restriction on applying to both Foundation and Project competitions: While this decision by CIHR was outside the scope of the Working Group, it was welcome news, as it reflects the desire of the research community. It means that CIHR will need to recruit more reviewers, but the move is a fair one for those applicants who have not had a chance to apply for a Foundation grant until now.

Indigenous health research

CIHR has been working with an Appropriate Review Practices Reference Group on Indigenous Health Research to implement a policy of iterative peer review and conditional funding for applications in the Project Grant competition. Two members of the Reference Group also sat on the broader Peer Review Working Group to ensure alignment. To help with relevance review for the iterative process for the current Project Grant competition, a new text box was added in ResearchNet for applicants to provide information about how their application addresses the principles of the Tri-Council Policy Statement (TCPS 2) - Chapter 9 Research Involving the First Nations, Inuit and Métis Peoples of Canada and Indigenous partnering community/organizational ethical guidelines. CIHR will continue to work with the Appropriate Review Practices Reference Group on Indigenous Health Research to further articulate details of the iterative review policy and implementation process for Indigenous health research, both for the current Project Grant competition and for future CIHR launches including operating grants.

Participating in the peer review process

As CIHR launches the next Project Grant and Foundation Grant competitions, I strongly encourage those of you who have received grant funding to give back to the system by making yourselves available for peer review. High-quality peer review is the bedrock upon which health research funding decisions are made; simply put, the success of the overall system depends on the participation of experts such as yourselves. CIHR will be reaching out to many of you, asking you to complete a short survey to confirm your availability to participate as a reviewer in the upcoming competitions. You will also be asked to complete your Reviewer Profile on ResearchNet. This data will help CIHR staff, in partnership with Competition Chairs and Scientific Officers, with matching and assigning applications to reviewers.

In order to foster the highest possible quality of peer review, CIHR has started to engage with the College of Reviewer Chairs, who will work with CIHR to recruit Competition Chairs – researchers who will help manage clusters of applications, ensure qualified reviewers are assigned to all applications, and chair face-to-face meetings. I believe that these are all strong signs that CIHR is taking the quality of peer review very seriously.

Going forward

CIHR has assured me that the organization—guided by the CIHR Chief Scientific Officer, Dr. Jane Aubin, in partnership with Science Council, the College Chairs, Institutes Advisory Boards, and the Competition Chairs—will carefully monitor and evaluate progress throughout the course of the Project Grant competition to ensure that it is conducted in line with the changes discussed at the July 13 meeting with the research community and reflects the recommendations of the Working Group. In addition, CIHR will report back to the community with its findings.

Although we’ve reached the end of the mandate for this Working Group, we are passing the baton, so to speak, to the College and Competition Chairs. They will continue to work with CIHR, and I’d like to thank them in advance for their dedication. I want to thank both CIHR and the Working Group for their genuine hard work and commitment to ensuring the process would be fair to all applicants. I want to remind you one last time, if we do not all participate in the review process and take it very seriously, it will fail. It is up to us to make the review process a success. With such a low funding rate, this will never be easy and it is up to us to give our colleagues the most fair and transparent reviews. It has been a pleasure working with all of you, and I am looking forward to continuing to work with many of you as the Executive Chair of the College of Reviewers.

Paul Kubes, PhD
Chair, CIHR Peer Review Working Group

Date modified: