Terms of Reference for the CIHR Peer Review Expert Panel

1. Background

The Canadian Institutes of Health Research's (CIHR) mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system. Its vision is to position Canada as a world leader in the creation and use of knowledge through health research that benefits Canadians and the global community.

Parliament established CIHR to transform health research in Canada by creating a national agenda and supporting programs that: are responsive to the health needs of Canadians; embrace all relevant perspectives and disciplines; develop research capacity in neglected or emerging fields of health research; position Canada internationally as a strong participant in the global research community; and ensure that the results of research are converted more rapidly and completely into action to benefit Canadians and the global community.

The CIHR Act (S. 14) specifies that CIHR's Governing Council responsibilities include developing strategic directions, goals and policies (S.14.a); evaluating overall performance, including with respect to achievement of its objective (S.14.b); and, establishing a peer review process for research proposals made to the CIHR (S.14.d).

2. The Opportunity

Since CIHR was established in 2000, the organization has been committed to continuous improvement of its mechanisms of funding adjudication. To this end, it has recently reformed its investigator-initiated research funding programs and peer review processes. These reforms were developed in response to a specific recommendation of CIHR's second International Review Panel in 2011:

‘CIHR should consider awarding larger grants with longer terms for the leading
investigators nationally. It should also consolidate grants committees to reduce their number and give them each a broader remit of scientific review, thereby limiting the load'.

The reforms also stemmed from a request by the scientific community – highlighted in a 2010 IPSOS Reid poll – to fix a peer review system that was perceived as ‘lacking quality and consistency'. The 2012 evaluation of CIHR's Open Operating Grant Program also made recommendations around the need for program and peer review reforms.

CIHR was indeed facing increasing difficulty in force-fitting applications into standing committees. It had in fact reached a point where over 30% of the applications received, many of them interdisciplinary, were being turned down – for lack of expertise – by the committee to which they were first submitted.

These major reforms of the architecture and peer review of investigator-driven programs have been met with mixed reaction by Canada's health research community. A working summit held with the scientific community in July 2016 led to a course correction that, while responding to the request of the community for increased face-to-face – as opposed to on-line – review discussions, remained consistent with the overall objectives of the reforms.

In an open letter to the health research community as well as in a letter to the Presidents of the country's research-intensive universities (U15), CIHR also formally committed to seek feedback from an independent panel of international experts on the quality and efficiency of its new peer review processes. It was the intent of CIHR to proceed with such an evaluation at a later date, to allow for post-mortem analysis of both qualitative and quantitative data from a few rounds of competitions. However, given the intensity of the concerns expressed by the community, and the concurrent launch of an independent review of federal funding for fundamental science in Canada, CIHR felt that there was a need to proceed urgently with the evaluation.

3. Mandate

The Panel will examine the design and adjudication processes of CIHR's investigator-initiated programs in relation to the CIHR mandate, the changing health sciences landscape, international funding agency practices, and the available literature on peer review.

Specifically, the Panel will address the following questions:

1. Does the design of CIHR's reforms of investigator-initiated programs and peer review processes address their original objectives?
2. Do the changes in program architecture and peer review allow CIHR to address the challenges posed by the breadth of its mandate, the evolving nature of science, and the growth of interdisciplinary research?
3. What challenges in adjudication of applications for funding have been identified by public funding agencies internationally and in the literature on peer review and how do CIHR's reforms address these?
4. Are the mechanisms set up by CIHR, including but not limited to the College of Reviewers, appropriate and sufficient to ensure peer review quality and impacts?
5. What are international best practices in peer review that should be considered by CIHR to enhance quality and efficiency of its systems?
6. What are the leading indicators and methods through which CIHR could evaluate the quality and efficiency of its peer review systems going forward?

4. Membership

The Panel will consist of a Chair and up to 6 members.

Chair and Panel members were selected on the basis of:

• Expertise in the science of peer review: to benefit from the latest advancements in research evaluation and the “science of science funding”
• Ability to represent International funding agency perspectives: to promote the building of an international community of practice on peer review among our funding counterparts
• Areas of expertise: to balance across pillars of health research, research funding, and gender.

5. Roles and Responsibilities

The Peer Review Expert Panel

The Panel is intended to provide independent expert advice to CIHR's Governing Council.

The Panel is responsible for meeting its mandate by identifying and reviewing relevant data and resources (e.g., peer-reviewed literature, competition data), interpreting it through the lens of their collective expertise in peer review and coordinating the production of a final report and recommendations.

Panel Chair

The Chair will play a catalyst role in ensuring that the Panel functions efficiently, effectively and objectively in order to respond to the Panel mandate. He will ensure a positive, constructive and ethical environment in which CIHR's peer review innovations are evaluated.

To this end, responsibilities of the Chair are to:

• Ensure that panel discussions remain in line with its mandate;
• Ensure that the entire panel is involved and is provided with an opportunity to contribute to the final report and recommendations;
• Collaborate and work closely with panel members, the panel Secretariat and other CIHR staff, as needed, throughout the course of the deliberations;
• Participate in one face-to-face meeting, as well as panel teleconferences as needed;
• Collaborate with panel members to summarize the discussions and prepare the report;
• Ensure that the report is submitted on time; and
• Submit the final report to CIHR.

Panel Members

The responsibilities of Panel Members are to:

• Review relevant documents provided by CIHR and respect the timelines of the evaluation;
• Provide critical input to the panel's deliberations based on their area(s) of expertise;
• Actively participate in meetings, teleconferences and panel discussions;
• Collaborate with the Chair to coordinate the production of the final report; and
• Support the chair in his role.

Panel Secretariat

The Secretariat support to the panel is provided by the Programs and Initiatives Team of the Evaluation Unit within CIHR. The responsibilities of the Secretariat are to:

• Act as the hub for coordinating the panel's activities, including serving as a single point of contact for the panel's interactions with CIHR and providing operational support and advice to the Panel throughout its operation;
• Engage all relevant CIHR staff in the panel's work; and
• Activate CIHR resources to provide data and materials to support the panel's deliberations.

The Unit will appoint a project manager for this project who will be responsible for ongoing project management and quality control.

CIHR Management

• Ensure access to required resources and approvals for the Panel to complete its mandate;
• Provide access to high-quality and relevant information to the Panel in advance of its on-site meetings;
• Share the report with Governing Council;
• Develop a management response to the report, in collaboration with Science Council; and
• Publically release the Panel's final report and Management's response.

6. Participation

Service is voluntary; however, members will be reimbursed at Government of Canada rates for expenses related to their participation (e.g., attending meetings).

Members will be required to participate in meetings as follows:

Additional ad hoc teleconference meetings may be held as required and at the determination of the Panel Chair.

The Panel report will be made public once it has received Governing Council approval and has been translated.

7. Term

The Panel will be time-limited and struck only for the purpose of the review. The Panel's mandate will begin in Fall 2016. A final report will be submitted to CIHR in March-April 2017.

8. Conflict of interest

Panel members will be required to abide by the CIHR Conflict of Interest Policy and to submit a Conflict of Interest declaration when first appointed and, if required, thereafter for the duration of their membership on the Panel. 

August 15, 2016

Date modified:

2016-09-28