CIHR Training and Career Support Program Evaluation
Final Report

March 2021

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation through end of award reports and surveys, CIHR's Research Programs Portfolio – Program Design and Delivery, Initiative Management, Operations Support, and Science Policy Branches, CIHR's Financial Planning Unit, CIHR's Results and Impact Unit (Kim Gaudreau, Fraser Smith, Chris Manuel, Sobia Hafeez), CIHR's Funding Policy and Analytics Unit, and Observatoire des sciences et des technologies (OST).

The TCS Evaluation Team

Carmen Constantinescu, Angela Mackenzie, Alexandra Leguerrier (student), Kim Ford, Sarah Viehbeck, Shevaun Corey, David Peckham, Michael Goodyer, and Ian Raskin.

For more information and to obtain copies, please contact: evaluation@cihr-irsc.gc.ca.

Table of Contents

List of Tables

List of Figures

List of Acronyms

Acronym Meaning
ARC Average of relative citations
ARIF Average relative impact factor
CERC Canada Excellence Research Chairs
CFI Canadian Foundation for Innovation
CFREF Canada First Excellence Research Fund
CGS Canada Graduate Scholarship
CGS-M Canada Graduate Scholarship – Master's
CGS-D Canada Graduate Scholarship - Doctoral
CRC Canada Research Chairs
CRCC Canada Research Coordinating Committee
CS Clinician Scientist
CTS Career Trajectory Survey
DFSA Doctoral Foreign Study Award
DRF Departmental Results Framework
ECR Early Career Researcher
FEAR Fellowship End of Award Report
FGP Foundation Grant Program
FO Funding Opportunity
IIR (Program) Investigator Initiated Research Program
ISED Innovation, Science, and Economic Development
MRC Medical Research Council (UK)
MSFSS Michael Smith Foreign Study Supplement
NHMRC National Health and Medical Research Council (AUS)
NI New Investigator
NIH National Institutes of Health (US)
NSERC Natural Sciences and Engineering Research Council of Canada
OOGP Open Operating Grants Program
OST Observatoire des sciences et des technologies
PAA Program Alignement Architecture
PDF Postdoctoral Fellowship
PGP Project Grant Program
PIP Performance Information Profile
RPA (Program) Research in Priority Areas Program
SSHRC Social Sciences and Humanities Research Council of Canada
STIHR Strategic Training Initiative in Health Research
TCS (Program) Training and Career Support Program
T-SAP Strategic Action Plan on Training (also referred to as the Training Strategy)

Executive Summary

Program Overview

Since inception, CIHR has contributed to building Canadian health research capacity by offering a large variety of training and career support programs that have funded trainees and researchers across health disciplines. Between 2000-01 and 2016-17, CIHR delivered ~561 programs and funding opportunities (FOs), funding 8,337 awards. Over this period, CIHR invested $966M in direct training and career awards; however, agency-specific investments have decreased over time and more health research trainees are funded via Tri-agency awards. Currently, CIHR continues to provide training and career support through a variety of mechanisms and funding approaches (i.e., direct awards, indirect stipends, and training grants).

Evaluation Objective, Scope and Methodology

The CIHR Training and Career Support Program Evaluation is thematic, and assessed the relevance, design and delivery, and performance of direct awards under two Programs: Training and Career Support (TCS) and Research in Priority Areas (RPA). Direct awards in the current evaluation refer to funding support given directly to an individual trainee or researcher through an award program (e.g., Doctoral Research Award, Career Support Award) versus funding support provided directly or indirectly through a research or training grant (e.g., Open Operating Grant, Project Grant, or Strategic Training Initiative in Health Research Grant). The evaluation covered the period from 2000-01 to 2016-17, using a range of methods and sources to triangulate evaluation findings. The scope excluded indirect awards and Tri-agency programs (although results from completed evaluations were included to contextualize findings where relevant); therefore, key findings relate to CIHR's direct awards only. The objective of this evaluation was to: inform CIHR's training strategy (the Strategic Action Plan on Training; T-SAP) and the next CIHR strategic plan; and, meet the requirements of the Treasury Board of Canada Secretariat (TBS) under the Policy on Results and the Financial Administration Act.

Key Findings

Relevance

The evaluation concludes that CIHR investments in direct training and career support are largely driven by the CIHR Act, which has a unique focus to build capacity in Canadian health research, as well as the strategic plan, and federal government priorities. CIHR investments in its direct training and career support awards have decreased over time. Currently, CIHR provides most of its direct training and career support awards at the training levels, in particular at the Postdoctoral level, with limited career awards beyond research grants to support the transition to independent research careers. CIHR's direct training awards complement Tri-agency programs by providing awards in priority areas, opportunities for international study (Doctoral and Postdoctoral levels), and supporting trainees unique to the health research community (e.g., support for post-health professionals in addition to Postdoctoral trainees). CIHR's direct training and career support awards contribute to building health research capacity by supporting trainees who have pursued research careers, with the majority of surveyed direct award recipients working full-time in the academic sector (approximately three quarters across doctoral and postdoctoral levels). However, given that half of trainees are in non-tenured or non-tenure track positions within academia and up to one quarter are employed in sectors outside academia, this lends support to the need to ensure trainees are provided with opportunities to obtain transferable skills (e.g., professional and entrepreneurial skills).

Design and Delivery

The evaluation found that the majority of CIHR's direct training support is provided at the Postdoctoral level, whereas the majority of direct career support is provided through research grants. The evaluation concludes that CIHR needs to better define, align, and monitor training and career support activities and investments in relation to the CIHR Act and next strategic plan. CIHR's investments in direct training and career support award programs have decreased from $87M in 2000-01 to $45M in 2015-16. There are some funding differences based on sex and language: more females have been funded across training levels and more males have been funded at the career support levels, and females have received higher average award amounts at the Chair level. The majority of funded applications are in English and more than half of funded applications have a Biomedical focus. With respect to design features of current direct award programs, CIHR award amounts and duration are on par with Tri-agency and international awards at the Doctoral level; however, they are lower at the Postdoctoral level. At the Master's level, they are on par with the Tri-agency and lower than international awards.

The evaluation found that CIHR's direct training and career support programs (and Tri-agency programs) do not address the need to provide sustained support (across multiple levels) for scientific careers in health research, as outlined in the CIHR Act. Broadly, the current TCS Program, which includes direct awards only, does not effectively align with the objectives and priorities of the CIHR Act, the most recent CIHR Strategic Plan (2014-15 to 2018-19) in place during the evaluation period, and the training strategy (T-SAP), and lacks indicators that reflect the full range of CIHR's training and career support programs (given its heavy focus on Tri-agency programs). In particular, the current TCS Program has no description of how expected results are produced across all of CIHR's training mechanisms (direct and indirect) and there are gaps in performance measurement data. The objectives of current CIHR direct training awards are not clearly defined, beyond the brief descriptions in funding opportunities, and do not align with the objectives of the training strategy (e.g., leadership, multidisciplinarity, and transferable/entrepreneurial skills). Career support, on the other hand, is currently provided mainly through research grants rather than award programs, and there is a gap in capacity building objectives with the sunset of the Foundation Grant Program, as there are no specific objectives related to capacity building covered by the Project Grant Program.

Performance

The evaluation looked at the outcomes and impacts of CIHR's investments in direct training and career support related to trainees and researchers funding trajectory, productivity over time, training and mentoring, careers, as well as the provision of multidisciplinary training.

The evaluation provides evidence that CIHR is not providing sustained support through direct training and career support awards (both agency-specific and Tri-agency), as the majority of recipients were funded at only one level. The majority of awardees (86%) received only a single training or career support award and only 3.6% were funded at both levels. Thirty percent of recipients of one or more awards also received grant funding; however, the majority were career support award recipients who were required to also hold a grant (19% of the 30%).

The evaluation shows that directly supported trainees and researchers (through either CIHR agency-specific or Tri-agency awards) are more productive than the trainees and researchers who did not receive these direct awards. CIHR ECR award recipients were generally more productive than applicants and OOGP recipients at the ECR level. Funding at the Postdoctoral level has had a greater impact on recipients' productivity and career outcomes compared to funding at other training award levels. However, funding across both Doctoral and Postdoctoral levels was associated with the greatest productivity.

The evaluation shows that recipients of CIHR's direct training and career support awards have demonstrated positive career outcomes and that CIHR's direct training and career support awards have contributed to building health research capacity, most notably in the academic sector. The majority of CIHR award recipients are working full time in the academic sector, with half to two thirds in tenured or tenure-track positions. Postdoctoral award recipients were more frequently in higher ranked positions. Up to one quarter of Doctoral (16%) and Postdoctoral (25%) award recipients are working in sectors outside of academia. Career award recipients reported higher levels of career satisfaction, employment-related income, and career advancement compared with applicants.

The evaluation also showed that CIHR direct training awards offered opportunities to develop research skills and the majority of trainees were highly satisfied with these opportunities; however, the awards lack explicit objectives related to training and mentoring. Recipients of direct career support awards supervised and trained more trainees and research staff compared to unsuccessful applicants and Open Operating Program Grant recipients, and career support awards have had a small impact on recipients' time spent on research. CIHR and Tri-agency direct training and career support award recipients reported experiencing multidisciplinary collaboration to varying extents. However, it is not clear the degree to which career support awards enabled researchers to provide multidisciplinary training to their trainees.

Recommendations

The evaluation makes four recommendations aimed at improving the design and delivery and performance of training and career support programming at CIHR.

Recommendation 1:

CIHR needs to continue to provide funding that contributes to building health research capacity to meet the objective of the CIHR Act and the needs of trainees and the health research community. Specifically, CIHR should continue to:

  1. provide support for international study at the Doctoral and Postdoctoral levels; and,
  2. provide support at the Postdoctoral level for both Postdoctoral fellows and post-health professional degree recipients.

Recommendation 2:

CIHR needs to define and outline how it will achieve "the provision of sustained support for scientific careers in health research" as specified in the CIHR Act, across the full spectrum of training and career support mechanisms (both direct and indirect).

Recommendation 3:

CIHR needs to align investments and activities in training and career support to meet the objectives of the Act, the training strategy, and the next strategic plan.

Recommendation 4:

CIHR needs to improve the monitoring and performance measurement of all training and career support activities and investments (both direct and indirect).

Background

This is the first evaluation of the Canadian Institutes of Health Research (CIHR) Training and Career Support (TCS) Program. The evaluation is thematic given the breadth of programming as well as increased flexibility under the Policy on Results (2016), and assessed the relevance, design and delivery, and performance of CIHR's direct training and career support awards, covering the period from 2000-01 to 2016-17. Direct awards in the current evaluation refer to funding support given directly to an individual trainee or researcher through an award program (e.g., Doctoral Research Award, Career Support Award) versus funding support provided directly or indirectly through a research or training grant (e.g., Open Operating Grant, Project Grant, or Strategic Training Initiative in Health Research Grant). An evaluation of the relevance, design and delivery, and performance of CIHR's salary and career award programs was completed in 2012, while evaluations of the performance and relevance of CIHR's Strategic Training Initiative in Health Research (STIHR) were completed in 2008 and 2016.

Context

Since 2001, CIHR, as Canada's main federal funding agency for health research (in collaboration with the other Tri-agencies), has contributed to Canadian health research capacity building by offering a large variety of training and career support programs that have funded trainees and researchers across health disciplines. Health research training is a core priority for CIHR, and as such CIHR is committed to developing the next generation of leaders within and beyond the health research enterprise. Over time there has been a shift in how health research trainees have been funded, from agency-specific training awards to Tri-agency awards whereby more trainees are now funded through the Canada Graduate Scholarship at the Master's and Doctoral levels. From 2000-01 to 2016-17, CIHR invested $966M in agency-specific training and career support programs (based on competition fiscal years). In addition, for the same period, CIHR's portion of Tri-agency investments in training and career support programs was $2.1B, for an overall total of $3.1B. See Table 1: Overview of CIHR and Tri-Agency Training and Career Support Awards since 2000-01.

CIHR supports capacity building in two key ways:

The focus in the current evaluation is on CIHR's direct training and career support awards, versus direct or indirect support through research or training grants and Tri-agency awards. However, given that contributions to capacity building are also made via these other ways, and there has been a shift from less agency-specific support to more Tri-agency training support, where possible findings from previous evaluations of Tri-agency and training grant programs are incorporated for context.

The CIHR Act specifies that the objective of CIHR is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system. The Act also outlines specific sub-objectives related to capacity building, including: (j) building the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research. Although it is not clear how sustained support is defined by CIHR. For the purposes of achieving its objectives, one of CIHR's relevant powers and functions listed in section 5 of its Act is to: (b) foster the development and ongoing support of the scientific careers of women and men in health research.

Consistent with CIHR's previous strategic plans, the CIHR Strategic Plan (2014-15 to 2018-19), Health Research Roadmap II: Capturing Innovation to Produce Better Health and Health Care for Canadians, included a commitment to improve training and career support. Strategic Direction 1 focuses on promoting excellence, creativity and breadth in health research and knowledge translation by supporting leading researchers for advances in health and building a solid foundation for the future. Specifically, section 1.2 outlines a key priority to enable the success of the next generation of researchers, through programs such as the Vanier Canada Graduate Scholarship program and the Banting Postdoctoral Fellowships program. It highlighted that "collective action is needed to maximize Canada's approach to training and mentoring, so as to provide students and trainees with the right mix of experience and skills to succeed in the health-related academic and/or professional careers of the future." This section also acknowledges the need for the next generation of researchers to develop research and professional skills to enable them to adapt to changes in the health research enterprise, the need to expose trainees to diverse expertise and environments, and its plan to encourage leading researchers to develop innovative training environments that prepare training for research-related careers.

Additionally, CIHR developed its Strategic Action Plan on Training (T-SAP), referred to as the training strategy, in 2015 in order to support the achievement of the Strategic Direction 1 in Roadmap II. The training strategy focused on identifying the following challenges and gaps and intended to address the following priorities:

Several factors had an impact on the current evaluation. Starting in 2017, changes in the way CIHR classified its programs were brought about as a result of CIHR transitioning from the previous Program Alignment Architecture (PAA) to the Budget 2019 and Performance Information Profile (PIP) structure. This was in compliance with the new Policy on Results implemented by the Treasury Board of Canada Secretariat (TBS) on July 2016 and it was used to guide reporting in the Departmental Plan and Departmental Results Reports. As per the DRF, training and career support is provided through the following Programs: Training and Career Support (TCS), Investigator Initiated Research (IIR), and Research in Priority Areas (RPA) Programs, using different funding approaches (i.e., direct awards, indirect stipends, and training grants). Each of these Programs has a PIP that includes an overall description and performance indicators used to measure the Programs' effectiveness. For the purpose of this evaluation, the TCS Program refers to that within the DRF, whereas the training and career support programs refer to all the individual training and career support direct awards funded by CIHR since 2000-01.

It should also be noted that CIHR investments in direct training and career support were the focus in the current evaluation and indirect funding for training and career support provided via CIHR grants or other mechanisms, disbursed through the IIR or RPA Programs, was not in scope. The reason for this is discussed in more detail in the Description of the Evaluation section below. Therefore, key findings, conclusions, and recommendations of this evaluation are focused on direct training investments yet have implications for indirect funding of training and career support.

The Canadian and international research and training landscape is continuously changing with new generations of trainees and researchers, including increasing graduate student enrolment and graduation rates (Statistics Canada, 2020) and changing Government priorities, (e.g., recommendations from the Fundamental Science Review to reinvigorate and harmonize scholarship and PDF fellowship programs, and rationalize and optimize the use of current awards to attract international talent). Current Federal Government priorities that influence capacity building are described below. Therefore, the training and career support landscape at CIHR has necessarily evolved since 2000-01 in order to respond to the changing research ecosystem. CIHR training and career support programs operate within a broader Canadian funding environment, including programs delivered in collaboration with other federal government departments, as well as a wide range of provincial and not-for-profit training and career support awards. Specifically, Tri-agency programs were initiated by Innovation, Science and Economic Development (ISED), such as the Canada Graduate Scholarship (CGS) at the Masters and Doctoral level as well as Vanier Canada Graduate Scholarships (Vanier CGS) and Banting Postdoctoral Fellowships (Banting PDF), which largely displaced agency specific awards and CIHR direct investments in training. The current evaluation focuses on CIHR's agency specific direct investments in training and career support, while acknowledging and incorporating Tri-agency support where possible and relevant but excludes indirect support.

Program Profile

Training and Career Support at CIHR

Overall, between 2000-01 and 2015-16Footnote 1, CIHR funded 561 training and career support award programs and priority announcement funding opportunities (FOs). The majority of these awards were priority announcement FOs, targeted at specific areas of research, including only a few awards per FO. These are often institute-specific FOs that do not have their own performance measurement strategies. Thus, there is no performance measurement data for the majority of these programs, limiting the conclusions that can be drawn about the impact of most of the priority-driven awards. CIHR's larger investigator-initiated award programs (where researchers and trainees apply in the research area of their choice) have included the Doctoral Research Award (DRA), the Doctoral Foreign Study Award (DFSA), CIHR Fellowship, and the New Investigator (NI) Award. Over this period, there was also a shift from agency-specific awards to Tri-agency awards with the introduction of the Canada Graduate Scholarship (CGS), the Vanier CGS and the Banting Postdoctoral Fellowship as well as the Canada Research Chairs. Although CIHR continues to support health research trainees with agency specific awards, more trainees are now supported through Tri-agency awards.

For the purpose of this evaluation, the focus is on CIHR's agency specific programs and FOs and they have been grouped into six award levels that can be conceptualized as progressive steps for training and career support, namely: Master's, Doctoral, Postdoctoral, Early Career Researcher (ECR), Career Support (Mid-Senior), and Chairs (across career stages). Master's, Doctoral, and Postdoctoral levels make up training support while ECR, Mid-Senior Career Support, and Chair levels make up career support.

CIHR investments in training and career support

Between 2000-01 and 2015-16, CIHR's biggest investment has been at the Postdoctoral level ($390M) through both investigator initiated and priority driven fellowships. See Figure 1: CIHR and Tri-agency total Investments in training and career support (2000-01 to 2016-17), below. Looking at the evolution of these investments across the years, CIHR's investments in its own training and career awards have decreased at all levels, from $87M in 2000-01 to $45M in 2015-16; whereas CIHR's portion of investments in the Tri-agency programs (through ISED) increased from $133M to $295M, with the emergence of programs like CGS Master's and Doctoral, Vanier Canada Graduate Scholarship (CGS), Banting PDF, Canada Research Chairs (CRC), Canada Excellence Research Chairs (CERC), and Canada First Research Excellence Fund (CFREF). Tri-agency programs are initiated by and funded through ISED; however, each individual agency administers these programs and distributes its own portion of the funds received from ISED.

During the same period (2000-01 to 2015-16), almost two-thirds of CIHR's training and career support awards were investigator initiated (65%) and the remaining were priority driven (35%). These funding mechanisms align with the current TCS Program and RPA Program (under the Departmental Results Framework discussed above). Considering CIHR's portion of the Tri-agency investments together with its agency specific investments, the proportion of investigator initiated awards was even higher (80%; with 20% priority driven)Footnote 2.

Over this same period, CIHR funded 8,337 agency specific awardsFootnote 3 with the highest number being awarded at the Postdoctoral level followed by the Doctoral and career support levels (Master's = 126; Doctoral = 1853; Postdoctoral = 3,862; ECR = 1,046; Career Support = 1,315; and, Chair = 135). A total of 16,158 training and career support awards were funded considering both CIHR and CIHR's portion of the Tri-agency awards. The highest number were awarded at the Doctoral followed by the Postdoctoral levels (Master's = 3,493; Doctoral = 4,473; Postdoctoral = 4,002; ECR = 1,046; Career Support = 1,325; and, Chair = 1,819).

Definitions of training and career support

The following definitions of training and career support programs have been adapted for the purpose of this evaluation using relevant and available information from program descriptions, funding opportunities and CIHR's website where possible. The amount of information on program objectives varies greatly, with more detailed information provided for Tri-agency awards compared to agency specific awards. Tri-agency awards tend to have a more fulsome program theory including objectives, expected results and logic models, whereas agency specific awards tend to have a more limited description of the program and rarely include expected results in their funding opportunities.

Training programs refer to direct award programs or FOs that target persons enrolled in university at the graduate level (Master's and Doctoral level) as well as persons at the Postdoctoral level or those engaged in a fellowship, post-health professional degree. A common element among all target groups is that they are enhancing their research skills through involvement in research and are guided under the formal supervision of an independent researcher. Since 2000-01, CIHR has offered or delivered the following training programs (including Tri-agency programs): CGS Master's and Doctoral, CIHR Master's Awards, DRAs, DFSA, Vanier CGS, MD/PhD Studentship Awards, Banting PDF, CIHR Fellowship, Clinician Scientist (Phase I), and Strategic Training Initiative in Health Research (STIHR) along with a variety of priority driven awardsFootnote 4.

CIHR's current training award programs include: the DFSA; the CIHR Fellowship; a variety of Masters, Doctoral, and Fellowship Awards in priority areas (e.g., Master's Award: Dr. James Rossiter MPH Practicum Awards Program; Doctoral Research Award: Priority Announcement [Specific Research Areas, e.g., HIV/AIDS and/or STBBI]; Fellowship: Leukemia & Lymphoma Society of Canada/CIHR Clinician Scientist Fellow Award). Current Tri-agency award programs include: CGS Master's and Doctoral; the Vanier CGS; and, the Banting PDF.

The common objectives of CIHR training programs have been to provide recognition and funding to academic researchers and students early in their academic research careers, to offer them an opportunity to gain research experience, to attract and retain trainees nationally and internationally, and to provide a reliable supply of highly skilled and qualified researchers.

Training support has been provided by CIHR in three ways:Footnote 5

Note that direct awards were the only form of training support included in the current evaluation; however, findings from the STIHR training program grant were incorporated where relevant for context and were included in some lines of evidence (e.g., the funding trajectory and bibliometric analyses).

Career support programs refer to direct salary award programs or FOs that target persons meeting CIHR's definition of independent researcher at either the Early, Mid- and/or Senior career stages. Independent researchers are required to hold an academic or research position at an eligible Canadian Institution. Since 2000-01 CIHR has offered or delivered the following career support programs (including Tri-agency programs): NI Awards, Mid-Career Investigator Awards, Senior Investigator Awards, Clinician Scientist (CS) Phase II Awards, CIHR Research Chairs, CRC, CERC, CFREF, and a variety of priority driven awards across career stages.

CIHR's current direct career support award programs include: New Investigator Awards (offered only in priority areas), Clinician Scientist Phase II (no new awards, any investments are for previous commitments for those funded in Phase I), and some priority driven awards (Canadian Diabetes Association Awards: Diabetes Investigator CIHR Sex and Gender Science Chair). Current Tri-agency career support award programs include the CRC, the CERC, and the CFREF.

Overall, the objectives of the CIHR direct career support programs have been to provide salary support to researchers who have made outstanding contributions and have demonstrated leadership in their field, and to attract and retain researchers nationally and internationally. One of the main provisions for the career support programs was the protection of time dedicated to research: funded researchers were required to spend at least 75% of their work time conducting research for the duration of their award. This protected time was intended to bring about positive outcomes in terms of research productivity and training opportunities, ultimately leading to a strong research community and the retention of excellent researchers in Canada.

Description of the Evaluation

Evaluation Purpose and Scope

The evaluation of CIHR's TCS Program, committed to as part of CIHR's 2017-18 Evaluation Plan, was designed to meet the CIHR's requirements to the Treasury Board Secretariat (TBS) of Canada under the Policy on Results and provides senior management with independent, objective, and actionable evidence of:

The evaluation was also expected to inform CIHR's training strategy (T-SAP) and the next strategic plan; therefore, although this evaluation is retrospective, the current state of training and career support will be discussed given recent administrative changes, programmatic changes, and the initiation of the planning process for the next strategic plan. Findings from the evaluation have informed discussions about the design and development of future training programs throughout the course of the evaluation. Recommendations are focused on the current context and informing direct training and career support programming and, where relevant, indirect support going forward.

This thematic evaluation assessed the relevance, design and delivery, and performance of direct awards funded under the TCS Program and the RPA Program. The evaluation covers the period from 2000-01 to 2016-17 and excludes indirect support and Tri-agency programsFootnote 7. However, results from completed evaluations of Tri-agency and CIHR programs are included to contextualize and assess evaluation findings (e.g., CGS, Vanier CGS, Banting PDF, and STIHR). The decision to exclude indirect support was due mainly to the difficulties associated with data collection given timing and resources (i.e., the lack of available data to track information related to indirect funding, including recipient contact information and demographics, and the nature and value of this funding); however, the findings and conclusions relating to the design and delivery of training and career support programming at CIHR from this evaluation can apply to indirect support as well, and inform areas for future research. See Figure 2: Overview of Training and Career Support Landscape and Awards in Scope of the Evaluation (2000-01 to 2016-17) below for an overview of the programs included in the evaluation.

Methodology

Evaluation Approach

Consistent with TBS guidelines and recognized best practice in evaluation, a range of methods and sources were used to triangulate evaluation findings. These methods included: document and data review, an environmental scan of training and career support programs offered by other research funders in Canada and internationally, analysis of the CIHR Fellowship End of Award Report (FEAR), surveys with trainees and researchers, and a bibliometric study looking at the productivity of applicantsFootnote 8 and recipients of CIHR training and career support. In addition to this, the evaluation used an innovative method, a funding trajectory analysis, which tracked the complete funding history of applicants to both CIHR and Tri-agency awards between 2000-2001 and 2015-2016.

Evaluation Questions

The evaluation addresses the following specific questions identified during the evaluation design consultation process with CIHR Senior Management (Director General, Performance and Accountability; Director General, Program Design and Delivery; and Director General, Science, Knowledge Translation and Ethics), which took place in 2016-17.

Relevance

  1. Why does CIHR invest in direct training and career support in the way that it does?
    • 1.1. What critical needs are addressed by the current CIHR training and career support programs?
    • 1.2. How does CIHR's role compare to other national (NSERC, SSHRC and health charities) and international research funders?

Design and Delivery

  1. How does CIHR invest in direct training and career support?
    • 2.1. How do the design features of CIHR's training and career support programs support achievement of intended outcomes?
    • 2.2. How does CIHR's approach to training and career support compare to other national (NSERC, SSHRC, and health charities) and international research funders?

Performance

  1. What are the outcomes and impacts of CIHR's investments in training and career support related to:
    • 3.1. Trainees and researchers' funding trajectory?
    • 3.2. Trainees and researchers' productivity over time?
    • 3.3. Training and mentoring of funded trainees and researchers?
    • 3.4. Provision of multidisciplinary training?
    • 3.5. Sustaining the careers of funded trainees and researchers?

Limitations

The following limitations and mitigation strategies have been identified:

Evaluation Findings

Relevance

Key Findings:

CIHR investments in direct training and career support contribute to the achievement of objectives of the CIHR Act and its strategic plan (Roadmap II)

In terms of the need to provide training and career support, CIHR investments in direct training and career support are largely driven by the Act and the strategic plan. Specifically, sub-objective 4j of the CIHR Act: "building the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research". As well as Strategic Direction 1 of CIHR's Health Research Roadmap II as it relates to enabling the success of health researchers and professionals.

CIHR investments in direct training and career support align with Federal Government priorities

CIHR investments also respond to the need to support the next generation of researchers identified as a priority by the Federal Government. A number of recent Federal Government documents identified a continued need to support the next generation of researchers in all areas of health research, including:

The Fundamental Science Review Panel, led in 2017 to determine the strengths of Canada's approach and identify gaps in the research funding ecosystem, identified some challenges related to the training and career environment: lack of career support for ECRs, insufficient funding for Doctoral students and Postdoctoral fellows, and lack of coordination and collaboration among the four granting agencies (CIHR, NSERC, SSHRC, and CFI). The Panel made a series of recommendations for the overall Canadian funding environment, some of which were specifically targeted at the need to support early career researchers, reinvigorate scholarship and fellowship programs, attract international talent, and harmonize funding strategies across Canadian funding agencies (CIHR, NSERC, and SSHRC).

Following recommendations from the Science Review Panel, the Canada Research Coordinating Committee (CRCC) was mandated to achieve greater harmonization, integration, and coordination of research-related programs and policies and to address issues of common concern to the granting agencies and the Canadian Foundation for Innovation (CFI). Through its work plan, the CRCC committed to engaging with the research community in the development of new programming and initiatives in each priority area, two of them related to training and career support specifically:

Some of the progress of the CRCC was highlighted in the CRCC's Progress Report 2018-2019. It includes the launch of the New Frontiers for Research Fund and the inaugural exploration call for ECR's doing innovative, high-risk, high-reward interdisciplinary research (February 2019), adopting a Tri-agency ECR Action Plan, and allocating 250 new Tier 2 Chairs for ECR's to the CRC Program.

Canada's Science Vision supports several research based outcomes, one of which is fostering the next generation of scientists, including students, trainees and early career researchers. Budget 2018 made the single largest investment in fundamental research in Canadian history of nearly $4 billion over five years in research and in the next generation of scientists. Budget 2019 includes investments of $114M over five years to the federal granting agencies to support Master's and Doctoral students through the CGS as well as $210M to the CRC Program for ECRs.

CIHR's award support complements Tri-agency programs, the majority of training support is provided through Postdoctoral fellowships and the majority of career support is provided through research grants

Since 2000-01 CIHR has delivered an extensive number of direct award programs and FOs (561) across award levels. As described previously, the majority of these programs or FOs were priority announcements in targeted research areas, which often included a single FO and offered only one or a few awards each. Broadly, CIHR training programs complement Tri-agency programs by providing awards in priority areas, opportunities for international study (Doctoral and Postdoctoral levels), and supporting trainees unique to the health research community (e.g., support for post-health professionals in addition to Postdoctoral trainees).

At the Master's level, the majority of direct awards are provided through the Tri-agency CGS-M program. CIHR complements these awards by providing a few Master's awards in priority areas. At the Doctoral level, the majority of awards are provided through the Tri-agency CGS-D program and a limited number of elite awards are also provided by the Vanier CGS program. CIHR offers some Doctoral awards in priority areas that are taken up within Canada, as well awards to pursue Doctoral training outside Canada through the DFSA. At the Postdoctoral level, CIHR provides the majority of support through CIHR Fellowship awards, which focus on two types of Postdoctoral trainees in health research studying either in Canada or abroad: postdoctoral fellows and post-health professional degree researchers. At this level, only a limited number of elite awards are provided through the Tri-agency Banting PDF program (23 awards per year, 70 total across Tri-agencies) as compared to the number of CIHR Fellowships provided (approximately 200 awards per year). The majority of CIHR's agency specific training awards at the Master's and Doctoral levels were displaced with the introduction of the Tri-agency training awards (CGS, Vanier CGS and Banting PDF), whereas there is no equivalent Tri-agency support at the Fellowship level.

In terms of career support, researchers at all career stages are currently primarily funded through research grants (i.e., Project Grant Program). Previous direct career support award programs have been sunset (e.g., New Investigator Awards, Clinician Scientist Awards) and CIHR currently offers limited awards for career support, only in priority areas and mainly at the ECR level. The sunset awards had specific objectives to support researchers early in their careers by offering salary support and protected research time; however, they did not offer sufficient support for the transition to a research career as they required researchers to hold an academic appointment as well as separate funding (i.e., operating grant).

As outlined in the Roadmap II, Strategic Direction 1 was focused on promoting excellence, creativity and breadth in health research and knowledge translation, which was expected to be achieved through the new open funding schemes (i.e., the Foundation and Project Grant Programs) and by continuing to support the next generation of researchers and professionals through Tri-agency training programs. Specifically, the Foundation Grant Program was expected to provide long-term support to research leaders at any career stage and to encourage those research leaders to establish innovative training environments that prepare trainees for evolving research and research-related careers at home and abroad.

The rationale for the way in which CIHR invests in direct training and career support is not entirely clear. The complexity of the environment and the multifactorial nature of the decision-making process is acknowledged, and a combination of the following factors likely contributed to CIHR's approach to training and career support:

Evaluations of the Tri-agency programs (e.g., CGS, Vanier CGS, Banting PDF, CRC, and CERC) largely supported their continuation, with some suggested design and delivery modifications. The STIHR Evaluation recommended that future CIHR approaches to indirect training should be informed by the STIHR model and that there should be increased use of the STIHR model within CIHR's priority-driven funding. The 2012 CIHR Salary/Career Award Programs Evaluation recommended the replacement of New Investigator Awards with operating grants. However, given that the NI awards required recipients to hold an academic appointment and a research grant, it is not surprising that outcomes resulting from these awards were similar to those resulting from grants. Recommendations from the 2012 Open Operating Grants Program (OOGP) Evaluation, the program preceding the new suite of grant programs, were focused on design and delivery in order to inform the Foundation and Project Grant Programs.

CIHR contributes to health research capacity building by supporting trainees to pursue academic careers

CIHR training award programs contribute to building health research capacity by supporting trainees to pursue academic careers. Sub-objective 4j of the CIHR Act emphasizes building the capacity of the Canadian health research community through the development of researchers. A primary goal of CIHR training programs is to train future researchers for careers in academia, as many training programs specify the training of academic researchers as an objective (i.e., CIHR Fellowship, DFSA, and CGS-D). However, current challenges to obtaining academic positions exist, including reduced faculty hiring and an aging professoriate (Statistics Canada, 2017), coupled with increasing enrollment in graduate programs (Looker, 2016 [ PDF (2.4 MB) - external link ]; Mitchell et al, 2013 [ PDF (1.29 MB) - external link ]; Tamburri, 2010).

There is limited evidence available in the literature related to the proportion of health research trainees in Canada who go on to hold faculty positions. However, existing data on career outcomes for PhD graduates indicate that the likelihood of obtaining an academic position is low. According to the Statistics Canada National Household Survey, 2011, only 18.6 percent of PhDs were employed as full-time university professors, and fewer still held tenured or tenure-track positions. Data on employment from the University of Toronto, 2016, across all disciplines show that over half (5,700) of the 10,000 PhDs found a career in the university sector, and about one third (2,900) found positions as Associate/Assistant/Full Professors.

According to end of award reports and surveys of CIHR Doctoral and Postdoctoral direct award recipients, most trainees supported by CIHR are working in the academic sector following the period of funding (CIHR Doctoral – 70%; CIHR Fellowship – 82%). See Appendix A, Figure 3: CIHR Doctoral and Postdoctoral Award Recipients by Employment Status and Sector (2000-01 to 2015-16). Similar results were observed for the Tri-agency programs (Doctoral: CGS-D – 68%, Vanier CGS – 75%; Postdoctoral: Banting PDF – 92%).

There is a need for trainees to obtain transferable skills

Although CIHR effectively supports funded trainees to pursue careers in academia, a smaller number of trainees supported by CIHR direct awards at the Doctoral and Postdoctoral levels go on to secure tenure or tenure-track positions. Findings from CIHR's Career Trajectory Survey (CTS) indicate that of those who secured employment in the academic sector (Doctoral: 70%, Postdoctoral: 82%), only 45% of Doctoral trainees, and 66% of Postdoctoral trainees secured tenure/tenure-track positions. This means that 55% of Doctoral trainees and 34% of Postdoctoral trainees who were employed in the academic sector were in non-tenure or non-tenure-track university positions (e.g., Postdoctoral Fellow, Research Associate/Assistant,). See Appendix A, Figure 4: CIHR Doctoral and Postdoctoral Award Recipients by Positions within Academic Sector (2000-01 to 2015-16). In addition, there is a proportion of trainees employed in sectors outside of academia (Doctoral: 24%, Postdoctoral: 16%). Taken together, these findings indicate that there is a need for transferable skills.

The objectives of current CIHR programs are addressing a need to train academic researchers via Doctoral and Postdoctoral level awards. However, given the proportion of trainees who end up in positions outside academia or in non-tenured university positions, there is a need to provide skills training for trainees that can be transferred to non-academic careers. It is not clear whether this need is currently being addressed. The provision of transferable skills is also a priority identified by the T-SAP.

CIHR training programs provide opportunities for trainees to study internationally

Science is international, and the need to support international study is reinforced by the Fundamental Science Review (2017), Canada's Vision for Science (2018), and the Global Education for Canadians Report (2017).

CIHR training programs (Doctoral and Postdoctoral level) allow recipients to take up the award at international institutions, providing the opportunity to develop research skills and experience within or outside Canada. At the Doctoral level, the DFSA allows Canadian Doctoral students to study internationally. Prior to DFSA's inception in 2010-11, 13% of CIHR Doctoral awards were taken up internationally, compared with 25% following DFSA inception (71 awards for the period of 2011-12 to 2015-16: 21% - DFSA, 4% - Doctoral awards in priority areas). At the Postdoctoral level, CIHR Fellowships can also be taken-up outside of Canada. Since 2000-01, only 14% of Fellowship recipients studied abroad (509 awards, 2000-01 to 2015-16).

Tri-agency direct training support has limited opportunities for international study, including the Michael Smith Foreign Study Supplement (MSFSS), but these programs aim to attract international trainees at the Doctoral and Postdoctoral levels (Vanier CGS, Banting PDF) and offer international study opportunities at the Postdoctoral level (Banting PDF). There is no Tri-agency support for international study at the Doctoral level.

Few International organizations have specific objectives to support training in other countries

Internationally, at the Doctoral level, there are no direct award programs comparable to DFSA, which is designed exclusively to support students who are pursuing a Doctoral degree abroad. There are a few programs that allow the trainees to study internationally (i.e., NIH and Research Council of Norway)Footnote 9 and there are only a few direct award programs that have a specific objective to attract international trainees (i.e., Howard Hughes Medical Institute, and Federal Ministry of Education and Research)Footnote 10.

At the Postdoctoral level, there are two direct award programs that specifically encourage international study, both funded by multi-national organizations (i.e., European Commission and Human Frontier Science Program Organization)Footnote 11. In addition, there are a few direct award programs allowing international study (e.g., Medical Research Council, National Science Foundation)Footnote 12. A few programs allow in foreign trainees (e.g., Howard Hughes, Medical Research Council)Footnote 13.

The CIHR Act differentiates it from the mandates of other Canadian funding agencies

The CIHR Act specifies an emphasis on building capacity in the Canadian health research community, which differentiates it from the mandates of other Canadian funding organizations at the Federal, Provincial and Not-for-Profit levels. The other federal funding agencies also specific a capacity building objective, but in areas other than health research: NSERC focuses on building capacity in scientific discovery and innovation and SSHRC focuses on building capacity in social sciences and humanities.

Consistent with other funders, CIHR provides most of its training and career support awards at the training levels, while international organizations fund more programs at career support levels

Consistent with other funders, CIHR award support is focused on the training levels; however, most of CIHR's investments are at the Postdoctoral level (40%). In Canada, across all jurisdictions, direct training award programs were offered most commonly at the Postdoctoral levels (n = 53 programs), followed by Doctoral level (n = 49 programs), and Master's level (n = 35 programs). Similar to the Canadian organizations, the International organizations also offered support at a variety of levels, with programs offered most commonly at the Doctoral level (n = 62 programs), followed by the Postdoctoral level (n = 48 programs). Fewer organizations offered support in the form of programs at the Master's level (n = 18 programs). Canadian organizations offered a much higher number of direct award programs at the Master's level compared with International organizations, whereas International organizations offered higher numbers of awards at the Doctoral and Postdoctoral levels.

International funders fund more direct award programs at career support levels. In Canada, across all jurisdictions, career support award programs were offered most commonly at the ECR level (n = 27 programs) followed by the Mid-Senior Career Support level (n = 20 programs). No Federal organizations beyond the Tri-Agencies (CIHR, NSERC, and SSHRC) were offering career support programs at the time of the scan, and CIHR was the only Federal organization with awards for ECRs (although only a few are currently offered in priority areas). However, it is possible that other organizations support ECRs through other investigator-initiated research mechanisms outside of the scope of this evaluation (e.g., operating grants). International organizations offered a higher number of career support programs at the ECR (n = 42 programs) and Career Support levels (n = 48 programs). Direct career support awards provided by international funders focused on ongoing skill development and support for the transition to independent researchers (ECR); and the transition to a leadership role while producing high impact research (Mid- to Senior Career). The following are a few examples of international organizations offering direct awards at the career level: Medical Research Council (MRC; UK), National Institute of Health (NIH; US), National Health and Medical Research Council (NHMRC; AUS), Wellcome Trust (UK)Footnote 14.

Design and Delivery

Key Findings:

CIHR investments in training and career support have decreased since 2000-01

As previously mentioned, CIHR's investments in training and career award support has decreased overall, from $87M in 2000-01 to $45M in 2015-16. Decreases have occurred across all levels. Most of CIHR's investments in training and career support (total of $966M) were at the Postdoctoral level (40%), followed by the ECR level (30%). At the Postdoctoral level, where CIHR's training and career support direct award investment is greatest, CIHR's investments have decreased overall by 26%, from $27M (2000-01) to $20M (2015-16). At the Doctoral level, CIHR provides limited direct award support outside of Tri-agency awards, and its investments have also decreased overall by 86%, from $13.8M (2000-01) to $2M (2015-16). CIHR's direct investments in training award support have largely been displaced by Tri-agency investments across all levels (i.e., Masters, Doctoral, and Postdoctoral). See Figure 5: CIHR and Tri-agency Doctoral and Postdoctoral Award Investments (2000-01 to 2015-16), in Millions. Career support is provided mainly through grants, with minimal direct award support at the ECR level through priority driven awards. CIHR's investment in career support awards has decreased by 71%, from $28M (2000-01) to $8M (2015-16). See Appendix A, Figure 6: CIHR Investments in Career Support Awards (2000-01 to 2015-16). As outlined in Roadmap II, the next generation of researchers and professionals was expected to continue to be supported through Tri-agency training programs, and the new suite of open programs, specifically the Foundation Grant Program, was expected to provide long-term support to research leaders at any career stage and to encourage those research leaders to establish innovative training environments that prepare trainees for evolving research and research-related careers at home and abroad.

Equity, Diversity and Inclusion (EDI)

Both the terms "sex" and "gender" are used in this evaluation. Analyses utilizing administrative data are based on the reported variable of sex given that the Tri-agency Equity, Diversity, and Inclusion (EDI) Questionnaire had not been implemented yet and administrative data in EIS only contained information on sex until then. Analyses utilizing survey data are based on the reported variable of gender because the gender questions from the EDI Questionnaire were included in the surveys. Overall, 52% of the CIHR training and support awards are from male recipients and 47% from female recipients; 1% preferred not to answer (N = 8337).

There is an observable shift in funding by sex across the training to career levels with more females funded across training levels and more males funded at the career support levels. Specifically, more female recipients were funded at the Master's (57%) and Doctoral (60%) levels; whereas, awards were distributed equally across sex at the Postdoctoral level. More male recipients are funded across all career support levels (ECR: 60%, Mid-Senior Career Support: 72%, and Chairs: 66%). Similar results were found in a recent study by Burns, et al. (2019) looking at gender differences in grant and personnel award funding rates at CIHR based on research content area where women were significantly less likely than men to be awarded grants and NI personnel awards. See Appendix A, Figure 7: Number of CIHR Training and Career Support Awards by Level and Sex (N = 8337, 2000-01 to 2015-16).

In terms of sex differences in award amounts, CIHR award amounts were distributed equally between female and male applicants for all award levels, with the exception of the Chair level where female applicants received higher average amounts ($740,344) compared with male applicants ($459,660). See Appendix A, Figure 8: CIHR Average Award Amounts by Funding Levels and Sex (2000-01 to 2015-16).

A greater number of English applications (88%) were funded at all levels compared with French applications. A greater number of applications with a Biomedical focus (59%) were funded across all the levels, although this most notable at the Doctoral and Postdoctoral levels where more awards are funded in total.

CIHR's programs do not provide sustained support for scientific careers in health research

Although CIHR's training and career support generally supports the capacity development element of sub-objective j in the Act, it also stipulates that CIHR should offer "sustained support for scientific careers in health research". However, "sustained support" has not been operationally defined, with clear indicators describing how sustained support should be achieved.

For the purpose of this evaluation, sustained support was defined as the provision of support to individuals across training and career award levels (i.e., funding an individual across multiple levels of both training and career support awards). Sustained support could also be interpreted as the provision of options for a continuum of support at various levels of award support, not at the level of the individual; for example, available programming at Master's, Doctoral, Postdoctoral, and career support levels. Sustained support may also be viewed as a combination of award and grant funding across the training and career stages. As only direct awards were in scope for this evaluation, analysis of the extent to which CIHR has provided sustained support included only direct investments (awards and grants).

Regardless of the definition, evaluation evidence indicated that CIHR is not providing sustained support for scientific careers in health research. If we consider sustained support at the individual level, the funding trajectory analysis (further discussed in the Performance Section of this report) showed that 14% of recipients received direct awards at multiple levels of training and career support and only 3.6% received direct awards at both training and career support levels. Beyond that, 30% of award recipients also went on to receive grant funding; however, the majority of those were career support award recipients who were required to hold a grant (19%). Recall that at the career support level, CIHR currently provides the majority of support through research grants and Tri-Agency Chair awards (CRC and CERC), and only a few agency specific career support awards in priority areas (previous career support direct awards have been sunset).

At the ECR level, funding is currently only provided by the CIHR Project Grant Program (PGP), and a few ECR direct awards are provided in priority areas. Currently, it is not clear whether offering career support via research grant programs (e.g., PGP) addresses the needs identified by ECRs. Surveyed NIs and CSs (both recipients and applicants) identified that support is needed to conduct independent health research, improve their reputation as researchers, support protected time, and promote professional development training and career advancement. The challenges most frequently identified by both NIs and CSs were lengthy training time, limited funding opportunities, limited mentorship opportunities, and the need for work-life balance. Recent changes to CIHR's open grant programs have been made to address inequities experienced by ECRs in terms of low success rates. The PGP now awards grants to ECRs proportionate to the application pressure from this group.

In conclusion, regardless of the definition, CIHR is not providing sustained support for scientific careers as stipulated in the CIHR Act: whether at the individual level (individuals are not being supported across multiple levels, through direct awards or grants) or across a continuum of support at various levels (minimal direct award support is provided at the Masters, Doctoral and Career Support levels, and it is not clear whether grants are providing sufficient career support, particularly given low success rates for grants).

The TCS Program is not fully aligned with the objectives of the CIHR Act, Roadmap II, and T-SAP

The Performance Information Profile (PIP) describes the TCS Program as that which provides award funding directly to promising current and next generation researchers to support training or career development. Applicants at different career stages compete through a rigorous process and those with the highest potential for promising research careers are funded. The TCS Program, as described in the PIP, is not fully aligned with the part of sub-objective j in the CIHR Act related to the provision of sustained support for trainees and researchers. It is also not fully aligned with the Strategic Direction 1 of Roadmap II in relation to training and mentoring, to provide the right mix of experience and skills to succeed in health related academic and/or professional careers.

In addition, the indicators included in the TCS PIP are more reflective of the Tri-agency programs (Banting PDF, Vanier CGS, and CGS Master's and Doctoral) and do not fully represent the range of CIHR's agency specific training and career support programs (e.g., DFSA, CIHR Fellowships, priority driven awards). Specifically, the TCS PIP indicators focus on attraction and retention, productivity, and the attainment of leadership and research positions (relevant for Vanier CGS and Banting PDF). Although indicators related to productivity and research positions are applicable to CIHR programs, there is little reference to indicators relevant to the common objectives of CIHR's direct training awards such as the opportunity to gain research experience, and to provide a reliable supply of highly skilled and qualified researchers. Similarly, although some PIP indicators relate to the challenges identified in the training strategy (i.e., research leaders of tomorrow), some elements are not reflected (i.e., multidisciplinary and multisectoral research, highly-qualified personnel with priority-driven specializations).

CIHR developed the T-SAP in 2015 in order to support the fulfillment of Strategic Direction 1 in Roadmap II. It was focused on identifying gaps and challenges in order to identify potential areas needed for funding. The following are the three main areas included in the training strategy.

With respect to the T-SAP, the current TCS Program does not address the need for training and mentoring, particularly for multidisciplinary training, professional skills, and expertise in critical priority areas. Although most of CIHR's agency specific training and career support programs existed before the development of the T-SAP in 2015, there is limited evidence that any programmatic changes have been made to align current investments with the challenges identified in the training strategy. Key programs (e.g., STIHR, Clinician Scientist, MD/PhD Program) that were aligned with the training strategy (i.e., multidisciplinary training, providing training across knowledge sectors) have been sunset.

The TCS Program has no description of how expected results are produced across all training mechanisms and there are gaps in performance measurement data

Although CIHR's TCS Program is described broadly in the PIP, the Program does not have a clearly defined program theory (i.e., how the program is expected to produce results) that specify intended outcomes and how they will be achieved across all activities and investments related to training and career support at CIHR (direct and indirect).

There are currently gaps in performance measurement data, which present challenges for ongoing program monitoring as well as the assessment of intended outcomes. The challenges include the absence of End of Award Reports (EARs) for some agency specific awards such as the DFSA and priority announcements, as well as for some Tri-agency awards such as the CGS (both Master's and Doctoral). Currently, EARs are collected for recipients of the CIHR Fellowship, the Vanier CGS and the Banting PDF. In addition to the absence of data, there are limitations with existing PM tools. Specifically, the EARs are very long, there are inconsistencies in the structure of measurement scales across questions, there are errors in branching and coding that, taken together, impact the accuracy and reliability of the data as well as the resources needed to analyze and report on it. The EARs are not always implemented consistently (e.g., sent out at differing times points since award completion) and there is currently no mechanism in place to enhance compliance, again limiting the reliability and generalizability of the data. Lastly, despite the collection of a large amount of data through the EARs, there is a lack of clarity on what is done with the information (i.e., it is not regularly reported on). There is also currently no process in place to track and monitor indirect support provided to trainees and researchers; therefore, CIHR can not reliably assess or evaluate the impact of indirect funding on capacity building within the broader context of training and career support. Having performance measurement in place for both direct and indirect funding is necessary to allow CIHR understand and report on the entire spectrum of training and career support at CIHR.

Objectives of current CIHR training programs are not clearly defined and do not align with the objectives of the training strategy

The specific award programs and FOs also typically have limited descriptions or objectives outlined in the FO and are rarely accompanied by expected results. For example, information about the objectives of the CIHR Fellowship is communicated to applicants in the FO in the following way: "To provide recognition and funding to academic researchers; to provide a reliable supply of highly skilled and qualified researchers." The following description of the DFSA provided in the FO is: "To provide recognition and funding to students early in their academic research career, providing them with an opportunity to gain research experience; and to provide a reliable supply of highly skilled and qualified researchers."

Generally, the programs are meeting their limited objectives, by selecting the best candidates and supporting excellence in research (more details below in Performance). However, although these objectives relate broadly to the capacity building elements of the CIHR Act they are not fully aligned with sub-objective j, and none of the program objectives relate to the priorities outlined in the training strategy (i.e., Research Leaders of Tomorrow, Leaders Across Knowledge Sectors, and Experts in Critical Priority Areas). Specifically, CIHR training programs do not have objectives related to leadership, multidisciplinarity, and transferable/entrepreneurial skills, beyond the development of research skills. Career support awards also did not have objectives for training and mentoring. Award recipients (NI, CS) reported a lack of training and mentoring opportunities for themselves. Much of the training gained by researchers is implicit, as they experience learning opportunities through the training of graduate students. The majority of NIs were engaged in the following activities: student supervision (99%), mentoring (90%), course teaching (96%), reviewing thesis and papers (89%), and lab meetings (78%; n = 157). Again, CIHR programs that had and met objectives related to mentoring and multidisciplinarity have been sunset: MD/PhD Grants, Clinician Scientist Awards, and STIHR Grants.

Career support is currently provided mainly through research grants, and there is a gap in capacity building objectives

Currently, career support is provided mainly through research grants rather than awards. The Foundation Grant Program (FGP), which was recently sunset in April 2019, was intended to develop and maintain Canadian capacity in research and other health-related fields (among other objectives) through the long-term support of established research leaders undertaking innovative, high-impact programs of research. The programs of research were expected to include mentoring/training components. Although capacity building impacts are expected from the recipients of the Foundation Grants (prior to the program being sunset), there is a gap in capacity building objectives going forward given that the Project Grant Program is now CIHR's only investigator-initiated grant and it does not have explicit objectives directly related to capacity building.

CIHR Postdoctoral award amounts are lower compared with the Tri-agency and International awards

Consistent with CIHR, direct training and career support awards in Canada and internationally are funded through both investigator initiated (subject of application is driven by the researcher) and priority driven mechanisms (subject is in a targeted area).

In general, CIHR direct award amounts are on par (Master's and Doctoral) or lower (Postdoctoral) when compared with other national organizations; while award durations are on par at all levels. CIHR training awards at the Master's and Doctoral levels are on par with CGS in terms of amount (~$17.5k and 35k advertised amounts per year, respectively) and duration (1 year and 3 years, respectively). CIHR direct award amounts at the Postdoctoral level are slightly lower than Tri-agency awards ($60k vs. 70k advertised amounts per year); however, the Banting PDF is designed to support elite trainees. CIHR award duration at this level (i.e., Fellowship) is on par with Banting (2 years). Overall, recipients of CIHR direct training awards (DRA, CIHR Fellowships) were satisfied with the award amount and duration. CGS-D recipients were satisfied with award amount but suggested an increase in duration. CGS-M recipients suggested an increase in amount and duration. In general, CIHR direct award amounts and durations are on par with (Doctoral) or lower (Master's, Postdoctoral) than those of international organizations.

Performance

Key Findings:

The evaluation examined the outcomes and impacts of CIHR's investments in direct training and career support related to trainees and researchers funding trajectory, productivity over time, training and mentoring, as well as the provision of multidisciplinary training and career outcomes.

CIHR is not providing sustained support: the majority of training and career award recipients were funded at only one level

The evaluation included a funding trajectory analysis in order to look at how CIHR funds individuals across the levels of its agency specific and Tri-agency training and career support awards. The results of this analysis have shown that CIHR is not providing sustained support (across multiple levels of training and career support) for scientific careers in health research. It included applicants who submitted at least one application to one of CIHR's 561 different training and the direct career support award programs/funding opportunities between 2000-01 and 2015-16. The analysis included a total of 63,234 applications and 29,472 distinct applicants.

More than half of all applicants (56%) who applied for a CIHR direct training or career support award were unsuccessful. Of those that did receive a direct award, the majority were funded at only one level of support (86% of 12,541). The most common singular levels of support were at the training level; specifically, 25% at the Doctoral level, 23% at the Postdoctoral, and 22% at the Master's level.

A small number of recipients were funded at multiple levels. Specifically, 13% were funded at two levels and only 1% were funded at three levels. Of those funded at two levels, the most frequent combinations were within the training levels - Master's + Doctoral levels (n = 716, 2.4%) and Doctoral + Postdoctoral levels (n = 683, 2.3%). Very few recipients (3.5%) were funded at both training and career support levels.

The success rates vary across award levels, with higher rates overall for programs in which the applicants' host institutions are involved in the selection and nomination processes, such as the Chairs (86%), Master's (67%) and Mid-Senior Career Support (65%) levels, and lower for programs in which the researchers submit their applications directly to CIHR such as Doctoral (21%), Postdoctoral (19%) and Early Career Support (19%) levels. The application pressure is greatest at the Doctoral and Postdoctoral level.

Almost half (47%) of the applicants applied only once (successfully and/or unsuccessfully), followed by approximately one quarter who applied twice (26%), with the remaining quarter applying three or more times (27%; range 3-18). Of those who applied more than once, applicants tended to apply to the same award level. This occurred most frequently at the Early Career (59%), Doctoral (56%), and Postdoctoral levels (47%). Of those who were funded, the majority were funded between 1 and 4 years (range 1-16). Unsurprisingly, those funded for a longer period of time were mainly at the ECR, Mid-Senior Career Support and Chair levels.

Thirty percent of award recipients received grants, the majority of which were career support award recipients who were required to hold a grant

The funding trajectory analysis also examined the extent to which direct award recipients went on to receive a grant as a principal investigator or a collaborator. Across the levels of training and career support, 30% of recipients of one or more awards also received grant funding. However, the eligibility criteria for some career support awards confound this finding (e.g., New Investigator Award). The majority of those recipients who also got a grant (19% of 30%) included career support recipients who were required to hold a grant at the time of award receipt. This finding strengthens the conclusion that CIHR is not providing sustained support across multiple levels, including a definition of sustained support as consisting of a combination of awards and grants.

When looking at recipients of training awards, the likelihood of success in obtaining a grant increases with the award level. Trainees who received an award at the Postdoctoral level were most likely to obtain a CIHR grant: 27% of Postdoctoral award recipients also received grants, compared with 13% of Doctoral and 2% of Master's award recipients.

Award recipients were more productive than applicants

A bibliometric analysis of a sample of 3,000 applicants to Doctoral and Postdoctoral awards was conducted for this evaluation, using number of published journal articles, average of relative citations (ARC), and average relative impact factor (ARIF) as measures of productivity and impact. While other types of research outputs can be used as indices of productivity, journal article publications and citations are traditional and frequently used measures of research productivity. Results of the bibliometric analysis showed that recipients of CIHR and Tri-agency Doctoral and Postdoctoral direct training awards published significantly more papers than applicants, two years before receiving the award, indicating that the right candidates have been selected for training awards. More specifically, Doctoral award recipients (n = 576) published significantly more papers than applicants (n = 1,441) before the award (0.55, vs.0.37, respectively; p < 0.001). A similar trend was observed for the average of relative citations (ARC; 1.51 vs. 1.33, p < 0.01). Postdoctoral award recipients (n = 517) also published significantly more papers than applicants (n = 1,553) before the award (1.03 vs. 0.82, respectively; p < 0.001). Postdoctoral award recipients also had significantly higher ARC and ARIF scores than applicants during the period before the award, although the differences were not large (ARC - 1.55 vs. 1.37, p < 0.001; ARIF - 1.41 vs. 1.28, p < 0.001).

Similarly, recipients of Doctoral and Postdoctoral awards had greater research productivity 4 years after the award compared to applicants. Although it is likely that applicants secured other funding, receiving CIHR or Tri-agency awards had a greater impact on research productivity as measured by the average number of publications. More specifically, Doctoral award recipients (n = 576) published significantly more papers than applicants (n = 1,441) four years after the award (1.03, vs.0.69, respectively; p < 0.001). A similar trend was observed for the ARC (1.62 vs. 1.45, p < 0.01). Postdoctoral award recipients (n = 517) also published significantly more papers than applicants (n = 1553) before the award (1.39 vs. 1.08, respectively; p < 0.001). Postdoctoral award recipients also had significantly higher ARC and ARIF scores than applicants following receipt of the award, although the differences were rather small (ARC - 1.65 vs. 1.58, p < 0.001; ARIF - 1.46 vs. 1.36, p < 0.001).

Postdoctoral recipients had a greater average number of publications than Doctoral recipients

Results of the bibliometric analysis also showed that CIHR and Tri-agency training awards have contributed to an increase in Doctoral and Postdoctoral award recipients' research productivity throughout their degree (4 years after the award). Both Doctoral and Postdoctoral level award recipients had a significant increase in number of publications after receiving the award (Doctoral: 0.7 papers before the award vs. 1.04 papers after the award, n = 576, p < 0.001; Postdoctoral: 1.28 papers before the award vs. 1.43 papers after the award, n = 517, p < 0.001). The average annual number of papers authored by Postdoctoral recipients was observed to be higher than that of Doctoral recipients, as would be expected given that Postdoctoral trainees are more advanced in their research careers than most Doctoral trainees.

On the other hand, their scientific impact over the period of funding (2 years before and 4 years after the start of the award, as measured by ARIF and ARC) was not associated with funding (except for a slight increase of ARIF for those funded at the Postdoctoral level: 1.43 before the funding, vs. 1.49 after the funding). The rate of acknowledgment to CIHR funding increases notably after the competition for those successful applicants (Doctoral: 0.20 vs. 0.34, p < 0.001; Postdoctoral: 0.24 vs. 0.32, p < 0.001).

The research productivity of CIHR Fellowship recipients was comparable to the productivity of Banting fellows in health sciences (average of 1.5 vs. 1.6 papers per year, respectively) as shown in the previous evaluation of the Banting PDF Program (2015).

Funding through CIHR and Tri-agency direct training awards also contributed to an increase in the international collaboration rates for publications. At the Doctoral level, 30% of recipients reported engaging in international collaborations before the award, vs. 37% after the award (n = 576, p < 0.001); at the Postdoctoral level, this reported engagement was 38% and 43% before and after the award, respectively (n = 517, p < 0.001).

Funding across both Doctoral and Postdoctoral levels was associated with increased productivity

In terms of the impact of multiple levels of funding, receiving direct funding at both Doctoral and Postdoctoral award levels resulted in increased productivity (vs. funding at the Postdoctoral level only). Postdoctoral awardees who were also funded through direct awards during their Doctoral studies produced more papers, on average, than Postdoctoral awardees who were not funded by a CIHR or Tri-agency direct award during their Doctoral studies (1.73 vs. 1.33 papers, n = 432, p < 0.05). However, it should be noted that there was no relative increase in productivity observed for those who received funding at both Master's and Doctoral award levels (vs. those funded at the Doctoral level only).

CIHR ECR award recipients were generally more productive than applicants and OOGP recipients at the ECR level

At the career support level, surveyed recipients of NI awards reported producing a similar number of peer-reviewed journal articles where they were first author compared with applicants (M = 8.9, SD = 8.15, n = 112 vs. M = 8.8, SD = 13.8, n = 124). Recipients of the CS award (n = 15) produced an observably higher average of peer reviewed journal articles where they were the first author than did applicants (n = 5) who still completed clinician scientist training (M = 32.3, SD = 35.1 vs. M = 4.8, SD = 3.5); although these results should be interpreted cautiously due to the low number of responses. Recipients of NI awards (n = 134), compared with applicants (n = 147), reported a significantly higher average of peer-reviewed journal articles (M = 31.7, SD = 23.3, vs. M = 23.8, SD = 25.5; p < 0.01).

When the productivity of NI award recipients was compared with the ECR cohort of OOGP grant recipients, the average number of publications was higher for NI award recipients than OOGP grantees (M = 8.3, SD = 8.9, n = 623, End of Grant Report data for 2000-2013). This is not surprising given that eligibility requirements of the NI awards included holding a CIHR grant and an academic position. In other words, the NI award holders had the benefit of additional salary support on top of a grant. This finding is also consistent with findings from the Salary/Career Award Programs Evaluation (2012; covering salary/career awards over the 2000-2012 period) that recipients of NI awards published a significantly greater number of refereed journal articles than applicants during the period of the award (M = 19.8, SD = 18.5, n = 405 vs. M = 13.2, SD = 11.4, n = 233; p < 0.001). Conversely, in the 2012 evaluation, recipients of Senior Investigator awards, published a similar number of refereed journal articles compared with applicants (M = 27.3, SD = 19.5, n = 101 vs. M = 28.8, SD = 20.8, n = 41; p = 0.7).

CIHR is contributing to building health research capacity, the majority of award recipients are working full time in the academic sector

CIHR direct training and career support award programs at all levels aim to promote recipients' academic career outcomes; specifically at the training level, by providing recognition and funding to academic researchers and a reliable supply of highly skilled and qualified researchers (i.e., Fellowship, DFSA), and at the career level by providing funding for research and offering protected time for research (i.e., NI, CS).

CIHR direct training awards have contributed to trainees finding relevant research related employment in academia, with the majority working full time in Canadian universities (Fellowship: 82% from end of award reports and CTS study; Doctoral: 70%, from CTS study; MD/PhD program: 83% from Skinnider et al.; STIHR: 67% from 2016 evaluation). See Figure 3: CIHR Doctoral and Postdoctoral Award Recipients by Employment Status and Sector (2000-01 to 2015-16). Similar results were observed for recipients of Tri-agency awards, with the majority of trainees working in the academic sector (CGS Doctoral: 68%, Vanier CGS: 75%, and Banting PDF: 92%).

Half of Doctoral award recipients and two thirds of Postdoctoral award recipients employed in the academic sector reported being in tenured or tenure-track positions; Postdoctoral award recipients were more frequently in higher ranked positions

Half of Doctoral award recipients and two thirds of Postdoctoral award recipients employed in the academic sector had secured tenured or tenure-track positions (CTS, Fellowship End of Award Report [FEAR]). Of those employed in the university sector, a greater proportion of Postdoctoral direct award recipients were in tenured or tenure-track positions (66%) compared to Doctoral recipients (45%). Postdoctoral direct award recipients were also more frequently in higher ranked positions compared to Doctoral direct award recipients (22% vs. 9% were Associate or Full Professors; and, 44% vs. 35% were Assistant Professors; respectively).

The remaining trainees employed in the academic sector were non-tenured. Results from the CTS study showed that most CIHR direct training award recipients were in Postdoctoral or Research Assistant positions (Doctoral: 32%, Postdoctoral: 23%), or other teaching and research faculty positions (Doctoral: 14%, Postdoctoral: 8%). Of the STIHR trainees employed in the academic sector, 31% were research faculty, 23% were Postdoctoral fellows/associates, 14% were teaching faculty, and 10% were Research Assistants (2016 Evaluation). See Figure 4: CIHR Doctoral and Postdoctoral Award Recipients by Positions within Academic Sector (2000-01 to 2015-16).

Trainees identified the limited supply of relevant positions/strong competition as the greatest barrier to current employment (FEAR). Additional challenges identified by MD/PhD doctoral trainees (Skinnider et al.) included financial constraints given the length of physician-scientist training (median = 13.5 years), and the lack of protected research time.

Up to one quarter of Doctoral and Postdoctoral award recipients are working in sectors outside of academia

Although the majority of direct award recipients are working in academia, 16% of Doctoral award recipients and one-quarter of Postdoctoral award recipients are working in sectors outside of academia (e.g., government, industry). Of those in non-academic positions, approximately half were in senior, executive, or managerial positions with no observable differences between training levels (Postdoctoral: 50%, Doctoral: 48%).

Award recipients reported higher levels of career satisfaction, employment-related income, and advancement in their careers

CIHR direct career support awards have provided funding for researchers in the early stages of their careers, and have helped researchers advance through their careers. Approximately half of both NI award recipients and applicants advanced from Assistant to Associate Professor positions over the course of their award tenure, although this occurred for a significantly higher proportion of recipients (53% vs. 47%, respectively, p < 0.01). Similarly, the proportion of funded ECRs who advanced from Assistant to Associate Professor positions (56%) was greater than for applicants (38%; Salary/Career Awards Evaluation, 2012). For CSs, 10 of 16 CS recipients reported that they were in academic positions (of which six were Assistant Professors), and the remaining six worked for hospitals and other health care providers. Similar results were obtained during the 2012 Salary/Career Awards Evaluation. Specifically, a greater proportion of Funded ECRs (56%) advanced from Assistant to Associate Professor positions, compared to applicants (38%).

In general, surveyed NI researchers reported satisfaction with their current employment; however, award recipients were significantly more satisfied than applicants (M = 4.33 vs. M = 3.89 [out of 5], respectively, p < 0.001). Award recipients also reported significantly higher employment-related income compared with applicants (M = $170,806 vs. M = $143,681, both before deductions, p = 0.02).

NI and CS awards have had a limited impact on recipients' time spent on research

With respect to time spent on research, NI awards have had significant overall impact on time; however, recipients and applicants reported spending similar amounts of time on research (75% and 70%, respectively), indicating that the difference is limited and may not be practically significant. The lack of observed difference in time spent on research between recipients and applicants is most likely due to the eligibility requirements of the award (i.e., holding an academic position and an operating grant), resulting in similar characteristics associated with receiving funding from other sources. Recipients of and applicants to CS awards also reported spending a similar amount of time on research (62% and 64%, respectively). NIs and CSs identified the following challenges related to their careers: lengthy training time, limited job opportunities, and lack of work-life balance.

CIHR training award recipients have been provided with opportunities to develop research skills; however, the awards lack explicit objectives on training and mentoring

CIHR has provided training and mentoring opportunities through its Fellowship awards (e.g., research skills, professional skills, opportunities for collaboration and publications). In general, trainees reported high levels of satisfaction with the opportunities provided. Specifically, 83% reported improvement in their research leadership skills and 68% reported improvement in teaching leadership skills (although recipients only reported improving to a moderate extent in service leadership and professional leadership).

In contrast to CIHR direct award programs, general objectives related to training are included in some Tri-agency programs (i.e., CGS, Banting PDF). The CGS Evaluation (2016) found that CGS recipients had more opportunities for the development of research skills compared to applicants, and leadership skills were enhanced for Vanier CGS and Banting PDF recipients. However, there has been variability in the type of opportunities and the levels of skill development reported in these Tri-agency evaluations, and the definition and assessment of the leadership concept is challenging for both the Vanier CGS and Banting PDF.

Vanier scholars' supervisors rated the award holder they supervised as being exceptional at demonstrating leadership in research (62%), personal and/or professional leadership (51%), leadership in providing service to the academic community (31%), leadership in teaching (27%) and leadership in providing service to the larger community (21%). Almost all Banting fellows (96%), CIHR Fellowship recipients (100%), and applicants (93%) believed that their research leadership abilities had developed to a great extent or some extent as a result of their Postdoctoral training. Awards also helped the trainees dedicate more time to research (Banting PDF: 71%, CIHR Fellowship: 74%) with the exception of MD/PhD Program (only 36% of graduates reported dedicating 50% or more of their time to research). However, trainees noted a lack of development opportunities related to professional and teaching skills.

The STIHR program involved a large number of trainees (3300+) and mentors (approximately 3000); and the majority of STIHR programs implemented transdisciplinary training programs (93%), integrated training on the ethical conduct of research (88%), knowledge translation (91%), and professional skills (92%). Over 80% of respondents surveyed about the MD/PhD program were satisfied with their training, and felt it had helped their career (Skinnider et al., 2017).

Recipients of career support awards supervised and trained more trainees and research staff compared to applicants and OOGP recipients

NI award recipients trained and supervised more trainees and research staff than applicants (M = 30.7, SD = 20.6, n = 154 vs. M = 24.9, SD = 19.1, n = 158; p < 0.05). CS award recipients also trained and supervised more trainees compared with applicants who still completed clinician scientist training (M = 26.5, SD = 19.1, n = 19 vs. M = 9.7, SD = 8.3, n = 6). However, the results for CSs should be interpreted cautiously due to the low number of responses.

Training opportunities for students have also been provided through grants; however, recipients of direct career support awards supervised and trained more trainees and research staff compared with OOGP funded researchers. The OOGP Evaluation (2012) found that funded researchers trained an average of 8.6 trainees and research staff per grant. OOGP funded researchers at the ECR level (based on end of grant data for the period of 2000-2013) trained an average of 13.9 trainees and research staff per grant (SD = 20.7, n = 666; end of grant data as of 2018).

Other national and international funders have objectives and/or eligibility requirements related to training and mentoring in some of their award programs (e.g., Nova Scotia Health Research Foundation, NIH, Welcome Trust), although the majority of award programs offered by other funders do not necessarily have these objectives explicitly stated.

CIHR training and career support programs lack objectives related to multi- or interdisciplinary training (consistent with other Canadian and International funders)

No current CIHR direct training award programs have specific objectives for multidisciplinary training despite the fact that CIHR's Roadmap II (i.e., Strategic Direction 1) and the training strategy (i.e., Research Leaders of Tomorrow) explicitly mention multi- or interdisciplinarity as a goal. The recently sunset STIHR program included objectives related to multidisciplinarity, and almost all STIHR programs (93%) implemented multidisciplinary training. Trainees reported noticeable improvement in multidisciplinary/transdisciplinary research skills (75%) and networking opportunities (74%).

CIHR (i.e., Fellowship) and Tri-agency direct training award programs offered some opportunities for trainees to interact with researchers in other disciplines, and award recipients reported experiencing multidisciplinary collaboration to varying extents. More than one third of CIHR Fellowship awardees interacted at least monthly with researchers in other disciplines in Canada (40%) and outside Canada (36%). In addition, 61% of Fellowship recipients also felt that they improved their leadership skills in interdisciplinary research.

It should be noted that few national and international organizations have explicit objectives for multidisciplinary training. Only 11 of 373 national and international direct award programs in the Environmental Scan specifically referenced multidisciplinary training in their objectives. Some of these organizations are: Alberta Innovates, Wellcome Trust, New Brunswick Health Research Foundation, National Science Foundation, Human Frontier Science Program Organization.

CIHR direct career support programs also do not have explicit objectives related to multidisciplinarity and it is not clear the extent to which these programs enable researchers to provide multidisciplinary training to their trainees (other than STIHR which had an explicit objective for NPIs to provide this type of training). However, many career award recipients interacted with researchers in other disciplines in Canada and internationally. Tri-agency career support programs (CRC, CERC) specify various disciplines for research in their objectives and have been successful in fostering collaborations to some extent.

Conclusions and Recommendations

Relevance

CIHR investments in direct training and career support are largely driven by the Act, the strategic plan, and Federal Government priorities

The evaluation concludes that CIHR investments in direct training and career support are largely driven by the objectives and priorities of the CIHR Act and the strategic plan (Roadmap II) as well as Federal Government priorities related to the need to support the next generation of researchers (e.g., Fundamental Science Review, Federal Budgets, Canada's Science Vision).

Since inception, CIHR has invested in direct training and career support by offering an extensive number of programs across all levels, the majority of which were priority announcements in targeted areas. Currently, CIHR provides most of its direct training and career support awards at the training levels, in particular at the Postdoctoral level, complementing Tri-agency award programs (e.g., CGS-M, CGS-D, Vanier and Banting). CIHR offers limited career support beyond research grants to support the transition to independent research careers. The current agency-specific direct career support awards are mainly priority driven awards, focused at the early career level. CIHR's career support investments through Tri-agency programs include the CRC, CERC, and CFREF.

CIHR investments in direct agency-specific training and career support have been decreasing, whereas overall investments in training through both agency-specific and Tri-agency programs have not. Overall, there has been a decrease in CIHR's investment in Tri-agency direct award support through the CRC's and CERC's. In other words, more trainees and researchers are being supported through Tri-agency award programs than CIHR's agency specific awards. Investment and programming decisions have been informed by previous evaluations (e.g., the salary/career award evaluation), Federal Government priorities, the development of the training strategy, and changes in grant programming through the reforms (career support and capacity development was incorporated in the Foundation Grant Program, which was expected to support leading researchers at any career stage and included a requirement for a training and mentoring plan as part of the application).

CIHR is contributing to building health research capacity as per the Act, most notably within the academic sector

CIHR direct training award support complements Tri-agency programs, supporting the needs of the health research community, through the provision of direct awards in priority areas, opportunities for international study (Doctoral and Postdoctoral levels), and support for Postdoctoral level trainees unique to the health research community (e.g., support for post-health professionals in addition to Postdoctoral trainees).

CIHR contributes to building the capacity of the Canadian health research community by supporting trainees to pursue academic careers, as the majority of CIHR trainees supported through direct funding are working full time in the academic sector. However, there is a need for trainees to obtain transferable skills, as only 45% of Doctoral trainees and 66% of Postdoctoral trainees who work in academic sector secured tenure/tenure-track positions and there are limited academic positions. In addition, 24% of Doctoral and 16% of Postdoctoral award recipients are working in sectors outside of academia, needing these transferable skills.

CIHR's training award support is consistent with other Canadian and International funders

The CIHR Act focuses on building capacity in the Canadian health research community, which differentiates it from the mandates of other Canadian organizations (Federal, Provincial, and Not-for-Profit). Consistent with other funders, CIHR direct award support is focused on the training levels with most of its investments at the Postdoctoral level. International funders fund more direct award programs at career support levels, whereas CIHR offers few ECR direct awards in priority areas.

Design and Delivery

The majority of training support is provided at the Postdoctoral level whereas the majority of career support is provided through grants

The evaluation found that CIHR currently provides most of its direct training and career support awards at the training levels, utilizing resources to target support at the Postdoctoral level, fulfill key priorities identified by the Federal Government (e.g., opportunity for international study given that science itself is international), and the needs of the health research community (e.g., Postdoctoral fellows and post-health professionals). CIHR provides limited career awards beyond research grants to support the transition to independent research careers.

More female recipients are funded at the Master's and Doctoral levels, while direct awards are distributed equally across sex at the Postdoctoral level and more male recipients are funded at the career support levels. CIHR award amounts are relatively equal between female and male recipients across award levels, except at the Chair level where female recipients receive higher average award amounts but fewer awards in total. The majority of funded applications are in English and more than half of funded applications have a Biomedical focus.

In terms of how CIHR compares to other national and international research funders, CIHR direct training award amounts and duration are on par with other Tri-agency and international direct awards at the Doctoral level; however, they are lower at the Postdoctoral level. At the Master's level, they are on par with the other Tri-agency direct awards and lower than international direct awards. CIHR supports international study at the Doctoral level via the DFSA and at the Postdoctoral level via the CIHR Fellowship; whereas, Tri-agency programs (Vanier and Banting) promote the attraction of international trainees and only a few international organizations have specific objectives to support training in other countries.

CIHR needs to better define, align, and monitor training and career support activities and investments in relation to the CIHR Act and the next strategic plan

The evaluation concludes that CIHR's direct training and career support award programs (and Tri-agency programs) do not address the need to provide sustained support (across multiple levels) for scientific careers in health research, as outlined in the CIHR Act. It is currently unclear how sustained support has been interpreted, implemented, and how this objective will be achieved. Few recipients are funded at multiple levels of training and career support, and almost all are funded at the training level only. It is currently not clear whether grants are providing sufficient career support across all career stages, given the recency of the reforms and the sunset of the Foundation Grant Program as well as low success rates for grants.

Evidence from the evaluation indicates that the current TCS Program does not effectively align with the objectives and priorities of the CIHR Act, strategic plan (Roadmap II), and the training strategy (T-SAP), and lacks indicators that reflect the full range of CIHR's training and career support award programs. Furthermore, there is no description of how CIHR's training and career support award programs are expected to produce results (i.e., theory of change) across all mechanisms that provide training and career support. The direct award programs themselves are also not fully aligned with the objectives of the CIHR Act, Roadmap II, and the training strategy (T-SAP), specifically with respect to sustained support for research careers and transferable skills. Given the limited information included in the funding opportunities with regard to objectives, it is currently not clear whether CIHR training and career support programs effectively support the achievement of intended outcomes. There are also gaps in performance measurement data including the absence of end of award reports, not only for some award programs (e.g., DFSA) but also for the large number of priority driven funding opportunities that fund few awards each. There are also limitations in the implementation of existing performance measurement tools and use of existing performance measurement data (e.g., response rates, inconsistent timing in launch of end of award reports, structure of questions and length of end of award reports). Taken together, these factors limit the ability to effectively assess whether the awards are fully meeting objectives. In addition, performance measurement mechanisms are needed to capture data for indirect training and career support as part of the CIHR grants. This would ensure that data is available for CIHR to monitor and report on all mechanisms of training and career support.

Current direct training and career support programs lack specific objectives for training, mentoring, or multidisciplinary training (identified as priorities in the strategic plan and training strategy) and those direct award or training programs with specific objectives related to training, mentoring, and multidisciplinary training have been sunset (e.g., STIHR). In addition, there is a gap in capacity building objectives with the sunset of the FGP.

Performance

Sustained support for scientific careers in health research is not being provided

The evaluation provides evidence that CIHR's training and career support programs (and Tri-agency programs) have not provided sustained support for scientific careers in health research as outlined in the CIHR Act. The results of a funding trajectory analyses have shown that few recipients (14%) have been funded at multiple levels across the award levels of training and career support, and only 3.6% were funded at both training and career support levels. Furthermore, when looking at support via grants in addition to direct awards, almost one-third (30%) of recipients of one or more awards, at one or more levels, also received grants. However, the majority of these recipients received career support awards (19%) and were required to hold a grant at the time of application or receipt. Taken together, sustained support, whether conceptualized at the individual level or via opportunities for support across the spectrum of levels, is not being provided by CIHR.

Supported trainees and researchers are more productive, with the most productive being those funded at both the Doctoral and Postdoctoral levels

The evaluation shows that recipients of training and career awards (both CIHR and Tri-agency) were more productive (in terms of research publications) than applicants. Postdoctoral recipients had a greater average number of papers than Doctoral recipients and cumulative funding across both Doctoral and Postdoctoral levels was associated with increased productivity (vs. Postdoctoral funding only). In addition, recipients of training awards had higher international collaboration rates than those who did not receive awards.

CIHR training awards contributed to building health research capacity, most notably in the academic sector

CIHR direct training award programs contributed to the research careers of award recipients, with the majority working full time in the academic sector (>75%). Approximately half of the Doctoral direct award recipients (46%) and two thirds of Postdoctoral direct award recipients (67%) working in the academic sector were in tenure-track positions. Of those, Postdoctoral award recipients were more frequently in higher ranked positions (e.g., Associate and Full Professor). At the training levels, CIHR and Tri-agency funding at the Postdoctoral level has had a greater impact on recipients' research careers – in terms of research productivity, advanced academic positions, and the receipt of CIHR grants (27% vs 13% for Doctoral award recipients).

CIHR direct career support award programs contributed to supporting the research careers of funded researchers. Award recipients reported higher levels of career satisfaction and employment-related income, as well as advancement in their careers.

CIHR training award recipients have been provided with opportunities to develop research skills; however, the awards lack explicit objectives on training and mentoring

The evaluation found that although CIHR direct training award recipients have been provided with opportunities to develop research skills and the majority were highly satisfied with these opportunities, the awards lack explicit objectives on training and mentoring and are not aligned with the priorities in the most recent strategic plan (Strategic Direction 1 of Roadmap II) or the training strategy during the period under review. In addition, programs with specific objectives related to training, mentoring, and multidisciplinary training have been sunset. Direct career support awards also lack objectives for training and mentoring, and award recipients identified a lack of personal development and mentorship opportunities. Recipients of direct career support awards contributed to the training and mentoring of trainees. Specifically, they supervised and trained more trainees and research staff compared to applicants and OOGP recipients. Currently, career support is provided mainly through research grants; however, with the sunset of the FGP (which included objectives related to capacity building and was intended to support researchers at all stages), there is a gap in career support with a lack of specific objectives focused on capacity building.

CIHR training and career support programs lack objectives related to multi- or interdisciplinary training (consistent with other Canadian and International funders)

With respect to the provision of multi- or interdisciplinary training, the evaluation found that CIHR direct training and career support programs lack specific objectives and are not aligned with the priorities of Roadmap II (Strategic Direction 1) or the training strategy. However, few national and international organizations have explicit objectives for multidisciplinary training. CIHR and Tri-agency direct training and career support award recipients reported experiencing multidisciplinary collaboration to varying extents, although these awards also lacked explicit objectives related to multidisciplinary training and it is not clear the degree to which career support awards enabled researchers to provide multidisciplinary training to their trainees.

Improved performance measurement is needed to better assess the performance of training and career support awards

Given the direct training and career support programs' lack of clearly defined objectives and expected results, as well as gaps in performance measurement data, it is hard to assess the effectiveness of the programs. The performance measurement gaps include the absence of End of Award Reports for some agency-specific as well as Tri-agency awards (e.g., DFSA, CGS), limitations in the implementation of existing performance measurement tools (e.g., timely data collection), and use of existing performance measurement data (e.g., a lot of data is collected and not used beyond evaluation). In addition, the TCS Program does not have objectives related to priorities in the CIHR Act and training strategy and has indicators that reflect Tri-agency programs and not the full range of CIHR training and career support programs (e.g., DFSA, Fellowship). The TCS Program should have clearly defined objectives and expected results, as well as available and reliable data. CIHR needs to ensure performance measurement mechanisms are in place for all awards in order to enable effective program monitoring and reporting. A similar approach should apply to indirect training and career support funding through CIHR grants to ensure reliable data is also captured for this type of support.

Recommendations

Recommendation 1:

CIHR needs to continue to provide funding that contributes to building health research capacity to meet the objective of the CIHR Act and the needs of trainees and the health research community. Specifically, CIHR should continue to:

  1. provide support for international study at the Doctoral and Postdoctoral levels; and,
  2. provide support at the Postdoctoral level for both Postdoctoral fellows and post-health professional degree recipients.

Recommendation 2:

CIHR needs to define and outline how it will achieve "the provision of sustained support for scientific careers in health research" as specified in the CIHR Act, across the full spectrum of training and career support mechanisms (both direct and indirect).

Recommendation 3:

CIHR needs to align investments and activities in training and career support to meet the objectives of the Act, the training strategy, and the next strategic plan.

Recommendation 4:

CIHR needs to improve the monitoring and performance measurement of all training and career support activities and investments (both direct and indirect).

Bibliography

  1. Burns,K., Sharon E. Straus, Kuan Liu, Leena Rizvi, Gordon Guyatt. (2019). Gender differences in grant and personnel award funding rates at the Canadian Institutes of Health Research based on research content area: A retrospective analysis.
  2. CIHR Act (2000).
  3. CIHR's Strategic Action Plan on Training (2015).
  4. CIHR 2018-2019 Departmental Plan.
  5. Canada Research Coordinating Committee.
  6. Canada's Science Vision (2018).
  7. Looker, D.E. (2016). 42nd Statistical Report [ PDF (2.4 MB) - external link ]. Canadian Association for Graduate Studies.
  8. Mitchell, J.S., Walker, V.E., Annan, R.B., Corkery, T.C., Goel, N., Harvey, L., Kent, D.G., Peters, J., Vilches, S.L. (2013). The 2013 Canadian Postdoc Survey: Painting a Picture of Canadian Postdoctoral Scholars [ PDF (1.29 MB) - external link ]. Canadian Association of Postdoctoral Scholars and Mitacs.
  9. Skinnider, M.A., Squair, J.W., & Twa, D.D. (2017). Characteristics and outcomes of Canadian MD/PhD program graduates: a cross-sectional survey [ PDF (491 KB) - external link ]. CMAJ Open, 5(2), E308-314.
  10. Statistics Canada (2017). Full-time teaching staff at Canadian universities, 2017/2018.
  11. Statistics Canada (2011). National Household Survey, Catalogue no. 99-012-X2011035.
  12. Tamburri, R. (2010). Give us the Dirt on Jobs. University Affairs/Affaires universitaires.
  13. The Fundamental Science Review Panel (2017).

Appendix A - Tables and Figures

Table 1: Overview of CIHR and Tri-Agency Training and Career Support Awards since 2000-01

Award level/Program Type Number of programs/FOs Source Program lifespan Number of awards Total CIHR and Tri-Agency Investments
Masters 29     3493 $61,684,181
CGS Master's 2 Tri-Agency 2003-Ongoing 3367 $58,539,476
CIHR Master's 27 CIHR 2003-Ongoing 126 $3,144,705
Doctoral 151     4473 $377,532,036
CGS Doctoral 2 Tri-Agency 2003-Ongoing 2173 $211,849,258
CIHR Doctoral 143 CIHR 2001-Ongoing 1718 $86,541,703
MD/PhD Program Studentship 4 CIHR 2001-2012 101 $8,877,380
Vanier 1 Tri-Agency 2008-Ongoing 447 $65,555,475
Other Award* 1 CIHR 2001-2015 34 $4,708,220
Postdoc 225     4002 $408,644,166
Banting Postdoctoral Fellowship 1 Tri-Agency 2010-Ongoing 140 $19,088,592
CIHR Fellowship 218 CIHR 2001-Ongoing 3807 $383,412,684
Prize 2 CIHR 2001-2013 43 $5,420,334
Other Award** 4 CIHR 2001-2016 12 $722,557
Early Career Support 73     1046 $279,994,013
New Investigator 73 CIHR 2001-2014 (IIR/Ongoing PDR) 1046 $279,994,013
Mid-Senior Career Support 60     1325 $397,016,912
Clinician Scientist I 3 CIHR 2001-2015 117 $17,163,384
Clinician Scientist II 11 CIHR 2001-Ongoing 127 $29,557,424
Exchange*** 6 CIHR 2001-2010 167 $979,169
Mid-Career Investigator 21 CIHR 2001-2015 134 $34,992,939
Senior Investigator 9 CIHR 2001-2008 39 $13,738,256
Other Award**** 10 CIHR 2014-2017 741 $300,585,741
Chair 23     1819 $1,579,460,316
CERC 1 Tri-Agency 2010-Ongoing 8 $71,014,008
Chair 20 Tri-Agency 2001-Ongoing 135 $75,712,539
CRC Tier 1 1 Tri-Agency 2001-Ongoing 729 $975,955,623
CRC Tier 2 1 Tri-Agency 2001-Ongoing 947 $456,778,147
Grand Total 561     16158 $3,104,331,624

Source: EIS Data, CIHR, as of 2017-12-21

Figure 1: CIHR and Tri-agency total Investments in training and career support (2000-01 to 2016-17)

Figure 1 long description
  CIHR's Programs CIHR's Portion of the Tri-Agency Programs
Masters $3,144,705 $58,539,476
Doctoral $100,127,303 $277,404,733
Postdoc $389,555,575 $19,088,592
Early Career Support $279,994,013  
Career Support $117,303,672 $279,713,240
Chair $75,712,539 $1,503,747,778

Source: EIS Data, CIHR, as of 2017-12-21

Note: All investigator initiated and priority driven awards are included.

Figure 2: Overview of CIHR's Training and Career Support Landscape and Awards in Scope of the Evaluation (2000-01 to 2016-17)

Figure 2 long description

Master's

  • CIHR Master's AwardFootnote *Footnote **
  • Canada Graduate Scholarship (CGS) - Master's
  • Strategic Training Initiative in Health Research (STIHR)

Doctoral

Postdoctoral

Early Career

Mid-Senior Career

Figure 3: CIHR Doctoral and Postdoctoral Award Recipients by Employment Status and Sector (2000-01 to 2015-16)

Figure 3 long description
  Doctoral (N = 122) Fellowship (N = 1069)
Employed in Academia 70% 82%
Employed Outside Academia 24% 16%
Unemployed 6% 2%

Source: Fellowship End of Award Report (2018) and Career Trajectory Survey (2017) data

Figure 4: CIHR Doctoral and Postdoctoral Award Recipients by Positions within Academic Sector (2000-01 to 2015-16)

Figure 4 long description
  Fellowship (N = 871) Doctoral (N = 85)
Assistant/Associate/Full Professor 66% 45%
Postdoc Fellow 16% 28%
Faculty (teaching, research) 8% 14%
Research Associate/Assistant 7% 4%
Other 3% 9%

Source: Fellowship End of Award Report (2018) and Career Trajectory Survey (2017) data

Figure 5: CIHR and Tri-agency Doctoral and Postdoctoral Award Investments (2000-01 to 2015-16), in Millions

Figure 5 long description

Doctoral

Year CIHR's Programs CIHR's Portion of the Tri-Agency Programs Total
2000-2001   13.8 13.8
2001-2002   12.3 12.3
2002-2003   7.3 7.3
2003-2004 13 8.3 21.3
2004-2005 7.1 10.7 17.8
2005-2006 10.4 9.1 19.5
2006-2007 18.5 6.2 24.7
2007-2008 19.1 6.1 25.2
2008-2009 43.1 2.9 46
2009-2010 27.5 3.4 30.9
2010-2011 25.4 4.2 29.6
2011-2012 25.5 4.3 29.8
2012-2013 24.3 6 30.3
2013-2014 21.3 1.4 22.7
2014-2015 21.7 2.1 23.8
2015-2016 20.7 2 22.7

Postdoctoral

Year CIHR's Programs CIHR's Portion of the Tri-Agency Programs Total
2000-2001   27.5 27.5
2001-2002   21 21
2002-2003   26.1 26.1
2003-2004   32.3 32.3
2004-2005   29.4 29.4
2005-2006   27.4 27.4
2006-2007   26.4 26.4
2007-2008   21.1 21.1
2008-2009   22.6 22.6
2009-2010   25.2 25.2
2010-2011 3.3 23 26.3
2011-2012 25.5 23.2 48.7
2012-2013 24.3 24.3 48.6
2013-2014 3.2 19.8 23
2014-2015 3.1 19.8 22.9
2015-2016 3.1 20.8 23.9

Source: EIS Data, CIHR, as of 2017-12-21

Note: All investigator initiated and priority driven awards are included.

Figure 6: CIHR Investments in Career Support Awards (2000-01 to 2015-16)

Figure 6 long description
  Dollars Invested
Years of Investment 2000-01 2001-02 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08 2008-09 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 Grand Total
Career Support $17,434,992 $17,569,445 $15,451,447 $6,539,090 $6,836,099 $6,776,955 $7,780,878 $3,900,886 $4,481,417 $5,116,587 $4,270,937 $3,814,468 $4,225,532 $2,579,443 $2,568,105 $7,957,392 $117,303,672
Chair $10,720,272 $2,190,000 $1,217,126 $1,864,378 $1,451,661 $2,078,290 $2,028,043 $18,919,262 $4,322,790 $487,500 $2,641,557 $575,000 $14,268,531 $175,000 $12,773,128   $75,712,539
Early Career Support $17,687,112 $21,041,672 $21,611,455 $18,897,737 $19,542,581 $23,940,372 $20,243,431 $21,271,097 $18,823,036 $14,044,674 $14,644,343 $14,540,197 $13,957,555 $14,195,000 $11,415,000 $14,138,750 $279,994,013

Source: EIS Data, CIHR, as of 2017-12-21

Figure 7: CIHR Investments in Career Support Awards (2000-01 to 2015-16)

Figure 7 long description
  Number of Awards
Award Levels Female Male
Masters 72 54
Doctoral 1098 731
Postdoctoral 1924 1914
Early Career Support 422 623
Career Support 360 940
Chair 46 88

Source: EIS Data, CIHR, as of 2017-12-21

Figure 8: CIHR Average Award Amounts by Funding Levels and Sex (2000-01 to 2015-16)

Figure 8 long description
Award Levels Gender Dollars Invested
Master's Female $22,573
Male $28,138
Doctoral Female $56,226
Male $51,332
Postdoctoral Female $101,927
Male $100,111
Early Career Support Female $265,500
Male $269,106
Career Support Female $95,546
Male $87,670
Chair Female $740,344
Male $459,660

Source: EIS Data, CIHR, as of 2017-12-21

Appendix B

Methodology

Consistent with TBS guidelines and recognized best practice in evaluation, a range of methods and sources were used to triangulate evaluation findings. These methods included: document and data review, an environmental scan of training and career support programs offered by other research funders in Canada and internationally, surveys with trainees and researchers, and a bibliometric study looking at the productivity of applicants and recipients of CIHR training and career support. In addition to this, the evaluation used an innovative method: the Funding Trajectory Analysis.

Data Collection and Analysis Activities

Document and Data Review

Relevant CIHR and Government of Canada documents (e.g., Fundamental Science Review 2017, Federal Budgets 2018-2019, and Canada's Science Vision, and the CIHR Act and Strategic Plan – Roadmap II) were consulted to provide context as well as to help address some evaluation questions related to relevance, design and delivery, and performance. These documents included past evaluations of CIHR (STIHR, Career and Salary Award) and Tri-agency programs (e.g., CGS, Vanier, Banting); as well as related literature, generated within CIHR (e.g., CIHR's Career Trajectory Survey of award recipients) and by outside organizations (e.g., survey of Canadian MD/PhD recipients).

Environmental Scan

An environmental scan of provincial, national, and international organizations that provide training and/or career support was undertaken to gather information about key features of design and delivery and identified needs and challenges of these programs relative to those offered by CIHR. An initial list of 63 organizations was proposed, with 40 ultimately included in the scan (organizations were excluded if they did not offer training or career support programs, or did not have sufficient information available on these programs). Features such as program objectives, funding mechanism (e.g., PDR vs. IIR), award amount, and award duration were recorded and compared, both against CIHR's programs, and nationally vs. internationally.

Surveys and End of Award Reports

Surveys were developed and administered for both recipients and applicants to two of CIHR's career support programs: New Investigator Award (n = 181 recipients, n = 202 applicants, N = 989) and Clinician Scientist Salary Award (n = 22 recipients, n = 15 applicants, N = 93). The surveys included questions related to applicants' experience with and perceived need for the program, outcomes and outputs, and research staff and trainees involved in the research program during the award tenure (or equivalent timeframe, for applicants). A parallel set of questions for recipients and applicants within each survey allowed for comparison between these cohorts. Unfortunately there was a very low response rate for the Clinician Scientist Award and therefore generalizability of results is limited and results should be interpreted with caution.

Survey data were also utilized from CIHR's 2017 Career Trajectory Survey (CTS), which targeted recipients of at least one direct, agency specific CIHR training award at the graduate level and beyond (i.e., CIHR Masters, Doctoral, or Fellowship) from the 2000-2013 competition years. This survey tracked award satisfaction and was primarily focused on outcomes of the awards, thus contributing to the findings on recipients' career trajectories. The sample size for recipients at the Master's level was too low to merit discussion of outcomes at this level (n = 8), but data from recipients at the Fellowship (n = 641) and Doctoral levels (n = 122) were analyzed and discussed where appropriate.

Available data were included from the FEAR, distributed to all 940 CIHR Fellowship recipients who had completed their fellowship between 2013 and 2018. The FEAR included questions about award experience and employment outcomes. Items from the FEAR were mapped to relevant evaluation indicators wherever possible. Fellowship recipients who completed the FEAR (n = 434) did not overlap with those in the CTS.

Funding Trajectory Analysis

The objective of the funding trajectory analysis was to better understand how CIHR funds the training and career development of Canadian health researchers. The analysis included a total of 63,234 applications and 29,472 applicants over 16 annual cohorts who submitted at least one application to the CIHR training and career support programs from fiscal years 2000-2001 to 2015-2016. The analysis tracked the complete funding and application history of each applicant from their first application to a program, until the end of the period covered. Included in the funding history for each applicant were all applications, successful or unsuccessful, to training and career support programs by award level (Master's, Doctoral, Postdoctoral, Career Support), and also whether applicants secured grants during the evaluation period.

The funding history of each applicant was determined through six steps corresponding to the six broad funding levels covered by training and career support programs (Master's, Doctoral, Postdoctoral, ECR, Mid-Senior Career Support, Chair). For each step, a code was assigned indicating whether the applicant 1) has not applied at this funding level, 2) has applied but was unsuccessful, or 3) has applied and was successful (including those who declined awards).

The funding trajectory analysis identified 146 different paths (i.e., combinations) of applications and funding status at one or more award levels. However, the majority of applicants (97%) are represented by only 35 different funding paths. Please note that a path can, and often does cover a single award level, and can include unsuccessful applications only.

Bibliometric Data

A bibliometric study was conducted for this evaluation by the Observatoire des sciences et des technologies (OST) of Université du Québec à Montréal. The study included the publication output of a sample of 3,000 applicants to Doctoral and Postdoctoral awards, drawn from the total population of 29,472 applicants from the funding trajectory analysis, who applied to CIHR and Tri-agency training and career support programs for the period 2000-01 to 2015-16. Using publication data for these individuals for the period from 1998-2016, this study addressed the evaluation question related to the scientific productivity and scientific impact of training and career support program applicants. Productivity and scientific impact were measured using the following indicators: number of papers published, number of inter-institutional and international collaborations, acknowledgement of CIHR funding, average of relative citations (ARC), and average relative impact factor (ARIF) of journals.

Limitations

The following limitations and mitigation strategies have been identified for this evaluation:

Limitations Mitigation Strategies Impact of Mitigation Strategies
  • Indirect funding for training and career support was not covered by this evaluation.
  • Only direct training and career support awards are covered in the present evaluation. Although a considerable number of trainees and researchers are funded through indirect sources, the process of obtaining the contact information for indirectly supported trainees (which is not collected currently) is very resource intensive and was considered to be outside the scope of the present evaluation.
  • Thus, the total value and breadth of training and career support provided through CIHR is currently underestimated. However, conclusions related to design and delivery based on direct programming may be applicable to indirect support mechanisms as well. Future research is needed to assess the outcomes and impacts of indirect career, involving institutions themselves and mining of the Form 300 data in addition to existing data from NPIs. CIHR needs to consider building measures related to indirect support into existing performance measurement tools to enable its inclusion in future evaluations.
  • There has been a wide variety in training and career support award programming at CIHR and the approaches and mechanisms for funding have evolved over time.
  • There are 561 training and career support programs that have been launched since 2000/01, of which many have less than 10 awards and some have only one award. The awards offered have varied over time, and most have limited end of award reporting.
  • To mitigate this challenge, these programs were rolled up in the analysis and reported according to award support level (e.g., Postdoctoral). Data collection was focused on certain larger programs where end of award reports were missing (e.g., New Investigator and Clinician Scientist Awards); however, it was not possible to implement them for all priority driven programs given the number of them.
  • Broadly the objectives of the programs/FO's were consistent, therefore conclusions are presented across programs by level. However, it is possible that these conclusions do not generalize to all programs and FO's.
  • Contribution vs. attribution
  • Trainees and researchers may receive multiple and sometimes simultaneous awards and grants, from a variety of sources, making it impossible to attribute recipients' outputs and outcomes exclusively to CIHR programs. This can include additional direct and indirect funding, as well as support and experiences within their graduate programs or careers. Therefore, reported outcomes such as career satisfaction and advancement may also be influenced by indirect support or graduate/career experiences.
  • Thus, conclusions from this evaluation speak to CIHR's contribution to trainee and researcher outcomes and impacts.
  • Lack of definitions for training and career support, IIR, and PDR research
  • Operational definitions were developed for the evaluation using all relevant and available information and through consultations with program staff.
  • Where appropriate, programs were categorized using CIHR's definitions of training and career support (for the environmental scan and the trajectory analysis).
  • Few organizations in the Environmental Scan provided definitions of training and career support, or definitions of IIR versus PDR funding as they relate to their training and career support programs. Few identified their programs overtly as providing training and career support.
  • Classifying programs under a single set of definitions for training and career support increased consistency in reporting of programs. However, in the absence of definitions or overt categorization of a program as training or career support, our labeling may not have fully reflected other organizations' definitions and approach to training and career support and it is possible that these programs may have been categorized differently in the past.
  • Limited data for the Environmental scan/ Inconsistency across organizations
  • The environmental scan was completed using publicly available information from organizations websites and data availability was limited in some cases.
  • Some design indicators were reported with caution or omitted due to limited or incomplete data. Additionally, some organizations reported information differently (e.g., quantifying the number of awards) although efforts to standardize and verify information were undertaken and a conservative approach to interpretation was adopted. Categorization may not have fully reflected other organizations' definitions and approach to training and career support.
  • Use of secondary data
  • Multiple sources of secondary data were included, that used different methods of data collection and analysis. This included end of award reports for the CIHR Fellowship, survey data from CIHR Doctoral and Fellowship awardees for the Career Trajectory Survey, the MD/PhD survey conducted by Skinnider and colleagues (2017), and previous evaluations of Tri-agency programs.
  • Wherever possible, the original data from secondary sources was analyzed. If it was not possible to analyze the original data, then attempts were made to verify the findings from the source of these data.
  • The best attempts were made to verify the accuracy of these data sources and ensure applicability for the current evaluation. Multiple sources of data were triangulated so as not to solely rely on secondary data wherever possible.
Date modified: