Canadian Institutes of Health Research Salary/Career Award Programs - Evaluation Report

CIHR Evaluation Unit 2012

Table of Contents

• Key Findings, Conclusions and Recommendations
• 1. CIHR Salary and Career Awards Program Profile
  • 1.1. Program Context
  • 1.2. Policy and Program Objectives
  • 1.3. Program Description
• 2. Evaluation Scope and Design
  • 2.1. Evaluation Objectives and Scope
  • 2.2. Methodology
  • 2.3. Limitations
• 3. Achievement of expected outcomes
  • 3.1. Protected time to conduct research
  • 3.2. Researcher productivity
  • 3.3. Research Outcomes
  • 3.4. Training and supervision
  • 3.5. Obtaining other sources of funding
  • 3.6. Retention and sustaining a career in health research
  • 3.7. Career trajectory
  • 3.8. Collaboration and participation in committees/networks
  • 3.9. Opportunities for improvement in program design/delivery
• 4. Efficiency and Economy
  • 4.1. Satisfaction with program delivery
  • 4.2. Consistency between program objectives and criteria used by peer reviewers
  • 4.3. Partnerships
• 5. Addressing the Needs of Canadian Health Researchers
  • 5.1. Health researchers' need for salary support and the extent to which CIHR meets this need
  • 5.2. Universities and research centres' needs for salary support and the extent to which CIHR addressed them
  • 5.3. Availability of CIHR salary awards
• 6. Alignment with Government Priorities, Roles and Responsibilities
• 7. Roles and Responsibilities in Delivering Salary Support
  • 7.1. Coordination role for CIHR
  • 7.2. Environmental scan of salary support programs
• Bibliography
• Appendix 1: Health researchers needs for salary/career awards by CIHR theme and career stage
• Appendix 2: Protected time to conduct research by CIHR theme and career stage
• Appendix 3: CIHR salary/career awards holders research productivity outputs by CIHR theme and career stage
• Appendix 4: CIHR salary/career awards holders research productivity for open and strategic funding
• Appendix 5: Students and research staff involved in the research program by CIHR theme and career stage
• Appendix 6: Extent to which the CIHR salary/career award covers researchers' salaries for the period of the award

Key Findings, Conclusions and Recommendations

Key findings

1. Performance

1.1 Protected time to conduct research

• Funded CIHR salary support awardees do not spend a significantly greater proportion of their time conducting research than they did before they held their award; 77% of their time is spent on research activities compared with 71% before receiving the award.
• This finding is, however, in-line with the amount of time required to be protected for research under the terms of the salary award (75%).
• Non-funded salary award applicants who did not receive a CIHR award spend a similar amount of time on research activities as those who are funded (74% vs. 77%).
• There is currently no definition in the program documentation for salary awardees that states which activities should be included in their protected time for conducting research. There is also no requirement for award holders to report on the time they spend on such activities after the award has been made.

1.2 Researcher productivity

• Salary support awardees publish a greater number of refereed journal articles over the period of their award than non-funded salary award applicants; 21.3 articles compared with 15.5 for non-funded researchers. Salary awardees funded by open and strategic competitions publish similar numbers of journal articles (21.6 and 20.9 each respectively).
• An acknowledged limitation of this evaluation is that bibliometric analysis was not conducted to assess the scientific impact of publications. It is anticipated that this methodology will form part of future evaluations of these programs.
• Statistical analysis was conducted to assess the extent to which the number of papers published by researchers can be linked to other factors. The number of trainees involved in a research program was found to have a limited but statistically significant influence. Other factors tested (dollar value of grants held; dollar value of awards held; percentage of time spent on research) did not emerge as significant influences.
• The annual production of refereed journal articles by salary awardees in-line with that for Canada Research Chair holders (5.9 annually per researcher for each group).
• In terms of other research outputs, CIHR salary awardees produce a greater average number of books/book chapters (2.1 vs. 1.4); reports/technical reports (1.4 vs. 0.8); and Master's thesis/doctoral dissertations (2.3 vs. 1.8) compared with non-funded salary award applicants.

1.3 Research outcomes

• Research conducted by CIHR salary awardees has resulted in a wide range of outcomes from knowledge creation through new methods, theories or practices to the development of spin-off companies.
• A greater proportion of funded salary awardees report that their research resulted in such outcomes compared with non-funded salary award applicants. There are few significant differences between the outcomes reported by salary awardees funded through open competitions and those receiving strategic award funding.
• These findings can be compared to the research outcomes resulting from operational grants; the proportion of CIHR Open Operating Grant Program funded researchers claiming each type of outcome is similar to that for salary awardees over a five year period.

1.4 Training and supervision

• CIHR salary awardees are more likely to have involved a greater number of students and research staff in their research programs than non-funded CIHR salary award applicants.
• As is noted below, salary awardees tend to leverage larger amounts of funding; it is likely that the findings on training and supervision relate to this factor.

1.5 Obtaining other sources of funding

• Funded CIHR salary awardees are no more likely than non-funded salary award applicants to subsequently receive CIHR Open Operating Grant funding (59% vs. 58%). Salary awardees do however receive a greater number of CIHR operational grants than non-funded applicants (2.4 vs. 1.9) and a larger average amount of funding ($1.02m vs. $671K).
• Both funded and non-funded applicants are highly likely to receive funding from organizations other than CIHR; 90% of salary award holders and 86% of non-funded applicants received operating funds from provincial, not-for-profit and other funders.
• One in two (55%) non-funded salary support applicants subsequently received salary support funding from organizations other than CIHR.

1.6 Retention and sustaining a career in health research

• A large majority of funded and non-funded salary award researchers had remained at the same institution as when they applied for a salary award (86% funded; 83% non-funded). Fewer than one in twenty of either group had left Canada to conduct research or left the field altogether.
• University Vice Presidents see salary awards as just one factor among many in retaining researchers. Access to research infrastructure and the prestige of an institution are identified as other key factors.

1.7 Career trajectory

• Salary support award holders appear to have a more accelerated career trajectory than non-funded applicants, particularly in the early career stages.
• While similar proportions of each group were assistant professors when they applied for an award (63% of funded; 68% of non-funded), a significantly greater proportion of funded applicants subsequently become associate professors (56% vs. 38%).
• Key informant interviews with University Vice Presidents support the finding that CIHR salary support award holders are likely to reach tenured positions quicker and have accelerated career progression.

1.8 Opportunities for improvement in program design/delivery

• Many researchers state that their award does not cover their salary. It can, however, be noted that the awards are intended to be a contribution to salary rather than to cover the full cost.
• The clinician scientist awards represent a particular challenge in terms of the amount of their award, and this is reflected by the large proportion (65%) who claim that the award covers their salary only to some extent/to a small extent. As doctors and dentists, this group of researchers face a greater opportunity cost for accepting an award, in view of the salaries they can attract as clinicians.
• To address these issues, key informant stakeholders suggested: reducing the proportion of dedicated research time required by clinician scientists as the nature of their work requires regular practice to maintain clinical skills; and increasing the amount of the award. A high success rate of applications vs. awards (76%) reflects a need to increase the attractiveness of awards to this group.

1.9 Efficiency and economy

• There are high levels of satisfaction with program delivery among a majority of funded and non-funded researchers.
• Observations of the peer review process show that all of the main criteria for selecting applications and most of the sub-criteria (71%) were discussed for each applicant.
• There has been an ongoing decline in partnership contributions to the awards between 1999 and 2010; this also reflects a decrease in CIHR corporate funding for the awards during this period. Some partners feel that CIHR does not do enough to publically recognize their contributions; they do, however, express overall satisfaction with their relationship with CIHR.

2. Relevance

2.1 Extent to which the CIHR salary programs address the needs of Canadian health researchers

• Researchers identify that the most important factor for them in applying for a salary award is to develop/maintain a research program. Other important factors are: ability to obtain other research funds; recognition and improving profile; and career advancement.
• Open and Strategic New Investigator awards attract significant levels of applications and have a success rate of approximately 20%; these programs represent the majority of applications (93%).

2.2 Alignment with government priorities and the role and responsibilities of delivering salary support to health researchers in Canada

• The awards are aligned with CIHR's strategic plan priorities (attracting and retaining the best early career researchers). The awards also align with the People Advantage of the Government's Science and Technology Strategy in developing, attracting and retaining highly skilled people.
• An environmental scan of salary/career support programs in Canada shows that several funders are cancelling or suspending their programs. Three of the fourteen provincial and not-for-profit organizations in the scan do not have active competitions planned for 2011-12.

Conclusions

Delivery and program outcomes

• The salary/career awards programs can be viewed as achieving their expected outcomes in a range of respects.
• CIHR salary awardees produce a greater number of peer reviewed publications and other research outputs than non-funded CIHR salary award applicants. They advance more quickly in their careers and are generally more successful in leveraging further grants and awards from CIHR and other funders.
• With regard to program delivery, there are high levels of applicant satisfaction with the application and peer review processes. The salary awards are generally highly competitive and the peer review process of selection is viewed as an assurance of quality by University Vice Presidents and other stakeholders.
• There are, however, areas where the salary/career award programs are not operating as intended. Protecting time for research is an integral part of the program theory, yet holding an award does not substantively increase the amount of time a researcher spends on research activities. Furthermore, this requirement is currently neither clearly defined nor reported on after the award is received. This brings into question whether the protection of time is an appropriate goal for the salary/career programs.

Program design

• This evaluation takes place in the context of CIHR's ongoing Roadmap implementation of reforming and redesigning the agency's programs. One design consideration is whether operating grants could potentially be used to replace some or all of the salary/career awards.
• These evaluation findings show that researchers receiving CIHR salary/career awards achieve similar research outcomes to those who obtain CIHR operating grants. The ability to develop and maintain a research program is identified as the most important priority for those applying to salary awards, as would also be expected with an operating grant. The prestige that University VPs associate with receiving a CIHR salary/career award can equally apply to success in a similarly competitive operating grant competition.
• The current design of the salary/career awards programs stipulates that new investigators must also hold an operating grant to be eligible to receive an award. These researchers are therefore required to apply to more than one program. There could be potential efficiencies in the application and peer review processes if an operating grant program targeted at new investigators replaced this existing model.
• Among those salary award applicants who are not funded by CIHR, the vast majority remain in Canada, continue in health research and obtain other salary and operational funds. The organization scan of salary award providers shows that while three funders are no longer providing these, other organizations continue to provide these awards.

Recommendations

1. In the context of CIHR's program redesign, consider whether operational efficiencies and equivalent or greater research impacts could be achieved by replacing the new investigator open salary awards with operating grants targeted specifically at this group.
2. If salary/career award programs are retained at CIHR, the current design issues with protected time should be addressed.
   • One option would be to better specify how protected time for researchers can be measured, monitored and reported to demonstrate compliance with the terms of the award.
   • A second option would be to remove the guidelines on the target amount of time to be spent on research as a condition of holding a salary award. Other approaches that could be more effective at achieving the same objective should be considered, for example, specifying a maximum number of teaching hours, clinical hours or administration.

Management Response

Management agrees with the findings and recommendations of this evaluation. CIHR is committed to investing in world class health research and more specifically to train, retain and sustain a healthy research foundation. New investigators play an important role in creating a sustainable foundation for Canadian health research. CIHR's New Investigator program accounted for 58% of the total financial commitments for salary programs from 2000 to 2010.

This evaluation was timely given the current reforms of CIHR's suite of open programs. The challenge will be to ensure that new investigators continue to be given a fair and equal opportunity for CIHR support. Management intends on using the results of this evaluation to influence these reforms. To address the recommendations noted in the report, the Research Portfolio Management Team along with the Health Research Roadmap Implementation Team will implement the following management actions:

1. CIHR Salary and Career Awards Program Profile

1.1. Program Context

The Canadian Institutes of Health Research (CIHR) contributes to Canada's health research capacity by providing a range of salary/career awards that target researchers at different stages of their careers. As is outlined in CIHR's strategic plan, the Health Research Roadmap, a key goal for the agency is to train, retain and sustain outstanding health researchers (CIHR, 2010). This includes a focus on attracting and retaining the best early career researchers, or new investigators, as well as other targeted groups such as clinician scientists.

The CIHR salary support programs operate within a wider Canadian funding landscape of similar awards. This includes large flagship federal funding programs such as the Canada Research Chairs Program^{Footnote 1}, as well as a wide range of provincial and not-for-profit salary awards.

It is important to consider CIHR's salary/career award programs within their wider context. The organizational scan presented as part of this evaluation shows that several funders have in fact discontinued their salary/career awards. Evaluation findings also show that salary award funding is rarely a zero-sum game; researchers who are unsuccessful in their applications to CIHR are frequently provided with salary support by other provincial or not-for-profit organizations.

1.2. Policy and Program Objectives

Salary/career awards programs are intended to provide investigators who hold research appointments at Canadian universities and institutions with the opportunity to develop and demonstrate their independence in initiating and conducting health research through the provision of a contribution to their salary.

Salary/career awards are offered through both open and strategic competitions. In an open competition, applicants apply to conduct investigator-driven health research in an area of their choice. In strategic competitions, CIHR offers salary/career support to investigators whose research areas are aligned with priorities identified by CIHR and its thirteen institutes.

Program theory is a model of how an intervention, such as a program, contributes to a chain of results and finally to intended or observed outcomes (Funnell & Rogers, 2011). It can be noted that the existing literature on the program theory of salary/career awards is extremely sparse. While some limited literature is available in relation to capacity building in health research more generally, even here there is a need for greater empirical evidence. As Bates et al (2006) note: "Capacity building is a poorly defined and understood concept" (p1).

The key features of the program theory for salary/career awards are shown in Table 1. Many of these are premised on the researcher having 'protected time' to conduct research (instead of teaching or other activities). This protected time is intended to bring about positive outcomes in terms of research production and training, ultimately leading to a strong research community and the retention of excellent researchers in Canada. A logic model for the salary/career awards, along with a detailed description of its components, can be found in Appendix 2 of this report.

1.3. Program Description

As of 2010, the following CIHR salary/career award programs were active:

• New Investigators – Open and Strategic;
• Clinician Scientists Phase 2 – Open;
• Clinical Investigators – Strategic;
• Investigators – Strategic;
• Mid-Career Awards – Strategic;
• Research Chairs – Strategic.

As of October 2010, CIHR's salary/career award programs commitments have accounted for $354.4 million since 2000.

As shown in Table 2 below, New Investigator awards are by far the largest of the salary/career awards programs. The New Investigators program represents the largest total investment, accounting for approximately 58% of the total salary programs financial commitments for the period 2000-2010. The Investigators program (12% of financial commitments), CIHR Research Chairs, (9% of financial commitments), and Clinician Scientists Phase II (4% of financial commitments) represent the majority of the remaining available funds.

Table 2 also shows that many of these programs have been discontinued over the last ten years, including all of the senior investigator awards. A detailed description of each of the salary/career award programs can be found in the appendices to this report.

The relative importance of the salary/career awards within CIHR's overall expenditures has decreased over time. As shown in Figure 1 below, in 2000-2001, salary awards represented 8.6% of CIHR's grants and awards expenditures; in 2009-2010 these programs accounted for only 3.7% of total grants and awards spending. This decrease can be attributed to the elimination of the Senior Investigator and Open Investigator awards in June 2003, and other agency-wide budgetary adjustments.

In keeping with the decline in budget allocation, the number of awards has also been decreasing. The number of new paid awards in 2009/10 is less than half of the 2000/01 total, decreasing from 181 awards in 2000/01 to 82 in 2009/10.

Figure 1: Percentage of annual salary awards expenditures as a proportion of the total CIHR grants and awards budget^{Footnote 2} and number of newly paid awards

Figure 1 long description

Source: CIHR Electronic Information System (EIS) database

2. Evaluation Scope and Design

2.1. Evaluation Objectives and Scope

The evaluation objectives were designed in consultation with CIHR management and in alignment with the TB Directive on the Evaluation Function^{Footnote 3}. The following objectives were identified as a priority for this evaluation:

Performance

1. To assess the extent to which the CIHR Salary Programs achieved their expected outcomes and identify opportunities for improvement to program design/delivery;
2. To determine the degree of efficiency and economy associated with the CIHR Salary Programs.

Relevance

3. To determine the extent to which the CIHR Salary Programs continue to address the needs of Canadian health researchers;
4. To determine the degree of alignment of the CIHR Salary Programs with Canadian government priorities;
5. To determine the role and responsibilities of CIHR in delivering salary support to health researchers in Canada.

A complete list of evaluation issues, questions and their corresponding indicators can be found in the appendices to this report.

2.2. Methodology

Consistent with TBS guidance and recognized best practice in evaluation, a range of methods were employed to triangulate the evaluation findings. To ensure that the evaluation findings and conclusions are robust and credible, the evaluation used multiple methodologies and utilized both quantitative and qualitative evidence.

The following lines of evidence and data sources were used:

1. CIHR Salary Support Program database analysis and document review – a review of documents and administrative data was conducted to ensure a thorough understanding of the historical development of the program.
2. Organizational Scan- a scan of Canadian health research funders that offer salary support, to assess the degree of duplication or overlap of CIHR Salary/Career Award Programs.
3. Literature review- a review of the literature to explore salary award program theory and alternative program design and delivery.
4. Field observation of the Peer Review process- field observation to assess the degree of consistency between program objectives and the review criteria used by peer reviewers to assess applications.
5. Quantitative survey – online surveys administered between October 28^th and November 30^th, 2010 ( a period of approximately 5 weeks):
   • Survey of funded salary awardees (N=1007): All researchers who received a CIHR Salary Award between 2000 and 2009 and who kept their award for a period longer than one year were invited to participate in the survey; valid response rate of 53%.
   • Survey of non-funded CIHR salary award applicants (N=777): Researchers who applied to the CIHR salary support programs, who were fundable (rated over 3.5 in the peer review process) but who did not receive an award due to budgetary constraints; valid response rate of 39%
6. Qualitative interviews with a range of stakeholders – interviews were conducted with the following stakeholder groups:
   • CIHR senior management, Institute and program staff (N=9) – key informants were selected according to roles, responsibilities and familiarity with CIHR salary/career awards;
   • Vice Presidents of Research in Canadian universities and research centres (N=20);
   • Representatives of other Canadian health research funding agencies and NGOs (N=6) – sample based on the results of the organizational scan;
   • Partners (N=4) – partners chosen based on their financial contributions to salary programs.

Further methodological details, the survey questionnaires and interview guides used for data collection and information on the data analyses that were conducted can be found in the evidence binder for this evaluation that accompanies this report.

2.3. Limitations

In keeping with best practices in program evaluation, the limitations of this study are noted below, together with the strategies that were employed to mitigate these.

• Limited program baseline data: This evaluation relied heavily on data collected at a single point in time using mainly self-reported data through interviews and surveys. In order to mitigate this limitation, triangulation of data was ensured where possible using other lines of evidence.
• Population issues: Some CIHR salary funded researchers have received more than one salary award. In these cases, their most recent grant was selected to be a focus of the survey. However, the number of previous awards and the amounts received were considered in the data analysis. Additionally, the responses of non-funded salary award applicants may be negatively biased against the CIHR salary/career awards programs. However, researchers funded by CIHR may be equally biased in favor of the CIHR salary/career awards programs.
• Sampling issues: In general, the salary awardee and non-funded salary award applicant respondent groups are comparable; they have a similar distribution among awards category, starting year, and institution size. However, self-selection bias (i.e., non-response bias) is an inherent limitation of the online survey and would likely have occurred in the two surveys undertaken for this evaluation.
• Data quality issues: Some funded and nonfunded researchers applied for or received a salary award in early 2000. Their capacity to answer the questions regarding short- and mid-term outcomes of their research or regarding the application and decision process at that time is limited. For the cases in which the information relied on respondents' recall of situations dating back ten years, potential inaccuracies may have occurred and, for some questions, respondents indicated that the numbers or answers they provided were approximate. Conversely, researchers who applied for or received a salary award in 2008 have not completed the award term.
• Methodological limitations: As with any program evaluation, there are a range of methodological limitations to the quantitative and qualitative methods used. The response rates for the quantitative survey of funded and non-funded researchers were relatively high (53% and 39% each respectively); these samples have margins of error of 3.1% (funded) and 4.8% (non-funded) at the 95% confidence level. Full details on the survey methodology and sampling can be found in the evidence binder for this evaluation. Where qualitative methods were used (key informant interviews), it should be noted that qualitative findings are not generalizable to the full population of each group that was sampled.
• Cost effectiveness analysis: A recognized limitation of this evaluation is that a full cost-effectiveness study was not included in the methodology to assess metrics such as the cost per award delivered. Other aspects of efficiency and economy are included, in accordance with Treasury Board core issues; these provide evidence on program delivery, the peer review process and leveraging of partnership funding. It is anticipated that future CIHR evaluations will include cost-effectiveness studies as part of the methodology.

3. Achievement of expected outcomes

This section details the extent to which the Salary Support Programs have achieved their expected outcomes. The following areas are assessed:

• Protected time to conduct research;
• Researcher productivity;
• Research outcomes;
• Training and supervision;
• Obtaining other sources of funding;
• Retention and sustaining a career in health research;
• Career trajectory;
• Collaboration and participation in committees/networks;
• Opportunities for improvement in program design/delivery.

3.1. Protected time to conduct research

A key objective of the CIHR Salary Awards is to ensure that funded researchers spend 75% of their time (no less than 30 hours per week) conducting research.

A first finding is that while the salary award program documentation states that 75% of researcher time should set aside for research, there is no definition in the program documentation as to exactly what falls within this category. This makes it difficult to accurately assess the extent to which this condition is actually being met. There is also no requirement for award holders to report on the amount of time they spend on research either during the period they hold the award or at the end of this.

To mitigate the issue with definitions of research, the findings in Table 3 below show both a 'broad' and a 'narrow' interpretation of what constitutes research activities. Consultations with stakeholders resulted in two definitions: a broad definition which includes the top three categories in Table 3 (research activities; administrative tasks associated with research; training and supervision of students/research staff) and a narrow interpretation which includes only the first category (research activities).

Regardless of which definition is used, holding an award does not appear to have a substantive impact on how salary funded researchers distribute their time between different types of activities (see Table 3:

• With the broad definition, researchers report spending 77% of their time on research activities compared with 71% before they held their award, and somewhat less time on teaching duties and clinical activities;
• With the narrow definition, researchers spend 47% of their time on research activities compared with 44% before they held the award.

If the broad definition of research is used, funded CIHR salary award holders spend a similar proportion of their time on these activities as non-funded applicants; 77% compared with 74%. A greater difference is observed if the narrow definition is used: funded researchers spend 47% of their time on research compared with 35% for the non-funded group.

The similarities between these groups are perhaps unsurprising if the wider context of Canadian research funding is considered. As is detailed in this report, many researchers who did not receive CIHR salary/career award funding still obtain this form of funding from other organizations. It should also be remembered that salary awards are only one source of support for researchers; many also concurrently hold operating grants to conduct their research.

3.2. Researcher productivity

A key indicator used to measure outcomes of the CIHR salary awards is the research productivity of funded researchers. The hypothesis is that researchers who spend more time conducting research rather than on other activities such as teaching duties will produce more publications (e.g. articles, books or book chapters) and engage in more activities that promote their research such as conferences or presentations.

3.2.1. Refereed journal articles

CIHR salary awardees publish a greater number of refereed journal articles than those in the non-funded salary award applicant comparator group:

• The average number of refereed journal articles per funded researcher for the entire award duration (five years) was 21.3;
• The average number of refereed journal articles per researcher for the same period was 15.5 for applicants who were not funded.

In terms of other benchmarks, salary funded researchers' productivity in publishing refereed journal articles is similar to that of Canada Research Chair (CRC) holders: an annual average of 5.9 publications per researcher for both programs^{Footnote 4}. This comparison should of course be put in context; the CRCP is one of the most prestigious salary awards available to Canadian researchers, particularly for those who are awarded Tier 1 chairs.

A comparative analysis of the number of referred journal articles published by CIHR salary award holders funded by an open competition compared with those receiving a strategic salary award showed no significant differences between the groups:

• The average number of refereed journal articles per researcher funded with a CIHR salary award through an open competition for the entire award duration (five years) was 21.6;
• The average number of refereed journal articles per researcher funded with a CIHR salary award through a strategic competition for the entire award duration (five years) was 20.9.

Statistical analyses of these data were conducted to assess the extent to which the number of papers that researchers publish can be linked to other factors.

First, correlation analysis was used to test associations between the number of publications and the following variables: percentage of time spent on research; dollar amount of awards held; dollar amount of grants held and number of trainees involved in the program. These variables were found to be positively correlated with the number of publications produced.

Multiple regression analysis was then used to test the influence of these variables on the number of publications produced by researchers. Results showed that only the number of trainees had a statistically significant influence; this influence was very limited, explaining only eight percent of the variability in the number publications produced.

An acknowledged limitation of all of these analyses of publication outputs is that while the findings allow us to assess the productivity of researchers, we are unable to assess the scientific impact of their publications, for example through a citation analysis. Bibliometric analyses are frequently used to assess the contributions scholars make in their research publications to advancing valid scholarly knowledge (e.g. Moed, 2005). It is recommended that future evaluations of the salary support programs include this methodology as a line of evidence.

3.2.2. Other research outputs

CIHR funds diverse health research communities in the biomedical, clinical, population health and health systems and services fields and it is important to acknowledge that peer reviewed journal articles are not always the best measure of output for all groups. Some social science researchers may, for example, focus to a greater extent on publishing books or book chapters than on peer reviewed journal articles.

As illustrated in Figure 2 below, CIHR salary awardees produce a greater average number of books/book chapters (2.1 vs. 1.4), reports/technical reports (1.4 vs. 0.8) and Master's thesis/doctoral dissertations (2.3 vs. 1.8) compared with non-funded applicants.

Figure 2: Average research outputs of funded and non-funded salary award applicants for the duration of a salary/career award

Figure 2 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.3. Research Outcomes

Funded health research can result in a wide range of outcomes, from knowledge creation through new methods, theories or practices to the development of spin-off companies. From 2011 onwards, CIHR's new end of grant/award Research Reporting System (RRS) will be capturing these outcomes for its funded researchers. The salary support questionnaire administered to both funded and non-funded researchers for this evaluation used these same measures from the RRS to assess the research outcomes of each of these groups.

As is shown in Table 4 below, a greater proportion of CIHR Salary Award recipients reported that their research resulted in these outcomes when compared with those who were not funded:

• Research findings/knowledge creation, replication and adaptation of research findings, and new or changed policy and program were all claimed by significantly more funded than non-funded salary award applicants;
• The exception to this is the greater proportion of non-funded salary award applicants who reported that their research resulted in intellectual property claims.

CIHR's Open Operating Grants Program (OOGP) provides a useful benchmark for contextualizing how the outcomes of salary support funded and non-funded salary award applicants compare to other sources of funding. Data from the Research Reporting System for the OOGP shows that many of the research outcomes are in-line with those reported by Salary Support awardees.

There are many caveats here when comparing grants to awards, although in most cases the duration of OOGP grants is the same as that for salary awards (five years). It should also be noted that salary support holders frequently hold a range of other funding in additional to their award, including operating grants from CIHR and other funders.

The difference in the proportions of salary award holders and OOGP grant holdings claiming an outcome relating to a new or changed policy/program may be attributable to differences in the demographic profile of the health researcher in each group. A greater proportion of salary award holders are health systems/services and population health researchers when compared with those in the OOGP grant sample.

An analysis of research outcomes reported by CIHR open salary award holders compared with those who received a strategic award showed few significant differences between these groups (see table in appendices for details).

Table 5 shows the differences in reported outcomes when analyzed by CIHR's four research themes. As would be expected given their research focus, some themes are far more likely to report certain outcomes than others.

3.4. Training and supervision

It is expected that salary award holders involve other junior researchers in their research project in order to pass on their knowledge and skills; this increase in research staff is assumed to increase their productivity. During stakeholder interviews, VPs Research confirmed the expectation that the funded researchers should attract and retain trainees for their research programs.

As shown in Table 5 below, there are no statistically significant differences between the number of students trained by CIHR salary awardees and non-funded salary award applicants.

When assessing the number of research assistants, trainees and postdoctoral fellows and master's students involved in the research program, CIHR salary awardees tend to have greater involvement compared with non-funded salary award applicants (see Table 6).

3.5. Obtaining other sources of funding

A key indicator used to assess program outcomes for new investigators in particular is their subsequent capacity for leveraging and obtaining other sources of research funding. The program theory is that holding a CIHR salary award will both allow the researcher to develop a research program and also add to their prestige, enabling them to be successful in other similar competitions.

3.5.1. Operating grants and awards from CIHR

CIHR's Open Operating Grants Program provides a significant amount of funding to Canadian health researchers for investigator-driven open research (around $500m annually). It is also competitive, attracting applicants with higher than average scientific impact scores for their previous publications and with an annual success rate of around 20% (approximately one in five applicants are funded). The agency also offers a range of other operating grants to health researchers including knowledge translation, training, team and catalyst grants.

Figure 3: Percentage of salary award funded and non-funded researchers obtaining CIHR operational funding

Figure 3 long description

Source: CIHR Electronic Information System (EIS) database

Figure 3 shows the proportion of funded and non-funded salary award applicants who receive CIHR operating grants via the OOGP or other types of grant. The chart shows the triangulation of two quasi-experimental design methods; a pre-post award comparison and a counterfactual comparison between the funded and non-funded groups. As can be concluded from Figure 3:

• Both groups of researchers are more likely to obtain operational funds over time (prior to applying vs. after applying/receiving award). This is to be expected, given that many of the researchers in both groups are new investigators who are progressing in their careers;
• The non-funded salary award applicants are initially significantly less likely to obtain CIHR operational funding, either OOGP or other operational grants. However, over time, this group 'catch-up' with their funded counterparts in their ability to obtain CIHR operating funds;
• This finding points to a conclusion that the highly competitive salary awards are selecting researchers who are also successful in obtaining operating funding;
• It could also be hypothesized that holding a CIHR salary award does not have a significant impact on a researcher's subsequent ability to obtain operating grants, given that both the funded group and the non-funded group become similarly likely to hold these.

Figure 3 does not tell the whole story however; while salary award funded researchers are no more likely to receive CIHR operational funds than their non-funded counterparts, they do receive a greater number of grants and a larger total amount of funding:

• Salary awardees received an average of 2.4 CIHR operational grants after obtaining a salary award compared with 1.9 for non-funded salary award applicants;
• Salary awardees receive an average of $1.02m of CIHR operational grant funding after obtaining the salary award compared with $671K for non-funded salary award applicants.

It cannot be determined with certainty whether the greater number of grants and higher average funding received by funded salary award holders can be attributed to the impact of the program. This could for example reflect a selection bias of the program intake; the peer review committees select the most able applicants who then go on to receive more grants and operating dollars.

3.5.2. Operating grants and awards from other organizations

Outside of CIHR, the majority of CIHR salary awardees (90%) and non-funded salary award applicants (86%) received operating grants from other organizations.

As shown in Figure 4 below, aside from CIHR itself, provincial organizations are the most frequent providers of other operating funding, representing 25% of the total number of grants received from other organizations by both funded and non-funded salary award applicants. Foundations are also a key player in this landscape; 21% of salary awardees and 19% of non-funded salary award applicants received research grants from this source.

Figure 4: Research grants received by funded and non-funded researchers from organizations other than CIHR

Figure 4 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.5.3. Salary awards received from organizations other than CIHR

As noted earlier, CIHR provides salary/career award funding as part of a wider landscape of other organizations offering these awards. This has an important implication when considering the future of the salary award programs; if CIHR were no longer offering some salary awards, what might be the implication for Canadian health researchers?

A greater proportion of non-funded salary award applicants (55%) received salary funding from organizations other than CIHR, compared with CIHR salary awardees (40%). This is perhaps to be expected given that those not funded by CIHR awards have greater financial drivers to seek this financial support elsewhere. It should also be noted that researchers holding CIHR Salary Awards cannot apply to other sources of federal funding for salary awards.

Figure 5 shows that aside from CIHR, provincial organizations are the main source of salary award funding in Canada, representing 62% of the other awards received by the funded researchers and 68% of the other awards received by the non-funded salary award applicants (see Figure 5).

Figure 5: Salary awards received by funded and non-funded researchers from organizations other than CIHR

Figure 5 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

The average total salary award amount (per researcher) received from provincial organizations by non-funded researchers is higher ($320,539) than the amount received by CIHR funded researchers from provincial organizations ($226,328).

It can be concluded from this data that many non-funded CIHR salary applicants are able to find salary support funding from other organizations and that they receive a significant amount of such funding. As is later detailed in the organization scan however, several funders are no longer offering salary support awards. Retrospective data on past funding may not therefore be indicative of the salary awards that might be available to Canadian health researchers in future.

3.6. Retention and sustaining a career in health research

One of the goals of the CIHR salary career awards is to contribute to attracting and retaining researchers in the health research field and in Canada. As is noted in CIHR's Health Research Roadmap, this has particular relevance to attracting and retaining the best early career researchers, including the many new investigators funded with salary awards.

3.6.1. Retention of health researchers

Table 7 shows that a large majority of researchers remained at the same institution or university they were at when they applied for a CIHR salary award. There is no evidence to suggest that either the funded or non-funded researchers are leaving Canada in large numbers; fewer than one in twenty funded or non-funded researchers were pursuing research outside Canada.

Only 25% of CIHR salary award funded researchers declared that it is likely (to a moderate or great extent) that they would have left Canada if not for the CIHR salary awards. Thirty-five percent of funded researchers stated that having received a CIHR salary awards influenced (to a moderate or great extent) their decision to stay in Canada after their last award's term.

In terms of the perspective of University VPs and other key informants on the role of the CIHR salary awards in the retention of researchers:

• Salary support is viewed as just one factor among several others in a decision to leave or remain in Canada to conduct research. Researchers also consider the access to research infrastructure and the prestige of the institution in their decision;
• The very best researchers are most at risk of leaving Canada in search of more prestigious universities and research centres that offer better research infrastructure, regardless of any support toward salaries. Some key informants indicate that the National Institutes of Health (NIH) in the United States are the most important source of funding competing with CIHR salary and other research supports.
• CIHR salary support is not perceived to have a significant impact on repatriating Canadian researchers or attracting researchers from other countries;
• Salary support factors more heavily at the early stage of a researcher's career, when there are fewer opportunities. When new investigators establish a research program in Canada, they become less likely, over time, to leave what they have started and head to another country.

Some key informants gave examples of provincial salary programs which seem to increase retention in those provinces. For example, FRSQ new investigators can benefit from an income tax exemption for five years once they get an appointment. This may allow researchers to establish a research program and they become less likely to move abroad to conduct their research in future. Also, FRSQ award recipients cannot undertake a portion of their research in another country.

3.6.2. Sustaining a career in health research

The majority of the CIHR salary awardees (70%) declared that receiving the salary awards influenced their decision to continue their career in health research (to a moderate or great extent). Key informants who were interviewed also stressed the importance of salary support in allowing researchers to sustain a career in health research and build up a research program.

However, set against these findings, it can be noted that only 1% of non-funded salary award applicants were no longer pursuing a career in health research when surveyed. The conclusion can be drawn here that while the salary awards may influence a decision, they are not an important factor in this regard. Only a tiny minority of non-funded CIHR salary award applicants abandon their health research careers.

Forty-five percent of non-funded CIHR salary award applicants did not receive any salary award during this period, either from CIHR or other funders. It is therefore likely that operational funding had an impact on sustaining the careers of this group, given that 86% of non-funded salary award applicants did receive operational grants after being unsuccessful in a salary award competition.

3.7. Career trajectory

A significant element of the program theory of salary awards for new investigators in particular is that receiving salary support at an early stage of a research career can have benefits in the longer term. Indeed, key informants articulated the view that the awards should increase the likelihood of reaching tenure at a university or sustaining employment and a research program until such a position became available.

As one key informant summarized:

"They have a greater ability to progress through the ranks faster. They get promoted faster and they get tenure faster. They get more grants and more capacity to do more research faster."

This qualitative assessment can be compared with career trajectory data on both the funded and non-funded applicant groups. Findings show a significant difference in the early stage career progression between salary awardees and non-funded salary award applicants (see Figure 6):

• Salary awardees who were assistant professors at the time of their application for an award (63%) moved in significantly higher proportions to more senior positions as associate professors (56%) or professors (16%);
• A similar proportion of non-funded salary award applicants were assistant professors (68%) when they applied for an award; however fewer subsequently became associate professors (38%).

Only 28% of the funded salary awardees remained assistant professors, compared with 48% of the non-funded researchers. No statistically significant differences were found for the other categories. It should be noted that most of the researchers in an assistant professor position at the time they applied for funding had applied for or were holding a New Investigator award among both the funded (97%) and non-funded researchers (91%).

Figure 6: Career progression of CIHR salary awarded funded and non-funded researchers

Figure 6 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.8. Collaboration and participation in committees/networks

Holding a CIHR salary award appears to have a positive effect on the level of collaboration with other researchers for a large proportion of the salary awardees. As shown in Figure 7:

• Many salary awardees indicated that they collaborated "more often" or "far more often" with other researchers in Canada (65%), with other researchers within the same institution (56%) and with national knowledge users/outside of the academic sector (42%);
• In terms of collaborations with partners outside Canada, many salary awardees indicated that they collaborated "more often" and "far more often" with other researchers internationally (63%) and with international knowledge users/outside of the academic sector (33%).

Confounding factors must be acknowledged here; it is possible that differences in career stage between applying for the award and completing the survey, or length of time in research has a significant impact on collaboration.

Figure 7: Salary awardee collaboration/participation in networks/committees

Figure 7 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

3.9. Opportunities for improvement in program design/delivery

From the perspective of researchers and University VPs research, the value of the award (typically $60K annually) emerged as a significant issue. As is shown in the findings below, the salary support provided does not generally cover the costs of a researcher salary. However, it should be remembered that the award is intended to be a contribution to the salary cost rather than to cover the total amount. This issue is more significant for clinician scientists than for other groups given their higher salaries and the financial opportunity cost of conducting research.

Calls for an increase in the size of awards by those receiving them should also be put in context. As was highlighted in the interviews with senior management at CIHR, increasing the amount of award funding for this program would either result in a corresponding decrease in the number of awards provided or reducing the budget available for other grants and awards, given finite resources at the agency. From a management perspective, a key design question is whether delivering salary award funding, either through open or strategic competitions, is the most effective mechanism of achieving the agency's overall objectives.

3.9.1. Amount of the awards

Figure 8 shows researcher opinions on the extent to which the CIHR salary /career award covered their salaries for the period of the award:

• Overall, one in three researchers (32%) felt that the award covered their salary to a great extent; 29% who stated that their salary was covered to some extent/to a small extent.
• By contrast, a majority of clinician scientists (65%) claim that their salary was only covered to some extent/to a small extent.

Figure 8: Extent to which CIHR salary awards cover researchers' salaries for the period of the award

Figure 8 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

Many key informant stakeholders expressed concern regarding the current amount offered by the CIHR salary awards in that:

• The award provided by CIHR does not cover current salary costs;
• The relative value of the awards combined with the applicant success ratio for New Investigators in particular may influence some researchers not to apply.

It can be noted, however, that despite these perceptions, application pressure has remained high for many of the salary awards. There is also no evidence to suggest that the awards are not attracting high calibre applicants.

In terms of the clinician scientist awards, the survey findings are supported by the perceptions of the key informants interviewed:

• A salary award of $60,000 represents a small portion of a clinician scientist's total salary, although this does vary greatly depending on their area of research;
• The disparity between salary support funding and clinician salaries is advanced as a reason why some clinicians are deterred from applying for salary support awards.

By contrast with the findings for other salary awards, the low application pressure and high success rate for clinician scientists compared with new investigators gives support to this conclusion.

Key informants suggested lowering the proportion of dedicated research time required by clinician scientists and increasing the amount of the awards for clinicians to increase the attractiveness of CIHR salary awards for this group of researchers.

Another significant reason given for relatively low application pressure for clinician scientist awards is the importance placed upon maintaining clinical skills. Key informants suggest that if the value of awards were increased, this would increase the incentive to apply for many clinician researchers.

Key informants were divided as to what should be done if additional funds were available for salary awards. Some key informants argued that funding should be directed at more generous awards to bring them in-line with the costs of research. Other key informants suggested that additional funds should increase the number of awards granted.

It was recognized by CIHR senior management in their interviews that CIHR salary/career award amounts currently fall below research costs, but that this is a reflection of a wider flat funding environment across the board. It was noted that this type of funding must be delivered in the context of partnerships between institutions and CIHR; institutions must also provide financial commitments to provide sustainable salary funding.

3.9.2. Open or strategic funding

Interviews with CIHR senior management highlighted a general view that strategic salary support funding would be most likely to make a significant contribution to capacity development. This was particularly the case for funding clinician scientists, where several of those interviewed felt that a shortage of these researchers made it essential to continue to fund this group.

It was also recognized that clinician scientists have specific needs based on the strong alternatives they have to conducting a research career, and therefore an imperative to provide adequate stipends to support this group. Senior managers felt that consideration should be given to expanding the type of clinicians eligible to apply for the awards.

The importance of supporting new investigators and their entry into research was mentioned by many senior managers. A range of options were discussed in this regard, including whether providing operating funding programs specifically for new investigators would achieve similar objectives to the salary support programs. Some concerns were raised that if this option were taken, it would be important to ensure protected time for research. This was raised as a particular consideration for the social sciences, where the teaching load was felt to be higher.

It can however be noted that the survey findings show salary awardees to be spending a similar amount of time of research before and during holding an award. Moving towards providing operating funds for the new investigator group may not therefore have significant implications in this regard.

3.9.3. Duration of the awards

Opinion was divided among VPs research and other key informants around the optimal duration for the awards:

• Some key informants believe the current five-year duration of the award is adequate;
• Others argue for a longer duration to enable universities and research centres to retain researchers, and provide more stability for researchers;
• Some key informants argue that the current five-year duration is simply too short and acts as a disincentive to pursue a research career in the health field. According to this view, researchers spend too much time on grant applications and renewals without any guarantee of sustained funding.

3.9.4. Holding awards concurrently with operational funds

As previously stated, some key informants point out that it is ineffective to support researchers if they are not otherwise supported in their research through operating funds. Researchers require research infrastructure, equipment, and trainees, as well as salary support. CIHR salary /career awards could therefore be combined with operational funding and research infrastructure funding to ensure full coverage.

Recent CIHR salary/career award program alterations initiated at CIHR in 2011 have responded to this need; salary award holders are now required to hold concurrent operational funding in order to qualify for a salary award.

3.9.5. Quotas for awards

Some key informants suggest that CIHR could award salary support quotas to various institutions who would administer the funding. This was viewed as decreasing the administrative burden for CIHR. This is also the approach taken for the Canada Research Chairs Program (CRCP).

However, other key informants stressed that the savings reclaimed through this strategy would be minimal. Furthermore, VPs Research, partners and CIHR management generally agreed that implementing a quota system could compromise the reputation of the peer review committees and have a negative impact on the prestige associated with the awards.

3.9.6. Peer Review Committees

Some key informants suggested that the peer review process could be updated to promote greater cost-effectiveness. CIHR management agrees that the peer review process could be streamlined in a variety of ways. Several key informants suggested a range of alternatives such as: utilizing online forums instead of face-to-face meetings and the consolidation of a variety of peer review committees into one (CFI, SSHRC, NSERC, CIHR).

4. Efficiency and Economy

As with all CIHR-administered grants or awards, a key element of efficient and economical program delivery is the management of the competition process. This includes the process for researchers to apply for an award, the work of the peer review committees in assessing the applications and making a decision, and post-award management.

4.1. Satisfaction with program delivery

Figure 9 on the following page shows high levels of satisfaction among applicants with the application and decision-making processes. These results can be benchmarked against a recent survey of all CIHR researchers conducted for the agency's 10^th Year International Review.

The findings are comparable; for example, in this survey, 19% of respondents note dissatisfaction with the clarity of application instructions and 21% of researchers and applicants report dissatisfaction on the timeliness of posting results.

While satisfaction levels with program delivery are high among applicants, a range of areas were identified where improvements could be made. It can be noted that many of these relate to only a minority of the salary award applicants surveyed:

• The electronic process should be simplified - this can be cumbersome to complete due to issues with the CV module and the required forms and attachments. This can lead to researchers unintentionally omitting pieces of information (Mentions - Funded: 11%, Non-funded: 10%)
• Greater clarity is needed in regards to: instructions for applying, eligibility guidelines, the role of the institution, the difference between a research project versus a program of research, rules for foreign-trained applicants and how applicants are assessed (Funded: 8%, Non-funded: 14%);
• More transparency is needed in the decision process — in particular, reviewers should provide more constructive and meaningful feedback that details the basis for the decisions made and how evaluation criteria were interpreted and weighted so that applicants can strengthen their next application (Funded: 11%, Non-funded: 14%);
• Greater timeliness is needed when releasing the decision results with adherence to the original stated timelines to receive these (Funded: 8%, Non-funded:14%);
• Members of the peer review committee should be selected more carefully for eligibility and to avoid conflicts of interest and ensure that they understand applicants' field of expertise; additional, external expertise might be appropriate (Funded: 7%, Non-funded: 5%).

Figure 9: Satisfaction with the application and decision-making process (funded and non-funded researchers)

Figure 9 long description

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

Key informants felt that CIHR's peer review process is highly respected by the research community, and, as such, selection by a peer review committee is typically a mark of prestige and quality for researchers and research institutions. CIHR's peer review process was identified throughout the interviews with university VPs research and other stakeholders as reinforcing and improving the reputation of salary awardees.

When partner funding organizations were invited to share their opinion on the application and decision process, they reported that delays in competition processes can cause significant challenges for industry to carry on with research projects that depend on timely funding.

Additionally, some university VPs Research perceive that the competitions for salary awards sometimes favour larger universities with greater research capacity and are less favourable to health research in the humanities and social sciences. Some key informants suggested a quota system to mitigate a perceived bias; however other key informants recognized that although CIHR's salary awards are very competitive, this adds to the prestige of the awards.

4.2. Consistency between program objectives and criteria used by peer reviewers

A field observation was conducted to assess the degree of consistency between the salary support program objectives and the review criteria used by peer reviewers to assess applications. A high degree of consistency signifies that the program is being delivered as intended and that the applications selected reflect the program objectives.

Each application is reviewed according to three main criteria:

• Applicant track record;
• Research proposal; and
• Institutional support.

There are also specific sub-criteria within each of these main criteria that are intended to guide the peer review committees' discussions. Taken together these criteria and sub-criteria reflect the objectives of the salary/career awards.

In February 2010, a sample of four peer review sessions were observed by two evaluators for the following peer review committees:

• Allied Health Professionals (AHP);
• New Investigators "A" (NIA);
• Health Research salary "A"(HAS); and,
• Health Professional Awards (HPA).

In all the four sample peer review committee sessions that were observed, all of the main criteria and at least 71% of the sub-criteria were discussed for each applicant. From this, it can be concluded that the peer review committees assessed applications in accordance with program objectives.

The observation form used by the evaluators to assess the consistency between the peer review committees' discussions and the peer review criteria can be found in the evidence binder.

4.3. Partnerships

CIHR partners with a variety of organizations to deliver salary awards; provincial and national health funding agencies, charitable organizations, industry, hospitals and universities and research institutes. Since 1999, CIHR has leveraged a total of $68,581,141 in partner financial contributions ranging in size from $7,102 to $6,645,767.

Figure 10 demonstrates a decline in partnership contributions to salary support awards from 1999 to 2010 and in particular from 2002-2003 onwards. It is noted that partnership contributions fell in accordance with the decrease in CIHR corporate funding dedicated towards Salary Awards over this time period.

Figure 10: Partner financial commitments for CIHR salary awards (open and strategic) (1999-2010)

Figure 10 long description

Source: Electronic Information Systems (EIS) Partnerships database

A few of the key informants interviewed suggested that CIHR could do more to develop partnerships and leverage funding in this program space. According to some partner organizations, it may become progressively difficult for CIHR to partner with industry on salary awards due to a paradigm shift in the priorities of these organizations. Some felt that industry would be increasingly focusing its financial support on strategic areas of research rather than on researchers.

Some partner organizations also expressed concern in regards to CIHR's recognition of partner contributions. According to these key informants, CIHR does not always do enough to publically recognize the contribution of their partners. This said, most partners who were interviewed reported overall satisfaction with their relationship with CIHR.

Interviews with CIHR management included suggestions that the agency should continue to examine how sustainable partnerships are facilitated, including introducing more effective communications strategies.

5. Addressing the Needs of Canadian Health Researchers

5.1. Health researchers' need for salary support and the extent to which CIHR meets this need

As shown in Figure 11 four key factors were identified by the stakeholders involved in this evaluation (researchers, university VPs, partner organizations and CIHR management) as being important to researchers applying for CIHR Salary Awards. Each of these elements is described in further detail below.

Figure 11: Health researcher needs for salary/career awards

1. Ability to develop or maintain a research program

Surveyed researchers identified that the most important factor for them in applying for salary support was the ability to develop or maintain a research program. Other stakeholders also acknowledge that salary support is necessary in order to allow researchers to dedicate the majority of their time (a minimum of 75%) to their research while reducing their time spent on other activities (e.g. teaching). Several interviewees from CIHR's Senior Management stressed that this 'protected' research time is particularly important for New Investigators, who are often burdened with activities, such as teaching, that can detract from their research time.

2. Ability to obtain other research funds

Both funded and non-funded researchers perceive the ability to obtain other research funds as one of the most important factors in salary support funding. Interviews with stakeholders and CIHR Senior Management highlighted that researchers require solid research infrastructure and operational funding in order to be successful, and that salary support is less effective if researchers do not have sufficient access to operational funds. Thus, in order for salary award funding to be effective in advancing research careers, it must be coupled with operational funding.

3. Recognition - improving profile and reputation as a researcher

Researchers require 'recognition' in order to increase their research profile and obtain funding from a variety of sources. CIHR's peer review process was identified throughout the stakeholder interviews as reinforcing and improving researchers' reputations; as the peer review committees are well respected in the research community, being selected as a salary award recipient greatly enhances the perceived quality of the candidate and their research.

4. Career advancement - improving prospects for promotion

CIHR Salary Awards were described by some survey respondents and interviewees (other stakeholders and CIHR senior management) as being essential to the career progression of researchers. Many key informants pointed out that some universities that will not hire health researchers if they do not qualify for a salary support award. Other universities support researchers for some time—two to five years—on a contractual basis, to allow them to build up their curriculum, after which, if they do not have a salary award of some kind, the institution will stop supporting them.

The differing perspectives of universities and research funders should be considered here. A number of research institutions claimed that they would not have sufficient financial resources to continue supporting some of their researchers without salary awards. However, in other stakeholder interviews, the role of health research funders in providing salary support of this type was questioned. Some interviewees felt that research institutions should be in a position to support their researchers without relying on salary awards as an ongoing funding stream that is sometimes built into the budgets and financial planning of an institution.

Figure 12 presents the importance given by salary awardees to each of these factors and the extent to which CIHR salary/career awards met the needs represented by these factors.

Figure 12: Health researcher needs for salary/career awards

Figure 12 long description

Note: The mean corresponds to the average value of the responses.
Importance: 1=Not important at all, 2=Not important, 3=Somewhat important, 4=Important, and 5=Very important
Needs met: 1=Not at all, 2=To a small extent, 3=To some extent, 4=To a moderate extent and 5=To a great extent
There is a strong correlation between the two ratings.

Source: Survey of CIHR funded salary/career award researchers and non-funded applicants

As would be expected, the specific needs of different groups of salary award holders vary somewhat according to their career stage and whether open or strategic funding is received.

5.1.1. New Investigators

Key informants stress that although helpful at all career stages, salary support is somewhat more critical at the beginning of a researcher's career. While senior researchers typically secure funding more easily than New Investigators based on their reputation and experience, New Investigators depend on funding from programs such as the CIHR Salary Awards to "break into the system."

Supporting this, surveyed New Investigators indicated that CIHR salary awards met their needs in term of their "career advancement and improvement of their prospects for promotion" to a greater extent than Mid- Career Investigators.

5.1.2. Clinician Scientists

As noted earlier in this report, the Clinician Scientist Phase II program, offered through CIHR's suite of salary awards, is of particular interest to stakeholders as it is intended to address the unique needs of clinician scientists. Throughout the key informant interviews, stakeholders described the needs of clinician scientists as differing from those of other health researchers in two predominant ways:

1. The protected time that they receive as a result of obtaining an award, and;
2. The amount of the salary award.

The nature of clinician scientists' work requires regular practice in order for them to maintain their clinical skills, making it especially challenging for them to devote 75% of their time to research and only 25% to clinical duties. Also, clinical environments such as hospitals can put significant pressure on these scientists to spend as many hours as possible on clinical duties - particularly those attached to teaching hospitals.

Second, the salary of clinicians is generally far higher than that of other researchers'. This diminishes the relative impact of salary support awards; the awards may cover the bulk of other researchers' salaries, but represent a small portion of the salary of clinician-researchers. It also diminishes the incentive to apply for such awards.

"You have to buy that time and the award does not cover it. You certainly don't get 75% of a clinician scientist's time [with] a CIHR new investigator award."

5.2. Universities and research centres' needs for salary support and the extent to which CIHR addressed them

Key informants in universities and research centres identified a range of needs for salary support. At an overall level, they acknowledge there is 'never enough' support for research and researchers; universities and research centres are always looking to expand their research programs and infrastructure in order to remain competitive. Nevertheless, they feel that CIHR salary awards address most of the needs of universities and research centres, albeit to a varying extent.

5.2.1. Support for operations and infrastructure

While universities and research centres are able to put a portion of their public funding toward supporting their research operations and infrastructure, they must typically solicit additional funds to put towards these ends. As a general rule, a well-supported research capacity and research infrastructure enables institutions to remain competitive, to attract prominent researchers, and to leverage additional funds, among other benefits.

• Some institutions can more easily attract researchers with salary awards or those who show a high potential of receiving a salary award, whereas other institutions struggle in this area. Specifically, some key informants from smaller universities and from universities without medical schools expressed that they feel disadvantaged in the pursuit of salary awards;
• Some stakeholders noted however that not all universities have the capacity and research facilities to support high quality research; research institutions must themselves ensure that they offer a supportive research environment if they want to attract researchers with salary awards;
• Some University VPs commented that CIHR salary awards also help institutions gain support for operations and infrastructure in that the protected research time allocated to funded research likely increases their productivity, which may have a positive impact on the reputation and prestige of the university;
• This benefit may, in turn, help the university attract funding from other sources, such as private donations, to further expand their research capacity and research infrastructure.

5.2.2. Attract, hire, and retain researchers

Universities and research centres compete at the national and international level to attract and retain researchers. To this end, CIHR salary awards help to reduce institutions' financial burden, allowing them to reallocate or reinvest some of their resources toward improving infrastructure and increasing research budgets and capacity, which contributes to the attraction and retention of new and prominent researchers.

Key informants pointed out that some universities only hire or offer tenured positions to researchers who receive external salary support. Consequently, the more salary awards offered, the more researchers hired by universities. In other cases, universities hire researchers without external salary support on the condition that they apply for a salary award during their employment.

5.3. Availability of CIHR salary awards

An analysis of CIHR administrative data for the period 2000/2001-2008/2009 for the application process of the currently offered salary awards suggests that there is a continued demand for salary support among health researchers in Canada.

Both Open and Strategic New Investigator awards attract significant levels of applications and have a success rate of approximately 20%.^{Footnote 5} It is important to note that these two programs also comprise the bulk of the salary support applications, representing 93% of total applications. The success rates for the other five programs range between 45% for CIHR Research Chairs and 76% for Clinical Scientists Phase II.

Figure 13: Application pressure for CIHR salary/career award programs

Figure 13 long description

Source: CIHR Electronic Information System (EIS) database

The availability of CIHR salary awards was also discussed though interviews with key stakeholders:

• There is general agreement that exclusivity heightens the prestige associated with CIHR salary awards;
• However, stakeholders generally favour a higher success ratio via an increase in the number of awards;
• Furthermore, while key stakeholders recognize that additional funding for CIHR salary awards is unlikely, many would favor funding a larger number of awards rather than offering larger sums to successful award holders, in the event that additional funding became available.

Some key informants felt that although the responsibility to support social science health research had been transferred from the Social Sciences and Humanities Research Council (SSHRC) to CIHR, CIHR had not made the necessary adjustments to reflect this change in its suite of salary awards.

Some interviewees from CIHR's Senior Management responded to this perception by emphasizing the importance of ensuring that each of CIHR's themes are represented on salary award Peer Review Committees, and suggested tailoring the eligibility criteria for salary awards to be more inclusive of social science researchers.

6. Alignment with Government Priorities, Roles and Responsibilities

CIHR Strategic Plan – Health Research Roadmap

Key informants argue that it is the responsibility of the federal government to support research to increase Canada's competitiveness, enhance the productivity of Canadians, and ultimately increase Canadians' standard of living. The Government's Science & Technology Strategy sets out a comprehensive, multi-year science and technology agenda.

CIHR's salary/career award programs support the following sections of this strategy:

Knowledge Advantage

Canada must build upon our research and engineering strengths, generate new ideas and innovations, and achieve excellence by global standards.

People Advantage

Canada must grow its base of knowledge workers by developing, attracting, and retaining the highly skilled people we need to thrive in the modern global economy.

CIHR's strategic plan, the Health Research Roadmap, sets out four strategic directions to achieve CIHR's long-term goals. These strategic directions enable CIHR to carry out its full mandate in all its complexity, show leadership within the wider health research community, and demonstrate accountability and results to Canadians.

By attracting and retaining the best early career researchers through the New Investigator program, salary awards align directly with the first strategic direction of the Roadmap, Invest in World-class Research.

7. Roles and Responsibilities in Delivering Salary Support

CIHR is widely considered by stakeholders to be the main source of salary support for health researchers in Canada. However, key informants identified a range of other sources of salary support which are available to health researchers in Canada. This includes delivery of salary support programs to support health research by provincial and not-for-profit agencies.

7.1. Coordination role for CIHR

Key informants state that there is an opportunity for CIHR to do more in terms of coordinating, positioning, and aligning its activities with other organizations that support health research in Canada. Interviews with CIHR management supported conducting an environmental scan to ensure that the salary support landscape is fully understood. There may also be subsequent opportunities to discuss complementarity between programs at suitable forums attended by agencies providing salary support programs.

Most key informants believe that the existence of other programs is not a sign of duplication or overlap with CIHR salary programs but is often complementary:

• Researchers cannot simultaneously hold a CIHR salary award and another federally funded award with a salary support component; they must decline one or the other;
• CIHR salary support programs do not reach all eligible health researchers; there are not sufficient funds to do so and application pressure shows that demand outstrips supply;
• Salary support programs provided by the various federal, provincial and non-profit agencies have different parameters which distinguish them from one another (e.g. targeted domain of health, monetary value, prestige associated, targeted career stage).

7.2. Environmental scan of salary support programs

Table 8 on the next page gives details of 14 provincial and not-for-profit organizations offering salary support programs in Canada.

To summarize the overall characteristics of these organizations:

• Six are provincial (AB, QC, ON, BC) and eight operate across Canada;
• All organizations offer 'junior awards'; five organizations offer mid-career awards and four organizations offer these for 'senior' researchers;
• The term of the awards varies between one and seven years; and
• Annual stipends paid to researchers range between $20,000 & $160,000.

It should be noted that three of these fourteen organizations do not have active competitions planned for 2011-12; two of these organizations have cancelled or suspended salary award programming. Interviews with key informants support this data, in that these suggested a trend away from salary awards and towards other approaches to funding.

Bibliography

Bates, I., Akoto, A.Y., Ansong, D., Karikari, P., Bedu-Addo, G., Critchley, J., Agbenyega, T. and A. Nsiah-Asare (2006).Evaluating Health Research Capacity Building: An Evidence-Based Tool. PLoS Med 3 (8), e299. DOI: 10.1371/journal.pmed.0030299

Bryar R.M. (2003). Focus. Practitioner research: an approach to developing research capacity in primary care. Nursing Times Research 8 (2): 101–114.

CIHR Mission – Retrieved on October 2009 from:
http://www.cihr-irsc.gc.ca/e/7263.html

CIHR Career Awards: Consultation Report – April 2006 - Retrieved October 2009 from:
http://www.cihr-irsc.gc.ca/e/31458.html

Research Net – New Investigator Salary Award Request for Applications (RFA): 2008-2009 - Retrieved October 2009 from:
http://www.researchnetrecherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=519&view=seedlist&org=CIHR
&type=AND&resultCount=25&sort=program&all=1&masterList=true&language=E

CIHR - Understand Peer Review – Retrieved October 2009 from:
http://www.cihr-irsc.gc.ca/e/37790.html.

2008-2009 CIHR Grants and Awards Guide – Retrieved October 2009

Charles F. Reynolds III, M.D., Paul A. Pilkonis, Ph.D. , David J. Kupfer, M.D., Leslie Dunn, M.S., Harold A. Pincus, M.D., (2007). Training Future Generations of Mental Health Researchers: Devising Strategies for Tough Times. Academic Psychiatry. 31(2): 152-159.

Funnell, S. C. & Rogers, P. J. (2011). Purposeful Program Theory. San Francisco, CA: John Wiley and Sons.

Moed, H. F. (2005). Citation Analysis in Research Evaluation. Dordrecht, Netherlands: Springer.

Segrott, J., M. McIvor and B. Green (2006). Challenges and strategies in developing nursing research capacity: A review of the literature. International Journal of Nursing Studies, 43(5): 637-651.

Task Force on Career Support: Final Report – Retrieved October 2009 from:
http://www.cihr-irsc.gc.ca/e/25943.html

Treasury Board of Canada – Policy on Evaluation - Retrieved October 2009 from:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024&section=text#cha1

Treasury Board of Canada –Directive on Evaluation Function – Retrieved October 2009 from:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681

Appendix 1: Health researchers needs for salary/career awards by CIHR theme and career stage

Importance given to each of the factors by funded researchers and the extent to which the CIHR salary/career awards met the needs represented by these factors:

Biomedical theme (N=264)

Biomedical theme (N=264) long description

Clinical theme (N=98)

Clinical theme (N=98) long description

Health system and services theme (N=55)

Health system and services theme (N=55) long description

Social, cultural and population health theme (N=77)

Social, cultural and population health theme (N=77) long description

New Investigators (N=426)

New Investigators (N=426) long description

Investigators (N=84)

Investigators (N=84) long description

Clinician Scientists (N=20)

Clinician Scientists (N=20) long description

Appendix 2: Protected time to conduct research by CIHR theme and career stage

Time spent by CIHR funded researchers on different tasks:

Appendix 3: CIHR salary/career awards holders research productivity outputs by CIHR theme and career stage

Appendix 4: CIHR salary/career awards holders research productivity for open and strategic funding

Appendix 5: Students and research staff involved in the research program by CIHR theme and career stage

Appendix 6: Extent to which the CIHR salary/career award covers researchers' salaries for the period of the award

No differences were found across CIHR themes.

New Investigators (N=426)

New Investigators (N=426) long description

Investigators (N=84)

Investigators (N=84) long description

Date modified:

2015-06-16