Evaluation of CIHR's Knowledge Translation Funding Program

Acknowledgements

This report was authored by Robert McLean, Lead Evaluator and Joanne Tucker, Junior Evaluator.

The evaluation was carried out by an Evaluation Working Group which included: Robert McLean (Chair), Ian Graham, Martha MacLeod, Jacqueline Tetroe, Joanne Tucker, Christopher Manuel, Adrian Mota, Yumna Choudhry, Stephanie Coen, Ryan McCarthy, Andrea Smith, and Kwadwo Bosompra.

Table of Contents

• Executive Summary
• Program and Evaluation Description
  • KT at CIHR, and the CIHR KT Funding Program
  • Evaluation purpose
  • Evaluation approach
  • Evaluation questions
• Key Findings
  1. Achievement of expected outcomes
  2. Program design and delivery
  3. The role of CIHR in KT
• Methodology
• References
  • Appendix A – Organizational Scan Participants
  • Appendix B – Survey Statistics
  • Appendix C – Funding Opportunity Descriptions
  • Appendix D – Evaluation Matrix
  • Appendix E – Case Study Readings
  • Appendix F – Glossary
  • Appendix G – Evaluation Working Group
  • Appendix H – Evaluation Recommendations Crosswalk

Executive Summary

Background

The following report presents key findings and recommendations resulting from an evaluation of the Canadian Institutes of Health Research (CIHR) Knowledge Translation (KT) Funding Program. Commercialization programming, although an important aspect of CIHR's KT strategy, is not a part of this study as it is being examined in a separate ongoing CIHR evaluation. For this evaluation, the KT Funding Program comprises six standalone CIHR funding opportunities.

• Partnerships for Health Systems Improvement (PHSI);
• Knowledge to Action (K2A);
• Knowledge Synthesis (Synthesis);
• Dissemination Events (DE);
• The KT Supplement (KTS); and,
• KT science – whose applications are reviewed by the Knowledge Translation Research (KTR) panel of the Open Operating Grant Program (OOGP).

When the evaluation was launched in early 2011, its objectives were twofold. Firstly, the evaluation was designed to assess the performance and impact of the KT Funding Program and identify areas for improvement. This was the first undertaking of such a study of KT at CIHR. Secondly, it was designed to fulfill CIHR's responsibility to the Treasury Board of Canada under the 2009 Policy on Evaluation. In the interim, CIHR initiated a reform of its open grants competition and its peer review processes. As a part of this reform, the decision was made by CIHR to integrate the KT Funding Program into new open schemes of funding. This evaluation – although not designed to inform the integration of the KT funding opportunities into the open grants competition – provides information that can be used to support evidence-based decision-making during this reform process.

This evaluation employed multiple lines of inquiry including: literature, document, and electronic information system (EIS) reviews; an international environmental scan; quantitative surveys; in-depth qualitative interviews; and case studies.

Key Findings

Achievement of outcomes

• All CIHR KT funding opportunities studied in the evaluation have performed well against CIHR's existing measures of success. KT funding opportunities have produced KT outputs (e.g., websites, decision aids), academic outputs (e.g., journal articles, scholarly books) and have engaged highly qualified personnel (HQP) (e.g., students, post-doctoral fellows). Although the OOGP and iKT funding opportunities have different objectives, are designed differently, and may support different types of research, the chart below provides iKT results and the OOGP reference point to allow interpretation of these figures.

  	HQP (students, post-docs, etc.)		Academic (journal articles, books, etc.)		KT (websites, decision aids, etc.)
  	per grant	per 100K investment	per grant	per 100K investment	per grant	per 100K investment
  iKT	6	6	4	4	17	16
  OOGP	9	4	10	4	16	6

• The evaluation identified the existence of a meaningful partnership between researchers and knowledge users (KUs) as a catalyst for increasing both the relevance of research and the use of research. Both researchers and KUs report that CIHR's iKT funding opportunities supported such partnerships. Evaluation data demonstrates:
  • the involvement of partners in research happens more often in iKT structured grants;
  • these grants are more likely to influence the behaviour of KU partners, and;
  • these grants lead to the creation of real-world applications.
• Data indicate that KT funding opportunities contribute to the fulfillment of the CIHR mandate in a way that is complementary to "investigator-driven" research funded through the OOGP. KT funded researchers report that they contribute more often to improving the health of Canadians, strengthening the health care system, and the creation of health services and/or products, whereas OOGP funded researchers report contributing more often to the creation of new health knowledge.
• A substantial amount of post grant activity is undertaken by CIHR KT funded researcher-KU partnerships. Activities such as further research, public engagement, policy development, and commercial ventures were cited as examples that continued long past grant expiration.
• Researchers from all four pillars of health research have used end of grant KT funding opportunities (DE and KTS). The evaluation identified end of grant KT funding opportunities as a timely (through an expedient application review process, accessible three times a year) and flexible (due to the broad range of researcher needs supported) means of facilitating the dissemination of findings to the most relevant KUs. In addition, researchers who have used end of grant KT funding opportunities report these opportunities fill a gap in CIHR's funding suite.

KT design and delivery

The evaluation identified key elements responsible for success within the KT funding opportunities. Along with these key elements, related challenges were uncovered. The identification of these factors is useful for understanding what can facilitate and what can impede KT. These elements are potentially relevant across alternative funding designs that include the objective of enabling KT.

• Key elements for success within KT funding opportunities are:
  • engaging KUs in and throughout the research process;
  • assuring commitment and buy-in from partners (not necessarily financial);
  • working with the right expertise (within both the researcher and KU contexts),
  • tailoring and timing the dissemination of results to the audience(s), and;
  • engaging both researchers and KUs in the review of funding applications for iKT research.
• Challenges to achieving success in KT are:
  • the substantial effort required to do iKT research (i.e., engaging KUs in a meaningful way);
  • timing research with KU needs;
  • submitting a KU's non-academic curriculum vitae to CIHR; and,
  • describing the parameters of a research partnership in a grant application.
• Performing iKT research and conducting KT of research findings is not well aligned with the performance measures used by universities to judge the success of individual researchers. Specifically, producing non-traditional research outputs and spending additional effort on partnering and dissemination activities receives limited recognition. This misalignment has created a systemic tension between performing KT and advancing a career as a university-based researcher.

The role of CIHR in KT

• Empirical evidence suggests a lag or often complete failure in moving research-developed evidence into practice (e.g., AHRQ 2001; Mitton et al. 2007).
• Current Government of Canada policy, including Budget 2012, articulates a strong desire for publicly funded research to be made more applicable to society at large. Evidence gathered in this evaluation validates that KT and the KT funding opportunities at CIHR are well-matched to this objective.
• Data collected through interviews and document reviews in this evaluation's international environmental scan of health research funders, illuminated that CIHR is considered to be a global leader in KT theory and KT funding models; areas that are becoming increasingly important to health research funding agencies worldwide.
  • In five of six global regions reviewed in this evaluation, the number of health research funders who include KT in their mandate has increased since 2008; in the remaining region, the number mandating KT has remained the same.
• KT funding opportunities reach a broad range of researchers, and are particularly well utilized by the following three groups.
  • New investigators
  • Pillars III and IV researchers
  • Female researchers

Recommendations

CIHR's KT Funding Program has performed well in terms of meeting expected outcomes. Moreover, it has helped to position CIHR for success in an area that is of increasing significance to health research funders across the world.

The following recommendations have been developed to support continued KT success for CIHR in the newly proposed Foundation and Project Schemes of research funding, as well as across CIHR strategic initiatives.

Appendix H of this report provides a detailed crosswalk of key elements for sustained KT success with detailed sub-recommendations and considerations for any mainstreaming process.

Recommendation 1

CIHR should invest the required resources to sustain its role in enabling KT. Given the decision by CIHR to integrate the KT funding opportunities into the open research suite, both of the proposed Project and Foundation Schemes of research should retain the key elements identified in the evaluation as responsible for the success of the current standalone iKT and end of grant KT funding opportunities. (See Appendix H for sub-recommendations and detailed cross-walk with each identified key element.)

Recommendation 2

To ensure the continued success of KT and address funding balance across all fields of research within the broad suite of open programs and strategic initiatives, CIHR should develop performance measures which accurately monitor and assess the integration of the KT Funding Program into the open research schemes. Data collected in this evaluation should be used as a baseline for the future study of CIHR success in KT under the newly proposed Project and Foundation Schemes of research. This will allow CIHR to measure whether or not the objective of mainstreaming – that there will be an increase in iKT research – has occurred.

Recommendation 3

For university-based researchers, performing iKT research and conducting KT of research findings is not well aligned with the performance measures used by universities to judge their success. CIHR should initiate dialogue with the academy and researcher community in order to draw attention to and work toward mitigating this tension.

Management Response

Program and Evaluation Description

KT at CIHR, and the CIHR KT Funding Program

At the Canadian Institutes of Health Research (CIHR), knowledge translation (KT) is defined as:

a dynamic and iterative process that includes synthesis, dissemination, exchange, and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products, and strengthen the health care system. (CIHR, 2012)

In more simple terms, KT at CIHR is about turning health research into improved health. KT has been an important aspect of CIHR's vision and work since the organization's inception in 2000, and is written into the Parliamentary Act which created CIHR (GOC, 2000).

The KT program at CIHR has evolved significantly since the year 2000. Today, this program encapsulates a diverse set of activities including funding opportunities, partner engagement activities, outreach, and research. Although it lists many ongoing and dynamic activities, Figure 1 provides a static snapshot of the CIHR KT program.

Figure 1 – The CIHR KT program

Through a more strategic lens, CIHR supports four interrelated but different types of KT. These are: synthesis, end of grant KT, integrated KT research, and KT science. The unique funding opportunities which make up the KT Funding Program are designed to encourage these four types of KT.

Synthesis is the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. It is a family of methodologies developed to determine what is known in a given area or field and what the knowledge gaps are. The underlying principle is the support of timely and accurate scientific knowledge being available to those who work in real-world settings requiring such evidence. Knowledge synthesis studies may be useful to policy-makers, industry, clinical and medical practitioners, amongst others. In some cases, knowledge synthesis research can be conducted with the participation of knowledge users (KUs) throughout the research process. CIHR refers to this approach as integrated KT. The Knowledge Synthesis funding opportunity (FO) is designed to support synthesis research, which is performed in an integrated KT (iKT) approach.

End of grant KT describes the process where the researcher develops and implements a plan for making knowledge users aware of the knowledge that was gained during a project. End of grant KT includes the typical dissemination and communication activities undertaken by most researchers, such as KT to their peers through conference presentations and publications in peer-reviewed journals. End of grant KT can also involve more intensive dissemination activities that tailor the message and medium to a specific audience, such as summary briefings to stakeholders; interactive educational sessions with patients, practitioners, and/or policy makers; media engagement; or the use of knowledge brokers to name a few. The commercialization of scientific discoveries is another form of end of grant KT, but as a specific strategy it is not being explored in this evaluation study. Commercialization is the focus of a separate CIHR evaluation which is currently underway. The Dissemination Events (DE) and Knowledge Translation Supplement (KTS) provide funding for end of grant KT.

Integrated KT research, involves engaging knowledge users through the entire research process. By doing iKT research, researchers and knowledge-users work together to shape the research process by collaborating to determine the research questions, deciding on the methodology, being involved in data collection and tools development, interpreting the findings, and helping disseminate the research results. This approach is designed to produce research findings that are more likely to be relevant to and used by end users. This approach is similar to those used in collaborative research, participatory, action-oriented research, co-production of knowledge, and Mode 2 knowledge production. The Knowledge Synthesis, Partnerships for Health Systems Improvement (PHSI), and Knowledge to Action (K2A) all support iKT.

KT science or research (also known as implementation science) is the study of the process of KT and the use of knowledge. KT science explores the factors which facilitate and hinder the sharing of knowledge between creators and users. While it often addresses issues such as the efficacy and effectiveness of certain KT strategies, it may also involve the development of new KT theory or practice. The KTR (knowledge translation research) panel of the Open Operating Grant Program (OOGP)¹ is the panel that reviews KT science grant proposals received by CIHR.

Evaluation purpose

When the evaluation was launched, its objectives were twofold. Firstly, the evaluation was designed to assess the performance and impact of the KT Funding Program and identify areas for improvement. Secondly, it was performed to fulfill CIHR's responsibility to the Treasury Board of Canada under the 2009 Policy on Evaluation which requires that all federal spending be evaluated to investigate the value for money of the expenditure.

In the interim, CIHR initiated a reform of its open grants competition and its peer review processes. As a part of this reform, the decision was made to integrate the KT Funding Program into new open schemes of funding and across CIHR strategic initiatives. This evaluation – although not designed to inform the integration of the KT funding opportunities into the open grants competition – provides information that can be used to support evidence-based decision-making in this reform process.

Evaluation approach

To evaluate CIHR's KT program in a resource-constrained environment, the direct object of evaluation was limited to a sample of six KT funding opportunities (Synthesis, PHSI, K2A, DE, KTS, and KT science). For the purposes of this evaluation these are called the KT Funding Program. These were highlighted as particularly important by CIHR senior management, are financially material (approx. 64% of KT program financial commitment), and provide a full representation of the four types of KT (synthesis, end of grant KT, iKT research, and KT science). A full description of each funding opportunity – including financial outlays, application rates, and success rates – is available in Appendix C of this report.

This evaluation employed an integrated, participatory approach. Evaluators, KT staff, CIHR senior management, CIHR Institute representation, and external researcher representation worked together to design and conduct this study (see Appendix G).

An early step in the design of this evaluation was the construction of a logic model (Figure 2) that was used to articulate the KT Funding Program Theory² and also helped inform the development of the evaluation questions.

Figure 2 – Logic model for the KT Funding Program

Figure 2 long description

Through the evaluation process – particularly the many formative discussions and interviews with other funding agencies, researchers, CIHR senior management, and KT staff – it became clear that underlying the KT Funding Program Theory were three critical and ordered steps: the involvement/engagement of KUs within research, the influence/influencing of KUs through the research process, and finally, KUs taking real world action that is informed by research created knowledge. This simplified version of the KT Funding Program Theory is illustrated below.

Evaluation questions

Eight evaluation questions³ were developed to guide this study:

1. To what extent are KT funding opportunities achieving their expected outcomes?
2. How effective is the KT Funding Program mix in achieving CIHR's expected outcomes? (iKT , End-of-grant KT, KT science, Synthesis)
3. To what extent have KT funding opportunities reached a broad and diverse range of knowledge-users?
4. What factors facilitate or inhibit the achievement of KT funding opportunity outcomes?
5. To what extent are KT funding opportunities being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness?
6. What role is there for CIHR in enabling/promoting iKT research, synthesis, end-of-grant KT, and KT science?
7. What would be the effect on CIHR-funded researchers and knowledge-users if the KT Funding Program no longer existed? What would be the effect on the improvement of health, more effective health services and products, and the strengthening of the healthcare system?
8. What are the unanticipated outcomes, positive or negative, resulting from the KT Funding Program?

Key Findings

This part of the report presents the key findings of the evaluation study and is organized into three sub-sections. Sub-section 1 presents findings related to the success of the KT Funding Program in achieving its expected outcomes. Sub-section 2 presents findings related to program design and delivery. Sub-section 3 examines what role exists for CIHR in enabling/promoting KT. Given the broad nature of evaluation questions 7 and 8, these are covered in each section where appropriate, rather than as a separate chapter of analysis.

1 - Achievement of expected outcomes

Evaluation Questions

1. To what extent are KT funding opportunities achieving their expected outcomes?
2. How effective is the KT Funding Program mix in achieving CIHR's expected outcomes?
3. To what extent have KT funding opportunities reached a broad and diverse range of knowledge-users?

This sub-section reports on key findings from collected data. Specifically, it presents findings related to the achievement of expected outputs and outcomes for the KT Funding Program. The presentation of a KT Funding Program Theory enriches the assessment by identifying additional areas of investigation that are more relevant to the purpose and objectives of KT. These additional indicators are examined using currently available data. Key findings related to how KT funding opportunities work to achieve outcomes are discussed. The findings include the importance of meaningful partnerships, the value of supported end of grant KT, and the sustained benefits of partnering.

1.1 – Traditional measures of research funding success

CIHR has traditionally measured KT success in terms of the production of KT outputs (a predefined type of research output) and a defined set of research outcomes. Data on these outputs and outcomes are collected through an end of grant survey known as the Research Reporting System (RRS), which until very recently was only administered to OOGP⁴ funded researchers. To better understand the success of KT funding programs against these traditional measures, several of the questions posed in the RRS survey were replicated in this study's surveys of KT funded researchers.

1.1.1 - Research outputs

Traditional research outputs span three categories: academic outputs (e.g., peer reviewed papers, book chapters); engagement of highly qualified personnel (HQP) (e.g., students, post-docs), and KT outputs (e.g., conference presentations, webinars). The academic output and HQP categories, although not specific KT priorities, are seen as traditionally important measures of CIHR success. Data from the RRS survey is included to provide a point of reference for interpreting these data⁵.

Table 2 presents the individual iKT funding opportunities, the KT science funding opportunity, iKT as a group, the OOGP, and the OOGP pillars II-IV. It should be noted – here and throughout this report where OOGP and KT data are presented together – that the strategic design of the KT funding opportunities and OOGP are different, the two have different objectives, and the nature of the research supported may be different. Therefore, any comparative analysis between the two should be interpreted with these caveats in mind.

In terms of CIHR's traditional measures of success, a typical iKT grant resulted in a mean of six trainees, four academic outputs and approximately 17 KT outputs. On average, an OOGP funded research project produced approximately nine trainees, 10 academic outputs, and 16 KT outputs (Table 2).

Using the common denominator of dollar amount invested (we employed a 100K base), results indicate that iKT funded projects produce greater numbers of KT outputs than do the OOGP funded projects. The unexpected findings, however, are that iKT funded projects also do better at engaging HQP and in two out of three cases (PHSI and Synthesis) at producing academic outputs than do OOGP funded projects. OOGP funded projects do produce greater numbers of academic outputs than do K2A funded projects. KT science grants lag behind both the OOGP and iKT in each of HQP, academic, and KT outputs using this measure.

The results of this analysis could be interpreted as indicating that iKT funding opportunities represent a low-risk, high-return investment opportunity to CIHR. This same hypothesis was raised by several knowledge users (non-researchers) interviewed as a part of this study. Such an assertion from this group gives additional context and weight to this finding.

1.1.2 – Research outcomes

Apart from outputs directly resulting from the KT Funding Program, a wider range of research outcomes could also be linked to KT grants.

In Table 3, data are ranked by order of frequency of iKT researchers reporting "yes" to having produced the research outcome. In this format it provides an indication of the type of research outcomes being produced most commonly from grants obtained from the iKT funding opportunities. The most common of these were research findings being cited by others, the introduction of a new tool, technique, instrument or procedure and a new research method.

Overall, the data relating to traditional CIHR measures of research outputs and outcomes identified in this evaluation confirm that the KT Funding Program is creating and disseminating knowledge.

1.2 - An appropriate KT evaluation approach; KU involvement, influence, and action

There is an extensive amount of academic literature which describes how the involvement of KUs in the research process will increase the uptake of research results and improve the relevance of research findings (Israel et al. 1998; Lomas 2000; Cargo & Mercer 2008; Macaulay et al. 2011). With a grounding in this theory, CIHR hypothesizes that KU engagement in research can lead to applications, such as informed decision-making, or the development of new tools or techniques, and that ultimately, this is will lead to health improvements at the individual and systems levels.

However, there is significant lack of evaluative work done on KT, especially KT funding interventions (Cordero et al. 2008; Tetroe et al. 2008). To address this challenge, a key step in assessing KT program performance in relation to expected outcomes was the development of a logic model (see Figure 2 on p.4). This logic model provides a visual representation of the KT Funding Program Theory, and accordingly, it provides a guide for identifying metrics of success.

A range of both quantitative and qualitative methods of investigation were employed in order to investigate and learn about the occurrence and intricacies of this KT Funding Program Theory. Results of this analysis are presented in the following sub-sections.

To begin this analysis, Table 4 provides quantitative survey data which provides a basic illustration of the correlation between involving and influencing KUs. The quantitative indicator used as a proxy for action is not included in Table 4 as the format of the survey question used to measure action did not allow for comparison across KU groups (as is done in Table 4). Data related to action is discussed in sub-section 1.2.3. Data related to involvement and influence is discussed in greater depth in sub-sections 1.2.1 and 1.2.2. Qualitative data is presented in each section, as appropriate, in order to corroborate and delve deeper into quantitative survey findings.

1.2.1 Involving and engaging KUs with research

To examine KU involvement in the research process, recipients of iKT grants (Synthesis, PHSI, K2A) were asked to report on involvement of various types of KUs at various stages of their research projects⁶. Results are presented in Table 5 along with reference data from the OOGP.

Overall, results indicate that iKT funded projects – grants which have a requirement for KU involvement in order to be funded – are more likely to have involved KUs in, and throughout, the research process than are researchers funded through the OOGP. IKT funding opportunities also tend to involve a greater variety of KU groups in research.

1.2.2 Influencing KUs with research

IKT researchers were also asked to assess the degree of influence that their research had on KUs. Table 6 presents survey responses related to influence.

Table 6 demonstrates that iKT researchers report influencing a diversity of KU groups with their research. In contrast to OOGP researchers and OOGP researchers from strictly pillars II-IV, iKT researchers were more likely to report influencing: listed study stakeholders, health system/care practitioners, patients and consumers of health care, health care managers, health care professional organizations, federal/provincial representatives, community/municipal organizations, as well as consumer groups and charitable organizations. OOGP researchers were more likely to report influencing other researchers/academics, industry, and media. ⁷

Additional noteworthy findings arise from these data. For instance, the KU categories where the OOGP researchers create greater influence than the iKT researchers are researchers/academics, industry, and media. Researchers/academics are well understood to be users of research and for this reason it is not surprising to see this category score well on the OOGP. However, the higher OOGP rates for the industry and media KU categories may be indicative of best practices for influencing industry and media KUs. The lesson seems to be: to best influence industry and media, use end of grant KT.

That is to say, a comparison of data in Table 6 with that in Table 5 illustrates that iKT researchers engage industry and media at higher rates, albeit marginally, throughout the research process with the exception of the end of grant KT stage. At the end of grant KT stage, OOGP researchers engage these KU categories more often. Although this finding may be a function of industry and media being more interested in research results funded through the OOGP, it may also demonstrate that researchers believe they can create influence in industry and media with end of grant KT activities. Those researchers who wish to influence industry and media, and likewise CIHR, should it wish to increase influence with these two groups, ought to consider and explore this finding further.

To probe deeper into these findings, we conducted in-depth interviews with both researchers and KUs. These interviews explored the issue of how influence happens in the research process, and how it happens best. This analysis is presented in the following subsection.

1.2.2.1 Meaningful partnerships as a catalyst for KT

In-depth, qualitative interviews, with both researchers and KUs, were used to further investigate and better understand not only if but also how iKT research might, and might not, lead to influence and eventual impact (n=29). Both funded researchers and KUs continually highlighted the concept of a "meaningful partnership" as a catalyst for creating influence and eventually action. This opinion was consistent among both researchers and KUs, and across each KT funding opportunity, which is particularly interesting as some grants do not require partnering (i.e., end of grant KT and KT science). The key finding of these interviews, however, was the contextual indications of how a "meaningful partnership" can crystalize. These are described in the following image.

Another significant finding from these interviews, reported by both funded researchers and KUs, was that building these relationships is the most difficult aspect of doing research. More specifically, it was cited as the most costly and time-consuming activity related to doing a research project. This finding is elaborated on in greater depth in section 2 of the key findings section of this report.

1.2.3 Action informed by research

If knowledge users are involved in and influenced by research, the KT Funding Program Theory suggests that KUs would be compelled into action to apply the research findings. A proxy measure of action used in this evaluation was researchers' perceptions of their contribution to fulfilling the CIHR mandate and the KT program's ultimate outcome of: improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system. A major avenue by which researchers can contribute to the achievement of CIHR's mandate is through inducing stakeholders to apply their research findings, and thus, this metric was used to assess the extent to which relevant stakeholders have taken action. It is recognized that a measure which directly assessed action taken by the KU would enhance this analysis, however, no such quantitative measures are collected by CIHR. To mitigate this data limitation, this evaluation undertook case studies and in-depth interviews to triangulate and validate the quantitative data presented below. Indications of research induced action from case studies and in-depth interviews with KUs are provided in the following sections of the report.

Figure 3 below illustrates the perceived contribution made by each KT funding opportunity to each part of CIHR's mandate.

Figure 3: Reported contribution to CIHR mandate: KT funding opportunities

Figure 3 long description

Figure 3 illustrates that the varied KT funding opportunities contribute quite similarly to the different parts of the CIHR mandate. Generally speaking, KT grants do best at creating the transformative outcomes of strengthening health care systems, creating more effective health services and/or products, and improving health for Canadians. They perform less well at creating new health knowledge and translating knowledge from research into real world applications. The finding that KT funding opportunity researchers report having translated knowledge into applications at lower rates than they report having produced the aforementioned transformative outcomes has no obvious explanation, but one interpretation is that these researchers identify their work as contributing to specific areas of health impact identified in CIHR's mandate, and doing so indicates that the KT has occurred.

When interpreting Figure 3 above and Figure 4 below, it is important to note that respondents in this evaluation were given an additional option of "may in the future" when identifying the contribution of their project to CIHR's mandate. It is quite possible that giving respondents the option to indicate that their research "may" contribute to CIHR's mandate "in the future" resulted in respondents understating present results. Table 7 is provided to mitigate this data interpretation limitation by presenting data for the "may in the future" category. Table 7 provides a detailed overview of how iKT funded projects (Synthesis, PHSI, K2A) and OOGP funded projects rank across the spectrum of CIHR's mandate.

Figure 4: Reported contribution to CIHR mandate: Synthesis, iKT, end of grant KT, KT science and OOGP

Figure 4 long description

The key finding of this analysis is the complementary role that projects funded through the KT Funding Program play in relation to projects funded through the OOGP. Figure 4 illustrates that, according to researchers, achievement of the CIHR mandate is realized through both OOGP and KT funding, and both sets of funding excel where the other is less effective. Together, these two funding streams produce a balanced approach toward the fulfillment of CIHR's mandate.

In summary, the findings presented in Figures 3 and 4 and Table 7 support the hypothesized KT Funding Program Theory – the notion that by being involved in research KUs are influenced to apply findings in action⁸. Further exploration, through in-depth and methodical statistical testing, of the relationships between the quantitative data points used as indicators of each step in this KT Funding Program Theory (i.e., Involve, Influence, Act) would be valuable to enhancing the understanding of the strengths, weaknesses, and nuances of this process.

1.3 The strategic value of the mix of KT funding opportunities within the KT Funding Program

In-depth interviews with researchers and KUs elicited data which support the strategic value of the mix of KT funding opportunities within the KT Funding Program, and also provide indications of how the breadth of the KT funding opportunity mix contributes to CIHR achieving the full breadth of its mandate. Two over-arching issues related to mix were identified:

• Respondents believe moving research into action requires multiple approaches. Specifically, respondents indicated the diversity of KT funding opportunities offers researchers and KUs options to design research projects that are the most appropriate to their research problem and that match partners' needs and available resources.
• Almost all researchers noted that the current range of funding opportunities allows them an opportunity to pursue projects that are outside the scope of traditional research grants. They noted the value of getting funding to pursue projects such as knowledge synthesis studies, participatory and collaborative research projects, as well as the rapid dissemination of findings that result from any type or style of research project. Researchers argued it would be unlikely such a diverse range of non-traditional research or researchers (i.e., KUs) could be supported without the strategic design of the KT funding opportunities. Specifically, that the OOGP did not fund these types of research projects or researchers as it currently operates.

The following two sub-sections will elaborate on findings related to two key aspects of the KT Funding Program mix: end of grant KT and iKT.

1.3.1 End of grant KT (DE and KTS)

As is described in section 1 of this report, end of grant KT is supported by CIHR through two funding opportunities, Dissemination Events and the KT Supplement. The parameters of these funding opportunities are provided in detail in Appendix C. Data presented earlier in this report provides KTS and DE results in terms of achievement of outcomes and impact. Here, we further the analysis to look for any significant divergence between the two, and look closer at how these funding opportunities achieved results.

To begin, end of grant KT (both KTS and DE) funding was highlighted by researchers as a particularly valuable and beneficial opportunity. Survey data shows that:

• 74% of KTS recipients and 81% of DE recipients reported that the grant led to outcomes that would not have occurred had they not received the grant. A further 16% of KTS and 5% of DE grant holders reported that it was too early to tell.
• 99% of respondents reported that both KTS grants and DE grants fill an important gap in CIHR's collection of funding opportunities.
• 98% of respondents from both KTS and DE indicated that they would apply for the grant again and would encourage other researchers to do so.

Qualitative evidence gathered from KTS and DE researchers provides an explanation of why these grants were so highly valued in the research community.

To investigate divergence between DE and KTS an assessment was undertaken of which stakeholders each program was intending to engage and which stakeholders were being influenced. Figure 5 presents results of this analysis.

Figure 5: Intended audience vs. influenced audience; KTS and DE

Figure 5 long description

Figure 5 illustrates some interesting discoveries. For one, for every KU group except health practitioners, and in the case of DE consumer/municipal organizations and industry, the funding opportunities are reported to create influence at greater rates than was initially intended. This finding highlights the ability and potential of end of grant KT activities to influence KUs. Researchers experience also shows that certain KU groups such as health practitioners are more difficult to influence through end of grant KT methods, and others such as industry and consumer/municipal organizations are more reactive to research findings through KTS than DE.

Data also show that aside from industry and other researchers/academics, DE grants are intended to reach more KU audiences than KTS grants. The industry and researcher/academic KU groups are both traditional Pillar 1 audiences, and this data indicates the KT value the KTS tool may hold for this group of researchers. As an additional note in regard to KTS, for the first four KU categories (health systems practitioners, patients, managers and professional organizations), not one respondent replied "not at all".

Generally speaking, Figure 5 illustrates that DE and KTS show some individual strengths and weaknesses in terms of reaching and influencing KU audiences. However, both funding opportunities are reported to create greater than intended influence in the majority of cases.

1.3.2 iKT partnerships and leveraging

Here we present a supplementary look at how the strategic design of the three iKT funding opportunities attribute to each performing differently in terms of leveraging external investments – a CIHR objective.

To begin, Figure 6 shows how K2A and Synthesis funding opportunities leverage partner money and in-kind contributions in grant proposals. PHSI grants attract the largest amount of contributions because of their unique partnership criteria, which require that a minimum of either 20% or 30%, depending on the province or territory, of the grant budget must come from external partner sources (i.e., non-CIHR funds). A review of EIS records on funded PHSI grants confirmed this, so PHSI data is not included in Figure 6. However, both Synthesis and K2A grants have also leveraged partner money and in-kind contributions in grant proposals, although this is not a formal requirement of the funding opportunity.

Figure 6: Synthesis and K2A researchers who reported partner financial and/or in kind contributions

Figure 6 long description

Source: KT Evaluation Survey responses 2011

While the above figure suggests that iKT grants effectively leverage partner contributions on grant proposals, we performed further analysis to examine the average amount of money leveraged per grant. Figure 7 shows that PHSI grants attract the largest amount of partner contributions.

The findings presented in Figure 7 may be attributable to a variety of factors. Firstly, a PHSI partnership criterion, as discussed above, requires leveraging money from partners. It is also important to consider the partners that are engaged through the PHSI funding opportunity. PHSI partners are primarily provincial governments and/or provincial health research funders. According to reported partner contributions across iKT grants, these organizations appear to have the capacity and political will to provide larger in-kind and monetary contributions.

While K2A and Synthesis grants share the common goal of moving knowledge into action by linking researchers and KUs, these funding opportunities attract partners that contribute smaller amounts, such as Canadian universities and research centers as well as other Canadian organizations. While these grants do not leverage as much total contribution as PHSI, they do provide opportunities for smaller organizations to participate in partnered health research. In-depth interviews undertaken with KUs and researchers from K2A and Synthesis grants revealed that forming linkages with diverse partners is a source of fresh ideas and perspectives, and this is asserted to fuel innovation. In short, the true value of these contributions should not be discounted because they are not financially equivalent.

Finally, PHSI attracts more partners on average per grant than other iKT funding opportunities.

1.4 The sustained benefits of partnered research

In-depth interviews and case studies elicited a further discovery about KT funding opportunity outcomes and impact, which can be summarized as the sustained benefits of partnered research. It was reported by both researchers and KUs that additional, post-funding activities occurred between project partners that would not have occurred without the initial grant funding. For instance, it was reported that many of the partnerships created through KT funding evolved over time into sustained long-term collaborations, including instances where a partnership either spawned continued research on a topic or research on entirely new topics with the same collaboration. In addition, networks created through CIHR funded partnerships were cited to have induced new relationships and collaborations. In essence, partnerships formed as a result of CIHR KT funding lasted and continued to produce transformative health impacts well beyond the sphere of the funded project.

Similarly, about half of the KU and researcher interviewees reported direct outcomes related to capacity building and ownership resulting from the researcher-KU partnership. Anecdotes from funded partnerships are provided to illustrate these concepts.

On the KU side, capacity building was described as KUs using research, or even doing their own research in the future. For example, in one project the KUs returned to their local organizations and provided training to staff on how to use research to improve their roster of services. In another case, KUs returned to the research team to ask them for additional research to be used to develop new health policy and programs. Given the success of this process, the researchers reported that KUs from other regions, across Canada, have contacted them to apply the research findings to their local context. Ownership was described by KUs as a situation where, following involvement in the research process, they found greater value in the knowledge created, and were more likely to utilize this knowledge as a result. As an example, a KU reported that their in-depth understanding of research results allowed them to advocate, with an evidence-base, for changes in the way pharmacists receive up-to-date information via handheld devices.

On the researcher side, capacity building was described as learning how to increase research relevance for partners and society at large. For example, in one project researchers described how the presence of KUs in the design of information management systems improved the efficacy of their product because they were constantly exposed to end-user needs and perspectives.

1.5 Summary findings

Review of the achievement of CIHR KT funding opportunities illustrates how the KT Funding Program has produced expected KT outputs, academic outputs, and has engaged HQP.

The KT Funding Program Theory, as reflected in this evaluation's logic model, is a guide for advancing efforts to evaluate KT, and particularly partnered research funding, more comprehensively. It should also support and inform the design of more appropriate KT-related performance indicators for ongoing data collection at CIHR.

The KT Funding Program is largely successful in achieving expected outcomes, including contributing to the achievement of the CIHR mandate. Substantial quantitative and qualitative evidence supports this finding. Quantitative researcher survey data indicates that the involvement of partners in research not only happens more often in projects funded through iKT funding opportunities (where partnerships are a requirement) than in projects funded through the OOGP, but that iKT-funded researchers are also more likely to report influence with KU partners, and they report that their research leads to real-world action. Qualitative data presented in this section supports these findings and suggests reasons for how this happens. The existence of a "meaningful partnership" is identified as a specific catalyst of impact. What "meaningful partnerships" look like in reality is explored further in the next sub-section of this report, where five examples of impact are presented in order to further explore and elucidate this finding.

Respondents identified end of grant KT funding opportunities as an important means of timely and appropriate dissemination of findings to the most relevant KUs. Nearly every researcher surveyed (99%) reported that the DE and KTS funding opportunities fill a gap in the overall CIHR suite of funding. Furthermore, end of grant KT is highlighted as an important device for OOGP researchers performing KT and several distinct KU groups: industry, media, and other researchers/academics.

An exploration of differences in design factors between iKT funding opportunities related to partnering and financial leveraging requirements provided interesting findings. Specifically, the requirement on the PHSI funding opportunity for partner financial contributions on grant applications (20–30% of funding must come from partners) was found to lead to more funds leveraged per grant, to funds being leveraged on more grants, and to more partners per grant. However, who partners are and how they interact with researchers, was highlighted as an important consideration in every method of this evaluation. The fact that the PHSI funding opportunity attracts more partners and money does not guarantee that the impact of the funding opportunity is optimal to that of K2A and Synthesis. In fact, the above investigation of researcher cited impact showed the PHSI, K2A and Synthesis funding opportunities to achieve quite similar results. Then too, the fresh and unique perspectives brought to a research project by the partners who are capable of participating (i.e., without the 20–30% requirement) on K2A and Synthesis grants have been suggested by researchers to be a driver of innovation.

Finally, the post-funding activities undertaken by CIHR-funded partnerships were identified as a lasting and sustained benefit of each KT funding opportunity. "Capacity building" by both KUs and researchers as well as the "ownership" of research findings by KUs were cited as the underlying causes of this ultimately positive outcome.

2 - Program design and delivery

Evaluation Questions

4. What factors facilitate or inhibit the achievement of KT funding opportunity outcomes?
5. To what extent are KT funding opportunities being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness?

This sub-section of the report presents identified facilitators and hindrances of CIHR KT success. It provides context as to why the results and impacts identified in the previous key findings sub-section are occurring, and in what ways they are being obstructed. Evidence from each line of inquiry has been examined, and triangulated key findings are presented.

2.1 Facilitators of success

In order to examine factors that induced and encouraged program success, a pathway case study analysis was undertaken. In this method, cases are chosen which exhibit a common starting point and a known end outcome – in our situation, an exemplary outcome (Gerring 2007). Such case study analysis provides detail about causal factors of change or the active ingredients of success within the funding opportunities.

To pick cases for analysis, a subset of the Evaluation Working Group undertook a comprehensive selection methodology. This process included review of nominated cases from KT program managers based on anecdotal reports of exceptional achievement of outcomes, a review of survey and interview responses related to outcomes achievement, and a review of end of grant reports submitted to CIHR by funded project teams. Following this, a narrowed set of projects was independently reviewed and scored by members of the Evaluation Working Group. Independent scoring was then aggregated to select final projects for case study analysis.

Summary narratives and key findings representing each KT funding opportunity are presented below⁹.

NPI: Dr. Esther
Suter

PDM: Grace
Mickelson

Synthesis: Can lnterprofessional Collaboration Provide Health Human Resources Solutions? A Knowledge Synthesis

Research team: Dr. Esther Suter, Ms. Grace Mickelson (KU), Dr. Siegrid Deutschlander, Ms. Zahra Nurani (KU), Ms. Jana Lait, Dr. Elizabeth Louise Harrison, Dr. Sandra Jarvis-Selinger, Dr. Lesley Ann Bainbridge, Ms. Sheila Achilles (KU), Dr. Christine Anne Ateah, Dr. Kendall Ho, Dr. Ruby Emily Grymonpre
CIHR Investment: $100,000

Overview

Between the shortage of health care providers and the lack of communication and collaboration amongst them, there is a clear need to change how health services are delivered and the way in which providers interact with one another. lnterprofessional Collaboration (IPC) has been identified as a way to address this need. With the support of a Synthesis grant the research team set out to answer the following questions: (1) In what ways do IP interventions at the post-licensure level influence IPC and workplace quality, provider satisfaction, reuutrnent, retention and turnover? (2) In what ways do IP interventions at the pre-licensure level influence graduate employment choice? (3) Is there any evidence of cost benefits of IP interventions?

Pathways to outcomes

Involving KUs throughout the project lifecycle is essential to performing an impactful synthesis

Having KU involvement at project outset was critical. The project team stressed that the findngs world not have gained the same traction in the KU community without the involvernent of the right knowledge users at the right stages of the research process. The formal iKT structure requred in the grant application (for example, requiring a specific Principle Decision Maker) solidified the importance of the process for the team.

Forming a KT sub-group within the project team created focus and drive for meaningful research

Tasking a smaller intemal group with continuous attention to the KT strategy was a fruitful practice. This team ensured that at each stage of the project KT was occurring or being planned for.

Team members matter

The individual expertise and experience of team members was important to project success. However, the team experience emphasized that perhaps the most important characteristic of a team member - in order to move research into action - is the connections they hold with external knowledge user and knowledge user networks. The result was the broadening of knowledge uptake.

Impacts

Project team members emphasized that the Synthesis project made KUs better proponents of evidence-informed change within their organizations and networks. Accordingly, the completed Synthesis study has made a significant and momentous contribution to the field of IPC and Health Human Resources (HHR) in Canada. Provincial organizations in Manitoba, Saskatchewan, Alberta, and B.C. have used the synthesis to improve HHR priorities, with the consequence of real improvement in the quality of health products, services, and outcomes for these provinces as a result of more integrated and coordinated HHR.

NPI: Dr. Judith
Bartlett

PDM: Dr. Catherine
Louise Cook

Knowledge to Action: Need to know too

Research team: Dr. Judith Bartlett, Dr. Catherine Louise Cook (KU), Ms Sheila Carter, Dr. Patricia Martens (KU), Ms. Rose Neufeld (KU), Dr. Brenda Elias, Dr. Julianne Sanguins. Research staff: Mr. Nathan Hoeppner
CIHR Investment: $200,000

Overview

Despite representing a significant part of the population of Manitoba, very little research in Manitoba had considered Metis health needs separately from other Aboriginal groups. A data linkage study called the 'Metis Health Status and Health Service Utilization' was conducted to address this gap in knowledge. Using a holistic approach, the team set out to meet the following objectives: 1) Facilitate Manitoba Metis Federation (MMF) and Manitoba Health (MH) Regional Health Authorities (RHA) to develop and sustain a knowledge network; 2) Implement and evaluate a novel KT model between the University of Manitoba, MMF and MH; and 3) Utilize this KT model to accelerate the translation of the Metis Atlas study outcomes in adapting MH programs and services to better meet the needs of Manitoba Metis.

Pathways to outcomes

Involving stakeholders with the right skills from across the Manitoba health care landscape

The project brought stakeholders from all levels of health policy and programming to the table. During the application preparation the team carefully identified individuals with the best skills and organizations best positioned to facilitate putting knowledge into action. This resulted in a team with a wealth of expertise that spanned the Manitoba health care landscape.

Using a knowledge translation framework

K2A projects require the utilization of a KT approach that is supported by empirical evidence. Through the use of a framework, participants negotiated their roles across the span of the project. The use of a KT model demonstrated that all parties were working towards the same end-goal: to better the health of Manitoba Metis.

Involving high-level KUs

Having KUs with sufficient organizational power on the research team ensured that knowledge flowed from research to its applied use in an efficient way. Additionally, having sufficiently empowered knowledge users involved meant the pathway of information flow was clear and concise, and very little was lost or misconstrued along the way.

Impacts

As the project expanded across the province, all seven other MMF Regions and 10 of the 11 RHAs have engaged with the implementation team to be involved in the KT model to inform Metis-specific health policy and programming in their region. Accordingly, the province funded the MMF as a key member of the researchers' implementation team to keep this process underway. Furthermore, requests for more research from Knowledge Networks spearheaded additional studies on chronic diseases, which will be used to inform future policy and programming. In essence, for the first time, MMF Regions and associated RHAs are engaged in collaborations aimed at improving Metis-specific health outcomes in their regions. They have negotiated a common-ground relationship that resulted in better understanding the health needs of Metis in the region, and have set up a sustainable knowledge-to-action framework that enables ongoing policy and program change.

NPI: Dr. Shoo Lee

PDM: Dr. Eizabeth
Whynot

PHSI: National health partnership for reducing infections in NICU

Research team: Dr. Shoo Lee, Dr. Eizabeth Whynot (KU), Ms Elaine Orrbine (KU), Dr. K.J Barrington, Dr, G Ross Baker, Dr. Joanne Langley, Dr. Arne Ohlsson, Dr. Nicola Shaw
CIHR Investment: $100,000

Overview

Hospital acquired infection is one of the leading causes of mortality and morbidity in neonatal intensive care units (NICU). Despite efforts by individual NICUs to reduce infection rates using Continuous Quality Improvement (CQI), there has been little to no significant change in the rate of hospital acquired infection in NICUs. An evidence-based quality inprovement method called the Evidence-based Practice for Improving Quality (EPIQ) was developed and evaluated and was shown to reduce hospital acquired infection. Following the successful implementation of EPIQ, the research team, with the support of the PHSI grant, started research with the following objectives: to test the generalizabiity of the EPIQ model to all Canadian NlCUs; to determine whether quality improvement gains using the EPIQ model are sustainable; and to establish and evaluate a national system that will last beyond this study.

Pathways to outcomes

Building organizational change using institutional staff

In traditional cases, organizational change is advocated by senior management, enforced by administration, and facilitated by external consultants (these are called top-down approaches). The PHSI team believed that sustainable change could be built from the bottom-up. This organic process ensured that when the official faciitation process ended, the agents of change remained within the organization.

Involve everyone who has a vested interest

In order to produce sustained change, the team involved everyone that had a stake in the intervention - from hospital administrators to the housekeeping staff. From the outset, all staff particpated in focus groups led by these clinical teams that helped them identify the barriers to change in their institution. After this informiation was gathered, the team identified effective change processes that met their unique institutional needs. This ensured that all staff bought into the process, and understood the impetus for change.

Obtain financial commitments from partners

The institutions provided support in the form of in-kind donations of staff hours from participating NICUs. This ensured that staff would have the time, resources, and support to carry out the work required throughout the duration of the project. It also created incentives for meaningful engagement from the provinces.

Impacts

This study had significant inpacts on health policy and decision making and has the potential to fundamentally change the current paradigm for quality improvement efforts in NICUs across Canada. The original eleven NICUs continued to have reductions in infection rates. It was clear that the EPIQ process was sustainable as the behavior and culture of the NlCUs changed to continue to meet the challenges of caring for premature infants. The team continues to build upon the knowledge gained from the PHSI grant. They are currently embarking on EPIQ III, a study that has increased the scope of previous studies.

NPI: Dr. Katherine
Boydell

KT science: Producing and communicating knowledge differently: Articulating the pathway to mental health care through artistic expression

Research team: Dr. Katherine Boydell, Ms. Brenda Gladstone, Ms. Angela Kertes, Ms. Elaine Stasilius, Ms. Bramilee Dhayanandhan
CIHR Investment: $74,953

Overview

First episode psychosis is an important period for intervention to prevent future impairments and disabilities. Many youth do not seek help during this critical period, and current social stigma that is associated with issues of mental illness are thought to be responsible for the lag in time it takes to seek help. Efforts to educate and inform populations about first episode psychosis and mental health have been largely unsuccessful. Arts-informed methods provide a new, accessible medium where participants are challenged to engage with mental illness. This CIHR-funded project aimed to accomplish two objectives: to better understand the science of knowledge generation and translation using art-based methods, and to further the understanding of first episode psychosis education and awareness from the perspective of a small group of young people who had experienced first episode psychosis and accessed care.

Pathways to outcomes

Using a participatory (iKT) and iterative process

The research design was iterative, meaning that it was built with the participation of all involved parties and flexible to the needs of the participants. Success was attributed to the integrated knowledge creation that took place. The design decisions for the mural were made by the participants, giving them complete control over the final product- a process that provided the research team with rich data. The process reflected an understanding of the emotional process that the participants went through to produce a product that effectively communicated their stories.

Team experience and skills

The research team believes that their combination of expertise contributed significantly to the success of the project. The team worked together for many years and had extensive experience in first episode psychosis in young people as well as strong methodological skills.

Leveraging project-specific resources

The funding mechanism for KT science grants is designed to be flexible to accommodate new and innovative projects. The KTR Panel does not require an official partner on the grant; however, the grant assisted them in effectively leveraging additional resources which contributed to the success of the project.

Impacts

This project acted as a stepping-stone toward a more evidence-based field of practice in arts-based methodologies. Today, the team continues to engage in research that explores the value of arts-informed methods, including collaborating with an intemational consortium that focuses on how arts-based knowledge translation works. Dr. Boydell presented this research at national and intemational conferences and was asked to publish on the topic in Nature magazine. Several participants, as well as the research team, continue to be involved in the evolution of this project as it travels to rural schools in Ontario and PEI.

NPI: Dr. Sherry
Lynn Grace

KT Supplement: Promoting Inpatient Cardiac Rehabilitation Referral

Research team: Dr. Sherry Lynn Grace, Dr. Caroline Chessex, Dr. Jack Ven Tu, Dr. Blair Joseph O'Neill, Dr. Robert Donald Reid
CIHR Investment: $100,000

Overview

Cardiovascular diseases are the leading causes of mortality and morbidity in Canada. Cardiac rehabilitation (CR) is an outpatient chronic disease management program designed to enhance and maintain cardiovascular health. It has been found that participation in CR programs reduces mortality and morbidity rates, however, these programs are poorly attended. The two leading reasons why patients do not access CR are referral failure and lack of provider endorsement. This KTS project aimed to increase referral and enrollment of indicated cardiac inpatients, based a on evidence-informed referral and enrollment targets established by the research team and endorsed by the Canadian Cardiovascular Society and the Canadian Association of Cardiac Rehabilitation. The project consisted of a multipronged KT approach that included educational components, audit and feedback systems, and informatics.

Pathways to outcomes

Tailoring both the message and the mechanism for the audience

KT pathways are not always linear and often researchers and KUs explore and implement varied approaches in order to achieve the best KT outcomes. To address this complexity, the team implemented three purposefully different KT models in order to bolster referrals and enrollment in cardiac rehabilitation.

End-of-grant implementation involves relationships with KUs

Although not a requirement for KTS funding, the research team actively engaged KUs throughout the project. In fact. the KTS funding was cited by the research team to enable the forging of important partnerships. By working with hospital administrators and team leads, the research team was granted access to frontline staff. The buy-in from top-level employees at the participating sites ensured that the KT models were accepted and implemented effectively. Clinicians and frontline staff also played integral roles in the successful implementation of the models.

Integrating feedback from KUs

Throughout the course of the project, the research team interacted with a myriad of institutional partners, ranging from hospital administrators to performance measurement units. This work informed the research in many ways, particularly by lending a nuanced perspective of the challenges and barriers to implementing KT models.

Impacts

This grant led to the effective implementation of KT models that enhanced CR enrollment and referral. Promotional and educational materials were created to inform patients of the services available at CR and these materials are available in the participating cardiac units, and translated into 4 languages. The team also developed a referral toolkit, which is available online to CR clinics across Canada.

Additionally, this project supported the development of the internationally endorsed Charter on Cardiac Rehabilitation. The newly formed Intemational Collaboration in Cardiovascular Prevention and Rehabilitation will present the drafted charter at the World Congress of Cardiology 2012 in Dubai.

(For additional details on each of the research projects highlighted in the above case narratives, links to further reading and resources are provided in Appendix E of this report.)

In summary, these case studies – which were purposefully selected based on their exemplary results – have been used to identify active ingredients of success within the KT Funding Program. Key elements can be summarized as: engaging KUs in and throughout the research process; assuring commitment and buy-in from partners (but not necessarily financial); working with the right expertise within both the researcher and KU contexts, and; tailoring and timing the dissemination of results to the audience(s).

In addition to these key elements, another ingredient of success identified in this evaluation is the inclusion of KUs in the review of research applications.

2.1.2 Merit Review as a facilitator of iKT research project success

Merit review is grant review process used at CIHR exclusively for iKT grant proposals. It is designed to increase the probability of the selection of projects that will excel both in scientific rigor (scientific merit) and real world impact (impact). Merit review requires that both researchers and KUs be grant reviewers.

Through interviews with both researchers and KUs, the strategic design of merit review (requiring both scientific merit and impact to be considered by researcher and KU committee members) was asserted to be a facilitator of eventual project success. Survey results show strong approval for merit review as over 82.3% of researchers were either satisfied or very satisfied with the merit review process.

In addition, variations of the Merit Review process are the standard at several high-profile private and public research funding agencies, including the Bill and Melinda Gates Foundation and a Government of Canada supported offshoot of the Gates Foundation: Grand Challenges Canada. Both organizations use non-researcher expertise in the review of grants in order to ensure the scalability (i.e. real world value) of funded research (Grand Challenges Canada 2010; Gates Foundation 2010).

For example, in their Global Health Strategy the Gates Foundation describes the objectives of their review process as follows:

"Our goal is to ensure that we are considering the widest range of funding opportunities and hearing diverse perspectives on the relative merit of those opportunities. The review process for all large grants involves input from a broad cross-section of outside experts, other funders, and other stakeholders. The vast majority of our grants, even many of the smallest, are shared with experts in an external review." (Gates Foundation 2010)

Further to this alignment with several innovative health research funders, during our interviews with other national research funding organizations anecdotal interest was shown in learning more about CIHR's Merit Review process. For instance, questions arose around how Merit Review could be implemented at the country level out of the stated desire of mimicking the process at their own national funding organization.

Additionally, the review of grants by potential users of research is very much in line with the post-Jenkins Report¹⁰ environment in which CIHR operates, including the direction for innovation laid out in the GOC Budget 2012. In this light, merit review presents significant value to CIHR as it moves forward in the current federal policy context.

Given the indications of the value of Merit Review uncovered in this evaluation, further analysis of the results and effectiveness of merit review would be both innovative and of high value to CIHR. Such analysis was not within the scope of this evaluation.

2.2 Hindrances of success

This evaluation also aimed to reveal any underlying challenges or weaknesses with/within the KT Funding Program. To perform this inquiry, a sequential mixed method approach was employed. Qualitative interviews probed researchers and KUs on four broad categories (internal, external, CIHR strategic, and CIHR program delivery) and through these interviews specific challenges under each of these categories were uncovered. Based on the range of issues unearthed in exploratory interviews, a series of survey questions were used to investigate the generalizability of these issues across the wider population. The most prevalent are discussed below, by category.

2.2.1 Internal to the funded project team

The additional effort required to involve KUs in research

The most frequently cited internal concern, by both researchers and KUs, was the amount of effort required to perform iKT research. Doing research in a partnership was considered to be more time consuming, expensive, and uncertain than doing the same project in a traditional research format. Moreover, creating and fostering a meaningful partnership – the type that is an identified driver of KT project success – was cited to require added effort and considerable tact. The additional burdens of time and finances are related, quite directly, to partnering activities such as more meetings and travel. Additional uncertainty was tied to matching the research to the KUs' timelines and the risk of a change in the KU environment that would affect the research process or results (e.g., staff turnover or changes in organizational mandates).

Related to each of the issues cited above is the finding that iKT research was considered to require diplomacy skills not typically associated with traditional investigator-driven research. For instance, negotiating roles and responsibilities, encouraging consistent and effective team member engagement, and ensuring the right researchers engaged the right KUs at the right instances (and vice versa) were all cited to require significant 'non-academic' skills. It is essential to note, however, that these skills are asserted by respondents to be vital for both researchers and KUs, as partnering is considered a dyadic process. Figure 9 below illustrates survey responses to a question developed around this topic. The results indicate this is a generalized challenge across the iKT funding opportunities.

Figure 9: Internal challenges

Figure 9 long description

2.2.2 External to the funded project team

Timing research to meet KU needs

Several challenges were cited as interferences from the external environment, which was defined for our purposes as that which is external to the funded project team. The first of these was timing research to meet the needs of the KU.

Both researchers and KUs identified matching the research production cycle with the knowledge use cycle as a challenge of doing knowledge translation. This finding was consistent across both iKT interviewees and interviewees undertaking end of grant KT not using an iKT approach. In simpler terms, delivering research in a timely manner to KUs – such as industry leaders, policy-makers and/or health care practitioners – is challenging.

Two solutions are, however, implicit to the current range KT funding opportunities. On the one hand, in-depth case study analysis shows that a key to partnering and knowledge translation success is the engagement of KUs in an iKT research approach. Specifically, having KUs aware and involved through the research process increases the meaning, utility, and timeliness of research findings. It should be noted that this finding is not unique to the CIHR environment and is well supported by empirical evidence and theory on collaborative research methods (Israel et al. 1998; Macaulay et al. 2011). Both KUs and researchers interviewed as a part of our evaluation argued that this type of research would not happen in the absence of a strategically targeted funding opportunity supporting partnered research.

On the other hand, the use of end of grant KT funding opportunities (KTS and DE) was considered an important tool for rapid research dissemination. The relatively open and flexible application process paired with the prompt turnaround of funding decisions helps researchers to share results with relevant KUs on demand, and in varied and appropriate ways. Another key benefit cited was the ability to engage KU groups who might not have been identified as relevant at the outset of a research project.

Doing KT in the university setting

In both interviews and case studies a tension was reported between working in a university setting and pursuing KT activities. Two specific causes of this tension were identified. The first was spending the necessary effort on partnering in an iKT research approach. The second was related to the creation of KT outputs and outcomes. This includes the development of end products, such as clinical care pathways, practical decision-aids, policy documents, websites, and commercialized products and services.

Universities generally value the creation of knowledge through traditional paths and the larger financial grants associated with such research. Indeed, researchers identified that academic tenure and career progression is often tied, explicitly and implicitly, to traditional knowledge creation products – including peer-reviewed publications, books, and book chapters – as well as to the attraction and retention of larger grants and awards. Thus, a direct conflict exists for the university-based researcher who pursues and performs KT and who seeks career advancement and security at his/her university.

Figure 10: External challenges

Figure 10 long description

Upon further investigation, this was not a unique research finding. Estabrooks et al. (2008) reach similar conclusions in an empirical study of university researchers. Of further interest is the research of Jensen et al. (2008) who found that scientists who engage with the public are just as likely to perform to excellent scientific standards. Further investigation of this issue, perhaps in collaboration with universities, may be of value to CIHR in order to improve incentives toward the pursuit of KT objectives.

2.2.3 CIHR strategic challenges

Length of time and amount of funds for iKT research

Closely related to the major internal challenge identified through this evaluation are the identified strategic challenges of iKT funding opportunity time-length and financial limitations¹¹.

Figure 11: CIHR strategic challenges

Figure 11 long description

Funded researchers express a desire for strategic changes to be made to iKT funding opportunities that would increase funding and length of grants. To consider this desire further, and to examine an additional indicator beyond researcher perspective, an analysis of grant holder behavior is presented below.

To begin, Figure 12 illustrates the difference between funding opportunity length and actual length of time spent on the same project by grant recipients.

These data indicate that researchers do take more time than they are allocated to complete their funded research projects, however, the reasons for this are less obvious and could be numerous. The qualitative and quantitative data gathered in this evaluation suggest that the considerable effort required to perform iKT research could be one significant underlying reason.

The desire to increase the financial contribution of these grants is also logically linked to the challenges of doing iKT research, and like the desire the increase time lengths this concept was reiterated through in-depth interviews with funded researchers. Examples cited of the explicit financial costs of partnered research surfaced in activities such as the co-design of the research process, consensus-seeking with partners, and the tailoring of research outputs to KU audiences. All of these activities can require significant travel, telephone conferencing, or simply the paid time of both researchers and KUs. Accordingly, a need for additional funds to finance these activities is perceivable. Figure 13 below indicates maximum and actual contributions made by CIHR to iKT research projects.

Figure 14 indicates that on average only the Synthesis funding opportunity disburses more funding than is listed on the grant RFA. This data is contrary to what is noted in Figure 11, where researchers suggest that the amount of funding available in the grants is a hindrance to success. In addition, survey data tells us that the vast majority of researchers (88% across iKT funding opportunities) have received the funding that they request in their original grant proposal.

Of the 12% that did not receive the amount they requested, Figure 14 indicates that a minority of these researchers were able to finish their grants in the anticipated timeframe, however, the majority did complete their project within the intended scope. This finding is verified in Figure 13 above. Accordingly, these data indicate that the issues of funding contributions and time-length of the iKT funding opportunities may well be intertwined.

Figure 14: Impact of a reduction in requested funding: iKT funding opportunities

Figure 14 long description

However, in terms of funding contributions from CIHR, this is not adequate evidence to fully judge whether the current funding ceiling, per funding opportunity, is sufficient or insufficient to perform research activities, as numerous confounding factors could be skewing this analysis. Furthermore, very little additional evidence is available for triangulation; mainly for two reasons. First, traditional research funding offered at CIHR is an inappropriate comparison, given it is not designed to encourage collaborative approaches to research. Second, data from the international scan of KT funding activities uncovered very little comparable programming, and none that had been evaluated, making it impossible to establish an evidence-based indication of adequate funding requirements for partnered research.

Finally, it should be reiterated that iKT funding opportunities did quite well at achieving expected outcomes (see the key findings presented in sub-section 1). This begs the question of whether funding increases would affect results proportionately, disproportionately (upward or downward), or if they would have any effect at all. Data collected in this evaluation do not answer this question.

2.2.4 CIHR program delivery challenges

CV and application issues

As discussed earlier in this report, researchers and KUs are satisfied with the processes of KT funding program delivery. Specifically, the merit review process for iKT grants and the timely administration of end of grant KT funding applications have been highlighted as contributing to, and facilitating, research project success. However, while program delivery has not been identified as a systemic barrier to program success, one opportunity for improvement arose through in-depth interviews and tested similarly in a survey of funded researchers. The issue is grant application for KUs and for researcher-KU partnerships.

Firstly, it was identified that for KUs – non-academic applicants who are encouraged and expected to apply to iKT funding opportunities – the application process is perceived by respondents to be tedious and to include many unrealistic criteria. For one, these partnering KUs face an overly strenuous and awkward task when trying to match their CV to the CIHR format requirements. During interviews, both KUs and researchers argued that CIHR could improve the application process by allowing non-academic CVs to be accepted from non-academic applicants. Additionally, a few interviewees noted that the CV module utilized by CIHR is actually different from other funding agencies, such as NSERC, and a common tri-agency approach would improve the efficiency of application.

Secondly, both researcher and KU interviewees noted that the application template is especially tiresome in requiring partnership details that are often simply unrealistic or unavailable. A common example cited was the requirement for listing the number of hours per week each KU was committing to the project. These requirements, which essentially predefine how the partnership will be structured, are inconsistent with findings of this evaluation which indicate that flexibility and idiosyncratic design are elements underpinning a meaningful, and therefore, successful partnership. Figure 15 below illustrates survey responses on this issue.

Figure 15: External challenges

Figure 15 long description

These findings, although certainly helpful for CIHR program delivery, are not unique to KT funding opportunities as KU and partnership criteria are the same for OOGP applications. In fact, researchers identified that the Merit Review process, which includes KUs in the review, helps facilitate the fair assessment of partnerships and partners. It should also be considered that, the respondents providing their views received funding from 2005 onward. Over this period CIHR has undergone multiple changes to the CV modules and funding opportunity application requirements. In this sense, goal posts are moving.

2.3 Summary Findings

In-depth case study analysis of highly successful KT projects offer insight into understanding key elements of success across the funding opportunities that support synthesis, iKT, end of grant KT, and KT science. The common themes to emerge include: engaging KUs in the research process, assuring commitment and buy-in from partners (not necessarily financial), tailoring dissemination to the audience, and working with the right subject area experts (within both the researcher and KU contexts). The factors highlighted in these cases correspond well with findings from other methods of the evaluation. As a triangulated set, these themes provide a guide to understanding which factors should be encouraged and supported in order to stimulate successful KT. Individually, the case study narratives play a valuable role as they provide real-world evidence to support CIHR KT Funding Program Theory, and in doing so, demonstrate the performance and impact of CIHR funded research.

This section has also presented Merit Review as a key element of iKT funding opportunity success, and, notably, the use of similar methods of application review by other funding agencies as a means of enabling the translation of research into action.

Hindrances to success are also identified in this section of the report, and included: the substantial effort required to do iKT research , timing research with KU needs, doing KT in a university setting, the length of time and the amount of funds provided for iKT research, and CV and application issues related to KU partners and partnerships.

The inhibiting factors identified are useful for understanding what may be slowing or diminishing the success of the KT Funding Program and the individual funding opportunities comprising it.

3 - The role of CIHR in KT

Evaluation Questions

6. What role is there for CIHR in enabling/promoting KT (iKT, synthesis, end of grant KT, and KT science)?

The over-arching purpose of this sub-section is to investigate the relevance of CIHR's position in promoting/enabling KT. To do this, this sub-section provides an overview of what CIHR aims to achieve through KT. Secondly, it presents a comparison of these aims to current Government of Canada (GOC) policy related to research and innovation. Thirdly, it presents key findings from an international environmental scan of KT funding to investigate the role CIHR plays within the Canadian and global contexts. Finally, it highlights data obtained through a scan of CIHR's EIS database and from evaluation survey results, both of which help to illustrate the use of KT funding opportunities by CIHR's client-base.

3.1 – What is CIHR pursuing through KT?

In health practice – including health products, services, and systems – a generally accepted desire exists for product and service provision to be informed by valid and reliable evidence. There is, however, an abundance of evidence indicating that health practice often lags behind knowledge and best practices established through health research (AHRQ 2001; Mitton et al. 2007). In this context, there is an increasing need for health research to be better linked to health innovation and its widespread application.

To address this issue, efforts have been made to promote the use of research-developed and tested evidence in practice. This concept of moving research into practice has become known by many names including KT (Straus et al. 2009; Tetroe et al. 2008). Since the creation of CIHR in 2000, the organization has been strategically involved in the effort to promote the use of evidence in practice through KT.

In other words, the KT program at CIHR has been offered with strategic intent. To support this, a precise definition of KT was developed and a set of activities, including standalone funding opportunities, were created and supported in order to increase the uptake of knowledge. Figure 1 in Section 1 of this report outlines the current KT program. Strategically, the program has included support for iKT, specific attention to innovative end of grant activities with knowledge users (KUs), and an acknowledgement of the needs of knowledge users for comprehensive syntheses of research evidence.

Although much evidence exists to support the need for KT, very little research exists that measures the performance and impact of KT interventions, especially when those interventions are funding mechanisms (Lavis et al. 2006; Tetroe et al. 2008; Cordero et al. 2008). The international environmental scan undertaken in this evaluation added further support to this finding. Of the 26 funding agencies who participated in the scan, not one had completed an evaluation study of their KT activities that would be comparable to those offered at CIHR. The need for such evaluative work is further demonstrated by Dalziel et al. (2012) in a recent systematic review of research impact studies. The authors find that this gap spans beyond the field of health research to impact studies across numerous research fields (Dalziel et al. 2012).

3.2 KT at CIHR and within the Federal Government context

The importance of KT to CIHR is clear. The concept is embedded in the CIHR mandate and written into the Parliamentary Act which created CIHR as it now exists:

The objective of CIHR is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system. (GOC Bill C-13 2000)

In addition, the desire for the application of research in society is an objective of significant importance to the Government of Canada (GOC Budget 2012; GOC 2012; Industry Canada 2007). As a dynamic and iterative process that leads to the ethically-sound application of knowledge, investments in KT enable researchers and KUs to move their discoveries and innovations into solutions to health challenges which in turn improve social welfare and economic competitiveness. Findings presented in this evaluation report validate that this is happening.

As such, these investments directly contribute to the vision outlined in the Federal Government's Budget 2012. Indeed, the GOC Budget 2012 indicates clearly that the Government supports investments in basic and applied research, and the translation of public research knowledge to applied uses. Chapter 3.1 of GOC Budget 2012 highlights "Knowledge Translation" as a specific area of importance to Canadians. Furthermore, as a research process, each KT funding opportunity directly aligns with priorities stated in the Federal Science and Technology Strategy, and specifically contribute to the Entrepreneurial Advantage and Knowledge Advantage priority areas outlines in the same Federal strategy (Industry Canada, 2007).

In sum, there does exist a close alignment between CIHR KT and current GOC plans and priorities for research and innovation.

3.3 The international KT context

In order to better understand the relevance and position of CIHR in promoting and enabling KT, an international environmental scan of research funding agencies was undertaken as a part of this evaluation. A full list of agencies examined in this scan is provided in Appendix A of this report. Figure 16 provides a contextual lens for positioning CIHR in KT, both nationally and internationally.

The environmental scan indicated that KT is known by many different names around the world. Representatives of numerous funding agencies spoke of 'alternative' descriptive terms used to define what is 'commonly' known as KT including: translational research, implementation science, knowledge transfer, and knowledge exchange. The definition of KT pioneered by CIHR has been adapted by both funding and non-funding agencies in Canada and abroad. For instance, it is the official definition employed at the World Health Organization (WHO 2006).

Another indicator of the significance of KT as a strategy and/or objective is presented in Figure 17 below. Examined in time-series¹², results of this scan provide telling indications of the growing importance of KT to research funders around the world.

Figure 17: Change in KT inclusion in mandate over time (2008 vs. 2011)

Figure 17 long description

Figure 17 shows that the inclusion of KT in the mandate of research funding agencies has either remained the same or increased in occurrence across the six regions/countries examined in this scan. All of the funding agencies contacted in 2011 offer some form of targeted KT funding. These findings indicate that CIHR is well aligned with a current global trend of promoting/enabling KT in health research funding. Considering the time-series nature of this data, and that CIHR has included KT in its mandate since 2000, it becomes apparent that CIHR has been ahead of the curve in promoting and enabling KT as a research funding agency strategy and objective.

The following statement by Australia's Chief Scientist helps to illustrate the importance and prominence of funding KT as a part of research funding around the world.

3.4 Who is using the KT Funding Program?

Another evaluative indicator of CIHR's role in promoting/enabling KT is an examination of program use. In other words, the CIHR KT Funding Program may only be as relevant as it is useful to CIHR's client-base. Although KUs are a required part of iKT funding opportunities, the lack of data on KUs in CIHR's EIS database required this analysis to focus only on researchers. Figure 18 provides an illustration of the demographics of KT researchers per KT funding opportunity. A reference group of CIHR open operating grant program (OOGP) researchers is included.

Figure 18: Researcher profiles (%)

In general, demographic data illustrates that KT grants are awarded to a different variety of researchers than is the case with the OOGP. For one, KT grants are held mainly by pillar II, III and IV researchers – whereas the OOGP is dominated by pillar 1 researchers. Secondly, KT grants are held by researchers of all experience levels, including new investigators (0-5 years experience) – the OOGP funds predominately more experienced researchers. Finally, across each KT funding opportunity, female grant holders outnumber male grant holders – OOGP data shows the reverse.

These demographic data highlight some further detail about who KT funding opportunity recipients are. Firstly, the majority of iKT researchers do report pillar III or IV as their primary research domain, however, 21% of KT researchers report pillar II as their primary research domain, and 9% report pillar I. The vast majority of pillar I KT funded researchers use end of grant KT funding opportunities DE and KTS¹³. Secondly, Figure 18 shows that although researchers awarded KT grants are often new investigators, many others are highly experienced; 46% report over 12 years of experience as an independent researcher and 27% report over 18 years of experience.

3.5 Summary findings

In addition, a review of researchers who have received KT grants shows that KT funding opportunities reach a broad range of researchers, and are particularly used by the three groups that are less represented in OOGP funded projects: female researchers, pillars III and IV researchers, and new investigators.

Ample evidence exists suggesting a lag or often complete failure in moving research evidence into practice (AHRQ 2001; Mitton et al. 2007). Since its inception through act of Parliament in 2000, CIHR has made KT its strategic response to this issue. A review of current GOC policy and strategic direction indicates a strong desire for research to be applicable to society, and the CIHR KT Funding Program is well suited to fulfill this aspiration.

An international scan of health research organizations illustrates the pursuit of KT by health research funding organizations is emerging around the world as an important strategic objective. It has also brought to light the leadership role CIHR has played in this process.

Methodology

To ensure that evaluation findings were robust and credible, and that valid conclusions were drawn about the performance of the programs, the evaluation used multiple methods and draws on both quantitative and qualitative evidence. A list of evaluation questions, indicators, and sources is available in Appendix B. For a comprehensive outline of the evaluation protocol, see: McLean et al. 2012.

References

Agency for Health Research and Quality. (2001). Translating research into practice (TRIP)-II. Retrieved February 11, 2012.

Canadian Institutes of Health Research. (2010). Knowledge Translation (KT) activities at CIHR. Retrieved October 18th, 2012.

Cargo, M., & Mercer, S. L. (2008). The Value and Challenges of Participatory Research: Strengthening Its Practice. Annual Review of Public Health, 29, 325–50

Cordero, C., Delino, R., Jeyaseelan, L., Lansang, M.A., Lozano, J.M., Kumar, S., Moreno, S., Pietersen, M., Quirino, J., Thamlikitkul, V., Welch, V.A., Tetroe, J., ter Kuile, A., Graham, I.D., Grimshaw, J, Neufeld, V., Wells, G., & Tugwell, P. (2008). Funding agencies in low- and middle-income countries: support for knowledge translation. Bulletin of the World Health Organization, 86(7), 524-534.

Chubb, I., (2012), Can Australia afford to fund translational research? Keynote Address to the BioMelbourne Network, April 3^rd 2012, Melbourne, Australia.

Dalziel, M., Roswell, J., Tahmina, T. N., & Zhao, X. (2012). Impact of Government Investments in Research & Innovation: Review of Academic Investigations. Optimum Online; The Journal of Public Sector Management, 42(2), 19–32.

Estabrooks, C.A., Norton, P., Birdsell, J.M., Newton, M.S., Adewale, A.J., & Thornley, R. (2008). Knowledge translation and research careers: Mode I and mode II activity among health researchers. Research Policy, 37(6-7), 1066–1078.

Gates Foundation. (2010). Global Health Strategy Overview [ PDF (252 KB) - external link ]. Retrieved May 24, 2012.

Gerring, J. (2007). Is there a (viable) crucial case method? Comparative Political Studies, 40(3), 231-253.

Government of Canada. (2000). Government of Canada bill C-13. Retrieved February 13, 2012.

Government of Canada. (2012). Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report [ PDF (4.6 MB) - external link ]. Retrieved April 15, 2012.

Government of Canada. (2012). Budget Plan: Budget 2012. Retrieved April 15, 2012.

Grand Challenges Canada. (2010). Integrated Innovation. Toronto, ON: McLaughlin-Rotman Centre for Global Health.

Industry Canada. (2007). Mobilizing Science and Technology to Canada's Advantage. Ottawa, ON: Publishing and Depository Services.

Israel, B.A., Schulz, A., Parker, E., & Becker, A.B. (1998). Review of community-based research: assessing partnership approaches to improve public health. Annual Review Public Health, 19,173-202.

Knowledge Translation Branch, CIHR (2010). Knowledge Translation 10 year Report (draft). Ottawa, ON.

Jensen, P., Roquier, J. P., Kreimer, P., & Croissant, Y. (2008). Scientists who engage with society perform better academically. Science and Public Policy, 35(7), 527-541.

Lavis, J. N., Lomas, J., Hamid, M., & Sewankambo, N. K. (2006). Assessing country-level efforts to link research to action. Bulletin of the World Health Organization, 84(8), 620-628.

Lomas, J. (2000). Using 'linkage and exchange' to move research into policy at a Canadian foundation. Health Affairs, 19(3), 236-240.

Macaulay, A., Sirett, E., Bush, P. (2011). Community-Based Participatory Research. In Oxford Bibliographies 2011. Oxford: Oxford University Press

McLean, R.K.D., Graham, I.D., Bosompra, K., Choudhry, Y., Coen, S.E., MacLeod, M., Manuel, C., McCarthy, R., Mota, A., Peckham, D., Tetroe, J., & Tucker, J. (2012). Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implementation Science, 57(7).

Mitton, C., Adair, C. E., McKenzie, E., Patten, S. B., & Perry, B. W. (2007). Knowledge transfer and exchange: Review and synthesis of the literature. The Milbank Quarterly, 85(4), 729-768.

Straus, S. E., Tetroe, J., & Graham, I.D. (2009). Defining knowledge translation. Canadian Medical Association, 181(3-4), 165-168.

Tetroe, J.M., Graham, I. D., Foy, R., Robinson, N., Eccles, M. P., Wensing, M., Durieux, P., Legare, F., Nielson, C.P., Adily, A., Ward, J.E., Porter, C., Shea, B., & Grimshaw, J.M. (2008). Health research funding agencies' support and promotion of knowledge translation: An international study. The Milbank Quarterly, 86(1), 125-155.

World Health Organization (WHO). (2006) Bridging the "Know–Do" Gap; Meeting on Knowledge Translation in Global Health. Geneva, Switzerland.

Appendices

Appendix A – Organizational Scan Participants

Appendix B – Survey Statistics

Appendix C – Funding Opportunity Descriptions

Synthesis

Background

Knowledge synthesis grants provide funding to researchers to produce scoping reviews or synthesis that meet the needs of decision makers or knowledge users in all areas of health using an iKT approach. They require that knowledge users should identify synthesis questions in collaboration with researchers so that the answers to these questions can inform policy, programs and practice. They are also expected to increase the capacity of researchers to identify new, relevant avenues for exploration that have not yet been investigated that respond to decision makers/ knowledge users' needs (CIHR 2010). Finally, synthesis grants are intended to promote the process of mutual learning between researchers and knowledge-users.

Launched first in 2004, CIHR invites all forms of knowledge synthesis. Qualitative, quantitative and mixed methods approaches are accepted, as well as syntheses of knowledge gained through observation, testing, or reviewing of texts. Scoping reviews are also accepted; these are projects that explore the literature available on a topic, identifying the key concepts, theories, sources of evidence, and gaps in the research (CIHR 2010). Synthesis funding opportunities fall under the iKT focus area. Applications go through a merit review process.

Resources

Competitions are launched twice a year by the Knowledge Translation Branch in partnership with various institutes and initiatives, along with external partners. The maximum amount awarded for a synthesis is $100,000 for one year. The maximum amount awarded for scoping reviews is $50,000 for one year.

Partnerships for Health System Improvement (PHSI)

Background

The first CIHR PHSI competition was held in 2005. The PHSI funding program supports teams of researchers and decision makers/knowledge users interested in conducting applied and policy-relevant health systems and services research that respond to the needs of health care decision makers. Partnerships can be project specific (partners that the researchers identify themselves with and with whom they negotiate) and/or competition specific (CIHR negotiated competition partnerships). This funding opportunity requires financial or in kind partner contributions (CIHR 2008).

Calls for applications for the PHSI program are launched annually. PHSI falls under the iKT classification and uses the merit review process to evaluate the applications (CIHR 2008).

Resources

The maximum amount awarded by CIHR for a single grant is $400,000 for up to three years (partnership contributions are in addition to the CIHR amount). A minimum of either 20% or 30%, depending on the province or territory, of the grant budget must come from external partner sources (i.e., non-CIHR funds). There is no limit to partner contributions, and in-kind contributions are recognized, especially where they reflect meaningful collaboration that will increase the likely success of the project. It should be noted that funding and contributions may be received from stakeholders who are not members of the grant team.

Knowledge to Action (K2A)

Background

K2A is designed to move knowledge into action by linking researchers and KUs and to increase the understanding of knowledge application through the process. By bringing both parties together, it is expected that research results will translate to actions that strengthen Canada's health care system and/or improve the health of Canadians.

K2A also aims to support the development, implementation and evaluation of cutting-edge KT research and approaches. As such, K2A establishes and strengthens common ground between the interests and expertise of the research community and the needs knowledge users. Applicants can request funding to support partnerships, knowledge and tools for implementation projects.

This program was first launched in 2005, and is currently launched annually. K2A falls under the iKT classification, and accordingly, all applications go through a merit review process.

Resources

The maximum CIHR contribution is $100,000 per year for up to two years. Applicants may increase funding for their proposal and further demonstrate the level of engagement of their partner(s) through cash or in-kind commitments, but a financial commitment from the partner is not a criterion for funding (CIHR 2007). Applicants are encouraged to apply for a renewal of their grant if they plan to scale up their implementation project, which might not be achievable within a two year time frame.

Dissemination Events (DE) & the KT Supplement (KTS)

Background

Meetings, Planning and Dissemination (MPD) grants are intended to provide support for meetings, planning and/or dissemination activities consistent with the mandate of CIHR and relevant CIHR Institutes, Initiatives or Branches.

From 2004–2007, the KSE (now KT) Branch ran a version of this program, called Workshop/Symposia/Conference Support in the branch, prior to the launch of the MPD pilot by the Research Portfolio in 2007. The successful pilot was transformed into the full current program in 2008. MPD offers three tools for funding: Dissemination events (DE), Planning Grants and the KT Supplement (KTS). This evaluation only covers DE and KTS¹⁵.

DE supports the organization of events focused on the communication of health research evidence. DE has three annual competitions (February, June and October) and undergoes a peer review process. Eligible activities for DE include:

• Education of groups such as patients, health professionals, community organizations, policy-makers, the general public;
• Education of stakeholders regarding partnership best practices;
• Knowledge dissemination that will inform practice, clinical care, policy and decision making;
• Publishing articles in open access journals not budgeted for in other applications, as part of a broader dissemination strategy.

KTS supports KT activities that follow a peer-reviewed grant/award where further dissemination is appropriate. KTS has three annual competitions (February, June and October) and uses a peer review process. Eligible activities for the KTS include:

• Development/maintenance/updating of websites;
• Production and distribution of written materials in various formats;
• Hiring of a knowledge broker or implementation facilitator/change agent;
• Development of plain language summaries;
• Development of knowledge exchange tools (e.g., educational CD-ROMs, decision support tools);
• Dissemination of research results through specialized publications as part of a broader KT strategy, and;
• Travel costs for a series of meetings/presentations (linkage and exchange activities) required to implement a broader KT strategy.

Resources

KTS and DE are non-renewable one year grants. However, multiple grants can be awarded to the same candidate in the same calendar year. KTS are funded up to $100,000, and DE are funded up to $25,000 (CIHR 2010).

KT science panel on the OOGP

Background

KT science is funded through the OOGP being assigned to the KTR panel for review. Funded projects must conduct research directed toward developing theory, evidence and innovation to define the determinants, implementation and uptake of health research evidence into practice. This includes grants which aim to improve KT to consumers, health practitioners and policy makers, to examine the role of organizations as KT vehicles, to determine how to improve knowledge uptake potential during the research process, to develop/evaluate KT tools and/or methods, to contribute to KT theory and to improve knowledge uptake (Knowledge Translation Branch 2010).

Resources

These competitions provide grant funding for up to five years and have no funding limit or specific requirements for team size or composition. In total, 85 KT science projects have been funded by the KTR panel since 2000 for a total expenditure of over $27,000,000.

Appendix D – Evaluation Matrix

Appendix E – Case Study Readings

Synthesis: Can Interprofessional Collaboration Provide Health Human Resources Solutions? A Knowledge Synthesis
Research leads: Dr. Esther Suter, Ms. Grace Mickelson (KU)

• Can Interprofessional Collaboration Provide Health Human Resources Solutions? A Knowledge Synthesis - Report [ PDF (799 KB) - external link ]
• Western Canadian Interprofessional Health Collaborative

Knowledge to Action: Need to know too
Research leads: Dr. Judith Bartlett, Dr. Catherine Louise Cook (KU)

• What is the comparative health status and associated risk factors for the Métis? A population-based study in Manitoba, Canada
• Manitoba Metis Federation

PHSI: National health partnership for reducing infections in NICU
Research leads: Dr. Shoo Lee, Dr. Elizabeth Whynot (KU)

• Evidence-based Practice for Improving Quality
• EPIQ Publications

KT Research Panel in the OOGP: Producing and communicating knowledge differently: Articulating the pathway to mental health care through artistic expression
Research NPI: Dr. Katherine Boydell

• Katherine Boydell
• Judging quality in arts-based health research: The case of the Ugly Baby [ PDF (997 KB) - external link ]

KT Supplement FO: Promoting Inpatient Cardiac Rehabilitation Referral
Research NPI: Dr. Sherry Lynn Grace

• Cardiac rehabilitation II: referral and participation

Appendix F – Glossary

Appendix G – Evaluation Working Group

Knowledge Translation Expertise

Ian Graham
Former CIHR VP of Knowledge Translation
Senior Scientist, Ottawa Hospital Research Institute

Martha MacLeod
Professor and Chair, School of Nursing, University of Northern British Columbia
Chair, CIHR Knowledge to Action Committee

Jacqueline Tetroe
CIHR Senior Advisor, Knowledge Translation

Adrian Mota
CIHR Manager, Process Stream

Yumna Choudhry
CIHR Senior Specialist

Stephanie Coen
Former KT Manager, CIHR Institute of Gender and Health

Ryan McCarthy
Former CIHR Director, Knowledge Translation

Andrea Smith
CIHR Manager, Major Initiatives

Evaluation Unit Project Team

Robert McLean
CIHR Lead Evaluator

Joanne Tucker
CIHR Junior Evaluator

Christopher Manuel
CIHR Junior Evaluator

Kwadwo (Nana) Bosompra
CIHR Evaluator

Appendix H – Evaluation Recommendations Crosswalk

Crosswalk of Evaluation Recommendation One with identified key elements of continued KT funding success

Recommendation 1

CIHR should invest the required resources to sustain its role in enabling KT. Given the decision by CIHR to integrate the KT funding opportunities into the open research suite, both of the proposed Project and Foundation Schemes of research should retain the key elements identified in the evaluation as responsible for the success of the current the standalone iKT and end of grant KT funding opportunities.

The following chart provides a crosswalk between what this evaluation has identified as key elements of KT success and important considerations for mainstreaming these into the open research suite. Both synergies and challenges are highlighted, and where appropriate recommendations for CIHR action are provided.

Date modified:

2013-09-24