Evaluation of the Regenerative Medicine & Nanomedicine Initiative
Final Report 2013
June 2013
Canadian Institutes of Health Research
160 Elgin Street, 9th Floor Address Locator 4809A
Ottawa, Ontario K1A 0W9 Canada
Canadian Institutes of Health Research
Acknowledgements:
This report was authored by Patrick MacGuire, Lead Evaluator, and Joanne Tucker, Junior Evaluator
The evaluation was carried out, from design to dissemination, by the Evaluation Working Group: Patrick MacGuire, Joanne Tucker, Eric Marcotte, and Susan Lalumière
Cover photo courtesy of National Research Council of Canada
Table of Contents
Executive Summary
CIHR’s Regenerative Medicine and Nanomedicine Initiative (RMNI) is a strategic investment of over $80M to support multi/transdisciplinary and high–impact research in the fields of regenerative medicine and nanomedicine. The initiative was launched in 2004 and has now ended, with the final funding competition held in 2010. CIHR’s decision to discontinue support for RMNI was made on the basis that research in the area of regenerative medicine and nanomedicine has matured to a point that it is highly competitive in the Open Operating Grant Program (OOGP) and can receive funding through the joint CIHR–NSERC Collaborative Health Research Projects Program (CHRP).
This evaluation assesses the performance of RMNI and the outcomes of CIHR’s investment in this initiative. The evaluation also examines the design and delivery of the initiative and provides findings that can inform future CIHR programming. In keeping with CIHR’s requirements to evaluate under the Treasury Board of Canada Secretariat’s 2009 Policy on Evaluation, issues related to relevance are also addressed including the continued need for RMNI and the sustainability of funding for researchers in the fields of regenerative medicine and nanomedicine.
Key Findings
This evaluation finds that RMNI has made a significant contribution to the Canadian health research enterprise in its targeted fields since its launch in 2004.
- RMNI–funded researchers have been responsible for 34% of Canadian publications in regenerative medicine and 21% of Canadian nanomedicine publications between 2004 and 2010. Furthermore, the scientific impact of the publications produced by RMNI–funded researchers, as measured by the Average of Relative Citations (ARC), was above the Canadian average and demonstrates that the initiative has been both selecting and funding research excellence over this period.
- The program design and delivery of the initiative are viewed as successful from the perspectives of researchers and RMNI’s partners and stakeholders.
- Successful applicants to RMNI express consistently high levels of satisfaction with the peer review process compared to CIHR benchmarks; and,
- Funding partners point to the initiative as an example of excellence in terms of the dedication and expertise of CIHR’s strategic initiative lead for RMNI.
- RMNI researchers have successfully leveraged grants and awards from CIHR and other research funders as a result of holding initiative funding. For every dollar invested in an RMNI catalyst grant, researchers leveraged $5.22 in other grants/awards; for team grants, this figure is $1.44 per dollar invested.
- For every $100K invested in an RMNI grant, 1.4 refereed journal articles were published and 2.1 research staff and trainees were supported.
- Around half of RMNI research grants (46%) resulted in patents/licenses, compared with a benchmark figure of 18% for CIHR’s Open Operating Grants Program (OOGP). Thirty–eight percent of RMNI grants resulted in intellectual property claims.
When considering the relevance and continued need for the initiative as well as the sustainability of funding for researchers in regenerative medicine and nanomedicine:
- The objectives of RMNI are in alignment with the Government of Canada's Science and Technology Strategy and CIHR’s 2009–2014 strategic plan and recent federal budgets continue to affirm the government’s commitment and role of CIHR in supporting advanced research and health research of national importance.
- RMNI–funded researchers expressed concern about future support for their projects and teams if the initiative were not renewed by CIHR. This was particularly the case with regard to multi/transdisciplinary research in regenerative medicine and nanomedicine. Partners were concerned that RMNI’s absence would result in a slowdown of research in both fields.
- International evidence shows that other countries continue to view regenerative medicine and nanomedicine as having critical importance. Worldwide publication activity in both fields has been increasing annually over time, particularly in nanomedicine and annual Canadian publication growth in both fields correlates highly with world trends. Regenerative medicine is, for example, a national health research priority in Singapore and the country is home to the Institute of Bioengineering and Nanotechnology, one of seven research institutes supporting public sector biomedical R&D. Singapore has attracted leading researchers from the United States and Europe to its institutions and laboratories, and in 2010, led the world in the scientific impact of publications in regenerative medicine and nanomedicine.
- This evaluation does however provide evidence that these once emerging fields have grown significantly. At a macro level, global publications in regenerative medicine and nanomedicine (combined) have grown from an annual rate of 3,381 in 2002 to 17,905 in 2010 – a growth rate of 430%. A similar picture can be seen in Canada where publications in regenerative medicine and nanomedicine have grown from 100 in both fields in 2002 to 628 in 2010, a growth of 530%. In contrast, Canadian health research publications increased from 15,679 in 2001 to 20,700 in 2009 – a growth of 32%.
- It is also apparent that RMNI–funded researchers have been successful in leveraging grants and awards after receiving initiative funding, including in CIHR’s highly competitive investigator–driven Open Operating Grant Program (OOGP).
- Among RMNI nominated principal investigators, 35 (52%) received OOGP grants (as a nominated principal investigator) and seven (10%) received Canada Research Chairs after having received RMNI funding.
- Nine (13%) RMNI nominated principal investigators had received support through the Collaborative Health Research Projects Program (CHRP) after being funded by the initiative, and there has been a $15M investment into the Centre for Commercialization of Regenerative Medicine (CCRM), which includes RMNI–funded researchers among its lead scientists and advisory group.
- Fewer than one in four RMNI–funded researchers say they would be unable to sustain their research program in the absence of future RMNI funding opportunities.
In addition to providing insight into the performance of the initiative itself, the following evaluation findings can inform decision making and assessments regarding current and future CIHR programming, particularly for programs aimed at funding multi/transdisciplinary research and research teams:
- The importance of ensuring good team coordination for large research teams through a hired project coordinator and/or trainee(s) and using regular videoconferencing to overcome barriers of geographical distance that can have a negative impact on team collaboration.
- Being involved in multi/transdisciplinary projects brings many benefits to research trainees; however, some trainees expressed concern that this can also potentially disadvantage future careers if academic institutions or employers are seeking those who have become more specialized in a single field.
- RMNI funding opportunities were intended to support the creation and enhancement of teams. Findings show that 83% of RMNI–funded teams involved some members who had worked together previously, although in no cases had all team members worked together previously. These findings demonstrate that few teams are formed purely as a result of responding to a funding opportunity; it is far more likely that existing teams who have worked together previously will be expanded or enhanced with new members.
- Both RMNI researchers and partners identified a need for an annual meeting for researchers with the purpose of making connections between research teams and exchanging information about research being conducted, including the management of research.
Conclusions
RMNI has made significant contributions to building Canadian research capacity and knowledge creation in the fields of regenerative medicine and nanomedicine. The program has been effectively designed and delivered.
The discontinuation of the initiative raises questions around the sustainability of research in these fields. Evidence from this evaluation is encouraging in that it shows a record of success among RMNI–funded researchers in leveraging other existing grants and awards. There may also be further opportunities for strategic funding for researchers in regenerative medicine and nanomedicine, for example through CIHR’s significant investments in networks for the Strategy on Patient–Oriented Research (SPOR) and the Epigenetics Strategic Initiative as well as funding for large scale research projects under the Personalized Medicine Initiative.
However, in the absence of RMNI funding, it will be important for CIHR to monitor Canada’s competitiveness in regenerative medicine and nanomedicine. If the country’s competitiveness declines, CIHR should assess the health of both fields in Canada through an examination of Canadian investment in this type of research as well as tracking the subsequent careers of RMNI principal investigators and trainees. CIHR should also offer direction to the regenerative medicine and nanomedicine research community on applying to other CIHR funding opportunities and initiatives providing support in these fields.
Recommendations
- Implement a communication strategy aimed at researchers working in the fields of regenerative medicine and nanomedicine that offers direction on applying to other CIHR funding opportunities and initiatives providing support in these fields.
- Conduct regular assessments with international benchmarks to determine the relative global position of Canada in the fields of regenerative medicine and nanomedicine in the absence of RMNI. If Canada’s competitiveness declines, ensure regular environmental scanning takes place to assess the ongoing health of both fields in Canada. This would include examining the Canadian investment in these research areas and tracking the subsequent careers of RMNI principal investigators and trainees.
- Ensure that future designs of programs relating to teams of researchers or networks take into account findings from the evaluation. It may, for example, be unrealistic to design programs that are expected to fund ‘newly formed’ teams, and there could be merit in including stronger requirements for dedicated research coordinators to aid success.
Management Response
Recommendation | Response (Agree or Disagree) | Management Action Plan | Responsibility | Timeline |
---|---|---|---|---|
1. Implement a communication strategy aimed at researchers working in the fields of regenerative medicine and nanomedicine that offers direction on applying to other CIHR funding opportunities and initiatives providing support in these fields. |
Agree |
CIHR will communicate the sunsetting of the initiative to both those funded by RMNI and in the wider community following the approval of this evaluation. A plan will be developed as to which stakeholders need to be informed and the channels used to inform them. One key element of this communication will be the alternative funding opportunities available to researchers, both in open and strategic CIHR programs as well as through other funders. A second element to communicate is that CIHR is aware of the importance of these fields and will be monitoring their ongoing progression relative to international benchmarks (see Recommendation 2). |
Chief Scientific Officer/Vice–President, Research and Knowledge Translation Portfolio |
Develop plan and communicate to the community – March – September 2013 |
2. Conduct regular assessments with international benchmarks to determine the relative global position of Canada in the fields of regenerative medicine and nanomedicine in the absence of RMNI. If Canada’s competitiveness declines, ensure regular environmental scanning takes place to assess the ongoing health of both fields in Canada. This would include examining the Canadian investment in these research areas and tracking the subsequent careers of RMNI principal investigators and trainees. |
Agree |
It is agreed that it will be important to put in place assessments to ensure that the sunsetting of this initiative will not have a detrimental impact on Canada’s performance in these key fields. As recommended in the evaluation, an approach to this will be to undertake a review of the extent to which those who were funded under this initiative are now receiving grants through CIHR’s Open Operating Grants program. Investment in these fields in other areas of CIHR’s strategic programming, for example, through the Epigenetics Initiative, can also be used. Analysis will include both dollar investment but also the types of projects funded and the number of former RMNI funded researchers receiving grants through these initiatives. More broadly, these types of approaches can be piloted for RMNI but could also work well for CIHR when considering the sunsetting of other strategic initiatives. The bibliometric scanning suggested would form part of this wider effort. |
Chief Scientific Officer/Vice-President, Research and Knowledge Translation Portfolio |
Initial assessment of RMNI researchers and projects relative to the Open Operating Grants Program and strategic investments to take place by end of Fiscal Year 2013–14. Produce a plan for regular environmental scanning of strategic investments by end of Fiscal Year 2013–14 |
3. Ensure that future designs of programs relating to teams of researchers or networks take into account findings from the evaluation. It may, for example, be unrealistic to design programs that are expected to fund ‘newly formed’ teams, and there could be merit in including stronger requirements for dedicated research coordinators to aid success. |
Agree |
As CIHR moves forward with its development of flagship Signature Initiatives, including the Strategy on Patient Oriented Research (SPOR), it will be critical to use the evidence from this evaluation to inform the future design of strategic programming. Those working on the design of such initiatives will ensure that all relevant evidence from this evaluation will be considered including the mechanics of how teams and networks form and the conditions required for their success. |
Chief Scientific Officer/Vice–President, Research and Knowledge Translation Portfolio |
Include as part of the design of new Signature Initiatives including SPOR – 2013/14 and ongoing |
Background
The Regenerative Medicine and Nanomedicine Initiative (RMNI)
The Regenerative Medicine and Nanomedicine Initiative (RMNI) is one of CIHR's major strategic initiatives. The goal of the initiative is to support multi/transdisciplinary and high–risk, high impact research approaches in the areas of regenerative medicine and nanomedicine. Between 2004 and 2010, RMNI has invested $82.3 Million in funding to researchers through two mechanisms: team grants and catalyst grants.
Within CIHR, RMNI is co–led by the Institute of Neurosciences, Mental Health and Addiction, the Institute of Genetics, and the Institute of Musculoskeletal Health and Arthritis. It also involves many of CIHR’s other Institutes and branches, as well as external partners such as the Juvenile Diabetes Research Foundation, the Canadian Space Agency, and the Stem Cell Network
RMNI Workshops and Meetings
RMNI has worked closely with a range of organizations such as Health Canada, the Natural Sciences and Engineering Research Council, and the National Research Council of Canada to sponsor workshops and meetings on topics of common interest. These meetings bring together experts and stakeholders from different domains, aligned along common themes relevant to the fields of regenerative medicine and nanomedicine, to form connections between fields, disciplines, and backgrounds. From 2003 to 2008, RMNI co–organized on average three workshops per year, with a total operational commitment of approximately $1 Million from all sources, including internal and external partners.
RMNI Competitions | 2004 | 2005 | 2006 | 2008 | 2009/10 | Total |
---|---|---|---|---|---|---|
RMNI Team Grants | $12.0M | $13.4M | $13.5M | $20.1M | $16.2M | $75.2M |
Team Grant Letters of Intent (LOI) | 25 | 59 | 38 | 52 | 61 | 235 |
Team Grant Relevance/Priority Reduction | 17 | 32 | 34 | 26* | 36* | 145 |
Team Grants Funded | 8 | 10 | 7 | 9 | 7 | 41 |
Team Grants Success Rate (Post LOI) | 47% | 31% | 21% | 35% | 19% | 28% |
RMNI Catalyst Grants | 300K$ | $1.2M | $1.2M | $1.9M | $2.3M | $6.9M |
Catalyst Grant Applications | 8 | 30 | 42 | 24** | 51** | 155 |
Catalyst Grants Funded | 5 | 8 | 8 | 7 | 8 | 36 |
Catalyst Grants Success Rate | 63% | 27% | 19% | 29% | 16% | 23% |
Total RMNI funds | $12.3M | $14.6M | $14.7M | $22.0M | $18.5M | $82.1M*** |
* LOIs reduced according to relative priority ranking, to maintain reasonable success rate ** RMNI partnered on Institute of Genetics Catalyst competition cycles *** RMNI also contributed funds through other funding competitions such as the Knowledge Synthesis Grant, Knowledge to Action Grant, and Seed Grant programs – total RMNI commitments are $82.3M |
RMNI Competitions | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | Total |
---|---|---|---|---|---|---|---|
CIHR Corporate (RMNI) | $75,000 | $75,000 | $50,000 | $30,000 | $17,269 | $10,957 | $258,226 |
CIHR Institutes | $40,000 | $0 | $5,000 | $5,000 | $25,000 | $25,000 | $100,000 |
External Partners | $70,000 | $60,000 | $65,000 | $65,000 | $45,000 | $76,000 | $381,000 |
Cost–recovery* | $0 | $40,000 | $50,000 | $55,000 | $80,000 | $25,000 | $250,000 |
Total RMNI funds | $185,000 | $175,000 | $170,000 | $155,000 | $167,269 | $136,957 | $989,226 |
* Hospitality costs paid in part or in total through workshop registration fees |
Photo: Image of Nanorods – Photo courtesy of Prof. Warren Chan, University of Toronto.
Evaluation Purpose
This evaluation is designed to provide valid, insightful and actionable findings about the performance of RMNI for CIHR’s senior managers, strategic leads and program management. As well, the evaluation will provide findings that can inform decision making and assessments regarding current and future CIHR initiatives and programs. The evaluation is also designed to meet CIHR’s requirements to the Treasury Board Secretariat (TBS) under the 2009 Policy on Evaluation. An evaluation of RMNI was requested by CIHR’s Scientific Council at their planning retreat in March of 2010.
Key Findings
Knowledge Creation in Regenerative Medicine and Nanomedicine
One of the primary objectives of CIHR’s Regenerative Medicine and Nanomedicine Initiative (RMNI) is to fund research projects with the potential for generating high impact results to ensure Canada’s strong and growing presence in the fields of regenerative medicine and nanomedicine. To help assess the extent to which this objective has been achieved, a bibliometric analysis1 was conducted on publications produced by the full population of RMNI–funded researchers (N=295) and non–funded applicants to RMNI (N=143), grouped according to RMNI funding received/applied for (team and catalyst grants), and the top 16 productive countries in the two targeted fields (including Canada). Publications relevant to regenerative medicine and nanomedicine were retrieved through a search query using specific U.S. National Library of Medicine Medical Subject Headings (MeSH). In addition, publications in core journals of regenerative medicine and nanomedicine were also included in the analysis (see Methodology section for a full description of the bibliometric analysis).
Global and National Publication Rates in Regenerative Medicine and Nanomedicine
To contextualize the performance of RMNI in terms of knowledge creation and scientific impact, it is useful to first briefly consider global and national publication rates for the fields of regenerative medicine and nanomedicine. Results presented in Figure 1 show a significant increase in the global number of publications in both fields over the period of 1997–2010, particularly in nanomedicine. Specifically, papers in both fields (combined) have grown from an annual rate of 3,381 in 2002 to 17,905 in 2010 – a 430% increase. Furthermore, results from a recent bibliometric study (Observatoire des sciences et des technologies, 2010) on ten research fields relevant to the mandate of RMNI’s co–lead Institute, the Institute of Neurosciences, Mental Health and Addiction, found that regenerative medicine and nanomedicine were among the fastest growing fields of those compared, both worldwide and in Canada.
As shown in Figure 2, publication activity in both fields in Canada has increased over time as well with growth rates similar to trends observed worldwide. A correlation analysis showed that annual publication production trends are highly correlated between the two fields as well as between Canada and the world (p<0.05). Canadian papers in both fields (combined) have grown from 100 in 2002 to 628 in 2010 – a growth of 530%. In contrast, the number of Canadian health research papers produced annually increased from 15,679 in 2001 to 20,700 in 2009 – a growth of 32%.
1997 | 1998 | 1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Regenerative medicine | 1,174 | 1,159 | 1,438 | 1,657 | 1,982 | 2,654 | 2,961 | 3,531 | 3,988 | 4,679 | 5,702 | 6,709 | 7,042 | 7,277 |
Nanomedicine | 39 | 41 | 67 | 94 | 248 | 738 | 1,170 | 2,297 | 3,978 | 5,252 | 6,580 | 7,898 | 9,071 | 10,909 |
Source: RMNI Bibliometric Data on World Publications
1997 | 1998 | 1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Regenerative medicine | 42 | 38 | 62 | 63 | 52 | 88 | 117 | 124 | 155 | 171 | 242 | 291 | 315 | 305 |
Nanomedicine | 1 | 0 | 1 | 2 | 3 | 13 | 37 | 76 | 148 | 175 | 242 | 282 | 281 | 335 |
Source: RMNI Bibliometric Data on Canadian Publications
Top Countries in Knowledge Creation and Scientific Impact
Figures 3 and 4 depict the publication volume (larger spheres indicate more publications produced), the Average of Relative Citations (ARC), and Specialization Index (SI) of the top 16 productive countries in regenerative medicine and nanomedicine for 2004–2010, the period of RMNI’s implementation (see Methodology section for full description of bibilometric indicators and Appendix for values by country). Overall, Canada ranks 9th on the total number of publications produced and 12th on specialization in each field – below the world average of one. For citation impact, Canada scored above the world average in each field ranking in the middle of the top 16 countries for regenerative medicine with a tie for 8th place (ARC of 1.15) and ranking 6th overall in nanomedicine (ARC of 1.04).
Results show that over the period 2004–2010, the United States was the world leader in knowledge creation in these fields, accounting for close to 40% of world publications produced in both regenerative medicine and nanomedicine. For citation impact, the United States leads in nanomedicine (ARC of 1.22) but ties for 4th in regenerative medicine behind Sweden (1st), Switzerland (2nd) and Singapore (3rd).
It is interesting to note that over the period of 2004–2010, Singapore has made significant contributions in terms of scientific impact (ranked 3rd in ARCs in both fields) and was the world leader in specialization in each field (particularly in nanomedicine with an SI of 3.21) based on production of a relatively low number of publications. A brief case study of Singapore’s approach to funding health research is presented in Figure 5.
Figure 3 – Scatterplot of ARC and SI for Top 16 Productive Countries in Regenerative Medicine 2004–2010
Source: Bibliometric Data on Top 16 Productive Countries
Figure 4 – Scatterplot of ARC and SI for Top 16 Productive Countries in Nanomedicine 2004–2010
Source: Bibliometric Data on Top 16 Productive Countries
Methodological Note: Regenerative Medicine and Basic Science Stem Cell Publications
Supported regenerative medicine fields within the scope of RMNI include tissue engineering, rehabilitation sciences as well as stem cell research pertaining to regenerative therapies. As the focus of RMNI was on translating regenerative approaches to health applications, papers on basic stem cell research were excluded from this analysis. However, a number of international comparator studies include all stem cell research under the regenerative medicine category. For example, a recent 2011 Thomson Reuters bibilometric study on global publications in regenerative medicine – that included basic science stem cell papers within their analyses – showed that these publications accounted for approximately 55% of the data set. This suggests that, had the RMNI bibliometric study included these papers, regenerative medicine publication volume would have doubled overall. The 2011 Thomson Reuters study also gives an opportunity to see how the inclusion of all stem cell research would have altered Canada’s ranking in the field among countries common to both studies – from 8th to 5th for citation impact (a tie with the United Kingdom); and from 9th to 8th for publication volume.
Although the Thomson Reuters study differs somewhat from the RMNI analysis in terms of the calculation and types of several bibliometric indicators used, the set of countries included and the time period covered, comparisons on relative rankings between countries common to both studies proved to be highly correlated (r=0.969, p<0.05 for publication volume rankings and r=0.864, p<0.05 for citation impact rankings) and supports the validity of the findings for the bibliometric analysis of regenerative medicine presented in this study. It also illustrates the relative strength of basic stem cell research in Canada, given the increased citation impact for Canada when these papers are included in the study.
Figure 5 – Singapore’s Approach to Health Research Funding
- Ranked 1st in specialization index and 3rd in average of relative citations in both regenerative medicine (~600 papers) and nanomedicine (~1,000 papers) over the period 2004-2010. For 2010, Singapore was the top ranked country in ARCs in both fields.
- Regenerative medicine is a national health research priority and the country is home to the Institute of Bioengineering and Nanotechnology, one of seven research institutes supporting public sector biomedical R&D. Singapore’s health research system is sustained by diverse funding sources across the public (63%) and private sectors (37%) with most funding schemes being commercialization driven.
- Provides an environment that is welcoming to intellectual property with favourable business and immigration conditions (low corporate taxes, 10 year tax exemptions and open immigration policies) that attract a high number of foreign investors and multinational companies.
- Has attracted leading researchers from the UK, USA, Sweden, Germany, and Japan to its laboratories and institutes.
- The Biomedical Research Council (BMRC) of the Agency for Science, Technology and Research (A*STAR), one of Singapore’s government funders, boasts a number of research facilities, most of which are available to all Singapore researchers regardless of research funding source, therefore reducing operating costs for basic research.
Source: Bibliometric data on Top 16 Productive Countries; Marjanovic & Chonaill (2010)
RMNI–Funded Researchers’ Contributions to Knowledge Creation
Having considered the global and national context, we turn now to the contribution of RMNI to knowledge creation in regenerative medicine and nanomedicine. As shown in Figures 6 and 7, RMNI–funded researchers were responsible for close to one–third of Canadian publications in both fields over the period of 2004–2010. In terms of each field, RMNI–funded researchers were authors on 34% of Canada’s total number of regenerative medicine publications and 21% of nanomedicine publications, percentages not unexpected given RMNI’s total overall funding success rate of 26%.
Canadian non-RMNI funded: 1053 Papers | RMNI team grant funded: 468 Papers | RMNI catalyst grant funded: 82 Papers | |
---|---|---|---|
Percentage | 66 | 29 | 5 |
Canadian non-RMNI funded: 1214 Papers | RMNI team grant funded: 210 Papers | RMNI catalyst grant funded: 115 Papers | |
---|---|---|---|
Percentage | 79 | 14 | 7 |
Source: Bibliometric Data on Canadian and RMNI–Funded Researchers (N=295)
It should be noted that the bibliometric analysis in this evaluation includes data for regenerative medicine and nanomedicine publications produced by researchers funded by RMNI at some point after the initiative’s inception. Although the period of publication used throughout the bibilometric analysis aligns with the overall lifecycle of RMNI (2004–2010), funded researchers may have published articles in either field prior to, or after having concluded, their RMNI grant. As a result, direct attribution between RMNI funding and publication data cannot be made.
Scientific Impact of RMNI–Funded Researchers Compared with Canadian Averages
Evidence from this evaluation demonstrates that RMNI has been attracting and funding excellent researchers in terms of scientific impact of their publications.
As shown in Figure 8, publications in regenerative medicine produced by RMNI–funded researchers generally have a higher scientific impact (based on ARC) than the average for Canadian health researchers in this field. Those funded through RMNI catalyst grants generally achieved higher average citation scores than both Canadian and team grant–funded researchers, particularly over the period 2009–2010. It should be noted that the overall average citation scores for Canada presented in Figures 8 and 9 were calculated based on publications with at least one Canadian author and as such also includes papers authored by RMNI researchers.
With regard to nanomedicine (Figure 9), publications produced by both RMNI–funded team and catalyst researchers generally achieved higher average ARC scores than those for Canadian health researchers over the period 2006–2010. Within this field, RMNI team grant researchers tended to outperform both Canadian and catalyst grant researchers in terms of their scientific impact, achieving a peak score in 2010 with an ARC of 2.24.
Group | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 |
---|---|---|---|---|---|---|---|
Canada | 1.02 | 1.12 | 1.15 | 1.05 | 1.14 | 1.24 | 1.23 |
RMNI team grant funded | 1.21 | 1.15 | 1.17 | 0.98 | 1.25 | 1.48 | 1.52 |
RMNI catalyst grant funded | 1.48 | 1.05 | 1.22 | 1.33 | 1.44 | 2.62 | 2.49 |
Source: Bibliometric Data on Canadian and RMNI–Funded Researchers (N=295)
Group | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 |
---|---|---|---|---|---|---|---|
Canada | 1.21 | 0.79 | 0.98 | 1.00 | 0.99 | 0.98 | 1.27 |
RMNI team grant funded | 0.93 | 1.14 | 1.69 | 1.45 | 1.45 | 0.76 | 2.24 |
RMNI catalyst grant funded | 1.36 | 0.49 | 1.21 | 1.25 | 1.46 | 1.11 | 1.57 |
Source: Bibliometric Data on Canadian and RMNI–Funded Researchers (N=295)
When the two fields are combined, both RMNI team and catalyst grant–funded researcher publications achieved an overall ARC score of 1.39 and 1.57 respectively, compared to an ARC of 1.11 for Canada over the period of 2004–2010 (p<0.052). Similarly, bibliometric data shows that researchers funded through CIHR’s Open Operating Grant Program (OOGP) produce papers with higher impact than Canadian health research publications. Specifically, supported papers3 published over the period of 2001–2009 by 1,125 researchers who received funding from OOGP between 2000 and 2007 resulted in an overall ARC score of 1.51, above the overall ARC of 1.24 for Canadian health research publications over the same time period4 (p<0.055).
Knowledge Creation and Impact of RMNI–Funded Compared to Non–Funded Researchers
Bibliometric results presented in Table 3 reveal that RMNI–funded researchers achieved higher citation impact scores than unsuccessful applicants in both fields although in only one case was a statistically significant difference observed. Furthermore, funded researchers published approximately 2.7 times the number of papers in total than unsuccessful applicants when grant type and research fields are combined. Of note, unsuccessful applicants are researchers who submitted RMNI applications that were rated through peer review as fundable but were never successful in receiving initiative funding.
For RMNI team grant–funded researchers, the citation impact of their publications in regenerative medicine over the period of RMNI’s implementation (2004–2010) was 1.29 compared to 1.27 for unsuccessful team grant applicants. For nanomedicine, team grant–funded researchers achieved an ARC of 1.48 compared to 0.96 for unsuccessful applicants. However, differences in ARCs between the two groups in either field were not statistically significant. Team–funded researchers also published 3.1 times more regenerative medicine publications and 1.8 times more nanomedicine papers than unsuccessful team grant applicants.
For catalyst grant researchers, the average citation score over the six year period was 1.88 for funded researchers compared to 1.00 for non–funded researchers in regenerative medicine (ARC differences between groups not statistically significant). For nanomedicine, RMNI–funded catalyst researchers had an ARC score of 1.28 compared to 0.62 for non–funded researchers (p<0.056). In terms of publication volume, funded catalyst researchers published 3.9 times more regenerative medicine papers and 3.7 times more nanomedicine publications than unsuccessful catalyst grant applicants.
|
Regenerative Medicine 2004–2010 | Nanomedicine 2004–2010 | |||
---|---|---|---|---|---|
ARC | Papers | ARC | Papers | ||
RMNI Team Grant Funded | (N=225) | 1.29 | 468 | 1.48 | 210 |
RMNI Team Grant Non–Funded | (N=98) | 1.27 | 152 | 0.96 | 115 |
RMNI Catalyst Grant Funded | (N=70) | 1.88 | 82 | 1.28 | 115 |
RMNI Catalyst Grant Non–Funded | (N=45) | 1.00 | 21 | 0.62 | 31 |
Source: Bibliometric Data on RMNI–Funded (N=295) and Non–Funded Researchers (N=143) |
Publications Produced from RMNI–Funded Research
Results presented in Table 4 show the average number of publications resulting from RMNI grants. Overall, RMNI team grants resulted in an average of 20.8 refereed journal articles published compared to 2.0 for catalyst grants (p<0.017) and an average of 3.4 books/book chapters compared to 0.2 for catalyst grants (p<0.017).7
Results should be viewed in context of differences between grant types: the majority of catalyst grants surveyed had expenditures between $140–$260k, were two to three years in duration, and involved one or two researchers. In contrast, team grants surveyed had expenditures of $1–2M, involved an average of eight researchers, and were mostly five years in duration at time of survey.
As the majority of nominated principal investigators surveyed indicated that their RMNI–funded research was biomedical (see Appendix for profile of surveyed research), a benchmark comparison on publications resulting from biomedical researchers funded through CIHR’s Open Operating Grants Program (OOGP) was conducted. Overall, OOGP–funded researchers produced an average of 8.1 refereed journal articles and 1.0 books/book chapters per grant. The average amount committed to an OOGP grant included in the benchmark comparison was just over $300k and the majority of grants (66%) were three years in duration. Of note, the OOGP grants were awarded over the period of 1991–2008 and hence historical and contextual factors such as variation in funding amounts over time may have a limiting effect on the comparability of OOGP and RMNI supported research.
RMNI Overall (N=26) |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
OOGP Grant (N=561) |
|
---|---|---|---|---|
Average number of: | Mean ± Std Dev | Mean ± Std Dev | Mean ± Std Dev | Mean ± Std Dev |
Refereed journal articles published | 12.1 ± 13.7 | 2.0 ± 1.2 | 20.8 ± 13.6 | 8.1 ± 8.8 |
Books/Book chapters published | 1.8 ± 3.1* | 0.2 ± 0.4 | 3.4 ± 3.6** | 1.0 ± 2.4 |
Reports/Technical reports published | 0.3 ± 3.6* | 0.0 ± 0.0 | 0.6 ± 1.2** | 0.2 ± 1.8 |
Source: Survey of RMNI–Funded Researchers; OOGP Research Reporting System Data * based on N=25 ** based on N=13 |
To account for the differences between RMNI team and catalyst grants in terms of dollars expended, duration, and number of researchers involved, the average number of journal articles published per grant was normalized by dividing the total number of articles per grant by grant expenditures and duration8 at time of survey as well as the number of researchers involved at time of application (Table 5). As a result, catalyst grant researchers produced an average number of 1.3 journal articles per $100k as compared to 1.4 articles for team grants (differences between groups not statistically significant). Additionally, 1.1 articles per year of grant were produced from catalyst researchers versus 4.4 for team grant researchers (p<0.017) and controlling for team size resulted in team grants producing 2.8 articles per researcher versus 1.4 for catalyst grants (p<0.017).9
Normalization applied to biomedical OOGP grants in terms of dollar amount committed (over grant lifespan) resulted in an average of 2.6 journal articles published per $100k while 2.3 journal articles were published per year of grant (Table 5).
RMNI Overall (N=26) |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
OOGP Grant (N=510; N=531) |
|
---|---|---|---|---|
Average number of: | Mean ± Std Dev | Mean ± Std Dev | Mean ± Std Dev | Mean ± Std Dev |
Refereed journal articles by $100k expended (at time of survey) |
1.5 ± 1.9 | 1.3 ± 0.8* | 1.4 ± 0.9 | 2.6 ± 2.8** |
Refereed journal articles by grant duration (at time of survey) |
2.8 ± 2.6 | 1.1 ± 0.6 | 4.4 ± 2.7 | 2.3 ± 7.1*** |
Refereed journal articles by researchers involved (at time of application) |
2.1 ± 1.5 | 1.4 ± 1.1 | 2.8 ± 1.6 | – |
Source: Survey of RMNI-Funded Researchers; OOGP Research Reporting System * based on N=10 grants with expenditures >= $100k ** based on N=510 grants with commitments of >= $100k *** based on N=531 grants at least 1 year in duration |
RMNI Return on Investment
A ‘return on investment’ analysis was conducted on a sample of 26 RMNI grants (based on dollar amount expended at the time of survey) on several key metrics: involvement of researchers, staff and trainees, refereed journal articles produced, and funding leveraged.
As shown in Table 6, in terms of return on investment per $100k, catalyst grant researchers produced 1.2 articles, involved 4.3 research staff/ trainees and leveraged 1.2 grants/awards. For every catalyst grant dollar invested, researchers were able to leverage $5.22 dollars in grants/awards. With regards to team grants, $100k invested resulted in 1.4 articles, 1.9 research staff/trainees and 0.3 leveraged grants/awards. For every dollar invested, team grant researchers leveraged $1.44 dollars in grants/awards.
Benchmark data on ‘return on investment’ for CIHR’s Open Operating Grants Program (OOGP) shows that biomedical researchers awarded OOGP grants over the period of 1991–2006 produced 2.8 articles and trained 2.6 research staff/trainees per $100k (Table 6).
This type of comparison between team and catalyst grants should be treated with caution given contextual factors and potential confounds related to differences between funding tools and areas of research supported. It does however provide insight into the results of return on investment at a basic level for a strategic initiative and its funding mechanisms and provides a benchmark for future CIHR evaluations and studies.
RMNI Overall (N=26) |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
OOGP Grant (N=440**) |
|
---|---|---|---|---|
Total: | ||||
Number of grants sampled | 26 | 12 | 14 | 440 |
Dollar amount expended (at time of survey) | $22,237,859 | $1,967,549 | $20,270,309 | $118,090,449* |
Number of researchers involved (at time of application) | 134 | 22 | 112 | – |
Number of research staff and trainees involved | 472 | 84 | 388 | 3,019 |
Number of refereed journal articles published | 315 | 24 | 291 | 3,300 |
Number of grants and awards leveraged | 89 | 23 | 66 | – |
Dollar amount of grants and awards leveraged | $39,366,929 | $10,270,288 | $29,096,641 | – |
Per $100K: | ||||
Number of research staff and trainees involved | 2.1 | 4.3 | 1.9 | 2.6 |
Number of refereed journal articles published | 1.4 | 1.2 | 1.4 | 2.8 |
Number of grants and awards leveraged | 0.4 | 1.2 | 0.3 | – |
Per 1$: | ||||
Dollar amount of grants and awards leveraged | $1.77 | $5.22 | $1.44 | – |
Source: Survey of RMNI-Funded Researchers; CIHR Administrative Database * Dollar amount committed ** 17 cases were excluded due to an indication of no response across all RRS categories fo research staff and trainees |
RMNI’s Influence on the Development of Regenerative Medicine and Nanomedicine
A majority of RMNI–funded researchers feel that the initiative has had a positive influence on the development of regenerative medicine (80%) and nanomedicine (55%) in Canada (Figure 10). Fewer expressed the same opinion about the initiative’s influence on both fields internationally (54% regenerative medicine; 35% nanomedicine). It should be noted that over one third of researchers indicated “don’t know/not applicable” responses in terms of RMNI’s influence on nanomedicine, both in Canada and abroad. Furthermore, the majority of funded researchers interviewed indicated that they were not sufficiently aware of the initiative’s influence outside of their own research and could only comment on general areas in which RMNI was impactful. As such, the data presented in Figure 10 should be treated with caution.
Figure 10 - Influence of RMNI on the Development of the Fields of Regenerative Medicine and Nanomedicine
Source: Survey of RMNI–Funded Researchers (N=72)
RMNI–Funded High Impact Research Projects
Photo: RMNI Case Study: Prof. John Pezacki (pictured in back) - Photo courtesy of CIHR.
The impacts of health research are far broader than what can be measured through bibliometric analyses of publications. To provide a more in–depth analysis of the wider impacts and benefits of RMNI–funded research, three case studies of projects that demonstrated high impact, breakthrough results are described in detail below.10
For each of the case profiles, the overview section provides the context and background of the research project including the research issue(s) being addressed as well as the objectives of the projects. The impacts section details the successes that were achieved, while the factors that led to success are presented under pathways to results. The role of RMNI funding provides insight into the importance of initiative support to these projects and researchers.
Understanding and Manufacturing Quantum Dots for Biological & Medical Imaging
Warren Chan
Associate Professor, Institute of Biomaterials and Biomedical Engineering
University of Toronto
Overview
Materials, systems, and particles smaller than 100 nanometers (nm) can have unique optical and electronic properties. These allow researchers to engineer new tools to probe biological systems, and to detect and treat diseases. Quantum dots are one type of nanotechnology that emit light of different colors by changing the dots size from 2 to 8 nm. As such, they may be ideal contrast agents for biomedical imaging of diseases in the body, tissues, or cells. Prof. Chan’s team originally set out to develop quantum dots for tumour targeting but their initial studies showed that targeting was inefficient, as only about 3% of qdots would enter the tumour. As a result, the team shifted focus to understanding how the surface chemistry, size and shape of a nanoparticle affect targeting. This advancement in knowledge would enable the rational engineering of nanoparticles to target diseases.
Impacts
Photo: Quantum Dots – Photo courtesy of Prof. Warren Chan.
The team was able to demonstrate that the size, shape, and surface chemistry of nanoparticles affect their tumour targeting efficiency. As well, the group demonstrated the need to quantify the amount of nanoparticles targeting tumors. While quantum dot technology was not translated, as initially proposed, the outcome of the research had a major impact in the nanomedicine community with one of the published studies having been cited over 1000 times and four other studies receiving over 100 citations since their publication. Additionally, Prof. Chan’s team received many requests for nanomaterials during the course of the project and the group created the Canadian biotechnology company Cytodiagnostics to commercialize nanomaterials. Currently, the company generates positive revenues and the team’s quantum dot nanomaterials are sold through Sigma–Aldrich globally as well as a number of distributors in many countries around the world.
Pathways To Results
A team of researchers with diverse expertise in biomedical engineering, medical biology, pharmacology, and pathology brought different perspectives to solving research problems. Strong leadership and support from Prof. Chan including the ability to change the team’s direction when needed as well as the willingness of collaborating professors to allocate time to mentor a core group of students (up to 42 in total) were factors that contributed to success.
Role Of Rmni Funding
RMNI provided a mechanism to fund a nanomedicine project in technology and health that would lie outside the boundary of most research funding programs in Canada. The work produced from the RMNI grant has been a major part of Prof. Chan’s career and accounts for approximately 30% of his total research publications.
Improving Corneal Transplantation through Tissue Regeneration & Femtosecond Laser Technology
Isabelle Brunette
Professeure titulaire, Département d'ophtalmologie
Université de Montréal
Overview
Dysfunction in the endothelial layer of the cornea leads to blindness and severe pain. Currently, endothelial dysfunction is the leading cause for corneal transplantation, being responsible for 42% of the 50,000 corneal transplantations performed every year in North America. The purpose of Dr. Brunette’s project, which involved the Département d'ophtalmologie at the Hôpital Maisonneuve–Rosemont in Montréal, QC, the Institut national de recherche scientifique (INRS) in Varennes, QC and the Laboratoire d’organogénèse experimentale (LOEX) in Québec, QC, was to improve the functional results of corneal transplantation for endothelial dysfunction through the use of tissue engineering and femtosecond laser technology.
Impacts
Photo: Microcavitation bubbles generated by femtosecond laser ablation in human cornea – Photo courtesy of Dr. Isabelle Brunette.
Through RMNI funding, Dr. Brunette’s team successfully built on a $20 Million Canada Foundation for Innovation award to develop and adapt a femtosecond laser into a surgical suite at Montréal’s Hôpital Maisonneuve–Rosemont. The laser cuts only the layer of donor tissue needed for transplantation (instead of the entire cornea as in standard transplantations) and with exact matching dimensions. The team also set out to use tissue engineering technology to grow patients’ own endothelial cells in culture for transplantation back into the patient’s eye. Preclinical studies are quite conclusive that the procedure will reduce the demand on eye banks for donor corneas and positively impact access to eye care due to decreased exclusion criteria for donor tissue. Furthermore, it will eliminate the risk for rejection since the patient’s own cells will be used to tissue engineer the corneal transplant. Dr. Brunette and her team are the only researchers in the world who have been successful in culturing cells from sick patients with Fuchs corneal dystrophy without genetic manipulation. The team was also the first to demonstrate the regenerative potential of these cells by using them to tissue engineer a new cornea that was successfully transplanted in a living eye. A socioeconomic benefits analysis conducted by a pharmacoeconomist from the group demonstrated that the proposed techniques may provide better results at a similar cost (due to improved clinical outcomes, improved recovery time, and reduced waiting times) compared to traditional corneal transplantation procedures. The results of the research have also been published in veterinary journals demonstrating how corneal transplant methods developed for humans can be applied to animal care. Dr. Brunette has received international attention for her work.
Pathways To Results
Success was credited to having access to staff working in the institutions where the research is being conducted and innovative researchers who could work in a trans/multidisciplinary environment. Collaboration between a range of expertise including clinicians, surgeons, and other researchers also helped advance the project.
Role Of Rmni Funding
The research could not have been carried out without a trans/multi–disciplinary team involving ophthalmologists, physicists, tissue engineers, economists, and clinicians and the RMNI team grant provided the resources necessary to bring these different skillsets together.
Applying CARS Spectroscopy to Improve the Study of the Molecular Determinants of Disease
John Pezacki
Senior Research Officer, Steacie Institute for Molecular Sciences
Overview
Early diagnosis has the potential to increase survival rates for people with chronic diseases. Sensitive molecular imaging techniques based on biophotonics can be used to detect the early signs of disease far before signs are present in a given tissue. Prof. Pezacki’s team, which involved researchers working in chemistry, biophotonics, and molecular imaging, set out to develop an innovative microscope using Coherent Anti–Stokes Raman Scattering (CARS), a non–linear optical spectroscopy that involves sending laser pulses down a microscope every femtosecond – one millionth of one billionth of a second – to "fingerprint" the molecular vibrations of cell components and create images of them. The CARS technique enables the study of the molecular determinants of disease without the use of dyes or other labeling agents that are invasive and destructive to cells and tissues. The project also involved an exploration of clinical applications of the CARS microscope in the early diagnosis of disease.
Impacts
Photo: Label–free multimodal CARS microscopy of an atherosclerotic rabbit aorta – Photo courtesy of Prof. John Pezacki.
During the initial phase of the team’s work, it was discovered that there was no suitable existing CARS hardware for their biomedical imaging approach. To deal with this issue, the team's technology development group, led by Dr. Albert Stolow, simplified a CARS technique developed at Harvard University and created a more cost efficient approach that could be applied beyond a controlled laboratory environment in locations such as hospitals, clinics and doctor’s offices. More than 20 peer reviewed papers have been published just from the team at the NRC alone and papers based on the research supported by the RMNI grant are still being published including a review in Nature Chemical Biology summarizing the recent applications of CARS microscopy towards understanding important diseases such as hepatitis C virus infections. The team’s success also led to a commercial collaboration with Olympus, a multinational company specializing in cameras, research and clinical microscopes. The team used an existing microscope that is sold by Olympus as a base for the development of their CARS microscope technology (which functions as an attachment). Of significance, Olympus had previously been unsuccessful in its attempts to add CARS functionality to its microscope. Since the team’s technology was compatible with an Olympus device, a partnership was created and the CARS microscope is now sold as an add–on to the Olympus FluoView FV1000–MPE microscope. An NRC–Olympus CARSLab Microscopy Facility was launched in 2009 to educate the wider biomedical community on the benefits of this technology and hence help to provide the Canadian health system with cost–effective state–of–the–art medical diagnostic technologies.
Pathways To Results
The sharing of expertise in a multidisciplinary team and the inclusion of students in the project due to RMNI funding (whom the lead researchers accessed through their adjunct professorships) were significant factors that contributed to success. The involvement of the end users – the clinicians – at the beginning who provided input on the development of the CARS microscope helped the team to better understand their end users’ needs and focus their efforts more efficiently.
Role Of Rmni Funding
Prof. Pezacki emphasized the importance of the RMNI funding mechanism as it enables multidisciplinary teams to be brought together more efficiently than having to apply for multiple grants to achieve the same results. The RMNI funding also enabled Prof. Pezacki to assume a leadership role in the collaboration at NRC and include students in the project. Overall, the team grant sped up the research process and strengthened the research outputs by bringing the innovators, the tool builders, and the end–users together at the start.
DK/NA | No extent at all (1) | 2 | 3 | 4 | Significant extent (5) | |
---|---|---|---|---|---|---|
Percentage | 4 | 0 | 3 | 14 | 29 | 50 |
Source: Survey of RMNI–Funded Researchers (N=72)
Although a total of six RMNI–funded projects were selected for case studies based on their high impact results (with three profiled in this report), a large majority of RMNI–funded researchers feel that their research resulted in major achievements (Figure 11). As is shown in Figure 12, a range of factors were identified by researchers as enabling or hindering the success of their projects. Several enabling factors identified are core to the program theory and intent of the initiative, including the multi/transdisciplinary nature of the research projects and the involvement of end users early in the research process. As one catalyst grant NPI noted: “le projet n'aurait pu voir le jour sans une approche multidisciplinaire. On a beaucoup appris l'un de l'autre [et] les expertises étaient complémentaires et essentielles.” Far fewer factors were identified by researchers as inhibitors to the success of their funded projects; those identified related to a lack of renewal funding and, in the case of a few team grant projects, having insufficient funds to engage researchers full–time or conduct knowledge dissemination activities.
Figure 12 – Identified Factors That Enabled and Hindered the Success of RMNI-Funded Projects
Factors Enabling Success
- Group of researchers collaborating as a team
- Multi/transdisciplinary nature of the research
- Personal qualities of team members
- Leveraging of additional funding and support
- Involvement of student trainees
- Involvement of end users early in the research process
- Development of partner/stakeholder relationships including industry support
Factors Hindering Success
In reference to RMNI team grants:
- Lack of renewal funding
- Insufficient funding amount to engage researchers full-time or carry out knowledge dissemination activities
Source: Interviews with RMNI-Funded Researchers (N=23); Interviews with RMNI Case Study Participants (N=29).
Research Team Collaboration
As outlined in CIHR’s 2009–14 strategic plan, the Health Research Roadmap (CIHR, 2010), one of the organization’s core values involves the promotion, encouragement, and appreciation of collaboration among researchers in Canada and internationally. As noted, one of the explicit objectives of RMNI is to fund the creation or further development of research teams undertaking collaborative trans/multidisciplinary research that will lead to enhanced approaches to understanding and resolving regenerative medicine and nanomedicine health issues.
RMNI funding opportunities were intended to support the creation and enhancement of teams (projects involving at least three researchers) and evaluation findings (Table 7) indicate that the majority of RMNI–funded teams were enhanced through funding as 83% involved some members who had worked together previously. According to interviews with RMNI team and catalyst nominated principal investigators11, in many cases, a prior history of collaboration among team members was viewed as a significant factor in the success of their project and in several instances the idea for their RMNI project originated from discussions between members of their team.
The meaning of ‘creating’ a team is also open to some interpretation. Findings show that in no case had the entire team worked together previously, demonstrating that RMNI has not simply been refunding existing teams that had already been formed for previous projects.
RMNI–Funded Teams Profile: | RMNI Overall (N=18) |
RMNI Catalyst Grant (N=6) |
RMNI Team Grant (N=12) |
---|---|---|---|
Previous experience of teams at the time of application to RMNI: | |||
The entire team worked together previously | 0% | 0% | 0% |
Some of the team worked together previously | 83% | 67% | 92% |
None of the team worked together previously | 17% | 33% | 8% |
Source: Survey of RMNI–Funded Researchers |
RMNI Multi/Transdisciplinary Research
Interdisciplinary teams have been described as “the defining feature of the scientific endeavor in the twenty–first century” (Kessel et al., 2008). The findings presented in Figure 13 confirm that from an RMNI researcher perspective, there is a value–add to this type of collaboration in terms of research outcomes.
For a majority of researchers, operating in a multi/transdisciplinary team enabled them to achieve research outcomes that would not have occurred without this type of collaboration (84%). Almost all RMNI–funded researchers would participate in this type of research again (97%) and encourage others to do so (95%) based on their experiences.
As researchers who have applied to undertake multi/transdisciplinary research, this group would be expected to have generally positive views on the benefits of that approach going into their project. The overwhelmingly positive responses at the end of the projects show that being funded by RMNI is likely to have reinforced these initial views.
RMNI researchers’ opinions on team collaboration were also generally positive with 86% reporting that their collaboration was effective and 80% indicating that their RMNI grant facilitated more collaboration with researchers from different disciplines than would have occurred through other grant funding (Figure 14).
Figure 13 – RMNI-Funded Researchers’ Opinions on Multi/Transdisciplinary Research
Source: Survey of RMNI–Funded Researchers (N=64*)
* RMNI–funded researchers involved in research teams (projects with three or more researchers involved)
Figure 14 – RMNI-Funded Researchers’ Opinions on Team Collaboration
Source: Survey of RMNI–Funded Researchers (N=64*)
* RMNI–funded researchers involved in research teams (projects with three or more researchers involved)
Key Factors for Effective Collaboration
RMNI–funded researchers and case study participants who were interviewed noted various factors that enabled and/or hindered effective team collaboration (Figure 15). For example, the importance of ensuring good team coordination for large teams through a hired project coordinator and/or trainee(s) was noted as extremely important since one of the key challenges faced in these projects was the amount of time required for NPIs to effectively manage their group.
As one RMNI team grant NPI, whose team involved many researchers and trainees, explained:
Additionally, using regular video teleconferencing to overcome barriers of geographical distance was also frequently mentioned by researchers as a key factor to ensuring effective collaboration.
Figure 15 – Identified Factors That Enabled and Hindered Effective Team Collaboration
Factors Enabling Effective Collaboration
- Regular group communication/meetings
- Coordination of teams via trainees and/or a hired project coordinator
- Setting common goals and clear objectives
- Personal qualities of team members
- Identification of appropriate team members
- Previous collaborations among team members
- Close geographical approximation of team members
- Use of videoconferencing software such as Skype to overcome geographical dispersion of team
- Presence of complementary expertise
- Mutual respect for team members and disciplines
Factors Hindering Effective Collaboration
- Required level of effort or adaptability across team members
- Required understanding of different disciplinary languages represented on the team
- Required time to manage the team
- Geographical dispersion of team members limiting face-to-face interactions
Source: Interviews with RMNI–Funded Researchers (N=23); Interviews with RMNI Case Study Participants (N=29)
International Collaboration Rates
Recognizing the importance of working at the international level, the CIHR Act states that “Canada should be an internationally acknowledged leader in contributing to the global advancement of health research" (Canadian Institutes of Health Research Act, 2000, p.1). To measure the extent of international scientific collaboration on regenerative medicine and nanomedicine publications authored by Canadian and RMNI–funded researchers over the period of 2004–2010, the number of papers with at least one author with a foreign country address was divided by the total number of papers to arrive at a percentage of international collaboration.
Results presented in Table 8 show that international collaboration rates for papers published by RMNI–funded researchers were higher than the world percentage in both fields but lower than Canada, particularly in nanomedicine. Both RMNI funded team and catalyst researchers achieved higher international collaboration rates in regenerative medicine than in nanomedicine. Canadian researchers achieved a high international collaboration rate of 56% in regenerative medicine (ranked 2nd in the world) but ranked lower in terms of nanomedicine with a collaboration rate of 41% (placing 8th among the top 16 productive countries in the field).
Regenerative Medicine 2004–2010 | Nanomedicine 2004–2010 | |
---|---|---|
International Collaboration Rate | ||
World | 17% | 19% |
Canada | 56% | 41% |
RMNI Team Grant Funded | 44% | 24% |
RMNI Catalyst Grant Funded | 52% | 22% |
Source: Bibliometric Data on Top 16 Countries and RMNI–Funded Researchers (N=295) |
Knowledge Translation
Knowledge translation (KT) is a fundamental part of CIHR’s mandate and is a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically–sound application of knowledge and can occur in a variety of ways, including the commercialization of research findings.
Results presented in Table 9 show that the most common type of knowledge user or stakeholder group involved in RMNI research was other researchers and academics (excluding study stakeholders) (46%) followed by study stakeholders (formally listed in the grant application) (38%), health system/care practitioners (35%), patients/consumers of health care (27%), and industry (27%). Furthermore, results show that these groups also had the greatest level of involvement across all stages of the research process for RMNI supported research.
In addition, the most common knowledge user or stakeholder groups involved in RMNI research having the greatest level of involvement were also the most common groups who were influenced to some or great extent by the results of the research (Table 10): other researchers/academics (73%), study stakeholders (46%), industry (38%), as well as health system/care practitioners (31%).
Photo: Scanning Tunnelling Microscope – Photo courtesy of National Research Council of Canada.
Stakeholders involved: | RMNI Overall (N=26) | Development of the research idea/question | Development of the protocol | Data collection phase/Project implementation | Interpretation of the results | End of grant KT activities | Other |
---|---|---|---|---|---|---|---|
Other researchers/academics (excluding study stakeholders) | 46% | 27% | 19% | 27% | 23% | 12% | 8% |
Study stakeholders (formally listed in grant application) | 38% | 19% | 23% | 27% | 19% | 12% | 8% |
Health system/care practitioners | 35% | 19% | 23% | 19% | 27% | 4% | 8% |
Patients/consumers of health system/care | 27% | 4% | 4% | 12% | 0% | 4% | 8% |
Industry | 27% | 12% | 8% | 8% | 8% | 12% | 8% |
Health system/care professional organizations | 15% | 4% | 0% | 8% | 4% | 0% | 4% |
Federal/provincial representatives | 15% | 12% | 4% | 0% | 0% | 8% | 0% |
Consumer groups/charitable organizations | 15% | 4% | 0% | 8% | 0% | 0% | 4% |
The media | 15% | 4% | 0% | 0% | 0% | 4% | 8% |
Community/municipal organizations | 8% | 0% | 4% | 4% | 0% | 4% | 4% |
Health systems/care managers | 4% | 4% | 0% | 0% | 0% | 0% | 0% |
Other (corporate partners) | 4% | 0% | 0% | 0% | 0% | 4% | 0% |
Source: Survey of RMNI–Funded Researchers |
Influenced to “some” or “great” extent: | RMNI Overall (N=26) |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
---|---|---|---|
Researchers/Academics (excluding study stakeholders) | 73% | 58% | 86% |
Study Stakeholders (formally listed in grant application) | 46% | 33% | 57% |
Industry | 38% | 25% | 50% |
Health system/care practitioners | 31% | 25% | 36% |
Patients/consumer of health system/care | 19% | 8% | 29% |
Federal/Provincial Representatives | 19% | 25% | 14% |
The Media | 15% | 17% | 14% |
Health system/care professional organizations | 12% | 8% | 14% |
Consumer groups/Charitable Organizations | 12% | 8% | 14% |
Community/Municipal Organizations | 8% | 8% | 7% |
Health system/care managers | 4% | 8% | 0% |
Source: Survey of RMNI–Funded Researchers |
Commercialization and Research Outcomes
RMNI grants resulted in a sizable proportion of commercialization–related outcomes in the context of CIHR benchmarks. Key achievements in terms of commercialization (Table 11) include the 46% of RMNI grants that produced patents/licenses and the 39% resulting in intellectual property claims. Benchmark data on biomedical research funded under CIHR’s OOGP between 1991 and 2006 reveals that 18% of grants resulted in patents/licenses while intellectual property claims resulted from 13% of OOGP grants. In addition, a large proportion of RMNI grants led to research related outcomes including research findings/knowledge creation (100%), new research method (92%), new theory (50%) and new practice (39%).
Percentage (%) of grants that resulted in: | RMNI Overall (N=26) |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
OOGP Grant (N=457) |
---|---|---|---|---|
Research findings/ Knowledge creation | 100% | 100% | 100% | 94% |
New research method | 92% | 92% | 93% | 60% |
New theory | 50% | 42% | 57% | 69% |
Patents/licenses | 46% | 25% | 64% | 18% |
New Practice | 39% | 33% | 43% | 18% |
Intellectual property claim | 39% | 33% | 43% | 13% |
Adaptation of research findings | 35% | 33% | 36% | – |
Replication of research findings | 23% | 8% | 36% | 51% |
Software/database | 15% | 17% | 14% | 7% |
Direct cost savings | 12% | 8% | 14% | 5% |
New vaccine/drug | 8% | 8% | 7% | 6% |
Spin off company | 8% | 0% | 14% | 5% |
Source: Survey of RMNI–Funded Researchers; OOGP Research Reporting System Data |
Capacity Development
Photo: Images of outgrowing axons from an injured mouse peripheral nerve, growing (from top to bottom) without (control group) or with a local electrical stimulation protocol – Photo courtesy of Dr. Douglas Zochodne, University of Calgary.
CIHR's mandate includes a duty to build the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research (Canadian Institutes of Health Research Act, 2000, p.5). CIHR supports capacity development directly through training grants and awards such as the Strategic Training Initiative in Health Research (STIHR) and the Vanier Canada Graduate Scholarships. Capacity development is also supported through funding for research projects that involve students, research staff, and technicians. For RMNI, the development of capacity in the fields of regenerative medicine and nanomedicine is one of the initiative’s key anticipated outcomes.
Involvement of trainees was identified as a key factor for success by RMNI–funded researchers and case study participants. A review of CIHR grant files for the 26 RMNI grants surveyed revealed that over half of RMNI funds expended were used to pay for the salaries of students and non–students.12 Specifically, 55% of RMNI funds expended from catalyst grants was used to support salaries for these individuals involved compared to 63% from team grants (for a total of 63% across all surveyed grants).
Results presented in Table 12 indicate that RMNI has contributed to the development of research capacity, particularly through its team grant funding mechanism (93% of team grants involved 11 or more staff/trainees). The average number of research staff and trainees involved in RMNI team grants was 27.7 compared to 7.0 for catalyst grants (p<0.0513). In contrast, benchmark data on biomedical OOGP grants awarded between 1991 and 2006 reveals that an average of 7.9 research staff and trainees were involved per grant. Furthermore, an average of 6.6 PhD students and 5.3 undergraduate students were involved per RMNI team grant (Table 13).
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
OOGP Grant (N=440*) |
|
---|---|---|---|
Percentage of grants with total number of staff/trainees involved: | |||
1–5 |
50% |
7% |
40% |
6–10 |
42% |
0% |
39% |
11–25 |
8% |
50% |
20% |
Over 25 |
0% |
43% |
1% |
Average number of staff/trainees involved |
7.0 (Std Dev=3.6) |
27.7 (Std Dev=17.4) |
7.9 (Std Dev=5.8) |
Source: Survey of RMNI–Funded Researchers; OOGP Research Reporting System Data * 17 cases were excluded due to an indication of no response across all RRS categories of research staff and trainees |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
|
---|---|---|
Average number involved by type: | Mean ± Std Dev | Mean ± Std Dev |
Research assistant(s)/technician(s) | 1.5 ± 0.9 | 3.6 ± 2.2 |
Undergraduate students | 1.5 ± 1.6 | 5.3 ± 3.4* |
Master’s students | 1.0 ± 1.2 | 5.1 ± 6.3 |
PhD students | 1.9 ± 1.4 | 6.6 ± 7.2 |
Postdoctoral fellows (post–PhD) | 0.9 ± 1.2 | 5.2 ± 4.7 |
Fellows (not pursuing a Master’s or PhD) | 0.0 ± 0.0 | 0.4 ± 0.7 |
Post health professional degree | 0.2 ± 0.6 | 1.9 ± 5.3 |
Source: Survey of RMNI–Funded Researchers * based on N=13 |
Multi/Transdisciplinary Research Training
According to CIHR’s 2009–2014 strategic plan (CIHR, 2010, p.13):
Over the next five years, CIHR will sustain a healthy research foundation by:
- Training, attracting and retaining the best talent in health research;
- Providing increased focus on trans–sectoral and multidisciplinary training; and
- Preparing young researchers for non–academic labour markets.
To solicit the opinions of trainees involved in RMNI–funded projects on the value of the multi/transdisciplinary training received as well as the influence that their training has had on career advancement, an online discussion forum was held with a total of 13 trainees representing seven RMNI grants (four team grants and three catalyst grants).
All trainees who participated in the forum indicated satisfaction with the training they received and would recommend multi/transdisciplinary training to others. Most trainees reported that training from multiple mentors exposed them to a wide variety of different perspectives which enriched their research by providing different points of view while a few trainees reported that this type of collaboration improved their ability to communicate their research in ways that were understandable to all members of their team. To further illustrate the type of experiences and skill sets achieved through involvement in multi/transdisciplinary projects, an RMNI catalyst grant NPI offered the following description of the nature of training received by a PhD student in statistics who was involved in their project:
Almost all trainees stated that the single greatest challenge in their research environment was learning new techniques but that this challenge was largely mitigated by the diversity found within their team and the multi/transdisciplinary expertise available on hand. While most trainees remarked that there were no distinct disadvantages to participating in this kind of training and that their experience was valuable, several mentioned that multi/transdisciplinary training isn’t necessarily valued by all employers and that it is sometimes difficult to find employment when industry or academia are looking for candidates whose work is based within one primary discipline. An evaluation of CIHR’s Strategic Training Initiative in Health Research (STIHR) similarly found that interdisciplinary skills were reported by respondents as valued, however, the marketability of these skills was unknown (CIHR, 2008).
RMNI trainees with career pursuits outside of academia suggested that a training model that required the acquisition of skills more suitable to the needs of the workforce including industry, through internships or other models being applied elsewhere (such as laboratory rotations at the start of training, a practice common to graduate programs in the United States), would be a more suitable and effective approach for trainees with similar career aspirations.
In terms of the influence of RMNI training on career paths and opportunities, most trainees mentioned that their career plans did not change as a result of their experiences but several respondents did note that they went on to pursue further involvement in multi/transdisciplinary research as a result of their training. One team grant trainee explained that:
Leveraging of Additional Funding and Support
One indicator used in the assessment of research outcomes for CIHR programs and initiatives is the extent to which funded researchers leverage additional and/or subsequent funding. As part of the evaluation, NPIs were asked whether their RMNI grant had contributed to the attainment of other grants and awards funding. The majority of NPIs surveyed (85%) indicated that their RMNI grant helped them and/or members of their team leverage other funding. Sources of funding obtained, in order of most common, were from the Natural Sciences and Engineering Research Council (NSERC), CIHR, the Canadian Foundation for Innovation (CFI), and provincial government organizations.
Results presented in Table 14 show that RMNI team grant researchers leveraged an average of 4.7 grants and awards versus 3.2 for catalyst projects (p<0.0514) and received approximately one quarter of grants and awards from CIHR. Of note, contextual factors related to the differences in funding mechanisms may account for differences in leveraging results such as team grants having longer durations and involving more researchers and trainees as compared to catalyst grants (see Tables 12 and 16).
Researchers who participated in interviews noted that RMNI funding facilitated publications and helped build their reputations within the fields of regenerative medicine and nanomedicine which enabled them to secure additional funding. A few researchers mentioned their RMNI grant had helped them establish preliminary results that supported the need for additional research while others explained that RMNI funding helped develop partnerships with industry and other researchers and moved their research into commercialization stages.
RMNI Overall (N=26) |
RMNI Catalyst Grant (N=12) |
RMNI Team Grant (N=14) |
|
---|---|---|---|
Average: | Mean ± Std Dev | Mean ± Std Dev | Mean ± Std Dev |
Number of grants/awards leveraged |
3.4 ± 3.2 | 1.9 ± 1.7 | 4.7 ± 3.7 |
Dollar amount of total grants/awards leveraged |
$1,514,113 ± $1,703,829 | $855,857 ± $1,393,495 | $2,078,332 ± $1,787,790 |
Percentage of: | |||
Total number of grants/awards leveraged from CIHR | 23% | 13% | 26% |
Total dollar amount of grants/awards leveraged from CIHR | 12% | 4% | 14% |
Source: Survey of RMNI–Funded Researchers |
RMNI Partnerships and Collaborations
Since its inception, RMNI’s goals and activities have been established through close collaboration and partnership with 11 CIHR Institutes and branches as well as 20 external organizations (see Appendix for full list of those involved). Collaboration with CIHR’s Institutes and branches was intended to reduce overlap and duplication in research funding through the sponsoring of joint funding opportunities.
For example, on RMNI team grant competitions, most funding partners typically contributed less than the average cost of one full grant. The multi–partnered competitions thus allowed all partners to leverage their funding significantly, with most partners identifying two to four relevant grants supported through each funding opportunity (i.e., the multidisciplinary nature of the research appealed to multiple partners). Even in cases where partners were making a significant contribution, it was more efficient from a program delivery perspective to run one large competition rather than several smaller ones. As noted by an RMNI–funded researcher and stakeholder, “ what RMNI did was that they were able to create a call that could pool money so you could get support through RMNI and there would be funding from a whole bunch of different agencies that, on your own [research] project, you would not be able to easily secure at one–time to support one project.”
External partners and stakeholders, such as other Canadian research funding agencies and government departments, provided not only funding to invest into research supported through the initiative but also support (both financial and in–kind) to organize joint workshops, meetings and symposia. As well, Canadian researchers (some funded by RMNI) working in the fields of regenerative medicine and nanomedicine provided expertise and advice in the design of the initiative.
To determine the success of RMNI’s external partnerships and collaborations, interviews were conducted with six individuals associated with RMNI who represented the following types of involvement: funding partners, workshop organizers, and content experts. RMNI partners/stakeholders all described their involvement in RMNI as successful, expressed satisfaction with the relationship and stated their intention to remain engaged with RMNI should the initiative continue.
One factor frequently identified by interviewees as having contributed to the success of their partnership was a positive relationship with the Associate Director of RMNI who was described as flexible, accessible, and collaborative. Other factors mentioned included strong management and administrative support of the initiative, RMNI’s clear mandate as well as having a history of prior partnerships with CIHR.
In terms of benefits received, funding partners reported that their RMNI partnership helped their organization access researchers, leverage their own funding and capitalize on CIHR’s established processes such as peer review. A few funding partners also stated how the partnership allowed their organization to develop relationships with other organizations and helped increase their profile among the research community and the general public. For example, one interviewee whose organization contributed funding through RMNI explained that:
Photo: 2010 RMNI Funding Announcement – Photo courtesy of CIHR.
While funding partners benefited from the partnership, they also noted that it had some risks. For example, there may not be any RMNI applications of interest to their organization or the applications of interest may not be successful in receiving RMNI funding. One of the interviewees commented that it would be helpful if RMNI had provided their organization with regular updates on the projects for which they were contributing funding and that there was a lack of clarity concerning what RMNI’s role or responsibilities were once a grant was awarded.
None of the interviewed partners reported having a similar partnership with other organizations or initiatives but many reported that their organization could benefit from a similar partnership model with other organizations. However, some doubted that the success achieved with RMNI could be accomplished elsewhere. They felt that this type of coordinated relationship required a level of commitment or reciprocity from both sides of the partnership and that not all organizations and/or initiatives are able (or willing) to do so. One interviewee also commented that the scope of RMNI was not something that existed elsewhere.
RMNI Workshops
Approximately $1 Million dollars of RMNI’s total investments was expended on holding a total of 14 workshops and meetings on topics of common interest in regenerative medicine and nanomedicine (see Appendix for full list of workshops held). These events were coordinated by RMNI in conjunction with several CIHR Institutes and branches as well as other government departments and agencies and were intended to bring together experts and stakeholders from different domains, aligned along common themes, to form connections between fields, disciplines, and backgrounds. Some RMNI workshops covered a specific theme and were by invitation only whereas other workshops were open to all interested in attending.
Approximately 33% of RMNI–funded researchers surveyed had attended at least one RMNI workshop. Results presented in Figure 16 indicate that the workshops were most useful in providing learning opportunities and presenting relevant information pertaining to research.
Figure 16 - Usefulness of RMNI Workshops for Researcher Attendees
Source: Survey of RMNI–Funded Researchers (N=24*)
* RMNI–funded researchers who attended at least one RMNI workshop
The majority of RMNI–funded nominated principal investigators interviewed revealed that they had not attended any of the workshops due to a lack of awareness of the events, not having enough free time to attend, a lack of interest, or someone else from their RMNI team had attended. RMNI nominated principal investigators who had attended described the workshops as beneficial in terms of networking, learning, and sharing ideas with other researchers. A few researchers interviewed described the workshops as not particularly useful to their research – they felt it was difficult to relate to the array of projects, and that workshops were driven by policies or the identification of priorities rather than by research topics.
Results should be interpreted with caution, however, as RMNI workshops varied in terms of purpose and theme and respondents were asked to provide their opinions on the overall usefulness of the event(s) they had attended in facilitating a number of outcomes, some of which may be more applicable than others depending on the nature of the workshop held.
RMNI partners/stakeholders interviewed who had been involved in the workshops indicated that the events helped open the dialogue between the different funding agencies and players involved in regenerative medicine and nanomedicine. As a result, interviewees indicated that there was increased coordination and consolidation of resources and policies/guidelines. This was also described as something that was new and unique to RMNI. For example, an RMNI partner/stakeholder stated:
When asked how RMNI could be improved, several NPIs interviewed commented that the workshops should be promoted more and both researchers and partners suggested holding an annual meeting for funded RMNI researchers with the aim to make connections between research teams and exchange information about research being conducted, including the management of research.
RMNI Design and Delivery
Researcher Satisfaction with RMNI Peer Review
Levels of satisfaction with the peer review process provide a researcher perspective on the efficacy of RMNI program delivery. Overall, RMNI–funded researchers15 reported a consistently high level of satisfaction and low level of dissatisfaction across five main aspects of the peer review process (Figure 17).
To situate the findings within the wider CIHR context, a comparison with results obtained through the 2011 CIHR International Review (IR) survey was undertaken. To ensure respondents were matched as closely as possible, a sample of IR respondents was selected based on the following criteria: respondents had indicated at least a biomedical (Pillar 1) research focus, were successful in their application for CIHR funding over the past five years, and whose application(s) for funding included team and/or catalyst grants. Of note, IR survey respondents were asked to respond with reference to all CIHR programs to which they applied. As a result, they could have submitted applications to other programs in addition to team and/or catalyst grants and therefore their opinions may not be directly attributable to either type of funding mechanism.
Results of the benchmark comparison (Figure 17) reveal that, apart from one aspect of the process, just over half of IR respondents reported satisfaction with CIHR peer review. Although results show a greater proportion indicated satisfaction, at least one third of IR respondents however were dissatisfied with each aspect of the process. In contrast, few RMNI–funded researchers were negative in their assessments of peer review and this finding provides further indication that the RMNI peer review process has been effectively delivered.
Figure 17 – RMNI-Funded Researchers’ Satisfaction with Peer Review
Source: Survey of RMNI–Funded Researchers (N=72); 2011 CIHR International Review Survey (N=152)
Figure 18 – Suggested Improvements to Peer Review
- For team grants, focus more attention on the funding history of an applicant as well as the project management elements (including the facilitation of team collaboration)
- Limit the number of applications a single researcher can submit to RMNI
- Make the instructions clear and concise
Source: Interviews with RMNI–Funded Researchers (N=23)
Generally, interviewed RMNI–funded researchers described the peer review process used to assess their application as satisfactory. Notwithstanding their overall satisfaction, interviewed researchers did express some concerns about certain aspects of the process, in particular the peer reviewers themselves. For instance, some felt that some reviewers may not always have a full understanding of the proposed project or noted that the peer reviewers' judgments were not consistent from one reviewer to the next. When asked about possible improvements, a few researchers suggested that the team grant review process focus more attention on an applicant’s funding history while one recommended a greater focus be placed on project management elements of team grant proposals (Figure 18).
Challenges in Reviewing RMNI Applications
To help determine whether there were challenges in the effective delivery of the RMNI peer review process in terms of reviewing multi/transdisciplinary and high risk grant applications, an interview was conducted with one of two RMNI peer review Committee Chairs and one of three RMNI Deputy Directors. Of note, the role of a CIHR Committee Chair is to provide oversight with respect to the review process; however the Chair does not provide ratings for applications. The responsibilities of a CIHR Deputy Director include directing the delivery of competition and scientific peer review services for an assigned set of CIHR funding programs.
In terms of challenges with peer review, the RMNI Chair noted that high risk grant applications submitted by new independent researchers without a prior history of grant funding were likely to be rated and ranked lower by the peer review committee and speculated that this was an issue that extended beyond RMNI to CIHR overall. Another challenge identified by the Chair concerns the effect that a lack of familiarity of committee members with smaller, emerging areas such as regenerative medicine and nanomedicine can have on the review process. According to the RMNI Chair, committee members who are less familiar with newer research areas and who are negative towards their view of an application tend to weight the assessments of the external reviewers16 less than those of the internal reviewers17. The RMNI Chair also suggested that if committees are reviewing multidisciplinary applications, the committee should reflect the multidisciplinarity of the types of applications they receive, and noted that the RMNI committee did not reflect this at the time he/she served as Chair.
Although recruiting multidisciplinary peer reviewers was noted as a challenge by the RMNI Deputy Director and that matching the expertise of a group of typically three peer reviewers (two internal reviewers and a reader18) to a grant application involving multiple researchers working in different areas is difficult, this was mitigated to the extent possible by obtaining complementary expertise through the use of reviewers with multidisciplinary backgrounds along with specialists and researchers residing outside of the country (who could provide a different perspective to a review). Furthermore, researchers funded by RMNI were recruited as reviewers to ensure that multidisciplinarity was reflected in the committee. As well, the use of RMNI–funded researchers as reviewers enabled assessments of not only the science of grant proposals but also the alignment of the proposed research to the objectives of the funding opportunity, the novelty of the research and the need for the research in Canada.
In terms of improvements to the peer review process, the RMNI Deputy Director suggested that, irrespective of feasibility and burden, an increase in the number of internal reviewers assigned to multidisciplinary team grant applications (up to five reviewers in total) would help ensure that sufficient expertise is available to cover the variety of disciplines involved.
RMNI Alternative Delivery Mechanisms
Considering that all RMNI–funded researchers interviewed felt that their projects were successful and that the vast majority of researchers were satisfied with the trans/multidisciplinary nature of their research and the RMNI peer review process (at least 74% reported satisfaction across various aspects), few researchers had suggestions for how the initiative could be changed or replaced.
Photo: Operating Theatre with Integrated Femtosecond Laser System – Photo courtesy of Dr. Isabelle Brunette, Maisonneuve-Rosemont Hospital.
Furthermore, several RMNI case study researchers noted that a particular strength of the initiative’s design is that applicants are not required to engage in private sector or industry partnerships as a condition of funding. One RMNI case study researcher noted that while Canada has created the infrastructure that allows commercialization to occur, there are few Canadian R&D companies to partner with due to the R&D branches of many organizations being located outside of the country. Furthermore, they noted that a considerable time commitment may be required to negotiate issues such as intellectual property rights.
The small number of interview participants (RMNI researchers and partners) who did provide some critical opinions about RMNI’s design commented that the targeted research areas of regenerative medicine and nanomedicine may be too broad in scope and questioned why both fields were chosen as priority areas for the initiative. Similarly, a few interviewees felt that RMNI lacked definition and that the initiative needed a short–term and long–term vision rather than trying to support all types of research such as high–risk, clinical and translational research.
In terms of suggestions for improvements to RMNI, funded researchers recommended additional support for research through increasing the funding envelope for the initiative, the success rate for competitions, and the number of funding competitions held. Of particular concern to RMNI researchers, particularly those involved in teams, was the need for continued funding. As one funded team grant researcher explained: “The prospects for me are that in two years our RMNI funding will expire and the year after that, the Stem Cell Network […] will also run to its conclusion. So, in three years from now, it’s quite possible that all of the networking that we’ve established may unravel without some sort of continued new funding to maintain it and to get to the next level.”
Of a total of 77 grants awarded, only one project was successful in receiving renewal funding from RMNI. To address this issue, several researchers suggested that funding be either continued through RMNI or through alternative programs with some suggesting the creation of a small scale funding program to keep RMNI projects moving forward and/or to support RMNI research that was completed by a single researcher until their next large grant.
Photo: Image of Macrophages Ingesting Quantum Dots – Photo courtesy of Prof. Warren Chan, University of Toronto.
A few funded researchers also noted that for some projects, results will not be available for a few years after the grant has ended. This was particularly salient for Catalyst Grant projects as these may require additional funding before evidence–based grants can be received from CIHR or other agencies/foundations. As one catalyst grant NPI noted: “When we have ideas which are new, we don’t have a cumulative result from a few years; it’s very difficult to get an operating grant from CIHR. So that RMNI [catalyst] grant is suitable to fund new ideas with novelty but not much results.”
Several researchers and a few RMNI partners suggested that CIHR increase the amount of support available to researchers through the development of partnerships with federal and provincial funding agencies such as NSERC and the Fonds de recherche du Québec Santé (FRSQ), other countries such as the United States where the California Institute for Regenerative Medicine (CRIM) co–funds international collaboration and Industry.
A few interviewed researchers commented about the possibility of using the CIHR Open Operating Grants Program (OOGP) as an alternative to RMNI. Generally, they felt that this would not be a viable alternative given the typical Operating Grants' size, limited funding envelope/low success rate, and scope (does not fund the same sort of non–conventional research). While it was not considered an alternative by RMNI–funded researchers, CIHR Operating Grant funding was described as complementary and the first step to developing research ideas and applying for larger team grants such as those offered through the initiative.
In terms of the possibility of having RMNI grant applications reviewed through relevant standing peer review committees under CIHR’s OOGP program in place of a separate committee for the initiative, the RMNI Deputy Director interviewed noted that for strategic initiative funding opportunities, peer reviewers are recruited based on the needs of the applications received in order to better match reviewer expertise to a proposal. In contrast, peer review committees used for the OOGP are standing panels held biannually and the Deputy Director stressed that the degree of variation in the types of RMNI applications received would require a reviewer recruitment period much longer than the six month duration between OOGP competitions. Furthermore, the RMNI Deputy Director indicated that the multidisciplinarity of RMNI applications renders them unsuitable for review by OOGP panels where biomedical research proposals submitted tend to be unidisciplinary.
Need for RMNI
In 2008, Canada’s Science, Technology and Innovation Council (STIC), with an endorsement by the Minister of Industry, recommended regenerative medicine as a Canadian health and life sciences sub–priority within the Government of Canada's Science and Technology Strategy (Science, Technology and Innovation Council, 2011). Furthermore, the CIHR 2009–2014 strategic plan (CIHR, 2010, p.16) states:
Photo: Image of Cells Growing on a Polymer Sphere – Photo courtesy of Dr. Christopher Yip, University of Toronto.
Through its partnerships and support of multi/transdisciplinary teams and high risk, high impact research projects at the interface of health and technology in the fields of regenerative medicine and nanomedicine, the objectives of RMNI are in alignment with Canadian and CIHR health research priorities. Additionally, the most recent federal budgets affirm the government’s commitment, and the role of the Canadian Institutes of Health Research, in supporting advanced research (Government of Canada, 2012, p.268) and health research of national importance (Government of Canada, 2011, p.153–154).
Publication Growth in Regenerative Medicine and Nanomedicine
Although regenerative medicine and nanomedicine were fields emerging at the time of RMNI’s implementation, bibilometric results presented earlier (Figures 1 and 2) provide evidence that these fields have grown significantly in terms of publication output since the initiative’s inception. At a macro level, global publications in regenerative medicine and nanomedicine (combined) have grown from an annual rate of 3,381 in 2002 to 17,905 in 2010 – a growth of 430%. A similar picture can be seen in Canada where publications in regenerative medicine and nanomedicine have grown from 100 in both fields in 2002 to 628 in 2010 – a growth of 530%. Growth rates for Canadian publication volume correlates highly with the rest of the world in both fields however its specialization still falls below the world average.
As revealed through the bibliometric analysis, RMNI–funded researchers have appeared as authors on approximately 34% of Canada’s total publication output in regenerative medicine and 21% in nanomedicine over the period of 2004–2010. Furthermore, publications from RMNI–funded researchers resulted in higher citation impact scores as compared to Canadian averages in both fields. If the initiative is not renewed and no alternative funding mechanisms are available to continue supporting research in these fields, its absence may have a considerable effect on Canada’s competitiveness worldwide, especially with another Canadian strategic funding initiative, the Stem Cell Network, reaching the end of its funding cycle in 2015.
Alternative Sources of Funding for RMNI Researchers
Nearly all RMNI–funded researchers and case study participants interviewed noted that the initiative appears to be filling a void and that their project would not have achieved the same level of success through other sources of funding or initiatives. In particular, researchers felt that currently, there is no or limited funding in Canada for research teams, high–risk or early stage research, and projects at the interface between technology and health.
As one RMNI team grant NPI noted:
Furthermore, one catalyst NPI stressed that “the type of research that we do is to make tools for imaging; they are very valuable tools (hundreds of requests for them each year). Yet, I have very little chance of being successful in a regular CIHR operating grant competition because we are not driven by biological questions; we are driven by the creation of tools to help other researchers.”
When asked whether they could obtain alternative sources of funding for their research in the absence of RMNI, 42% of funded researchers surveyed felt they could do so in Canada and half indicated that they would be able to sustain their research program if the initiative is discontinued (Figure 19).
An analysis of subsequent funding found that, after having received their RMNI grant, 35 NPIs (52%) went on to receive OOGP grants (as a nominated principal investigator) and seven (10%) were awarded Canada Research Chairs. Furthermore, the Collaborative Health Research Projects Program (CHRP) has subsequently funded nine (13%) RMNI nominated principal investigators, and there has been a $15M investment into the Centre for Commercialization of Regenerative Medicine (CCRM), which includes RMNI–funded researchers among its lead scientists and advisory group.
Although a sizable percentage of researchers indicated that they could obtain support and maintain their research if RMNI is not renewed (evaluation results show that half of RMNI–funded NPIs go on to receive CIHR operating funds), researchers and partners expressed concern that RMNI’s absence would result in the slowdown of research in both fields, a reduction in the amount of new technology produced within Canada as well as the disbanding or reduction of research teams supported including the trans/multidisciplinary linkages established.
Furthermore, RMNI funding partners expressed concern that the absence of the initiative would result in a loss of research in their field and would reduce the ability of their organization to leverage resources. Funding partners also expressed concern that the absence of RMNI would indicate to their stakeholders that research in regenerative medicine and nanomedicine is no longer perceived as important.
Figure 19 – RMNI-Funded Researchers’ Opinions on the Impact of the Absence of RMNI
Source: Survey of RMNI–Funded Researchers (N=72)
Conclusions and Recommendations
Conclusions
RMNI has made significant contributions to building Canadian research capacity and knowledge creation in the fields of regenerative medicine and nanomedicine. Over the period of 2004–2010, researchers supported through the initiative authored close to one–third of Canadian publications in both fields, produced more publications than unsuccessful applicants, achieved higher citation impact scores as compared to Canadian averages and achieved higher international collaboration rates than the world average in each field. At a national level, Canada ranked in the middle of the top 16 productive countries in each field in terms of papers produced and achieved higher citation impact scores than the world average. However, Canada’s specialization in regenerative medicine and nanomedicine was below the world average over the period of RMNI’s lifecycle (2004–2010).
RMNI funding also enabled the support of a considerable number of research staff and trainees. Furthermore, trainees report satisfaction with their involvement in multidisciplinary research. However, they expressed concern over the extent of the demand for individuals who work in more than one research field. RMNI funding also facilitated the conduct of multidisciplinary research collaboration that funded researchers felt was highly satisfactory and valuable and RMNI grants resulted in a sizable proportion of commercialization–related outcomes in context of CIHR benchmarks. Furthermore, a high percentage of RMNI–funded research involved and influenced stakeholders including other researchers/academics, industry and health system/care practitioners.
The program has been effectively designed and delivered as reflected by the level of satisfaction reported by funded researchers and partners/stakeholders. Funded researchers report a high level of satisfaction with the RMNI peer review process in light of CIHR benchmarks and partners and stakeholders viewed their participation in the initiative as successful and expressed an intention to remain involved if RMNI were to continue. Researchers had varying opinions on the usefulness of RMNI workshops; however partners and stakeholders felt the events were successful in facilitating collaboration and communication between funding agencies and key organizations involved in regenerative medicine and nanomedicine. The objectives of RMNI are in alignment with the Government of Canada's Science and Technology Strategy and CIHR’s 2009–2014 strategic plan and recent federal budgets continue to affirm the government’s commitment and role of CIHR in supporting advanced research and health research of national importance. Finally, the majority of researchers and partners/stakeholders indicated that the design of the initiative is appropriate and successful.
The discontinuation of the initiative raises questions around the sustainability of research in these fields. Evidence from this evaluation is encouraging in that it shows a record of success among RMNI–funded researchers in leveraging other existing grants and awards. There may also be further opportunities for strategic funding for researchers in regenerative medicine and nanomedicine, for example through CIHR’s significant investments in networks for the Strategy on Patient–Oriented Research (SPOR) and the Epigenetics Strategic Initiative as well as funding for large scale research projects under the Personalized Medicine Initiative.
However, in the absence of RMNI funding, it will be important for CIHR to monitor Canada’s competitiveness in regenerative medicine and nanomedicine. If the country’s performance declines, CIHR should assess the health of both fields in Canada through an examination of Canadian investment in this type of research as well as tracking the subsequent careers of RMNI principal investigators and trainees. CIHR should also offer direction to the regenerative medicine and nanomedicine research community on applying to other CIHR funding opportunities and initiatives providing support in these fields.
Recommendations
- Implement a communication strategy aimed at researchers working in the fields of regenerative medicine and nanomedicine that offers direction on applying to other CIHR funding opportunities and initiatives providing support in these fields.
- Conduct regular assessments with international benchmarks to determine the relative global position of Canada in the fields of regenerative medicine and nanomedicine in the absence of RMNI. If Canada’s competitiveness declines, ensure regular environmental scanning takes place to assess the ongoing health of both fields in Canada. This would include examining the Canadian investment in these research areas and tracking the subsequent careers of RMNI principal investigators and trainees.
- Ensure that future designs of programs relating to teams of researchers or networks take into account findings from the evaluation. It may, for example, be unrealistic to design programs that are expected to fund ‘newly formed’ teams, and there could be merit in including stronger requirements for dedicated research coordinators to aid success.
Management Response
Recommendation | Response (Agree or Disagree) | Management Action Plan | Responsibility | Timeline |
---|---|---|---|---|
1. Implement a communication strategy aimed at researchers working in the fields of regenerative medicine and nanomedicine that offers direction on applying to other CIHR funding opportunities and initiatives providing support in these fields. | Agree |
CIHR will communicate the sunsetting of the initiative to both those funded by RMNI and in the wider community following the approval of this evaluation. A plan will be developed as to which stakeholders need to be informed and the channels used to inform them. One key element of this communication will be the alternative funding opportunities available to researchers, both in open and strategic CIHR programs as well as through other funders. A second element to communicate is that CIHR is aware of the importance of these fields and will be monitoring their ongoing progression relative to international benchmarks (see Recommendation 2). |
Chief Scientific Officer/Vice–President, Research and Knowledge Translation Portfolio | Develop plan and communicate to the community – March – September 2013 |
2. Conduct regular assessments with international benchmarks to determine the relative global position of Canada in the fields of regenerative medicine and nanomedicine in the absence of RMNI. If Canada’s competitiveness declines, ensure regular environmental scanning takes place to assess the ongoing health of both fields in Canada. This would include examining the Canadian investment in these research areas and tracking the subsequent careers of RMNI principal investigators and trainees. |
Agree |
It is agreed that it will be important to put in place assessments to ensure that the sunsetting of this initiative will not have a detrimental impact on Canada’s performance in these key fields. As recommended in the evaluation, an approach to this will be to undertake a review of the extent to which those who were funded under this initiative are now receiving grants through CIHR’s Open Operating Grants program. Investment in these fields in other areas of CIHR’s strategic programming, for example, through the Epigenetics Initiative, can also be used. Analysis will include both dollar investment but also the types of projects funded and the number of former RMNI funded researchers receiving grants through these initiatives. More broadly, these types of approaches can be piloted for RMNI but could also work well for CIHR when considering the sunsetting of other strategic initiatives. The bibliometric scanning suggested would form part of this wider effort. |
Chief Scientific Officer/Vice–President, Research and Knowledge Translation Portfolio |
Initial assessment of RMNI researchers and projects relative to the Open Operating Grants Program and strategic investments to take place by end of Fiscal Year 2013–14. Produce a plan for regular environmental scanning of strategic investments by end of Fiscal Year 2013–14 |
3. Ensure that future designs of programs relating to teams of researchers or networks take into account findings from the evaluation. It may, for example, be unrealistic to design programs that are expected to fund ‘newly formed’ teams, and there could be merit in including stronger requirements for dedicated research coordinators to aid success. |
Agree |
As CIHR moves forward with its development of flagship Signature Initiatives, including the Strategy on Patient Oriented Research (SPOR), it will be critical to use the evidence from this evaluation to inform the future design of strategic programming. Those working on the design of such initiatives will ensure that all relevant evidence from this evaluation will be considered including the mechanics of how teams and networks form and the conditions required for their success. |
Chief Scientific Officer/Vice–President, Research and Knowledge Translation Portfolio | Include as part of the design of new Signature Initiatives including SPOR – 2013/14 and ongoing |
Methodology
Consistent with Treasury Board policy and recognized best practices in evaluation19, a range of methods – involving both quantitative and qualitative evidence – were used to triangulate evaluation findings. This was to ensure that the evaluation findings were robust and credible and that conclusions drawn about program performance are valid.
Bibliometrics: RMNI
A bibliometric analysis was conducted by Observatoire des sciences et des technologies (OST) on articles published between 1997–2010 in regenerative medicine and nanomedicine by 295 RMNI–funded researchers, 143 non–funded RMNI applicants – both groups including NPIs and co–grantees – as well as researchers worldwide. The bibliometric data presented in this study was drawn from the Canadian Bibliometric Database (CBDTM) built by the OST using Thomson Reuters’ Web of Science (WoS). Given that Thomson’s databases do not have a subject classification at the level of individual papers, OST used the U.S. National Library of Medicine Medical Subject Headings (MeSH), which relies on a controlled vocabulary to assign a medical domain to each paper indexed in the PubMed database.20 The following search queries were used to retrieve papers in Regenerative Medicine and Nanomedicine:
Regenerative Medicine MeSH Headings
Adult Stem Cells
Bioartificial Organs
Embryonic Stem Cells
Fetal Stem Cells
Liver, Artificial
Multipotent Stem Cells
Organoids
Pancreas, Artificial
Pluripotent Stem Cells
Regenerative Medicine
Skin, Artificial
Stem Cell Transplantation
Tissue Engineering
Tissue Scaffolds
Tissue Therapy
Totipotent Stem Cells
Regenerative Medicine Journals
Artificial organs
Biomaterials artificial cells and artificial organs
Biomaterials artificial cells and immobilization biotechnology
Biomaterials medical devices and artificial organs
Cell stem cell
Cloning and stem cells
International journal of artificial organs
Journal of artificial organs
Journal of tissue engineering and regenerative medicine
Neural regeneration research
Regenerative medicine
Stem cell reviews
Stem cells
Stem cells and development
Tissue engineering
Tissue engineering and regenerative medicine
Tissue engineering part a
Tissue engineering part b–reviews
Tissue engineering part c–methods
Transactions american society for artificial internal organs
Wound repair and regeneration
Nanomedicine MeSH Headings
Fullerenes
Lab-On-A-Chip Devices
Microfluidic Analytical Techniques
Microfluidics
Nanocapsules
Nanomedicine
Nanostructures
Nanotechnology
Nanomedicine Journals
Digest journal of nanomaterials and biostructures
IEE proceedings–nanobiotechnology
IEEE transactions on nanobioscience
IET nanobiotechnology
International journal of nanomedicine
Journal of biomedical nanotechnology
Microfluidics and nanofluidics
Nanobiology
Nanomedicine
Nanomedicine–nanotechnology biology and medicine
Nanotoxicology
Nature nanotechnology
The query using MeSH headings and journal articles retrieved 51,953 Regenerative Medicine papers while the Nanomedicine query resulted in a total of 48,382 papers retrieved.21 The total set of regenerative medicine and nanomedicine papers comprised 0.7% of all Canadian health research papers. Furthermore, 8% of all papers authored by RMNI researchers (funded and non–funded) were captured through the search query.
Indicators
Number of publications: The number of scientific papers with authors from a country, as found in authors’ addresses.
Average of Relative Citations (ARC): Based on the number of citations received by papers over a two–year period following publication year. Thus, for papers published in 2000, citations received between 2000 and 2002 are counted. Self–citations are excluded. Citations are counted until the end of 2010, which means that papers from 2009 have an incomplete citation window of only one year. The number of citations received by each paper is normalized by the average number of citations received by all papers of the same specialty — as defined by US National Science Foundation classification of journals22 — and research domain, hence taking into account the fact that citations practices are different for each specialty and domain. When the ARC is greater than 1, it means that a paper or a group of papers scores better than the world average of its specialty and domain; when it is below 1, those publications are not cited as often as the world average.
Specialization index (SI): The relative intensity of publication of a country in the areas of nanomedicine and regenerative medicine relative to the intensity of the world in the area. An SI value above 1 means that a country is specialized in the research area compared to the world average, while an index value below 1 means the opposite.
International collaboration rate: The relative intensity of scientific collaboration between countries. The rate is calculated by dividing the number of papers with at least one author with a foreign country address by the country’s total number of papers. A country’s international collaboration rate is generally determined by its size, i.e. larger countries collaborate less that smaller ones.
Network analysis: In order to visualize the collaborative ties between institutions or countries active in the research area, a network analysis was performed using the UCINET and Netdraw softwares. These softwares allow the creation of 2–dimensional networks of co–authored papers. The size of the edges (lines) between each of the nodes is determined by the number of co–authored papers between the two entities. A threshold of numbers of papers written in collaboration is fixed in each of the figures in order for the network to be clearer.
Bibliometrics: OOGP
Benchmark data on average of relative citation scores (ARC) was obtained from a bibliometric study conducted for the 2012 evaluation of CIHR’s Open Operating Grants Program by the Observatoire des sciences et des technologies (OST). The analysis included 20,287 supported papers23 published over the period of 2001–2009 by a randomly selected sample of N=1,125 researchers (from a population of 3,236) who received OOGP funding between 2000 and 2007. Citation impact data was also captured on all Canadian health research publications produced over the period of 2001–2009.
Quantitative Survey: RMNI–Funded Researchers
An online survey was conducted by EKOS Research Associates Inc. on a sample of RMNI–funded researchers who were awarded funding over the period of 2004–2009 that provided quantitative data used to address a range of evaluation issues and questions. Where appropriate, survey questions were matched to those found in both the 2011 CIHR International Review survey as well as CIHR’s Research Reporting System, an end of grant survey, to allow for benchmarking. A total of N=295 researchers were identified as having been funded by the initiative. Researchers only funded by RMNI in 2010 (N=18) were excluded due to these grants having just started at time of survey. Furthermore, catalyst grant co–grantees (co–principal investigators, co–investigators, and collaborators) appearing on grants with less than three researchers (N=13) were excluded as they were not eligible to respond to research team collaboration questions. Finally, two deceased NPIs, and four co–grantees involved in an RMNI grant with one recently deceased NPI were removed and N=8 NPIs and one co–grantee declined participation. The remaining population of funded researchers received an invitation to participate with a total of 72 respondents (26 NPIs and 46 co–grantees) completing the questionnaire.
Group | Population Number | Respondent Number | Confidence Interval (95% CL) |
---|---|---|---|
RMNI–Funded Researchers | 295 | 72 | ±10.1 |
RMNI Grants Overall | 77 | 26 | ±15.7 |
RMNI Team Grants | 41 | 14 | ±21.5 |
RMNI Catalyst Grants | 36 | 12 | ±23.4 |
Quantitative Survey: OOGP Research Reporting System
Data were drawn from reports submitted by biomedical researchers funded under the Open Operating Grants Program (OOGP) using CIHR’s end of grant Research Reporting System (RRS); a 2009 pilot study of the RRS that targeted grantees whose authorization to use funds expired between January 2000 and June 2008 (N=457); and data on publication outputs from the full launch of the RRS in 2011 (N=104 responses were included, all submitted by February 2, 2012). Before combining the data between the two RRS data sources, validation of responses was conducted, including checking for differences in demographic profile between respondents to the pilot and the full survey. No significant issues were identified in this regard.
Quantitative Survey: CIHR 2011 International Review
Data was analyzed from an online survey conducted by Ipsos Reid between November 5th and December 5th 2010 as part of CIHR’s 10th Year International Review to examine satisfaction with the peer review process among researchers. To help ensure respondents were matched as closely as possible to researchers supported by RMNI, a sub–sample of IR respondents (N=152) were selected by the Evaluation team based on the following criteria: respondents had indicated at least a biomedical (Pillar 1) research focus, were successful in their application for CIHR funding over the past five years and whose application(s) for funding included team and/or catalyst grants.
Document Review
This covered a review of relevant CIHR and Government of Canada documents as well as health research related reports. It also included previous evaluations and studies conducted by CIHR and its Institutes.
Discussion Forum
An online discussion forum was hosted through Interactive Tracking Systems Inc. (itracks) with a total of N=13 RMNI trainees. The forum took place over a period of five days with participants logging in to the forum once or twice a day to respond to moderator questions and engage in discussion with others. Key discussion areas included the value of training and mentoring received through RMNI, the value of a trans/multidisciplinary training environment, and career advancements.
Key Informant Interviews
Semi–structured in–depth qualitative interviews were conducted by R.A. Malatest & Associates Ltd. with a stratified random sample of RMNI–funded researchers (N=23) and purposive sample of N=6 partners and stakeholders (selected based on type and extent of involvement with RMNI) to provide evidence to address a number of the evaluation issues and questions. Two additional interviews were conducted by the CIHR evaluation team with an RMNI peer review Committee Chair and Deputy Director to address issues with assessing multi/transdisciplinary and high risk, high impact RMNI grant applications.
Case Studies
Based on discussion with RMNI management, six case studies of RMNI–funded projects demonstrating high impact breakthrough research were purposively sampled to provide detailed qualitative narratives of highly impactful research outcomes to address several evaluation issues and questions. Key informant interviews were conducted by R.A. Malatest & Associates Ltd. with a total of N=29 participants: RMNI principal investigators and researchers, students, stakeholders and knowledge users. Due to length considerations, three profiles of RMNI projects are presented in this report and were selected for inclusion based on their illustration of a multitude of varying high impact results.
Administrative Data Analysis
The CIHR Electronic Information System (EIS) is designed to collect and store data on all applicants to CIHR programs. Data obtained from EIS, along with RMNI program records, were used in the analyses of administrative and financial information pertaining to RMNI, its funded grants as well as an analysis of subsequent CIHR grants held by RMNI–funded researchers, for example.
Limitations
In keeping with best practices in program evaluation, the limitations of this study are noted below, together with the strategies that were employed to mitigate them.
Bibliometrics: RMNI
Bibliometric analysis has been criticized on the grounds that estimates of publication quality based on citations can be misleading and that citation practices differ across disciplines and sometimes between sub–fields in the same discipline (Ismail et al., 2009). This is a particularly salient issue for RMNI, with an objective to fund multi/transdisciplinary research in the fields of regenerative medicine and nanomedicine. To mitigate this, measures of other outputs are also used in this evaluation to assess knowledge creation as a result of the initiative. A case study approach is also taken to assess highly impactful research conducted as a result of RMNI funding.
Secondly, it should be noted that the bibliometric analysis in this evaluation included data for regenerative medicine and nanomedicine publications produced by funded RMNI researchers at some point after the initiative’s inception. Although the period of publication used throughout the bibilometric analysis aligns with the overall lifecycle of RMNI (2004–201024), funded researchers may have published articles in either field prior to, or after having concluded, their RMNI grant. As a result, direct attribution between RMNI funding and publication data cannot be made. With further development of CIHR’s Research Reporting System, where researchers list publications produced as a result of the grant that can then be linked directly to bibliometric data, this type of analysis should become available for future CIHR evaluations.
Bibliometrics: OOGP
The OOGP bibliometric analyses used to provide benchmark data for the evaluation is based on data for publications produced by OOGP researchers (NPIs) while supported by these grants. While this method is commonly accepted based on an assumption that these grants are a significant contribution to research output (e.g., Campbell et al, 2010), an outright attribution between grant and publication bibliometric data cannot be made. Also, due to budget limitations, a sample of OOGP researchers funded at least one year between 2000 and 2007 (N=1,125) was randomly selected from a population of 3,236 for analysis rather than selecting all researchers within that group. The total sample size was adequate for the analysis, and there is no reason to expect that the universe of all funded researchers would be different from the selected sample.
Quantitative Survey: RMNI–Funded Researchers
The foremost limitation with regards to the use of a survey methodology is that it relies largely on self–reported data and/or memory recall from respondents. Furthermore, RMNI awarded a total of 77 grants and although ones awarded in 2010 were excluded from the survey, data was captured on a relatively small number of grants (26 in total) which limits the ability to generalize results to the initiative overall as well as within its two funding mechanisms. Furthermore, the relatively low number of grants and researchers sampled prevents the ability to make statistical conclusions including observed differences between RMNI results and benchmark comparison data.
Quantitative Survey: OOGP Research Reporting System (RRS)
Data collection in the RRS end of grant survey ‘Pilot study’ involving OOGP funded researchers was halted before the fourth wave of invitations were sent out. Similarly, OOGP researchers responding to the current version of the RRS have until October of 2012 to complete their report, meaning that a full sample was not available. Among the completed reports, data quality checks are still ongoing and only the responses related to knowledge creation were available to be included in the analyses. Also, in relation to estimating the numbers trained/supported by OOGP, there could be double counting since trainees could be involved in multiple projects with different nominated principal investigators.
To mitigate against the possibility that the OOGP RRS samples may not be representative of the overall population of OOGP researchers, a comparison of demographic variables of the two RRS sets of data with the OOGP population was conducted. This suggested that the two incomplete samples were broadly representative of the overall universe of researchers. The variables compared were: pillar, language and region, with differences between the samples and the population of around 5%.
Quantitative Survey: CIHR 2011 International Review
To help ensure respondents to the 2011 CIHR International Review survey were matched as closely as possible to the sample of RMNI survey respondents, an IR sample was selected based on the following criteria: respondents had indicated at least a biomedical (Pillar 1) research focus, were successful in their application for CIHR funding over the past five years and whose application(s) for funding included team and/or catalyst grants. As IR survey respondents were asked to respond with reference to all CIHR programs to which they applied to, they could have submitted applications to other programs in addition to team and/or catalyst grants and therefore their opinions may not be directly attributable to either type of funding mechanism.
Case Studies
The sampling of projects to be included in the case study analysis was purposive with only exemplary cases being selected. Also, only a small number of projects were selected due to budget and timing constraints. As with all qualitative data, these findings are not generalizable to a wider population but are used instead for illustrative purposes only.
Appendix
Evaluation Questions | Indicators | Methods | Sources |
---|---|---|---|
Knowledge Creation in Regenerative Medicine and Nanomedicine | |||
1. To what extent has RMNI had an impact on the development of the research fields of regenerative medicine and nanomedicine in Canada and internationally? |
|
|
|
|
|
|
|
2. To what extent has RMNI supported research projects that have led to high impact results? |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Research Team Collaboration | |||
3. To what extent has RMNI facilitated effective trans/multidisciplinary research collaborations? |
|
|
|
|
|
|
|
3.1. What are the best practices and/or challenges/barriers to effective collaboration? |
|
|
|
Knowledge Translation | |||
4. To what extent have RMNI–funded researchers undertaken knowledge translation activities? |
|
|
|
Capacity Development | |||
5. To what extent has RMNI facilitated capacity development of trainees? |
|
|
|
|
|
|
|
5.1 To what extent does a trans/multidisciplinary research environment impact on the training and mentoring of trainees? |
|
|
|
5.2 To what extent have training and mentoring received through RMNI advanced the careers of trainees? |
|
|
|
Leveraging of Additional Funding and Support | |||
6. To what extent has RMNI enabled funded researchers to leverage additional or subsequent funding and in–kind support? |
|
|
|
RMNI Partnerships and Collaborations | |||
7. To what extent has RMNI been successful in establishing and maintaining effective partnerships? |
|
|
|
RMNI Workshops | |||
8. To what extent have RMNI workshops facilitated collaboration? |
|
|
|
|
|
||
RMNI Design and Delivery | |||
9. How satisfied are RMNI–funded researchers with the delivery of the peer review process? |
|
|
|
9.1 What are the challenges in peer reviewing RMNI trans/multidisciplinary team and high risk catalyst grant applications? |
|
|
|
10. What alternative delivery mechanisms could be used to fund and/or support researchers in regenerative medicine and nanomedicine? |
|
|
|
|
|
||
Need for RMNI | |||
11. What would be the impact on funded researchers and projects, trainees, CIHR and partners/stakeholders if the initiative is no longer funded by CIHR? |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
Relevance | |
---|---|
Issue #1: Continued Need for program | Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians |
Issue #2: Alignment with Government Priorities | Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes |
Issue #3: Alignment with Federal Roles and Responsibilities | Assessment of the role and responsibilities for the federal government in delivering the program |
Performance (effectiveness, efficiency and economy) | |
Issue #4: Achievement of Expected Outcomes | Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes |
Issue #5: Demonstration of Efficiency and Economy | Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes |
RMNI Evaluation Question | Core Treasury Board Evaluation Issue Addressed |
---|---|
1. To what extent has RMNI had an impact on the development of the research fields of regenerative medicine and nanomedicine in Canada and internationally? | Issue #4: Achievement of Expected Outcomes |
2. To what extent has RMNI supported research projects that have led to high impact results? | Issue #4: Achievement of Expected Outcomes |
3. To what extent has RMNI facilitated effective trans/multidisciplinary research collaborations? | Issue #4: Achievement of Expected Outcomes |
3.1. What are the best practices and/or challenges to effective collaboration? | Issue #4: Achievement of Expected Outcomes |
4. To what extent have RMNI–funded researchers undertaken knowledge translation activities? | Issue #4: Achievement of Expected Outcomes |
5. To what extent has RMNI facilitated capacity development of trainees? | Issue #4: Achievement of Expected Outcomes |
5.1. To what extent does a trans/multidisciplinary research environment impact on the training and mentoring of trainees? | Issue #4: Achievement of Expected Outcomes |
5.2. To what extent have training and mentoring received through RMNI advanced the careers of trainees? | Issue #4: Achievement of Expected Outcomes |
6. To what extent has RMNI enabled funded researchers to leverage additional or subsequent funding and in–kind support? | Issue #4: Achievement of Expected Outcomes |
7. To what extent has RMNI been successful in establishing and maintaining effective partnerships? |
Issue #4: Achievement of Expected Outcomes Issue #5: Demonstration of Efficiency and Economy |
8. To what extent have RMNI workshops facilitated collaboration? |
Issue #4: Achievement of Expected Outcomes |
9. How satisfied are RMNI–funded researchers with the delivery of the peer review process? |
Issue #4: Achievement of Expected Outcomes Issue #5: Demonstration of Efficiency and Economy |
9.1. What are the challenges in peer reviewing RMNI trans/multidisciplinary team and high risk catalyst grant applications? |
Issue #4: Achievement of Expected Outcomes Issue #5: Demonstration of Efficiency and Economy |
10. What alternative delivery mechanisms could be used to fund and/or support researchers in regenerative medicine and nanomedicine? |
Issue #1: Continued Need for program Issue #4: Achievement of Expected Outcomes Issue #5: Demonstration of Efficiency and Economy |
11. What would be the impact on funded researchers and projects, HQP, CIHR and partners if the initiative is no longer funded by CIHR? |
Issue #1: Continued Need for program Issue #2: Alignment with Government Priorities Issue #3: Alignment with Federal Roles and Responsibilities |
RMNI Partner and Stakeholder Organizations
RMNI has partnered with a number of CIHR Institutes, branches, and divisions on joint funding programs and collaborative workshops, meetings and symposia, including:
- Institute of Neurosciences, Mental Health and Addiction (RMNI co–lead)
- Institute of Genetics (RMNI co–lead)
- Institute of Musculoskeletal Health & Arthritis (RMNI co–lead)
- Institute of Aboriginal Peoples' Health
- Institute of Aging
- Institute of Cancer Research
- Institute of Circulatory and Respiratory Health
- Institute of Infection and Immunity
- Institute of Musculoskeletal Health and Arthritis
- CIHR Ethics Office
- CIHR Knowledge Translation Branch
RMNI also has served as a nucleus for engaging external partners. A number of voluntary health organizations (VHOs), non–governmental organizations (NGOs), government agencies, and Networks of Centres of Excellence (NCEs) have joined in sponsoring joint funding opportunities, including:
- ALS Society of Canada
- Canadian Space Agency
- Canadian Stroke Network
- Foundation Fighting Blindness
- Heart and Stroke Foundation of Canada
- Jacob's Ladder
- Juvenile Diabetes Research Foundation International
- Neuroscience Canada
- Ontario Neurotrauma Foundation
- Stem Cell Network
In addition to funding research, RMNI has worked closely with a large number of government departments and agencies to sponsor workshops and meetings on topics of common interest. CIHR's partners in these planning and development activities include:
- Alberta Heritage Foundation for Medical Research
- Alberta Ingenuity
- Canadian Federation of Biological Societies
- Environment Canada
- European Research Area and Canada Initiative
- Health Canada
- Industry Canada
- National Research Council Canada – National Institute for Nanotechnology
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Social Sciences and Humanities Research Council of Canada (SSHRC)
RMNI is also part of CIHR's contribution to the Cancer Stem Cell Consortium (CSCC). The CSCC is designed to enhance research into cancer stem cells, and its partners include the Ontario Institute for Cancer Research, Genome Canada, the Canadian Foundation for Innovation (CFI), the Stem Cell Network, and research organizations in Canada and California.
RMNI Workshop Name | Date | Location |
---|---|---|
NanoMedicine/NanoHealth Workshop | February 13–14, 2003 | Montreal |
Regenerative Medicine in Canada: Defining the National Strategy in Tissue Engineering and Artificial Organs | March 16–17, 2003 | Toronto |
Integrating the Physical and Applied Sciences into Health Research | September 19–21, 2003 | Vancouver |
Second Annual Nanomedicine Workshop | February 26–27, 2004 | Toronto |
NanoForum Canada 2004 | June 16–18, 2004 | Edmonton |
Third Annual Nanomedicine Workshop – What’s Nano about Bio? | March 14–15, 2005 | Edmonton |
NanoForum Canada 2005 | June 15–17, 2005 | Montreal |
Integrating the Physical and Applied Sciences into Health Research II | June 1–3, 2006 | Ottawa |
Fourth Annual Nanomedicine Meeting | June 19–20, 2006 | Edmonton |
NanoForum Canada 2006 | June 20–22, 2006 | Edmonton |
NanoForum Canada 2007 | June 18–20, 2007 | Waterloo |
Fifth Annual Nanomedicine Meeting | June 20–21, 2007 | Waterloo |
Canadian Workshop on Multidisciplinary Research on Nanotechnology: Gaps, Opportunities and Priorities | January 22–24, 2008 | Edmonton |
NanoForum Canada 2008 (joint Nanomedicine meeting) | May 28–30, 2008 | Edmonton |
Regenerative Medicine and Nanomedicine Initiative (RMNI) Logic Model
Regenerative Medicine and Nanomedicine Initiative (RMNI) Logic Model: long description
Profile of RMNI–Funded Research and Researchers
Survey results (Table 16) show that researchers and research supported through RMNI were mainly biomedical (CIHR pillar 1) and RMNI researchers funded under either mechanism tended to be mid to senior investigators. In terms of dollar committment, most catalyst grants tended to fall within the $150–300K range (83%) while team grants were mainly valued at $1.0–1.5M. Catalyst grants mainly involved one to three researchers at time of application while number of researchers involved in team grants ranged from 4–6 (57%) and 10–13 (21%).
RMNI–Funded Researchers and Research Profile: | RMNI Overall | RMNI Catalyst Grant | RMNI Team Grant |
---|---|---|---|
Number of years of experience as an independent researcher | |||
5 years or less | 7% | 13% | 5% |
More than 5 years but less than 10 | 31% | 38% | 29% |
10 or more years | 63% | 50% | 66% |
Primary research pillar of RMNI–Funded Researchers | |||
Biomedical | 93% | 94% | 93% |
Clinical | 3% | 6% | 2% |
Health system/services | 0% | 0% | 0% |
Social, cultural, environmental and population health | 4% | 0% | 5% |
Pillar(s) of RMNI–Funded Research | |||
Biomedical | 96% | 100% | 93% |
Clinical | 12% | 8% | 14% |
Health systems/services | 4% | 8% | 0% |
Social, cultural, environmental and population health | 4% | 0% | 7% |
Dollar Amount Committed | |||
$60k | 8% | 17% | 0% |
$150K | 23% | 50% | 0% |
$300k | 15% | 33% | 0% |
$1.0–1.5M | 38% | 0% | 71% |
$2.0–2.5M | 15% | 0% | 29% |
Number of researchers involved | |||
1–3 | 35% | 75% | 0% |
4–6 | 35% | 8% | 57% |
7–9 | 11% | 17% | 7% |
10–13 | 12% | 0% | 21% |
More than 15 | 8% | 0% | 14% |
Source: Survey of RMNI-Funded Research (N=26) and Researchers (N=72) |
Average Relative Citations | Publication Productivity | Specialization Index | ||||||
---|---|---|---|---|---|---|---|---|
Rank | Country | ARC | Rank | Country | Papers | Rank | Country | SI |
Regenerative Medicine | ||||||||
1 | Sweden | 1.38 | 1 | United States | 15,047 | 1 | Singapore | 2.21 |
2 | Switzerland | 1.37 | 2 | Japan | 4,052 | 2 | Israel | 1.55 |
3 | Singapore | 1.36 | 3 | Germany | 3,875 | 3 | South Korea | 1.52 |
4 | Israel | 1.29 | 4 | United Kingdom | 3,212 | 4 | Italy | 1.45 |
United States | 1.29 | 5 | China | 3,195 | 5 | Japan | 1.43 | |
5 | Netherlands | 1.23 | 6 | Italy | 2,461 | 6 | Netherlands | 1.37 |
6 | United Kingdom | 1.22 | 7 | South Korea | 1,772 | 7 | Switzerland | 1.31 |
7 | Spain | 1.18 | 8 | France | 1,689 | 8 | Sweden | 1.30 |
8 | Italy | 1.15 | 9 | Canada | 1,603 | 9 | Germany | 1.29 |
Canada | 1.15 | 10 | Netherlands | 1,344 | 10 | United States | 1.28 | |
France | 1.15 | 11 | Spain | 1,036 | 11 | United Kingdom | 1.02 | |
9 | Australia | 1.14 | 12 | Australia | 1,034 | 12 | Canada | 0.89 |
10 | Germany | 1.12 | 13 | Switzerland | 930 | China | 0.89 | |
11 | Japan | 0.90 | 14 | Sweden | 879 | 13 | Australia | 0.86 |
12 | South Korea | 0.78 | 15 | Israel | 657 | 14 | France | 0.78 |
13 | China | 0.59 | 16 | Singapore | 595 | 15 | Spain | 0.76 |
Nanomedicine | ||||||||
1 | United States | 1.22 | 1 | United States | 17,288 | 1 | Singapore | 3.21 |
2 | Netherlands | 1.18 | 2 | China | 7,157 | 2 | South Korea | 1.79 |
3 | Singapore | 1.12 | 3 | Germany | 3,603 | 3 | China | 1.68 |
4 | United Kingdom | 1.09 | 4 | Japan | 3,373 | 4 | Taiwan | 1.44 |
Switzerland | 1.09 | 5 | United Kingdom | 2,548 | 5 | India | 1.25 | |
Germany | 1.09 | 6 | South Korea | 2,464 | 6 | United States | 1.24 | |
5 | Australia | 1.08 | 7 | France | 2,083 | 7 | Switzerland | 1.18 |
6 | Canada | 1.04 | 8 | India | 1,766 | 8 | Germany | 1.01 |
7 | France | 1.01 | 9 | Canada | 1,539 | Japan | 1.01 | |
8 | Spain | 0.95 | 10 | Italy | 1,426 | 9 | France | 0.81 |
9 | China | 0.90 | 11 | Spain | 1,265 | 10 | Spain | 0.78 |
10 | South Korea | 0.87 | 12 | Taiwan | 1,251 | 11 | Netherlands | 0.76 |
11 | Italy | 0.83 | 13 | Singapore | 1,018 | 12 | Canada | 0.72 |
12 | Taiwan | 0.80 | 14 | Switzerland | 983 | 13 | Italy | 0.71 |
13 | Japan | 0.77 | 15 | Netherlands | 887 | 14 | United Kingdom | 0.68 |
4 | India | 0.74 | 16 | Australia | 841 | 15 | Australia | 0.59 |
Source: Bibliometric data on top 16 most productive countries |
References
Campbell D., Picard–Aitken M., Côté G. et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31; 66–83.
Canadian Institutes of Health Research Act, S.C. c. 6 (2000) [ PDF (260 KB) - external link ].
CIHR (2010). Health Research Roadmap: Creating innovative research for better health and health care.
CIHR (2008). Strategic Training Initiative in Health Research (STIHR) 2001–2006: Evaluation report.
Government of Canada (2012). Jobs, growth and long–term prosperity: Economic Action Plan 2012, The Budget Speech [ PDF (4.84 MB) - external link ].
Government of Canada (2011). The Next Phase of Canada’s Economic Action Plan: Low Tax Plan for Jobs and Growth [ PDF (4.84 MB) - external link ].
Ismail S., Nason E., Marjanovic S. & Grant J. (2009). Bibliometrics as a tool for supporting prospective R&D decision–making in the health sciences: Strengths, weaknesses and options for future development. Rand Corporation, Santa Monica, CA.
Kessel, F., Rosenfeld, P.L., & Anderson, N.B. (Eds.) (2008). Interdisciplinary research: Case studies from health and social sciences. Oxford. University Press.
Marjanovic, S., & Chonaill, S. N. (2010). Health and medical research in Singapore: Observatory on health research systems [ PDF (1.25 MB) - external link ].
Observatoire des sciences et des technologies (2010). Bibliometric analysis of INMHA–related research, 1997–2008.
Science, Technology and Innovation Council (2011). State of the nation 2010: Canada’s science, technology and innovation system.
Thomson Reuters (2011). A bibliometric analysis of regenerative medicine [ PDF (472.5 KB) - external link ].
U.S. National Library of Medicine (2011). Fact sheet: Medical subject headings.
Evaluation Unit
Resource, Planning and Management Portfolio
Canadian Institutes of Health Research (CIHR)
160 Elgin Street, 8th Floor
Address Locator 4809A
Ottawa, Ontario
Canada K1A 0W9
Phone: 1–613–941–2672
Toll Free: 1–888–603–4178
Fax: 1–613–954–1800
Web: Canadian Institutes of Health Research
Footnotes
- footnote 1
-
Bibliometric analysis is used to asess the extent of knowledge creation and scientific impact of research in a given field by measuring, among other things, the volume of publications produced as well as the relative frequency with which they are cited, respectively.
- footnote 2
-
Mann Whitney U test performed to test the distribution of ARC scores across the three groups (catalyst–funded, team–funded and Canada).
- footnote 3
-
Papers written by researchers while they were receiving funding from an OOGP grant published one year following the start of the grant (effective date) to one year following the end of the grant (expiry date).
- footnote 4
-
Average of Relative Citation (ARC) scores presented in this report excludes self–citations.
- footnote 5
-
Mann Whitney U test performed to test the distribution of ARC scores across the two groups (OOGP supported and Canadian health research papers).
- footnote 6
-
Mann Whitney U test performed to test the distribution of ARC scores across the two groups (catalyst–funded and non–funded).
- footnote 7
-
Separate Mann Whitney U tests were performed to test the distribution of (a) journal articles (b) books/book chapters and (c) reports/technical reports across the two groups (team and catalyst). To account for the possible effects of multiple testing (3 tests), the probability level for statistical significance was adjusted to p<0.05/3=0.017.
- footnote 8
-
Duration defined for RMNI as number of term years of grant at time of survey (2011); duration for OOGP is total number of term years per grant (all OOGP grants reported on through RRS had expired their authority to use funds).
- footnote 9
-
Separate Mann Whitney U tests were performed to test the distribution of (a) journal articles per year of grant, (b) journal articles per researcher and (c) journal articles per dollars expended across the two groups (team and catalyst). To account for the possible effects of multiple testing (3 tests), the probability level for statistical significance was adjusted to p<0.05/3=0.017.
- footnote 10
-
Due to length considerations, three profiles of RMNI projects are presented in this report and were selected for inclusion based on their illustration of a multitude of varying high impact results.
- footnote 11
-
CIHR defines a Nominated Principal Investigator as an individual who is responsible for the direction of the research.
- footnote 12
-
RMNI grant expenditures (as of May 2012) were analyzed using data captured through the Grants in Aid of Research Statement of Account (Form 300) for CIHR that reports on annual expenditures from CIHR funding investments. Student categories listed in Form 300 are Bachelors, Masters and Doctorate while non–student categories are Postdoctoral and a general category of “Other”. Of note, no explicit category exists for research assistants and technicians.
- footnote 13
-
Mann Whitney U test was performed to test the distribution of the average number of staff/trainees involved across the two groups (team and catalyst)
- footnote 14
-
Mann Whitney U test was performed to test the distribution of the average number of grants/awards leveraged across the two groups (team and catalyst).
- footnote 15
-
It should be noted that while surveyed and interviewed researchers were successful applicants to RMNI, it is possible they could have submitted applications that were not successful as well (either as a nominated principal investigator and/or co–grantee). To help control for this potential confound, researchers were asked in the evaluation to respond about the review process used in the assessment of a specific, successful RMNI grant application and not the process in general.
- footnote 16
-
Retained for specific applications only when specific expertise is required and who prepare a written review but do not attend the peer review committee meeting.
- footnote 17
-
Attend the meeting and are assigned to an application, prepare a written review and lead the review of a proposal during the meeting.
- footnote 18
-
Act as discussants and are not required to provide a written review.
- footnote 19
-
See, for instance, McDavid, J C., & Hawthorne, L.R.L. (2006). Program evaluation and performance measurement: An introduction to practice. Thousand Oaks, CA: Sage Publications. Also, Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (2004). Handbook of practical program evaluation. San Francisco, CA: Jossey–Bass.
- footnote 20
-
PubMed US National Library of Medicine National Institutes of Health
- footnote 21
-
Although OST’s database includes several types of documents, only articles, research notes and review papers are included, as these are the primary means of disseminating new knowledge
- footnote 22
-
More details on the classification scheme can be found at NSF
- footnote 23
-
Papers written by researchers (NPIs) while they were receiving funding from a CIHR Open Operating Grant Program (OOGP) grant published one year following the start of the grant (effective date) to one year following the end of the grant (expiry date)
- footnote 24
-
Although the publication window used to align with RMNI’s lifecycle was 2004–2010, the first RMNI funding competition (to award catalyst grants) was held in November 2003 and the next competition (for team grants) was held in May 2004. As such, it is possible that some 2004 regenerative medicine and nanomedicine papers produced by RMNI–funded researchers funded in these competitions could have been published prior to RMNI grant funds being received.
- Date modified: