Evaluation of the Open Operating Grant Program – Long Descriptions
Figure 1-1: Impact of supported papers produced by OOGP-funded researchers vs. OECD health research comparators (2001-2009)
2001
2002
2003
2004
2005
2006
2007
2008
Supported by OGP Funding
1.4
1.39
1.4
1.48
1.46
1.44
1.47
1.63
Canada
1.23
1.21
1.2
1.19
1.21
1.21
1.25
1.31
Australia
1.15
1.12
1.12
1.16
1.16
1.17
1.22
1.26
France
0.98
1.01
1
1.02
1.06
1.06
1.14
1.16
Germany
1.07
1.07
1.08
1.09
1.12
1.11
1.18
1.23
Netherlands
1.3
1.28
1.32
1.32
1.35
1.32
1.41
1.44
Switzerland
1.41
1.39
1.39
1.41
1.48
1.4
1.48
1.57
United Kingdom
1.21
1.25
1.24
1.25
1.27
1.26
1.3
1.41
United States
1.33
1.33
1.32
1.3
1.3
1.3
1.34
1.36
« Back to Figure 1-1
Figure 1-2: Scientific impact of OOGP-supported research papers (ARC)
Average Relative Citation
2000-2005 ARC score
1.44
2006-2009 ARC score
1.54
Overall ARC score (2000-2009)
1.5
« Back to Figure 1-2
Figure 1-3: Journal article productivity per year of grant duration by pillar of respondent
1st Research Theme/Pillar
Average number of publications per year of grant
Biomedical Research
2.34
Clinical Research
2.03
Health Systems & Services Research
1.31
Soc, Cultural, Enviro, Popln Hlth Research
2.38
Total
2.26
« Back to Figure 1-3
Figure 1-4: Publication behavior by grant duration - When do supported researchers publish?
CSY
CSY+0
CSY+1
CSY+2
CSY+3
CSY+4
CSY+5
CSY+6
CSY+7
CSY+8
CSY+9
All Durations (N=450)
7.52%
12.31%
16.77%
20.37%
17.53%
11.99%
6.37%
3.82%
2.12%
0.58%
5-year Grants (N=72)
10.81%
15.45%
15.31%
19.24%
17.28%
11.94%
6.18%
2.25%
0.84%
0.14%
3-year Grant (N=297)
6.93%
11.95%
17.73%
21.42%
17.09%
11.78%
5.66%
3.75%
2.54%
0.69%
« Back to Figure 1-4
Figure 1-5: Books/book chapters and reports published as a result of OOGP grants
1st Research Theme/Pillar
Books/Book Chapters Published
Reports/Technical Reports Published
Biomedical Research
.99
.24
Clinical Research
1.05
.01
Health Systems & Services Research
.08
.38
Soc, Cultural, Enviro, Popln Hlth Research
1.13
.55
Total
.96
.25
« Back to Figure 1-5
Figure 2-1: Impact of applicants for two years following competition by application status and Canadian papers in health fields by publication year (2000-2009) (ARCs)
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
Successful applications
1.69
1.5
1.38
1.46
1.57
1.55
1.56
1.51
1.74
1.64
Unsuccessful applications
1.43
1.47
1.32
1.45
1.52
1.39
1.4
1.44
1.53
1.58
Applications by researchers who are never funded
1.39
1.25
1.47
1.45
1.32
1.2
1.39
1.45
1.47
1.2
Canada
1.24
1.21
1.19
1.19
1.18
1.2
1.2
1.25
1.33
1.33
« Back to Figure 2-1
Figure 2-2: Average Relative Citations of supported papers of funded researchers by peer review committee percentile ranking and publication year (2001-2009) (ARCs)
2001
2002
2003
2004
2005
2006
2007
2008
2009
Always Top Ranked (n=74)
0.66
2.12
1.34
1.68
1.86
2.07
1.8
2.11
2.05
Sometimes Top Ranked (n=470)
1.65
1.47
1.57
1.55
1.54
1.61
1.64
1.75
1.7
Never Top Ranked (n=956)
1.4
1.24
1.21
1.41
1.4
1.22
1.35
1.58
1.5
« Back to Figure 2-2
Figure 2-3: Previously funded status of OOGP renewal applications with FRNs, by Competition (2000-2010)
First version
With more than one version
FY 2000-2001
48.97%
51.03%
FY 2001-2002
54.40%
45.60%
FY 2002-2003
55.56%
44.44%
FY 2003-2004
66.41%
33.59%
FY 2004-2005
66.84%
33.16%
FY 2005-2006
68.48%
31.52%
FY 2006-2007
75.44%
24.56%
FY 2007-2008
76.81%
23.19%
FY 2008-2009
76.17%
23.83%
FY 2009-2010
76.44%
23.56%
« Back to Figure 2-3
Figure 2-4: Concordance between independent and committee stage assessments
Bottom 20% at committee stage
60-80% at committee stage
40-60% at committee stage
20-40% at committee stage
Top 20% at committee stage
Was in top 20% at Independent stage
0%
1%
4%
22%
75%
Was in 20-40% at Independent stage
1%
6%
24%
52%
21%
Was in 40-60% at Independent stage
5%
21%
49%
22%
3%
Was in 60-80% at Independent stage
16%
55%
21%
4%
1%
Was in bottom 20% at Independent stage
78%
17%
2%
1%
0%
« Back to Figure 2-4
Figure 2-5: Origins of top 5% of committee stage applications
Top 20% at independent stage
20-40% at independent stage
40-60% at independent stage
60-80% at independent stage
Top 5% at committee stage
94.59%
4.69%
0.60%
0.12%
« Back to Figure 2-5
Figure 2-6: True Positive applications perform better than the other three comparison groups
Average Relative Citation
True negative
1.45
False positive
1.59
False negative
1.54
True positive
1.63
« Back to Figure 2-6
Figure 3-1: Percent of funded researchers saying research results have had impacts
Percent of respondents who said research had…
Total
%
Health system/care consumer level impacts
22.1%
Health system/care organization level impacts
26.0%
Health system/care provider level impacts
34.4%
Generated subsequent research by others
44.2%
Generated subsequent research by project team members
88.2%
« Back to Figure 3-1
Figure 3-2: Percent saying OOGP research has influenced stakeholders to a considerable or great extent
Community/municipal organizations
0.3%
1.7%
Federal and provincial reps
1.7%
Health system/care manager
1.7%
Health care professional orgs
2.3%
Media
2.5%
Patients/consumers
4.5%
Health system/care practitioners
6.2%
Industry
8.7%
Formal stakeholders
26.5%
Researchers/academics
61.4%
« Back to Figure 3-2
Figure 4-1: Average number of full-time equivalent trainees per year of grant by competition year
Year of competition
Competition year 2000
Competition year 2001
Competition year 2002
Competition year 2003
Competition year 2004
Total average per year
1.54
1.38
2.43
1.73
3.81
« Back to Figure 4-1
Figure 4-2: Proportion of annual expenditure on the OOGP by pillar 2000/01 – 2010/11
Expenditures (New methodology)
2000-2001
2001-2002
2002-2003
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
Biomedical
61%
74%
79%
81%
82%
82%
81%
80%
80%
79%
78%
Clinical
5%
7%
7%
7%
7%
7%
8%
8%
9%
10%
10%
Health Systems & Services
1%
2%
3%
3%
3%
3%
3%
4%
4%
4%
4%
Social, Cultural, Environmental or Population Health
2%
2%
4%
5%
6%
6%
7%
8%
7%
7%
7%
Not Specified
31%
14%
8%
4%
2%
1%
0%
0%
0%
0%
0%
« Back to Figure 4-2
Figure 4-3: Barriers and challenges to Pillar III/IV applicants in the OOGP
There are three broad categories of barriers and challenges faced by Pillar III/IV applicants to the OOGP. These are (a) diverse projects cutting across disciplines and methodologies; (b) higher proportion of applications rated as "unfundable" and lower average peer review scores; and (c) renewal behaviours.
« Back to Figure 4-3
Figure 4-4: Likelihood of an application being deemed non-fundable (odds ratios)
Likelihood
Theme IV
2.14
Theme III
2.15
Theme II
1.68
Theme I
1
« Back to Figure 4-4
Figure 5-1: Open Operating Grant Program: Number of applications and success rates, 2000-2010
Funded
Fundable, Not Funded
Not Fundable
Successe Rate (%)
2000-01
801
500
1,077
34%
2001-02
833
604
1,104
33%
2002-03
935
723
1,217
33%
2003-04
880
915
1,284
29%
2004-05
911
915
1,096
31%
2005-06
957
1,167
1,241
28%
2006-07
916
1,382
1,374
25%
2007-08
829
1,743
1,322
21%
2008-09
816
1,568
1,241
23%
2009-10
797
1,599
1,284
22%
« Back to Figure 5-1
Date modified:
2012-10-29