Open Reforms Pilots: Spring 2013 Fellowship Competition Report – Long Descriptions

Figure 1. Would you classify yourself as a:

For the career stage where reviewers were asked, Would you classify yourself as a: 20 respondent (14%) consider themselves as Early career scientist (<5 years as an independent researcher); 65 (45%) as Mid-career scientist (5-10 years as an independent researcher) and 60 (42%) describe themselves as Senior scientist (>10 years as an independent researcher).
Response Percentage Count
Early career scientist (<5 years as an independent researcher) 14% 20
Mid-career scientist (5-10 years as an independent researcher) 45% 65
Senior scientist (>10 years as an independent researcher) 42% 60
Total Responses 145

Back to report

Figure 2. Which research position(s) do you currently hold at this time?

For the question Which research position(s) do you currently hold, 49 are Professors (34%), 72 are Associate Professors (50%), 20 are Assistant Professors (14%), 14 are Researchers (10%), 6 are Clinicians (4%) and three selected other.
Response Percentage Count
Professor 34% 49
Associate Professor 50% 72
Assistant Professor 14% 20
Researcher 10% 14
Clinician 4% 6
Other (specify): 2% 3
Total Responses 145

Back to report

Figure 3. Which of the following is your primary research domain:

For the question Which of the following is your primary research domain, 115 reviewers classify themselves as Biomedical (79%), 14 as Clinical (10%), 6 as Health systems/services (4%) and 10 as Social, cultural, environmental and population health (7%).
Response Percentage Count
Biomedical 79% 115
Clinical 10% 14
Health systems/services 4% 6
Social, cultural, environmental and population health 7% 10
Total Responses 145

Back to report

Figure 4. Did you look at the other reviews of the applications assigned to you?

129 respondents (89%) looked at the other reviews of the applications assigned to them while 16 (11%) said they did not.
Response Percentage Count
Yes 89% 129
No 11% 16
Total Responses 145

Back to report

Figure 5. How much additional time in total (all applications) did you spend reading reviews of other reviewers?

Respondents indicated that 53 (41%) of them spent under an additional hour reading reviews of other reviewers, while 67 (52%) spent between 1 & 2 hours and 9 (7%) spent more than three hours.
Response Percentage Count
Under an hour 41% 53
1 to 2 hours 52% 67
3 hours or more 7% 9
Total Responses 129

Back to report

Figure 6. How often did the reading of other reviews help finalize your submitted score?

To the question How often did the reading of other reviews help finalize your submitted score, 10 respondents (8%) said never, 37 (29%) said rarely, 62 (48%) said occasionally, 14 (11%) said often, 4 (3%) said very often, while 2 (2%) answered always.
Response Percentage Count
Never (0%) 8% 10
Rarely (up to 25%) 29% 37
Occasionally (>25% to 50%) 48% 62
Often (> 50% to 75%) 11% 14
Very Often (>75%) 3% 4
Always (100%) 2% 2
Total Responses 129

Back to report

Figure 7. Did you initiate an online discussion thread?

104 respondents (72%) said they initiated an online discussion thread while 40 (28%) did not.
Response Percentage Count
Yes 72% 104
No 28% 40
Total Responses 144

Back to report

Figure 8. What factor(s) did you use to determine whether an online discussion was needed for an application?

To the question What factor(s) did you use to determine whether an online discussion was needed for an application, 90 respondents (87%) said scoring discrepancy, 2 (2%) said content clarification required, 3 (3%) said quality check, while 9 (9%) had other reasons.
Response Percentage Count
Scoring discrepancy 87% 90
Content clarification required 2% 2
Quality check 3% 3
Other: 9% 9
Total Responses 104

Back to report

Figure 9. Who should decide when an online discussion must take place?

When asked Who should decide when an online discussion must take place, 42 respondent (29%) said CIHR (e.g., using a decision rule within ResearchNet), 87 respondent (61%) said reviewers (at their discretion), and 14 (10%) didn’t know.
Response Percentage Count
CIHR (e.g., using a decision rule within ResearchNet) 29% 42
Reviewers (at their discretion) 61% 87
Don’t know 10% 14
Total Responses 143

Back to report

Figure 10. Did you participate in an online discussion?

126 respondents (88%) participated in the online discussion while 17 (12%) did not.
Response Percentage Count
Yes 88% 126
No 12% 17
Total Responses 143

Back to report

Figure 11. Did online discussion increase the amount of time required to complete the review of an application (on average) in this competition?

When asked Did the online discussion increase the amount of time required to complete the review of an application (on average) in this competition, 69 respondent (55%) said under an hour, 48 (38%) said one to two hours, while 9 respondents (7%) said 3 hours or more.
Response Percentage Count
Under an hour 55% 69
1 to 2 hours 38% 48
3 hours or more 7% 9
Total Responses 126

Back to report

Figure 12. How often did the online discussion help finalize your submitted score?

When asked How often did the online discussion help finalize your submitted score, 11 respondents (9%) said never, 31 (25%) said rarely, 47 (37%) said occasionally, 17 (13%) said often, 16 (13%) said very Often, while 4 (3%) said always.
Response Percentage Count
Never (0%) 9% 11
Rarely (up to 25%) 25% 31
Occasionally (>25% to 50%) 37% 47
Often (> 50% to 75%) 13% 17
Very Often (>75%) 13% 16
Always (100%) 3% 4
Total Responses 126

Back to report

Figure 13. How much time, on average, did you spend on:

On average, reviewers spent 78 minutes reading a single application (standard deviation of 67 minutes), 37.8 minutes writing the review of a single application (standard deviation of 26 minutes), 87 minutes reviewing a single application (standard deviation of 62 minutes) and 25 minutes participating in an online discussion for an application (standard deviation of 22 minutes).
Task Average in hours Standard Deviation
Reading a single application 1.30 hours (78 minutes) 1.12 (67 minutes)
Writing the review of a single application 0.63 hours (37.8 minutes) 0.43 (26 minutes)
Reviewing a single application 1.45 hours (87 minutes) 1.03 (62 minutes)
Participating in an online discussion for an application 0.42 hours (25 minutes) 0.37 (22 minutes)

Back to report

Figure 14. Was the peer review process efficient?

118 respondents (84%) said that the peer review process was efficient while 22 (16%) said it was not.
Response Percentage Count
Yes 84% 118
No 16% 22
Total Responses 140

Back to report

Figure 15. Was the technology employed to conduct the reviews user-friendly?

126 respondents (91%) said that the technology employed to conduct the reviews user-friendly while 13 (9%) said it was not.
Response Percentage Count
Yes 91% 126
No 9% 13
Total Responses 139

Back to report

Date modified: