Evaluation of the Vanier Canada Graduate Scholarships Program
Final Report 2014
Canadian Institutes of Health Research
160 Elgin Street, 9th Floor Address Locator 4809A
Ottawa, Ontario K1A 0W9 Canada
Acknowledgements
This report was authored by Patrick MacGuire. The evaluation was carried out by an Evaluation Working Group which, during the different phases of the evaluation, included: Patrick MacGuire, Michael Goodyer, Shannon Clark-Larkin, Nicole Michaud, Joanne Tucker, Heather Bell, Kwadwo (Nana) Bosompra, Christopher Manuel, Igor Tupitsyn, Marc-Étienne Joseph and Rachele Goulet.
Special thanks to all the participants in this evaluation and to Circum Network Inc., Nicole Michaud and Joanne Tucker for assisting with data collection and analysis.
Table of Contents
Executive Summary
The Vanier CGS program was created by the government of Canada in 2008 to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning. The program is implemented through the three Federal Granting agencies and invests approximately $25 million annually to support 500 Canadian and international doctoral students studying at Canadian universities.
This is the first evaluation of the Vanier CGS program and was led by the Canadian Institutes of Health Research (CIHR) in partnership with the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). The evaluation examines the relevance and performance of the program as well as its design and delivery in order to meet Treasury Board requirements for evaluation and inform the renewal of its Terms and Conditions which expire July 31, 2014.
Key Findings
Relevance
Alignment with Government Roles, Responsibilities and Priorities
- The Vanier CGS program is aligned to federal government and granting agency priorities and there is a clearly established role for the federal government in providing education-related financial support to doctoral students in Canada.
- The Vanier CGS program has remained a federal government priority since its inception and key informants interviewed noted that the impacts of graduate training are considerable and an important reason for the Government of Canada to support world-class doctoral students. The goals of the Vanier CGS program align seamlessly with the research capacity building priorities of the three Federal Granting Agencies to attract, train and retain high-quality talent.
Continued Need
- Evaluation findings indicate that the education-related financial need of students is being fully met by the Vanier award and that the majority of award holders who graduate are working in the university sector and the majority of those who have yet to complete their studies intend to work in the same.
- Data on projected levels of supply and demand for university professors and assistants in Canada over the next ten years indicates a balance between both, predicting that there will be potential employment opportunities in Canada for a sizable majority of students supported through this program. Given the career aspirations of Vanier scholars, the need for the program will be determined at least in part on the demand for professors in Canada. If the supply is greater than demand, there is a risk that the program will fund a large majority of students who eventually leave the country to seek employment elsewhere.
Performance
Attraction and Recruitment
- The role of the Vanier award in attracting and recruiting students from outside of Canada is hindered by the inability of institutions to guarantee receipt of the award, the high competitiveness of the program in relation to the relatively small number of scholarships awarded and the reluctance of doctoral supervisors to nominate students with whom they have not yet worked with and who are not already enrolled in the institution in which they are seeking Vanier support. As such, the ability of the program to attract and recruit students from outside of Canada has been limited due to the fact that foreign applicants face barriers in terms of obtaining nomination support from Vanier eligible institutions. Based on this, the program has not met its objective of attracting and recruiting world-class students to Canada because the majority of supported students were already studying in Canada.
- To date, the Vanier program has been mostly supporting students already studying in the institution in which they sought scholarship support (93%). In addition, 70% of Vanier award holders were already enrolled in doctoral studies at their Canadian institution at time of application.
- Of Vanier scholars surveyed, most (86%) reported living in Canada at the time of their application to the award and 13% were foreign students. In total, foreign Vanier scholars were attracted from 40 different countries worldwide with close to one-quarter (24%) coming from the United States.
- Although the Vanier CGS award is open to students worldwide, 8 out of 10 institutions were not able to identify enough suitable candidates to fulfill their nomination quotas at the end of the first allocation period. The shortfall of suitable applicants nominated was not due to a lack of interested candidates as institutions screened out a little over half of all applicants in the first stage of the selection process.
- As an incentive to enroll in doctoral studies in Canada, survey results showed that the existence of the Vanier CGS program appears to have played a moderate to high degree of importance in the decision of about half of award holders (53% for Canadians and 51% for foreign students). In addition, a small number of Vanier scholars (7%) reported that they would have pursued doctoral studies outside of Canada if they had not received the scholarship (although this is most likely due to a majority already being enrolled in doctoral studies at their Canadian institution when they applied).
World-Class Students
- University representatives generally agreed that students who applied and were nominated for a Vanier CGS award were usually world-class although some noted that, while awardees were world-class, not all nominees were considered as such. Current and former Vanier selection committee members who participated in an expert panel review concluded that the overall quality of both successful and unsuccessful applications was consistent over time and that their judgments in the panel were, for the most part, in agreement with those made by the granting agency selection committees.
- According to assessments provided by Vanier scholar supervisors, the majority of recipients outperformed other doctoral students on a variety of dimensions related to research, teaching and service. Furthermore, their ratings showed that the area they felt Vanier scholars are most displaying exceptional leadership in is in research (62%). Of particular importance given the program’s focus on leadership, 31% of supervisors indicated that Vanier scholars demonstrated an exceptional ability to lead/influence others.
Impact of Vanier CGS on Scholars
- Vanier scholars (78%) reported that the award had the strongest positive impact on their current financial situation (for those who have not yet graduated), the need for obtaining income during their studies (68%) and the time they were able to devote to their doctoral studies (66%).
- Evaluation findings show that few Vanier award holders incur debt related to their doctoral studies; however, the proportion of award holders who are debt free is similar to that of Canadian doctoral students in general.
- One-fifth (19%) of Vanier scholars reported working in a non-academic position during their doctoral studies indicating that the majority of award holders do not tend to seek non-academic employment while in school.
Training
- The majority of Vanier scholars are satisfied with their skill development and training opportunities related to their studies and 8 out of 10 award holders expressed some degree of satisfaction with their overall research environment. In terms of dissatisfaction, close to 2 in 10 reported feeling discontent with the teaching opportunities that were available to them.
- Over one-third (38%) of Vanier scholars reported some gaps in the training they received during their doctoral studies. The most often cited training gap was related to teaching skills (in part due to the lack of teaching assistantships offered to them as a result of holding the Vanier award) with the second most frequently mentioned being the lack of opportunities to develop the skills required to conduct and manage a research project.
Retention and Career Outcomes
- Survey results showed that 20% of Vanier recipients have completed their doctoral studies to date with 59% of graduates having applied for the award in 2008 and 38% in 2009. A majority (63%) of Vanier scholar graduates were living in Canada, with students in natural sciences and engineering (49%) less likely to live in Canada compared to students in health and social sciences and humanities (76% and 67% respectively). The most important factors influencing the decision to leave Canada were prospective employment and an actual employment offer. For Vanier scholar graduates who left Canada, the United States (23%) was the most common country they relocated to. Of those who left and did not relocate to the United States, the majority were residing in a European country. Furthermore, 8 out of 10 Canadian Vanier scholar graduates outside of the country indicated a moderate to high likelihood of returning to Canada whereas 29% of foreign Vanier scholars living abroad indicated a moderate to high likelihood of returning.
- Of those who have not yet completed their studies, the majority of Vanier scholars intend to work in the university sector with most expressing interest in pursuing postdoctoral and university professor/researcher positions. For graduates, 74% indicated being employed in the university sector with 51% working as postdoctoral fellows/associates. As to whether their doctoral training was useful in preparing them for their career, close to 8 out of 10 Vanier scholars confirmed that it was moderately to extremely useful.
- Of the Vanier scholar graduates who are living outside of Canada, 61% indicated that they are employed as a postdoctoral fellow or associate. For Vanier scholars who have not yet completed studying, 95% of Canadian students indicated an intention to remain in the country after graduation (compared to 86% for foreign students).
- For Vanier scholar graduates employed, two-thirds (67%) indicated they lead/influence others within their organization while a just over one-third (37%) stated that they lead/influence the strategic direction of their organization (although this finding should be viewed in context as half of Vanier graduates were employed as postdoctoral fellows/associates).
Vanier Alumni Network
- An expected output of the Vanier CGS program was the establishment of an alumni network of Vanier scholars. At the time of this study, the network had not yet been implemented however data obtained through the evaluation can help inform its development. Results showed that Vanier scholars would prefer to communicate with each other through email (57%), followed by a LinkedIn group (50%) and Facebook (46%). About one-quarter of Vanier scholars who provided their own suggestion indicated a desire to meet with other scholars in person through annual meetings and conferences. In terms of being kept informed of Vanier CGS-related updates and activities, periodic emails sent by Vanier program staff was preferred by most (74%) followed by a Vanier electronic newsletter (71%).
Impact of Vanier CGS on Canadian Universities
- University representatives interviewed often indicated that the presence of Vanier recipients enhanced the reputation of their faculty, program or laboratory, but that it had little to no impact on the reputation of the institution as a whole. Vanier scholar supervisors indicated that the program has had the greatest positive impact in terms of enhancing the reputation of their departments through enhancing their own reputation as a supervisor of doctoral students. The area in which the highest proportion of supervisors identified as having not been impacted by the Vanier CGS program is the reputation of Canada as a research environment. Finally, two areas in which a small minority of supervisors indicated the program has had a negative effect has been on the climate between doctoral students under their supervision and the climate within their research team or lab.
Prestige
- Evaluation findings confirm that award recipients, their supervisors and universities view the Vanier CGS program as a highly prestigious award in Canada. The extent of those who viewed the Vanier award as being highly prestigious outside of Canada was lower although more foreign students rated the award as having more prestige worldwide than Canadians.
Design and Delivery
- While the award amount is meeting the educational-related financial needs of almost all recipients, questions remain as to the incremental outcomes that result from the additional financial value of the Vanier award in comparison to other doctoral scholarship awards given the absence of comparator data at the time of this study. Survey findings show that almost all award recipients (92%) expressed the highest level of satisfaction for the dollar amount of the Vanier award (with proportions consistent across fields of study). As well, the majority of Vanier scholars reported having no debt related to their doctoral studies. When asked for suggestions for improvement to the program, some universities representatives and Vanier supervisors as well as a small percentage (5%) of award recipients felt that the Vanier award amount was too high. Some argued that a portion of the funding should be allowed to cover research and travel-related costs (e.g., to attend conferences) while others suggested that having more CGS awards would be preferable.
- With respect to the duration of the award, some university representatives, Vanier scholars and doctoral supervisors suggested that the three-year funding period is too short considering the average amount of time it takes to complete a doctoral degree.
- Opinions were generally positive regarding the university allocation system; however, it was mentioned that smaller institutions had more difficulty operating within this allocation system because of their limited resources and small quotas. It was also indicated that in some cases universities would be more comfortable with an annual allocation quota, because it would require institutions to nominate candidates each year which would mitigate a common situation whereby institutions conserve their allocations in each of the first two years of the cycle, passing up high-calibre candidates for the prospect of stronger candidates in the following year. In the final year of the cycle where many universities have unused quotas as a result, institutions will then put forward candidates that in some cases may not meet the criteria of the program in order to use up their remaining allocations. This conservatism is also a barrier to foreign candidates being put forward for nomination given the level of risk associated with students unfamiliar to institutions (the level of uncertainty and risk that universities had to manage when nominating new students was identified as the main challenge faced in the delivery of the program at the university level).
- Opinions concerning allocations specifically for foreign students were mixed. Key informant interviewees who were in disagreement did so on that basis that the Vanier scholarship should reward excellence, not origin, that they would not be able to fill international quotas (especially universities attracting few international students) and that there is a scarcity in student funding. University representatives in favour argued that the knowledge economy transcends borders while some Ontario-based universities explained that their provincial funding does not cover international students, and that having a portion of the Vanier scholarships reserved for this group would be a positive decision given their situation. Vanier granting agency selection committee members were also in favour of targeting foreign students and raised the suggestion of a direct applicant route; however, they agreed that significant changes to the Vanier CGS program design would have to be considered to allow for this.
- The assessment of leadership in the university and granting agency selection processes was identified as a significant challenge given its subjectivity and the reported inconsistency in evaluating it. Both university representatives and Federal Granting Agency selection committee members stressed the need for a clearer definition and scoring process for what they felt was the most difficult yet most important and defining feature of the program.
- In terms of the application and selection process, award recipients were generally satisfied with both and expressed the highest degree of satisfaction with the information available on how to apply to Vanier, the eligibility requirements and the information available on the program.
- Universities and Vanier selection committee members expressed concerns pertaining to the timing of the Vanier application process. For one, the deadline was thought to be too close to the start of the fall semester which leaves little time for the preparation and assessment of applications. Moreover, the time lapse between the application period and when the decision is made at the agency level discourages universities from presenting first-year doctoral nominees (which would include new or foreign students not already studying at the university), because they would have had to apply to the Vanier program almost a full year before starting their doctoral studies. It was also noted by current and former Vanier granting agency selection committee members that the Vanier CGS program’s application deadline did not work in favour of international applicants given internal institutional deadlines for graduate program admissions.
Vanier CGS Cost-Efficiency
- Findings indicate that the Vanier CGS program has been delivered by the Federal Granting Agencies in a cost efficient manner. Data shows that the program has reduced its administrative costs as a proportion of its total expenditures from 8.7% in 2009-10 to 2.2% in 2013-14. As well, the aggregate proportion of administrative costs to expenditures was below that of the Federal Granting Agencies for the period of 2009-10 to 2012-13.
Conclusions
Evaluation findings indicate that the program is supporting world-class doctoral students – the majority of whom are Canadian, with most Vanier scholars demonstrating exceptional leadership in the area of research. The Vanier CGS award is having the strongest positive impact on students’ financial situation, their need for obtaining income during their studies and the time they were able to devote to their education.
The evaluation found that the Vanier CGS program is considered to be a highly prestigious award in Canada and the presence of Vanier scholars enhances the reputation of a faculty, program or laboratory but has little to no impact on the reputation of the institution. The majority of Vanier scholars are satisfied with their training and skill development and most report that their training has been useful in preparing them for their career. Of the Vanier scholars who have completed their studies, the majority are employed in the university sector and are living in Canada.
Findings from the evaluation regarding the communication preferences of Vanier scholars can help inform the development of an alumni network of Vanier scholars, which is an expected output of the program that has not yet been implemented.
The evaluation identified key barriers that negatively impact its ability to attract and recruit students from outside of Canada: the inability of institutions to guarantee receipt of the award, the high competitiveness of the program in relation to the relatively small number of scholarships awarded and the reluctance of doctoral supervisors to nominate students with whom they have not yet worked with and who are not already enrolled in the institution in which they are seeking Vanier support.
Opinions on the design and delivery of the program were positive overall, with the majority of Vanier scholars reporting satisfaction with the award amount and duration, however the evaluation found that the three year nomination allocation system is contributing to the reluctance of institutions to nominate new students (including foreign candidates) which limits the ability of the program to attract students from outside the country and has had a negative impact on the quality of candidates put forward for nomination. As a result, many universities suggested that the three year allocation cycle be changed to annual nominations.
As the objective of the Vanier CGS program is to be an internationally competitive scholarship program that enables Canada to attract world-class doctoral students (a key feature which distinguishes it from other federal scholarship programs) and given the reluctance of institutions to nominate foreign candidates, it is important that a portion of allocations be targeted to international students outside of the country in order to facilitate their nomination and better enable the program to meet its objective in terms of attraction. As well, the program’s competition deadlines were viewed by many as being problematic in terms of attracting new (including foreign) students to Canadian institutions and, as such, should be reconsidered.
The interpretation and assessment of the leadership criterion was identified as a challenge and, in light of the program’s expected long-term outcome that Vanier scholars become leaders in Canada and abroad, there is a need to further develop and clarify the definition and assessment of leadership given the important role it plays in the selection process at the university and federal granting agencies.
While the award amount is meeting the educational-related financial needs of almost all recipients, questions remain as to the incremental outcomes that result from the additional financial value of the Vanier award in comparison to other doctoral scholarship awards given the absence of comparator data at the time of this study. There is a need therefore to pursue an additional analysis using comparator data obtained through the evaluations of other doctoral award programs to identify the incremental outcomes that result from the additional financial support offered through the Vanier scholarship.
Recommendations
- To enable the Vanier CGS program to better meet its objective of attracting and recruiting world-class doctoral students to Canadian universities, the following changes to the allocation and application processes should be considered:
- The three year allocation cycle should be changed to annual allocations with restrictions placed on carrying forward unused quotas.
- A portion of nomination allocations should be targeted for foreign students not already enrolled in the institution in which they are seeking Vanier support, with the amount calculated, at least in part, on an institution’s international student enrollment rate. The Vanier CGS program should monitor the extent to which the target of foreign student nominees is being met, per institution and in total, after each competition and use this information as part of the calculation to determine targets and allocations for the next competition year.
- The timing of the application deadlines for the Vanier CGS program should be re-considered in light of findings from the evaluation.
- To help improve the assessment of the leadership criterion in the University and Federal Granting Agency selection processes and increase the extent to which it is interpreted consistently, Vanier CGS program management should establish a clearer definition of what leadership is and how it should be evaluated – especially for foreign students. Suggestions for improvement obtained through the evaluation should be taken into consideration and any changes made should be communicated to universities and Federal Granting Agency selection committee members.
- To help inform the implementation of a Vanier Alumni Network, data obtained through the evaluation on the communication preferences of Vanier scholars for interaction with other award holders and program staff should be considered to ensure that Vanier CGS graduates are connected to the program and to other scholars after completion of their studies.
- Evaluation findings demonstrate that the Vanier program is meeting the educational-related financial needs of almost all of its recipients and that it is viewed as a highly prestigious award in Canada. However, the evaluation was unable to assess what incremental outcomes are associated with the higher value of the scholarship in comparison to the CGS and Federal Granting Agency doctoral awards as evaluations of those programs were in progress at the time of this study and comparison data was not yet available. The surveys used in these scholarship evaluations were designed to enable a comparative analysis of data across programs. As such, it will be important to undertake a further analysis to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes. Issues to be examined would include incentives to enroll in studies, financial situation of students, training (including skill acquisition), research productivity, retention rates and employment. Based on this, it is recommended that Vanier CGS program management consider the results of this analysis in the future planning and design of the program.
Background
Photo: 2011 Vanier CGS Awards Announcement - McMaster University. Photo courtesy of the Vanier Banting Secretariat.
Vanier CGS Program
Introduced in Budget 2008 (Department of Finance Canada, 2008, p. 114) with the first round of scholarships awarded in 2009, the Vanier CGS program was created to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning. Vanier scholarships are branded and promoted as a key instrument to achieve Canada’s Science and Technology Strategy (Industry Canada, 2007) goal of increasing the supply of highly qualified and globally connected graduates and build a People Advantage for Canada.
Vanier invests approximately $25 million annually to support 500 Canadian and international doctoral students studying at Canadian universities. As of 2014, approximately 1,000 scholarships valued at $50,000 per year for three years have been awarded through the program.Footnote 1 The Vanier program is administered through the three federal granting agencies: the Canadian Institutes of Health Research (CIHR), the Social Sciences and Humanities Research Council (SSHRC) and the Natural Sciences and Engineering Research Council (NSERC).
Figure 1.1 – Vanier CGS Award Details and Program Objectives
Award Details
- $50,000 annually
- 3 year duration
- Eligible to receive supplementary funding
Program Objectives
- Make scholarship initiatives such as the Vanier Program more internationally competitive
- Market Canada as a destination of choice for international students
- Enable Canada to attract world-class doctoral students
- Retain Canada`s own top doctoral students who are being sought by other countries
- Help build world-class research capacity
- Brand Canada and Canadian universities as a home for excellence in research and higher learning
- Be globally competitive and internationally recognized
Source: Vanier CGS Terms and Conditions; Vanier CGS Performance Measurement Strategy
The following are the key stakeholders for the Vanier CGS program:
- Industry Canada
- SSHRC, NSERC and CIHR
- Universities
- Department of Foreign Affairs, Trade and Development Canada (DFATD)
- Research Community
- Other Government Departments (i.e. Citizen and Immigration)
Vanier CGS Application and Selection Process
Vanier scholarships are awarded after a competitive selection process involving reviews conducted at eligible Canadian universities as well as by the federal research granting agencies.Footnote 2
University Review
Students must be nominated for a Vanier CGS award by a recognized Canadian university. Nominations are initiated in one of two ways: (1) the candidate informs the faculty of graduate studies at the selected university of their intent to apply or (2) the university initiates the nomination process by contacting the candidate. The scholarship liaison officer at each eligible Canadian university is responsible for coordinating the university review of their selected candidates and forwarding nominations for Vanier CGS awards to the appropriate federal research granting agencies in accordance with their university's allocationFootnote 3. In total, 1,800 nominations are allocated over three years to eligible institutions (200 per granting agency per year).Footnote 4
Federal Granting Agency Selection Committees
Once submitted to the appropriate granting agency, each nomination is evaluated by an agency-specific selection committee. Per year, each selection committee recommends the top 55 or 56 candidates (for a total of approximately 167 candidates among the three granting agencies) to the program’s Steering Committee based on the nominee's academic and research potential, as well as their leadership skills.
Steering Committee
The Steering Committee, which comprises the presidents of the three federal granting agencies and the deputy ministers of Industry Canada and Health Canada, makes the final decisions on funding. Each year, the Steering Committee approves approximately 167 Vanier scholars for funding based on the recommendations of the selection committees.
The Vanier Banting Secretariat
The Vanier Banting Secretariat was implemented in October 2012 and is responsible for providing administrative oversight to the Vanier CGS and Banting Postdoctoral Fellowships programs. The secretariat is located at CIHR and consists of an executive director, manager and program delivery staff from each of the federal granting agencies.
Figure 1.2 – Vanier CGS Logic Model
Evaluation Purpose and Scope
This study is the first evaluation of the Vanier CGS program and was led by the Canadian Institutes of Health Research (CIHR) in partnership with the Natural Sciences and Engineering Research Council (NSERC) and the Social Science and Humanities Research Council (SSHRC). The evaluation examines the relevance and performance of the program as well as its design and delivery in order to meet Treasury Board requirements for evaluation (see Appendix for a description of TBS requirements and for a cross-walk of evaluation questions with TBS requirements) and inform the renewal of its Terms and Conditions which expire July 31, 2014. The evaluation covers the time period from the program’s inception in 2008–2009 to the end of fiscal year 2012–2013.
As two immediate outcomes are expected to occur within four years after inception of the program (world-class students are attracted and recruited and world-class students are trained), the evaluation will assess the extent to which these two outcomes have been achieved as well as examine the progress made towards the achievement of longer-term outcomes.
Key Findings
Relevance
Need for Vanier CGS
Evaluation Question
6. Is there a continued need for the Vanier CGS program?
One indicator of continued need for the Vanier CGS program is determining whether its demand (in terms of number of applicants applying for nomination) has increased, decreased or stayed the same over time. Data on the total number of applicants indicates the demand for the program has decreased since the program’s harmonized launch through the federal granting agencies in 2009 (Figure 2.1) although the variance across most years is relatively minor.
Figure 2.1 – Number of Vanier CGS Applicants by Year
Source: ResearchNet
Another indicator of demand is the proportion of Vanier allocation quotas fulfilled over the period of the first allocation cycle (2010–2012). Under this period, a total of 52 Canadian universities received at least one nomination allocation. At the end of the cycle, only 21% of institutions fulfilled their allocation quota, which implies that most institutions were unable to identify enough high calibre candidates for nomination against the maximum amount set out by the federal granting agencies over a three year period. Furthermore, of the institutions who did not fulfill their quota, 8% did not submit a single nomination. Finally, of a total of 4,106 applicants since 2009, 2,120 or a little over half were screened out of the university nomination selection process (Figure 2.2). These results are concerning given that the Vanier CGS award is open to students worldwide yet 8 out of 10 institutions are not able to identify enough suitable candidates to fulfill their nomination quotas. As institutions screened out a little over half of all applicants in the first stage of the selection process, it can be concluded that unfilled quotas were not due to a lack of interested candidates to consider for nomination.
Figure 2.2 – Vanier CGS Application Data 2008–2012
Source: ResearchNet
*For the first competition of the Vanier CGS program in 2008, the application and peer review process was implemented differently across federal granting agencies. As such, data on unsuccessful applicants in 2008 is not equivalent and comparable to other Vanier CGS competition years and has been excluded from the analysis presented in Figure 2.2 as a result.
A final indicator of program demand examined through this study is the proportion of those offered the Vanier CGS award that declined. Data obtained from administrative records shows that, over the period of 2008–2012, only 22 (3%) successful candidates declined the Vanier award (Table 2.1). Given that Vanier recipients are ineligible to hold other federal granting agency scholarships concurrently, the small number of declines implies that Vanier CGS is the top choice for financial support for doctoral students amongst the slate of doctoral awards offered through CIHR, NSERC and SSHRC. This finding is not surprising given the relatively high dollar value of the Vanier scholarship.
2008 | 2009 | 2010 | 2011 | 2012 | Total | |
---|---|---|---|---|---|---|
CIHR | 1 | 3 | 0 | 2 | 1 | 7 |
NSERC | 1 | 2 | 2 | 1 | 0 | 6 |
SSHRC | 3 | 1 | 2 | 2 | 1 | 9 |
Vanier CGS Total | 5 | 6 | 4 | 5 | 2 | 22 |
Source: Federal Granting Agency Administrative Data
In terms of addressing the education-related financial need of doctoral students, evaluation findings indicate that the Vanier scholarship is sufficient for the majority of its recipients however some stakeholders feel that the award amount may be excessive. A survey of Vanier scholars revealed that only 5% were not concerned about their financial status at the time of their application to their doctoral program and close to half (43%) expressed a high degree of concern which implies the existence of a demonstrated need for financial support for Vanier scholars at the outset of their studies.
As to whether the education-related financial need of Vanier scholars is being addressed, of those who have not yet completed their studies, 8 out of 10 (82%) reported having no debt related to their doctoral program. For those who completed their doctoral studies, 9 out of 10 (90%) reported having no debt upon graduation. This finding holds true even when comparing Canadian and foreignFootnote 5 students (who can incur higher costs to study in Canada). As noted elsewhere in the report, the strongest positive impact that Vanier scholars felt the award had was related to the value of the award and its impact on their current financial situation (for those who have not yet graduated) as well as their need for obtaining income during their studies.
Although evaluation findings indicate that the Vanier award is having a positive impact financially on its recipients and, as noted elsewhere in the report, is viewed as being highly prestigious, other findings obtained through the evaluation raise uncertainty as to whether the value of the award is commensurate to the financial needs of its recipients. Specifically, the proportion of Vanier scholars who were debt free are in line with the overall proportion of Canadian doctoral graduates (88%) who have reported not owing any money related to their graduate studies (Statistics Canada, 2008). Although proportions of students debt-free are similar, it may be that the Vanier scholarship offsets the extent to which employment and financial support is sought by students as compared to Canadian doctoral students in general as the award was reported to have a very positive impact on recipients’ need for obtaining income during their studies.
Photo: 2009 Inaugural Vanier CGS Awards Announcement – Ottawa, Ontario. Photo courtesy of the Vanier Banting Secretariat.
As well, survey findings show that almost all award recipients (92%) expressed the highest level of satisfaction for the dollar amount of the Vanier award (with proportions consistent across fields of study). Finally, when asked for suggestions for improvement to the program, some universities representatives and Vanier supervisors as well as a small percentage (5%) of award recipients felt that the Vanier award amount was too high. Some argued that a portion of the funding should be allowed to cover research and travel-related costs (e.g., to attend conferences) while others suggested that having more CGS awards would be preferable. As comparator data on CGS and other Federal Granting Agency doctoral scholarship programs was unavailable at time of study (see methodology section for further details), it is important that further analysis be conducted to assess the incremental outcomes that are associated with the higher financial value of the Vanier award.
Evaluation findings indicate that a majority of Vanier recipient graduates are working in the university sector – a trend similar to the early stage career trajectory of Canadian doctoral graduates in general. According to a report from Statistics Canada based on data from the 2005 Survey of Earned Doctorates and the 2007 National Graduates Survey (Statistics Canada, 2011), a little over half (56%) of Canadian doctoral graduates living in Canada and the United States two years after graduation in 2005 were employed in educational services with the vast majority of them working in a university (87%). Data obtained through this evaluation shows that the majority of Vanier recipients who have not yet completed their degree intend to work in the university sector (79%) with 84% expressing interest in pursuing a position as a university professor/researcher. For Vanier scholars who have completed their studies, 74% indicated that they are employed in the university sector (with many working in postdoctoral positions).
As to whether there is sufficient demand in future years in Canada for the type of employment the majority of Vanier scholars intend to pursue after graduation, Employment and Social Development Canada (2011) predicts that supply and demand for university professor and assistant positions will be balanced over the coming years with shortages in some fields which is promising in terms of retaining Vanier graduates seeking employment in these positions. Specifically, the number of job openings is expected to be smaller over the period of 2011–2020 as compared to 2001–2010 with the number of job seekers expected to increase slightly. As well, according to projections from ESDC, most PhD graduates may eventually become university professors or assistants, which could cause a rapid rise in the number of job seekers. However, ESDC notes that based on projections and considering that labor supply and demand in this occupation were balanced, it is expected that supply and demand will continue to be balanced over the 2011–2020 period and there may be shortages in certain fields of study.
Given the career aspirations of Vanier scholars, the need for the program will be determined at least in part on the demand for professor positions in Canada. If the supply exceeds demand, there is a risk that the program will fund a large majority of students who eventually leave the country to seek employment elsewhere.
Evaluation Question
7. Does the Vanier CGS program align with government priorities and is it consistent with federal roles and responsibilities?
Alignment with Federal Government Priorities
Introduced in the Federal Government’s 2008 budget (Department of Finance Canada, 2008, p. 114), the Vanier CGS program was an initiative designed to attract the world’s best doctoral students to Canadian institutions. The program has remained a federal government priority since its inception and the most recent 2014 federal budget reiterated its role in Canada’s science, technology and innovation system and referenced the Vanier CGS program as one of several means of government support for post-secondary education (Department of Finance Canada, 2014, p. 75).
When it comes to the importance of the Vanier scholarship in addressing federal government priorities, some granting agency representatives interviewed through the evaluation thought that it depended on the nature of the goal that the program was to serve. According to them, if the goal was to create leaders and researchers, then there is a need for a separate leadership building program that is currently not being met (because research excellence and leadership do not necessarily come hand in hand).
Interviewees also thought that the Vanier program was part of a greater Government of Canada effort to brand itself as valuing research. Some argued that, while the Vanier program played an important role in meeting this goal, it was not an optimal tool to develop research capacity, due to the limited number of awards handed out per year. The role and value of the program therefore need to be assessed against a specific goal.
Compatibility with Federal Granting Agency Priorities
The goals of the Vanier CGS program align seamlessly with the research capacity building priorities of the three Federal Granting Agencies – CIHR, NSERC and SSHRC. CIHR aims to ‘invest in world-class research excellence through attracting and retaining the best international scholars and experts’ (CIHR, 2009). NSERC aims to attract, retain and develop highly qualified people in natural sciences and engineering (NSERC, 2014) while SSHRC strives to ‘make Canada a world leader in social sciences and humanities research and research training’ (SSHRC, 2013).
Alignment with Federal Roles and Responsibilities
The importance of the spill-over effects of graduate studies as a reason for the Government of Canada to support world-class doctoral students was underscored by key informant interviewees. In that sense, the impacts of graduate training were perceived as considerable and social impact comes from better educated students rather than patents and licenses. Therefore, they felt that the federal government’s involvement in supporting world-class doctoral students is a sign of its true ownership of the talent advantage priority.
Performance
Achievement of Immediate Outcomes - Capacity Development
Evaluation Question
1. To what extent has the Vanier CGS program achieved its expected immediate outcomes?
1.1. Has the Vanier CGS program attracted and recruited world-class students?
Attraction and Recruitment
Evaluation findings present strong evidence that the Vanier CGS program has not fully met its objective of attracting and recruiting world-class students. To date, the majority of awards have been given to students already enrolled in Canadian institutions at the time of application to the program. The ability of the program to attract and recruit students from outside of Canada has been limited due to the fact that potential international applicants face barriers in terms of obtaining nomination support from Vanier eligible institutions. As a result, the program has not met its unique objective (among federal granting agency scholarship awards) of attracting and recruiting the world’s best doctoral students to Canada.
Survey results reveal that the majority of Vanier scholars (86%) reported living in Canada at the time of their application to the award and that 13% of Vanier scholars were foreign students (Table 2.2). These results are concerning given that the proportion of foreign students receiving the award annually is lower than the international student enrollment rate of 22% for doctoral and postdoctoral programs in Canada in 2010 and the fact that international student enrollment has outpaced that of Canadian students over the period 1998 to 2010 (Statistics Canada, 2014). Of the number of foreign Vanier scholars, close to one-quarter (24%) were living in the United States when they applied. In total, foreign Vanier scholars were attracted from 40 different countries worldwide.
Figure 2.3 – % of Foreign Vanier Scholars by Year
Source: Survey of Vanier Scholars
2008 | 2009 | 2010 | 2011 | 2012 | Total | |
---|---|---|---|---|---|---|
Health Sciences | 3% | 8% | 3% | 11% | 10% | 7% |
Natural Sciences & Engineering | 19% | 8% | 14% | 11% | 16% | 13% |
Social Sciences & Humanities | 12% | 15% | 28% | 10% | 19% | 17% |
Vanier CGS Total | 11% | 10% | 16% | 11% | 15% | 13% |
Source: Survey of Vanier Scholars
Another important finding from the evaluation is that, at the time of application to the award, a total of 93% of Vanier scholars were already studying at the institution in which they were seeking Vanier support (approximately 70% of Vanier scholars were already enrolled in their doctoral program) (Table 2.3).
2009* | 2010 | 2011 | 2012 | Total | |
---|---|---|---|---|---|
CIHR | 66% | 68% | 63% | 82% | 70% |
NSERC | 58% | 63% | 64% | 70% | 64% |
SSHRC | 67% | 78% | 82% | 82% | 77% |
Vanier CGS Total | 64% | 69% | 69% | 78% | 70% |
Source: ResearchNet (N=669)
* data pertaining to the institution for the degree in progress of Vanier applicants was not available for 2008.
** applicants were determined to be already enrolled at the Canadian institution for which they were seeking Vanier CGS support at the time of application to the award if their degree in progress was a doctoral/PhD program and the institutions for degree in progress and degree sought were the same.
Gender (N=811) |
Mean Age (N=756) |
Language (N=830) |
Citizenship* (N=830) |
|||||
---|---|---|---|---|---|---|---|---|
Female | Male | English | French | C | R | O | ||
CIHR | 160 | < td class="text-center" headers="header30 header35 header42">11627 ± 4 | 266 | 14 | 216 | 12 | 52 | |
NSERC | 103 | 167 | 26 ± 3 | 257 | 20 | 180 | 11 | 86 |
SSHRC | 179 | 86 | 30 ± 6 | 214 | 59 | 167 | 12 | 94 |
Vanier CGS Total | 442 | 369 | 28 ± 5 | 737 | 93 | 563 | 35 | 232 |
Source: ResearchNet
* C=Canadian Citizen, R=Permanent Resident of Canada, O=Other Citizenship(s).
As part of the evaluation, key stakeholders of the Vanier program were interviewed including representatives from the Department of Foreign Affairs, Trade and Development Canada (DFATD), formerly Foreign Affairs and International Trade Canada (DFAIT), as well as deans of graduate studies and research liaison officers from 23 of the 52 Canadian universities under the first allocation cycle. Findings from this line of evidence helped provide insight into why the Vanier CGS program is mostly supporting Canadian doctoral students already enrolled in their studies.
Interview respondents doubted Vanier's ability to attract new international students (who are not already living in Canada), because of its high level of competitiveness and long competition timelines. DFATD interviewees mentioned that they were unable to attract new students with a Vanier award because they could make no guarantees that potential candidates would be successful in receiving it. University respondents also confirmed that they are unable to use the award to recruit as the number given out is too small and the process is extremely competitive which prevents them from making any guarantees to doctoral students (prospective or already enrolled) about receiving the award. As well, some university representatives believe that international students who are nominated tend to keep their options open, so it is less likely that they would come to Canada for their doctoral studies without a Vanier scholarship.
The role of the Vanier award as a recruitment tool for students (both domestic and international) is furthermore limited by doctoral supervisors’ reluctance to nominate students with whom they have not yet worked with, as would be the case if students were to be nominated during their first year of doctoral studies. The program timelines are such that between the application period and when the decision is made at the federal granting agency level, first year candidates would have had to apply to Vanier a full year before starting their doctoral studies. As such, supervisors have had very limited contact with the most recent cohort of students by the time the applications need to be submitted, and they are less likely to put forward students who are starting their doctoral studies than those who are in their second year, or who have worked with this supervisor during their Masters' (and who are therefore not new to the institution), or who have been students of close colleagues in other centres. Results from a survey of doctoral supervisors of Vanier scholars confirms this finding as only 9% reported having low to no familiarity with their award recipient at the time of their application to the scholarship.
Vanier CGS selection committee members who participated in the evaluation through an expert panel review agreed that international applications have been flagged as a longstanding concern by the granting agency selection committees because foreign students have more hurdles to overcome in terms of securing a nomination for a Vanier CGS award, given the need to establish support and connection to a Canadian institution. Concern was also expressed by selection committee members that the method of nomination at Canadian universities has been overly restrictive and detrimental to attracting world-class applicants, resulting in the narrowing of the potential pool. In addition, concern was raised by the panel that current timelines for application submission and award decision might be potentially limiting the applicant pool.
As an incentive to enroll in doctoral studies in Canada, evaluation findings show that the existence of the Vanier CGS program appears to have played a moderate to high degree of importance in the decision of Vanier scholars - Canadian and foreign - for about half of this group (53% for Canadians and 51% for foreigns). In addition, only a small number of Vanier scholars (7%) reported that they would have pursued doctoral studies outside of Canada if they had not received the scholarship. This finding is most likely due in large part to the fact that 70% of Vanier scholars (excluding those funded in 2008) were already enrolled in doctoral studies at the time they applied for the award (Table 2.3).
In making their decision to pursue doctoral studies in Canada, the reputation of their primary supervisor was identified by approximately 8 out of 10 Vanier scholars as having a moderate to high influence on their decision (this finding holds true for both foreign and Canadian students). The reputation of their university was also identified by the same proportion of award holders as having been a factor in their decision although fewer rated it as having high importance. Other important factors identified by the majority of Vanier scholars included the reputation of their department, the university being in Canada and the quality of life in Canada.
World-Class Students
To help address the issue of whether the program has been attracting and recruiting world-class students, the evaluation obtained the perspectives of current and former Vanier selection committee members through an expert panel review (divided into three sub-committees each representing health, natural sciences and engineering and social sciences and humanities) that examined a purposive sample of Vanier CGS nominees and recipients’ applications over the period 2008–2012, with a view to assessing their overall quality in terms of being world-class. The concept of world-class in relation to the Vanier program was deemed as a synonym for excellence by expert panel members.
Photo: 2009 Inaugural Vanier CGS Awards Announcement – Ottawa, Ontario. Photo courtesy of the Vanier Banting Secretariat.
In reviewing a total of seventy-five files per sub-committee, the panel concluded that the overall quality of both successful and unsuccessful applications was consistent over time and that their scores given as part of the panel reflected the judgments made by the selection committees (although a few cases were noted by members of the panel where their scoring differed from that of the selection committee in terms of applications that were funded but would not have been and those that were not but would have been). These findings should be interpreted within context, however, because the assessment of the overall results of the expert review panel was based on a small sample of Vanier CGS applications (with smaller subsets for each panel member). In responding to the issue of the overall quality of applicants, panel members expressed some reservation as to whether all Vanier CGS recipients were indeed world-class (although they noted that they lacked data on what candidates went on to do after their application to the award and thus were limited in terms of the evidence needed in making a sound decision on quality).
Further evidence on the quality of doctoral students awarded a Vanier scholarship was obtained through a survey of their supervisors where respondents were asked to rate their Vanier scholars on various research, teaching and service-related dimensions. In terms of leadership, findings showed that Vanier scholar supervisors rated the award holder they supervised/were supervisingFootnote 6 as being exceptionalFootnote 7 at demonstrating leadership in research (62%), personal and/or professional leadership (51%), leadership in providing service to the academic community (31%), leadership in teaching (27%) and leadership in providing service to the larger community (21%). In addition, results show that a higher percentage of supervisors rated their Vanier scholars as demonstrating exceptionalism on several research, teaching and service-related dimensions compared to other doctoral students whom they supervised. Of particular importance, 31% indicated that Vanier scholars demonstrated an exceptional ability to lead/influence others. These findings imply that the majority of Vanier award holders, while being rated higher than other doctoral students, are mainly exhibiting leadership in research; however according to supervisors, their leadership potential in other areas has not yet been fulfilled for the majority of recipients (Table 2.5).
% of Supervisors Providing Ratings of Exceptional For: | ||
---|---|---|
Dimensions | Vanier Scholar Supervised | Other Doctoral Students Supervised (N=239) |
Quality of research | 58% | 13% |
Productivity in the research environment | 52% | 10% |
Publication and knowledge dissemination record | 46% | 8% |
Ability to create a community of peers | 32% | 9% |
Ability to create collaborative relationships | 40% | 8% |
Ability to supervise and mentor | 25% | 10% |
Ability to manage their projects and research | 51% | 11% |
Ability to teach | 25% | 5% |
Ability to lead/influence others | 31% | 6% |
Contributions to the academic community | 24% | 5% |
Contributions to the larger community | 22% | 5% |
Source: Survey of Vanier Scholar Supervisors
University representatives interviewed were also asked to assess the extent to which those nominated and awarded were world-class. It was generally agreed the Vanier award had set expectations of excellence, and that therefore students who applied and were nominated were usually world-class. On the other hand, some interviewees did note that, while students who were awarded a Vanier were world-class, nominees, albeit all high-performance students, were not all world-class. They noted that the leadership criterion appears to be a key distinguishing factor: nominees may well all be excellent researchers but awardees distinguish themselves by also being or having a strong potential to be leaders in their environment. Said one university representative: "Of our general pool of students, world-class students are pretty rare, which is statistically logical. For those selected for the Vanier, we never had issues finding students who were at least very strong academically. The real issue came with meeting all the other criteria, mainly the leadership skills."
Awareness of the Vanier CGS Program
Survey results revealed that, for Vanier scholars both in Canada and outside Canada at the time of application to the award, the most common ways they learned about the program were through colleagues/friends/word of mouth, from being approached by a graduate program coordinator or from being approached by their doctoral supervisor. Although the frequencies of these responses changed slightly between those inside and outside Canada, these findings imply that Vanier promotional activities, the Vanier CGS website and university organized events play a lesser or more indirect role in raising awareness of the program.
University Promotion of the Vanier CGS Program and its Recipients
Due to limited resources, few universities reported doing any promotion of the Vanier program, whether in Canada or internationally, other than on their website. International recruitment is generally limited, if the universities do any at all. The universities who do recruit abroad tend to target specific countries, and do not believe that the Vanier award has a place in that recruitment discourse (due to not being able to guarantee receipt of the award). In terms of Vanier scholars, universities often promote these individuals internally while some have had their recipients featured in the local news. One factor hindering the ability of universities to promote its scholars is the delay between when the decision is made (and the student is notified) and when the federal granting agencies make their official announcement. This delay results in the universities being too late in promoting the news of the Vanier awards.
DFATD’s Promotional Activities
DFATD representatives report mentioning the Vanier scholarship whenever possible in their recruitment efforts, especially when the organization does not have any country-specific scholarships to offer. DFATD has promoted the Vanier and Banting programs through its Edu-Canada initiative and has distributed Vanier program brochures for use in Canadian embassies and high commissions, as well as numerous tradeshows. However, representatives interviewed from DFATD stressed that their organization can only promote the Vanier award to a certain extent, given its exclusivity and its prestige; it cannot be used as a mass-attraction tool. The Vanier award is also included on their website, which is said to be a very popular website for foreign students.
Missions abroad sometimes hold information sessions on scholarships, and they mention the Vanier scholarship among the options available to prospective students. However, DFATD representatives indicated that the organization does not have the resources to undertake specific recruitment abroad. Moreover, their mandate is to promote Canada as a research destination, not the Vanier scholarship itself.
The Vanier CGS Program’s Promotional Activities
Data obtained from a recent Vanier CGS communications report highlighted the various promotional activities undertaken by the program in 2012. Social media was an important promotional component of the Vanier program as it disseminated information through its Twitter and Facebook accounts. As of January 2013, there were 839 followers on Twitter, and the program has 779 Facebook fans. Since January 2012, the program published 163 tweets.
Evaluation Question
1.2. Have Vanier CGS recipients received training that has led to their research, leadership and academic development?
Survey findings demonstrate that the majority of Vanier scholars are satisfied with their skill development and training opportunities related to their studies. In terms of development, the majority of Vanier scholars surveyed reported a moderate to significant improvement of research skills for most of the research-related activities assessed. Areas where improvement in skills did not occur or were slight at best pertained to collaborative research with the private/government sector/government/not-for-profit (37%), international research collaborations (26%) and leading research projects (20%). In terms of teaching skills, improvement rates were not as high as compared to research skills. In particular, between 13% and 14% of Vanier scholars reported showing no improvement in areas such as developing course materials, developing teaching methods and teaching a university course.
For personal and professional skill development, close to half of Vanier scholars reported showing slight to no improvement for financial management, business management and personal/non-academic pursuits. One-quarter of Vanier scholars indicated that they showed little to no improvement in leading/influencing others within the larger community. However, the lack of skill development in some areas may not necessarily be an indication of training gaps as 86% of Vanier scholars reported being satisfied to very satisfied with the development of their academic and non-academic skills during their doctoral degree.
With respect to involvement in service-related activities during their doctoral studies (Figure 2.4), organizing or participating in volunteer activities was the most widespread activity reported on with 9 out of 10 Vanier scholars indicating some level of involvement. Overall, the majority of Vanier scholars reported being involved to some degree in service-related activities to their academic and larger communities.
Figure 2.4 – % of Vanier Scholars Reporting Involvement in Service-Related Activities
Source: Survey of Vanier Scholars (N=559)
% of Vanier Scholars Who Reported Interaction at Least Once a Month | |
---|---|
Researchers in your discipline in Canada | 79% |
Researchers in your discipline outside Canada | 51% |
Researchers in other disciplines in Canada | 57% |
Researchers in other disciplines outside Canada | 25% |
Source: Survey of Vanier Scholars (N=578)
In terms of training opportunities available to Vanier scholars during their studies, the majority reported satisfaction with the opportunities they received to develop their skills in research, teaching, service to the academic and/or larger community and personal/professional. The degree of satisfaction was strongest with respect to the research and personal/professional opportunities they received. In terms of dissatisfaction, teaching opportunities garnered the highest level of discontent at 19%. Finally, 8 out of 10 Vanier scholars expressed some degree of satisfaction with their overall research environment.
Research-Related Publications/Products | Average ± Std. Dev. |
---|---|
Articles written or co-written published or accepted in peer-review journals | 4.5 ± 6.0 |
Research papers, books, book chapters and technical publications authored or co-authored published or accepted for publication | 2.7 ± 6.7 |
Grey literature products written or co-written | 1.1 ± 2.6 |
Oral or poster conference presentations given | 9.2 ± 9.4 |
Oral or poster conference presentations at international conferences | 4.5 ± 5.2 |
Art installations, productions or exhibitions produced | 0.2 ± 1.2 |
Research tools produced | 0.7 ± 2.0 |
Tools for research-related activities produced | 0.6 ± 1.9 |
Patent applications filed | 0.1 ± 0.6 |
Patents granted | 0.3 ± 0.2 |
Other intellectual property claims filed | 0.1 ± 0.4 |
Other intellectual property claims granted | 0.0 ± 0.3 |
Source: Survey of Vanier Scholars (N=535)
With respect to the doctoral supervisors of Vanier scholars, survey results reveal that the majority have supervised one scholar (86%) up to this point and tend to be male (63%). In particular, supervisors of NSERC funded Vanier scholars are primarily male (82%) as compared to those supervising award recipients from the other two granting agencies (CIHR supervisors are 62% men while SSHRC supervisors are 59%). Additionally, Vanier scholar supervisors tend to be either a professor (63%) or associate professor (28%) with 23% having received a Canada Research Chair (1% were in receipt of a Canada Excellence Research Chair). Approximately one-third of Vanier scholars reported they had interacted with their supervisor(s) several times a week while only 11% indicated that they interacted once a month or less. Overall, 81% of Vanier scholars reported being satisfied or very satisfied with the supervision provided by their primary doctoral supervisor. In terms of interactions with researchers, close to 8 out of 10 Vanier scholars interacted at least once a month with Canadian researchers in the same discipline while only one quarter did so with researchers in other disciplines abroad (Table 2.6).
An important indicator of excellence achieved at the doctoral level is research productivity. Table 2.7 outlines the average number of research-related publications and products produced by Vanier scholars during the period of their doctoral studies. Productivity was greatest in terms of conference presentations given (including international presentations) and peer reviewed articles published or accepted. As well, approximately half of Vanier scholars report a new theory (53%), findings cited by others (47%) and a new research method (45%) had resulted from the research they have conducted during their doctoral studies (Figure 2.5).
Figure 2.5 – % of Vanier Scholars Reporting Research-Related Outcomes
Source: Survey of Vanier Scholars (N=498)
Evaluation findings show that universities contributed greatly in terms of providing financial support to Vanier scholars during their studies. In terms of their success in receiving awards and prizes in addition to the Vanier CGS award, close to 9 out of 10 Vanier scholars reported having received awards/prizes from their university (Table 2.8). With respect to employment-related financial support received, 19% of Vanier scholars reported working in a non-academic position during their doctoral studies indicating that most award holders do not need to seek non-academic employment while in school. Furthermore, close to 6 out of 10 Vanier scholars reported having obtained a teaching assistantship while close to one-third reported having received at least one research assistantship from a federal granting agency (Table 2.9).
Sources of Awards/Prizes | % of Vanier Scholars who Received Award/Prize |
---|---|
Awards/prizes from their university | 87% |
Awards/prizes from a Canadian provincial body | 38% |
Awards/prizes from a Canadian federal granting agency (excluding the Vanier CGS award) | 35% |
Awards/prizes from a non-Canadian organization | 20% |
Awards/prizes from a not-for-profit organization | 21% |
Awards/prizes from a private sector | 9% |
Other | 4% |
Source: Survey of Vanier Scholars (N=581)
To contextualize these findings, data from Statistics Canada’s survey of earned doctorates (Statistics Canada, 2008) showed that the two most frequent sources of financial support obtained by doctoral students during their graduate studies (which includes Master’s and Doctoral studies) was a fellowship or scholarship from the student’s own institution (65%) and a teaching assistantship from the student’s own institution (65%). Data obtained in the evaluation indicates that Vanier scholars have a higher success rate than Canadian doctoral students in obtaining university scholarships/fellowships and a similar rate for teaching assistantships – which is another measure of their excellence at the doctoral level.
Sources of Employment-Related Financial Support | % of Vanier Scholars who Received Support |
---|---|
Research stipend/assistantship paid from a CIHR Strategic Training Initiative in Health Research (STIHR) grant | 4% |
Research stipend/assistantship paid from an NSERC Collaborative Research and Training Experience (CREATE) grant | 2% |
Research stipend/assistantship paid from a research grant from a federal granting agency (CIHR, NSERC, SSHRC) | 34% |
Research stipend/assistantship paid from a different/unknown source | 43% |
Teaching assistantship | 60% |
Non-academic employment position | 19% |
Source: Survey of Vanier Scholars (N=645)
Evaluation findings indicate that the high dollar amount of the award results in a large majority of recipients not having financial debt related to their doctoral studies. For those Vanier scholars who have not yet completed their studies, 80% reported having no debt related to their doctoral program. For those who completed their doctoral studies, 90% reported having no debt related to their program upon graduation. This finding holds true even when comparing Canadian and foreign students (who can incur higher costs to study in Canada): 80% of foreign students who did not complete their studies were debt free (compared to 82% of Canadian students) while 92% of foreign graduates did not owe any money (compared to 90% for Canadians). These findings are consistent with data obtained through Statistics Canada’s 2005/06 survey of earned doctorates which showed that 88% of doctoral students did not have any debt related to their graduate studies (which also included costs related to their Master’s degrees) (Statistics Canada, 2008).
Impact of Vanier CGS on Universities
University representatives interviewed generally felt that the Vanier program had had little to no impact on the training offered at their institutions, given the limited number of awards distributed and received by each institution. As well, some felt it was difficult to assess attribution of changes to the university by the program since many award recipients are already enrolled prior to winning the scholarship. Some university representatives assumed that the Vanier program made a difference in the quality of the training offered at their institution, but they could not provide data supporting this idea. To some, the Vanier scholars did not make a greater difference than that made by CGS award holders. Others argued that having students of Vanier calibre at their institution inspired others and elevated the quality of the training.
However, evidence collected through the evaluation confirms the assumptions made by several university representatives interviewed. According to Vanier supervisors surveyed, award holders have had the biggest positive impact on their departments in terms of the quality of training provided (70%) and the quality of the research environment (67%). According to them, the aspect of their departments where Vanier scholars had the least positive impact was on the research equipment, instruments and facilities available (27%).
The presence of Vanier scholars at an institution was mentioned as a validation of the quality of the research conducted there: it is assumed that Vanier caliber students would have plenty of opportunities to study in various universities, but that when they select a Canadian institution and that their excellence is recognized by such a prestigious award, the quality of the research program is confirmed.
Evaluation Question
1.3. Have Vanier CGS recipients received training that meets their research and career objectives?
In terms of research and career objectives, Vanier scholars who had not yet completed their doctoral studies were asked what sector(s) they intended to work in after graduation. The majority (79%) indicated an intention to work in the university sector while close to one-third planned to work in the private sector (32%), government (37%) and not-for-profit (28%). As well, 7 out of 10 Vanier scholars indicated a moderate to high interest in pursuing a postdoctoral degree upon completion and 84% felt the same towards pursuing a position as a university professor/researcher. As to whether their doctoral training has met their needs, close to 8 out of 10 (78%) Vanier scholars noted that the training they received during their doctoral studies was moderately to extremely useful in preparing them for their career.
Approximately 38% of Vanier scholars reported some gaps in the training they received during their studies. The most often cited training gap was related to teaching skills. Because Vanier scholars are well supported financially due to the high dollar value of the award, it was felt by several award recipients that university departments may not allow or encourage award holders to receive teaching assistantships in order to ensure that other students have access to that funding. Without exposure to teaching and the skills required, several scholars reporting feeling unprepared for an academic career. The second most reported training gap related to the lack of opportunities to develop the skills required to conduct and manage a research project.
The CGS-Michael Smith Foreign Studies Supplement (CGS-MSFSS)
Vanier CGS award holders are eligible to receive a CGS Michael Smith Foreign Studies Supplement of up to $6,000 to study abroad for up to six months of their doctoral degree. A total of 14% of Vanier scholars surveyed indicated that they received the CGS-MSFSS (an almost equal distribution across Vanier scholars in health, natural sciences and engineering and social sciences and humanities). The majority of Vanier scholars reported that the CGS-MSFSS allowed them to accomplish their international research-related objectives, allowed them to establish relationships and networks within the country/countries they visited and enabled them to achieve outcomes they would not have achieved without the award. A little under three-quarters (68%) of Vanier MSFSS recipients indicated that the award amount was appropriate for their research needs and 71% confirmed that the award duration was appropriate.
Progress Towards Achievement of Intermediate Outcomes
Evaluation Question
2. To what extent has the Vanier CGS program progressed towards the achievement of its expected intermediate outcomes?
Retention of Canada’s Top Doctoral Students
According to results obtained through the survey of Vanier scholars, 20% of award holders have completed their doctoral studies with 59% of graduates having applied for the award in 2008 and 38% in 2009. Findings reveal that 63% of Vanier scholar graduates were living in Canada at time of survey and that students in natural sciences and engineering were more likely to leave at the end of their studies (Table 2.10). The country outside of Canada most frequently listed as a current residence for Vanier graduates was the United States (23%). Of those who left Canada and did not locate to the United States, 71% were residing in a European country.
These findings are in line with data on Canadian doctoral students in general as previous research has shown that slightly more than 12% of doctoral graduates end up living in the United States after graduation (Statistics Canada, 2011). As well, 83% of the graduates living in the United States intended to return to Canada. According to Statistics Canada (2008), 46% of Canadian doctoral graduates receive their post-doctoral training from an institution outside of Canada. Of the Vanier scholar graduates who are living outside of Canada, 61% indicated that they are employed as a postdoctoral fellow or associate – higher than the overall rate for Canadian doctoral graduates.
% of Vanier Scholar Graduates Who Stayed in Canada Post-Graduation | |||
---|---|---|---|
Foreign Students (N=12) |
Canadian Students (N=91) |
Total (N=103) |
|
Health Sciences | 50% | 77% | 76% |
Natural Sciences & Engineering | 50% | 49% | 49% |
Social Sciences & Humanities | 25% | 74% | 67% |
Vanier CGS Total | 42% | 66% | 63% |
Source: Survey of Vanier Scholars (N=103)
Of the Vanier graduates who left Canada, 8 out of 10 Canadian students indicated a moderate to high likelihood of returning to the country. The likelihood of foreign Vanier graduates returning to Canada was lower with 29% indicating a moderate to high likelihood of returning. Canadian Vanier graduates who left the country indicated that prospective employment (87%), an actual employment offer (81%) and research funding and equipment available (79%) were the most important factors that influenced their decision to leave. For foreign Vanier scholar graduates now living outside Canada, 86% indicated that prospective employment and an actual employment offer were the two most important factors in influencing their decision.
For Vanier scholars who have not yet completed their studies and who indicated a planned country of residence post-graduation, 95% of Canadians indicated an intention to remain in the country (compared to 86% for foreign students). Results from Statistics Canada’s survey of earned doctorates (Statistics Canada, 2008) showed that one fifth (21%) of Canadian doctoral students had planned to live outside Canada upon completion of their degree. Furthermore, for those intending to leave, the United States was the most frequently identified destination by both groups (9%) which is in line with results from previous research on doctoral students in Canada.
Over 90% of both Canadian and foreign students indicated that quality of life is the most important factor in their decision to stay in or leave Canada after they graduate. Location of spouse/family/friends (85%) and prospective employment (83%) were the next most important factors identified by Canadian Vanier award holders while prospective employment was listed as second for foreign students (88%) and research funding and equipment available was third (80%).
Vanier Scholar Post-Graduation Employment
Of the Vanier scholars who have completed their studies, 89% indicated that they are employed (83% full-time and 6% part-time). Employment rate post-degree is somewhat higher for Vanier scholars than the 85% employment rate (8% of which were part-time) reported by Canadian doctoral students living in Canada or the United States two years after their graduation in 2005 (Statistics Canada, 2011).
Sector | % of Vanier Scholars Who Are Employed |
---|---|
University | 71% |
Private sector | 11% |
Government | 11% |
Not-for-profit | 1% |
University & Government | 1% |
Private Sector & Not-for-profit | 1% |
Private Sector & Government | 1% |
Private Sector & University | 2% |
Source: Survey of Vanier Scholars (N=91)
Approximately three quarters (75%) of Vanier scholars employed indicated that they are working in the university sector and 6% indicated being employed in two sectors (Table 2.11). Employment data on Canadian doctoral student graduates show similarities to findings obtained in this study (although Vanier scholars are more likely to obtain employment in universities than doctoral students in general): a little over half (56%) of Canadian doctoral graduates living in Canada and the United States two years after graduation in 2005 were employed in educational services with the vast majority of those in education working in a university (87%) (Statistics Canada, 2011).
A little over half (51%) of Vanier graduates who are working in the university sector are doing so as postdoctoral fellows/associates. When asked how many postdoctoral fellowships have they held, 48% indicated one postdoc, 6% reported holding two while 46% indicated that they have not held any. Approximately 17% are working as research faculty, scientists, associates or fellows and 15% are teaching faculty.
Income Range | % of Vanier Scholars Reporting Level of Income |
---|---|
$5,000-$9,999 | 1% |
$15,000-$19,999 | 1% |
$25,000-$34,999 | 1% |
$35,000-$49,999 | 30% |
$50,000-$74,999 | 35% |
$75,000-$99,999 | 24% |
$100,000-$149,999 | 4% |
$150,000-$199,999 | 1% |
Source: Survey of Vanier Scholars (N=91)
Overall, the majority of Vanier scholars working in non-academic sectors are employed as researchers and are earning an average annual income (before deductions) between $35,000 and $99,999 (Table 2.12). Statistics Canada data on Canadian doctoral graduates showed that the median income for all graduates two years after graduation in 2005 was $65,000 while graduates at the 25th percentile were paid $48,387 and those at the 75th percentile were paid $79,000 (Statistics Canada, 2011). In terms of the match between education and employment, 81% of Vanier graduates stated that their current position was moderately to perfectly related to their doctoral degree.
For those who indicated working in a job that is not somewhat related to their studies, the most frequently mentioned reasons were a change in career or professional interests, a job in their degree field was not available and the location of the job. In terms of exerting a moderate to high degree of leadership and influence through their primary occupation, 67% of Vanier scholars indicated they lead/influence others within their organization while a much smaller proportion (37%) stated that they lead/influence the strategic direction of their organization (although it should be noted that many graduates are employed in post-doctoral positions).
Enhancement of the Reputations of Canadian Universities
University representatives interviewed often indicated that the presence of Vanier recipients would enhance the reputation of their faculty, program or laboratory, but that it would have little to no impact on the reputation of the institution as a whole. It was noted that a university's reputation was built on numerous factors, and that prestigious scholarships were only one of those. The presence of Vanier scholars at an institution was mentioned however as a validation of the quality of the research conducted there: it is assumed that Vanier caliber students would have plenty of opportunities to study in various universities, but that when they select a Canadian institution and that their excellence is recognized by such a prestigious award, the quality of the research program is confirmed. As with the impact of the Vanier program on training and research, universities were not able to provide data on any impacts to their reputation.
Vanier scholar supervisors indicated that the program has had the greatest positive impact in terms of enhancing the reputation of their departments through enhancing their own reputation as a supervisor of doctoral students (69%). The area in which the highest proportion of supervisors (14%) identified as having not been impacted by the Vanier CGS program is the reputation of Canada as a research environment.
Two areas in which a minority of supervisors indicated the program has had a negative effect has been on the climate between doctoral students under their supervision (7%) and the climate within their research team or lab (6%). This issue was identified by several university representatives interviewed who noted that the high funding level provided by the Vanier program has, in some instances, reportedly created tension, uneasiness and sometimes envy, given the large differences in revenue between unfunded PhD students, granting agency-funded students and Vanier awardees who are working at the same level on the same problems.
Enhancement of the Capacity of Universities to Attract the Best and Brightest
To some university representatives interviewed, the Vanier scholarship has influenced their institution’s capacity to attract the best and brightest students because of the funding level provided by the program, which is much greater than what it could otherwise offer. However, university respondents also brought up that the number of Vanier awards was too small to be used as a recruitment tool: the selection of university nominees is very stringent, and the award is extremely competitive, so universities are unable make any guarantees to PhD students, whether prospective or already enrolled, about receiving the award. The role of the Vanier scholarship as a recruitment tool for Canadian and foreign students is furthermore limited by the reluctance of supervisors to nominate students with whom they have not yet worked with (as previously discussed under evaluation question 1.1).
Progress Towards Achievement of Long-Term Outcomes
Evaluation Question
3. To what extent has the Vanier CGS program progressed towards the achievement of its expected long-term outcomes?
Recognition of the Vanier CGS Program as a Prestigious Award
Evaluation findings confirm that award recipients and their supervisors view the Vanier CGS program as a highly prestigious award in Canada. Results are lower however when looking at the issue of its prestige internationally although progress is being made towards achieving international recognition with more foreign students rate the award as having more prestige worldwide than Canadians.
Survey results reveal that Vanier scholars, in rating the prestige of all federal granting agency masters and doctoral scholarships and training grants, viewed the Vanier award as having a very high level of prestige (90%) – the highest level of agreement for that rating compared to all other funding opportunities rated. A similar opinion was also shared by their doctoral supervisors the majority of whom felt the award carried a very high level of prestige (88%) in Canada.
With respect to the prestige of the Vanier scholarship amongst the doctoral scholarship landscape outside of Canada, ratings by Vanier scholars were somewhat lower with 27% indicating the award held very high prestige and 30% rating it as having high prestige. Approximately 42% of supervisors felt that the award carried a very high prestige internationally (while 30% felt it had high prestige). Foreign students rated the prestige of the Vanier award outside of Canada more favourably than Canadian students. Approximately 38% of foreign students rated the Vanier scholarship as having very high prestige abroad while 30% viewed it as having high prestige.
In terms of the effect of the prestige of the award on students, it was felt by university representatives interviewed that the Vanier program's reputation may be more relevant for students interested in a career in academia, whereas it was unclear what weight the award had on the reputation of a doctoral student who was headed for a career in consultation or public administration, for instance. This was linked to the perceived notion that the private, non-academic sector is less preoccupied with awards when assessing the merit of job applicants than universities would be.
Efficiency and Effectiveness - Program Design and Delivery
Evaluation Question
4. Has the Vanier CGS program been implemented as intended?
Levels of satisfaction with the application and selection process provide client perspectives on the efficacy of the delivery of the Vanier CGS program - both at the university and federal granting agency level. Overall, Vanier scholars reported a consistently high level of satisfaction and low level of dissatisfaction across several aspects of the process (Figure 2.6). Award recipients were most satisfied with the information available on how to apply to Vanier (91%), the eligibility requirements (89%) and the information available on the program (88%). The issues of program delivery that a minority of Vanier scholars were most unhappy with pertained to the clarity of the university selection process (15%), the amount of work required to complete an application (12%) and the application form itself (10%).
Figure 2.6 – Opinions of Vanier Scholars on the Application and Selection Process
Source: Survey of Vanier Scholars (N=519)
In terms of the design of the Vanier CGS program, Vanier scholars surveyed indicated a high degree of overall satisfaction with the amount, duration and reputation of the award. Specifically, 92% of Vanier scholars expressed the highest level of satisfaction in regards to the award amount (with satisfaction levels consistent across fields of study). Results were somewhat lower for the reputation of the award with 83% indicated a high level of satisfaction (with Vanier scholars working in social sciences and humanities indicating a slightly lower degree of satisfaction).
Additionally, it was sometimes indicated by universities that the length of the Vanier funding was insufficient, given that in some disciplines it is rare for students to complete their doctoral studies in three years. As one university representative explained, "given that is it extremely uncommon for a doctoral student to complete a program in less than four years, some other institutions are offering four-year funding packages, which can put Vanier at a slight disadvantage."
Photo: 2013 Vanier Scholar Jennifer Shaw (SSHRC) – Simon Fraser University. Photo courtesy of Simon Fraser University.
Although only a small percentage of award holders expressed dissatisfaction with the length of the award, Vanier scholars did not express as high a level of satisfaction on this dimension (particularly students in social sciences and humanities) with 62% feeling very satisfied. This finding is not surprising given that the Vanier award covers three years of study and recent data shows that the average completion time for Canadian PhD students ranges from slightly under 15 terms – or five years, based on three terms per year – in the physical sciences and engineering, to a high of 18.25 terms, or just over six years, in the humanities. The mean time-to-completion was 15.4 terms in the health sciences and almost 17 terms in the social sciences (Tamburri, 2013). A suggestion to lengthen the duration of the award was made by several Vanier scholars when asked for ways to improve the program.
Although no key informant interviewees argued that the funding level for the award was too low, some argued that it was too high (regardless of the student's origin), especially in comparison to the salaries the postdoctoral fellows with whom the awardees work. It was suggested by interviewees that having more CGS scholars would be preferable to having very few Vanier scholars. It was sometimes perceived that going beyond the $35,000/year CGS doctoral award funding level had a lesser incremental impact on the student, whereas those funds would have more impact if they were used to fund more students at the CGS level. As well, they were concerned that it may make other students feel less worthy. The idea of turning CGS awards into "the silver medal" was viewed as detrimental in the long run. However, at the granting agency level, stakeholders felt that the funding amount was commensurate with their aspirations to attract the very best doctoral students and leaders world-wide to Canada.
Overall, university representatives were generally satisfied with the Vanier CGS program design and were supportive of the Vanier scholarship being open to international students, as this is their only national scholarship vehicle to attract such students. As one university representative noted, "universities don't have many resources to provide funding packages to students, and international students cost more than Canadian ones, so we now realize that without Vanier we wouldn't be able to get international students."
Photo: 2013 Vanier Scholar Zhihui Yi (NSERC) – École Polytechnique de Montréal. Photo courtesy of Natural Sciences and Engineering Research Council.
While some university representatives viewed the students' ability to apply to the Vanier program as well as other tri-agency scholarships simultaneously positively, it was sometimes viewed as a program flaw. To some, if a student was nominated for a Vanier (and was awarded it) but had already applied for a Canada Graduate Scholarship, for instance, it meant that their university had "wasted" a CGS application on that student.
From the perspective of Vanier scholars, the strengths of the program are the award amount (which many claimed allowed them to focus solely on their research, increase their productivity and pursue their own research that was more innovative and less traditional), the prestige of the award, the leadership component as well as international student eligibility.
The areas of suggested improvement (by order of most frequently cited) were:
- Enable scholars to connect with one another (e.g., online, conferences)
- Award duration is too short
- Lack of promotional activities (especially internationally)
- Application is difficult and time consuming
- Enable scholars to connect with others
- Promote scholars and their work
- Enable the use of award funds to pay for travel and research-related expenses
- Award amount is too high
According to Vanier scholar supervisors, the strengths of the program are that it provides adequate funding to doctoral students, supports excellence (according to many this is due to its leadership component) and is very prestigious. The most common area for improvement identified was that the Vanier award amount is too high. Specifically, it was mentioned that it would be in Canada’s best interest to award more grantees with less money. They noted that post-doctoral funding and salaries are much lower than the value of the Vanier award and can lead to disappointment or even discourage Vanier scholars from pursuing a post-doctoral position. As well, they claimed that the level of award creates a class-system amongst students in the lab where one person is earning almost double that of other students for completing the same work. Many also commented that the selection process for Vanier is relatively arbitrary as there is little actual difference between students at this level. Others cited that the high award amount can create conflict or negativity in the lab environment which affects productivity. Finally, Vanier scholar supervisors suggested that research and travel costs be allowable expenses under the Vanier scholarship.
Vanier scholars (78%) reported that the award had the strongest positive impact on their current financial situation (for those who have not yet graduated), the need for obtaining income during their studies (68%) and the time they were able to devote to their doctoral studies (66%). In terms of influencing the pace of their progress in their doctoral program, 77% indicated the award had an overall positive impact.
Vanier CGS Nomination Allocation System
Over the period of the first Vanier CGS allocation cycle (2010–2012), a total of 52 Canadian universities received at least one nomination allocation. At the end of the cycle, only 21% of institutions fulfilled their allocation quota. Of the 79% whose quota was unfulfilled, 8% did not submit a single nomination. G-13 institutions received the bulk of the allocations (71%) and provided nominations for 88% of their allocations (compared to non-G-13 universities who used 77% of their allocations).
Photo: 2013 Vanier Scholar Mohamed Soliman (CIHR) – University of Toronto. Photo courtesy of the Lunenfeld-Tenenbaum Research Institute, University of Toronto.
In terms of how institutions manage their allocations over the cycle, university representatives interviewed noted that earmarking nominations for each year was one way of planning for the funding cycle, though the earmarked allocations were followed loosely. In some instances, more nominations were submitted in the last year of the cycle, knowing that their students may not meet all the criteria, in order to use up their allocations. A review of Vanier CGS administrative data confirmed that the largest number of nominations submitted occurred in the final year of the first allocation cycle (Figure 2.2).
Key informant interviewees were generally at ease with the university allocation system, because it forces the universities as a whole to work harder to develop research capacity and obtain tri-agency funding. In this sense, the allocation system is viewed as stimulating; however, it was also mentioned that smaller institutions had more difficulty operating within this allocation system because of their limited resources and small quotas.
Some university representatives thought that the three-year allocation period allowed the universities to be flexible in their nominations, while highlighting the exclusivity and prestige of the scholarship. However, it was also indicated that in some cases universities would be more comfortable with a yearly allocation quota, because it would limit the chances of passing up on a high-calibre candidate early on in the funding cycle.
The idea was also brought up that the allocation system was skewed towards maintaining status quo in terms of resource allocations. Because the indicators used to determine the allocations are internal to the three granting agencies, there is little room for change. As one interviewee explained, "the rich get richer: it doesn't foster a growth or changes in the landscape of research training in Canada." On the other hand, the current indicators were underscored as being reliable, since the data were gathered by the granting agencies themselves.
It was in some cases suggested that taking the federal granting agencies' $20,000 doctoral student awards into consideration would improve the allocation system; it is possible that this suggestion was made in part because it would be at these universities' advantage. It was also suggested that the universities' success rate in the granting agency doctoral scholarship competition be taken into consideration, along with the number of doctoral students enrolled at each university or the timeliness of doctoral graduation.
Regarding the possibility of allocations being reserved specifically for foreign students, interviewees were sometimes in disagreement with the idea, indicating that the Vanier scholarship should reward excellence, not origin. It was often mentioned that cultural differences and discrepancies in opportunities rendered the leadership criteria more difficult to judge in foreign students (see below). Some universities spoke against quotas for international students because they feared that they would not be able to fill those quotas; this was especially true of universities attracting few international students. The scarcity in student funding was also an argument used against quotas for international students.
When university representatives spoke favourably of implementing quotas for international students, their argument was based on the principle that the knowledge economy transcends borders. Moreover, some Ontario-based universities explained that their provincial funding does not cover international students, and that having a portion of the Vanier scholarships reserved for international students would be a positive decision given their situation.
The equal split of the candidates among the granting agencies was generally viewed as appropriate; indeed, some were loath to favor one agency over the others. They also thought that having an uneven split would put interdisciplinary research at a disadvantage. In that regard, the suggestion was made that a portion of the scholarships be reserved outside of the granting agency envelopes for Vanier scholars conducting interdisciplinary research. On the other hand, some university representatives argued that given that NSERC and CIHR have more funding than SSHRC; it would make sense to allocate more grants to those agencies. At the same time it was also recognized that the higher demand for SSHRC funding should be taken into consideration.
Vanier CGS Nomination Selection Process
Faculty members, potential supervisors and graduate program coordinators were often relied on to identify students among those applying for scholarships as potential Vanier calibre, as they are closer to them. In some universities, it is possible for potential candidates to self-identify, in which case students need to submit documentation. Universities noted that challenges to identifying potential nominees included not being able to identify students of a high-enough calibre, students turning down invitations to apply due to the additional work required over the CGS award (and the low odds of being awarded) as well as some identified candidates not being eligible because they had received a CGS award in their first year of doctoral studies.
Photo: 2014 University of Toronto Vanier Awards Announcement – Toronto, Ontario. Photo courtesy of the University of Toronto.
Universities each follow their own process in selecting Vanier nominees and a wide range of processes are used to support candidates once they had been identified. At the university level, applications from Canadian and international candidates are reviewed together by an internal adjudication committee usually comprised of faculty members (which is sometimes the same committee as for the other tri-agency scholarships) and assessments are based on the Vanier selection criteria with leadership being emphasized. Some interviewees explained that among equally excellent academic dossiers, the leadership component would be the factor that would discriminate among candidates.
However, some university representatives indicated that their internal committees found the leadership criterion difficult to evaluate because they had less experience gauging an applicant's leadership (as opposed to academic excellence for instance, which they are used to assessing); the leadership criterion was perceived as rather subjective. It was suggested that more details be provided around the definition and operationalization of the concept (as well as further insight into the Vanier CGS selection criteria). To this end, some suggested that the universities should receive feedback on their nominations from the Federal Granting Agencies’ selection committees and felt that the Vanier program had a sense of secrecy to it.
Evaluating candidates' leadership skills also proved challenging for universities when it came to international students, because they didn't want to assume that international students’ cultures and governments did not offer the same opportunities for exercising leadership as does Canada. This was noted to be a factor in potentially limiting access to female foreign students in the Vanier program. Universities were generally struggling with the assessment of leadership in both Canadian and foreign students but particularly the latter, and it was suggested that the leadership criterion should be adapted for international students.
Program Delivery at the Granting Agency Level
Photo: 2013 Université Laval Vanier Awards Announcement – Quebec City, Quebec. Photo courtesy of the Vanier Banting Secretariat.
Universities expressed concerns pertaining to the timing of the Vanier application process. For one, the deadline was thought to be too close to the start of the fall semester, which is very busy for faculty members and administrative personnel, and leaves very little time for students to prepare their applications and for the university adjudication committee to review them and provide feedback. Moreover, the time lapse between the application period and when the decision is made at the agency level discourages universities from presenting first-year doctoral nominees, because they would have had to apply to the Vanier program almost a full year before starting their doctoral studies. This proved especially challenging when seeking to attract new and foreign students to a university.
It was also noted by current and former Vanier granting agency selection committee members that the Vanier CGS program’s application deadline did not work in favour of international applicants given internal institutional deadlines for graduate program admissions. In addition, concern was expressed that current timelines for application submission and award decision might be potentially limiting the applicant pool, given that a number of potential Vanier CGS recipients will be taking up other awards. The level of uncertainty and risk that universities had to manage when nominating new students was identified as the main challenge faced in the delivery of the program at the university level. Some universities also mentioned that the tardiness of the announcement was problematic for international students, because it did not leave them enough time to make travel and visa arrangements.
Vanier CGS selection committee members who participated in an expert panel raised the suggestion of a direct applicant route for international applicants who might not come to Canada without the scholarship and who are applying for awards in other countries around the world. Selection committee members agreed that significant changes to the Vanier CGS program design would have to be considered to allow for a direct application stream for international applicants in order to increase the number of international applications (and their quality). As well, it was also suggested that allocation of a portion of the applications quota and funds be considered for the submission of direct international applicants.
In terms of difficulties encountered in the selection process at the Federal Granting Agency level, committee reviewers agreed that the leadership criterion was the most challenging and contentious, and posed the greatest variance when reviewing applications during the expert review panel exercise convened as part of the evaluation. On this note, panel members agreed that the assessment of academic excellence and research potential were more straightforward but that leadership generated extensive discussions during Vanier selection committee meetings. They noted that leadership is fundamental to the Vanier CGS program and, as was reported by university representatives regarding their nomination selection process, is often used in the selection process to differentiate between candidates. It was noted that on Vanier CGS selection committees, leadership has been interpreted in different ways, however, the panel discussions let to a degree of consensus in that it should be assessed with the same sense of quality (in other words ‘world-class’) as the other selection criteria. Lastly, panel members agreed that leadership was fundamental to the Vanier CGS program.
However, although panel members agreed that the leadership criterion clearly distinguished the Vanier CGS from regular NSERC, CIHR or SSHRC doctoral awards, they also felt strongly that academic excellence and research potential were prerequisites of a Vanier CGS scholarship. On this note, panel members stressed that equal weight should be assigned to the academic, research and leadership potential criterions and that they be based on excellence/world-class. Furthermore, they recommended that applications with a score of three or below for any of the selection criteria be triaged out of the selection process, provided consensus is obtained from both reviewers.
Several suggestions were made by the Vanier evaluation expert panel on how to improve the assessment of the leadership criterion. They noted that in the context of a Rhodes scholarship, leadership is assessed based on initiatives that go above and beyond the actual and expected demands of a student. For example, some kind of initiative that brings good to the world or some kind of vision not directly connected to their studies (a leadership service component which is separate from the academic component). This could refer, for example, to a student excelling in music who then starts up an orchestra. They suggested that Vanier CGS applicants be required to clearly identify their key leadership accomplishments amongst those presented in the application, along with a brief rationale.
Panel members unanimously agreed on the need for a clearer tri-agency definition of leadership, noting that providing some good examples of what should be considered as strong leadership would be particularly useful such as making a distinction between leading an activity and being a participant.
Overall, most universities were satisfied with their interaction with the granting agencies and were generally pleased with the harmonization of the program because it lightened their administrative workload and spoke positively of the Vanier CGS common website. In terms of the Vanier Banting secretariat, not all universities had interacted with the group so they had less feedback to provide. Granting agency representatives viewed the Vanier Banting Secretariat as extremely important to optimal program delivery and viewed it positively because it allowed the three agencies to share a common vision and to collaborate more effectively. One downside noted was the risk of losing the flexibility to adapt the Vanier program to each agency, but this risk was thought to be limited and manageable.
Both the university and agency representatives unequivocally saw a value in having the funding competition managed centrally: centrality provides Vanier a prestige that it would not have if the universities managed the program independently. Federal Granting Agency staff recognized that the program was still young, and that improvements were still to be made.
Alternative Delivery Mechanisms
University representatives rarely had any suggestions for alternate or more effective program delivery mechanisms (aside from those already identified in other sections in the report). In terms of improvements, a one-year scholarship was suggested in addition to the current Vanier scholarships, in order to foster more international mobility; it was suggested that this increased support to foreign students wishing to come to Canada for a portion of their PhD would be beneficial.
With regards to the ability to use the Vanier award as a recruitment tool, it was suggested that the quota system be redesigned to provide universities with a given number of "guaranteed" awards for each agency. This would decrease the uncertainty of the award, therefore making it a more effective recruitment tool, in particular for international students. In the opinion of DFATD's representatives, the Vanier scholarship should be more effectively branded and potentially renamed to make the Canadian origin more prominent.
Vanier CGS selection committee members made several suggestions in terms of improving the delivery of the program (in addition to those previously mentioned in other sections of the report). First, it was agreed that some level of reporting (progress and/or final reports) should be requested from the Vanier recipients and their supervisors. They note that these reports would gather information on what has been accomplished during and at the end of the period of tenure of the award (It should be noted that an end of award survey for Vanier scholars was in development at the time of the evaluation). Selection committee members also suggested that universities should be persuaded to submit customized letters providing comments on the applicant, the quality of the infrastructure and the proposed supervisor and stressed the usefulness of personalized letters as part of the selection process. It was felt that this change would benefit the applicant, given that both the supervisor and the institution directly benefit from Vanier CGS funding. It was also proposed that the Vanier Banting Secretariat make the necessary refinements to their instructions for university nomination letters, and provide examples of excellent letters to institutions. In order to reduce workload for Vanier CGS selection committee members in assessing research potential, it was suggested that applicants be asked to provide the standard impact factor for publication journals, along with a sample of articles to be included with their application form.
The issue of the grade point average (GPA) assessment was also raised by expert panel members in terms of assessing the academic excellence of Vanier candidates, noting that some applicants with low GPA but with awards and distinctions would ultimately receive a medium score for academic excellence. Members noted that the current review criteria used by the Vanier CGS selection committees differed from the nomination criteria provided to universities. As such, instructions given to the Vanier CGS selection committees in terms of judging the GPA was based on the entire academic record, whereas universities were instructed to consider only the last two years of undergraduate studies. Panel members agreed that it would be useful to give universities the same guidance as selection committees.
Vanier CGS Alumni Network
Photo: 2012 Queen’s University Vanier Awards Announcement – Kingston, Ontario. Photo courtesy of the Vanier Banting Secretariat.
According to a performance measurement strategy developed for the Vanier CGS program in 2010, a Vanier Alumni Network was to be established during the second phase of the program with the purpose of ensuring that Vanier students are globally connected and have access to research collaboration opportunities in Canada and with international partners during their doctoral studies. At the time of the evaluation, an alumni network had not yet been implemented.
To help inform decision-making regarding the network, the evaluation sought the opinions of award holders in terms of their preferences for communicating with each other and with the Vanier secretariat. Results indicate that Vanier scholars showed a moderate to high interest in communicating with each other via email (57%), followed by a LinkedIn group (50%) and Facebook (46%). About 1 in 4 Vanier scholars provided their own suggestion with the majority indicating a desire to meet with other scholars in person through annual meetings and conferences. In terms of being kept informed of Vanier CGS-related updates and activities, periodic emails sent by Vanier program staff was preferred by most (74%) followed by a Vanier electronic newsletter (71%).
Evaluation Question
5. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient
Findings show that the Vanier CGS program has been delivered by the federal granting agencies in a cost efficient manner. Data shows that the program has reduced its administrative costs as a proportion of its total expenses over time - both at the aggregate program level and at the level of the Federal Granting Agencies (Tables 2.13 and 2.15). As well, the program’s costs are below that of the total for the federal granting agencies over the period of 2009-10 to 2012-13 (Table 2.14). It should be noted that costs outlined in the tables below refer to the direct costs of the program incurred by the federal granting agencies (specifically the divisions administering the program). Also included in the costs are PWGSC’s contributions to rental space (located under “accommodations costs”) as well as branding expenses paid by SSHRC’s communication division. Costs associated with the administration of the Federal Granting Agency’s selection committees are included under direct operations and maintenance in Table 2.13. For the most part, staff time and salaries were derived based on estimates given by program management/staff.
It is important to note that several aspects related to the implementation of the Vanier program have impacted on its administrative costs. First, a Vanier Banting secretariat was created in October 2012 to provide administrative oversight of both programs across the federal granting agencies. Prior to the secretariat, the Vanier and Banting programs were supported by staff housed at CIHR, SSHRC and NSERC with the total number of full-time or part-time staff exceeding 20 individuals. In contrast, the secretariat, which is housed at CIHR, involves a total of 9 staff (5 from CIHR, 2 from NSERC and 2 from SSHRC). Furthermore, in the Fall of 2012 the Vanier Banting Steering Committee made the decision to dissolve the Vanier Selection BoardFootnote 8. The decision to dissolve the board was made to improve harmonization of the program and increase efficiencies. Finally, the absence of a planned Vanier Alumni Network has influenced the degree of cost associated with the program’s communications activities.
2009-10 | 2010-11 | 2011-12 | 2012-13 | 2013-14 | |
---|---|---|---|---|---|
Vanier CGS Program Expenditures | |||||
Total Vanier CGS Admin Costs | $758,476 | $878,635 | $808,746 | $554,658 | $564,171 |
CIHR Vanier Admin Costs | $282,259 | $365,933 | $226,168 | $240,231 | $230,348 |
NSERC Vanier Admin Costs | $169,494 | $216,886 | $250,297 | $165,462 | $176,916 |
SSHRC Vanier Admin Costs | $306,723 | $295,816 | $332,281 | $148,965 | $156,906 |
Total Vanier CGS Awards Expenditures | $7,970,833 | $16,525,003 | $23,988,078 | $24,802,824 | $24,807,499 |
CIHR Vanier Awards Expenditures | $2,670,833 | $5,500,000 | $7,722,401 | $8,286,264 | $8,245,833 |
NSERC Awards Expenditures | $2,700,000 | $5,533,336 | $8,047,076 | $8,225,092 | $8,275,000 |
SSHRC Vanier Awards Expenditures | $2,600,000 | $5,491,667 | $8,218,601 | $8,291,468 | $8,286,666 |
Total Vanier CGS Total Expenditures | $8,729,309 | $17,403,638 | $24,796,824 | $25,357,482 | $25,371,670 |
CIHR Vanier Total Expenditures | $2,953,092 | $5,865,933 | $7,948,568 | $8,526,495 | $8,476,182 |
NSERC Total Expenditures | $2,869,494 | $5,750,222 | $8,297,373 | $8,390,554 | $8,451,916 |
SSHRC Vanier Total Expenditures | $2,906,723 | $5,787,483 | $8,550,882 | $8,440,433 | $8,443,572 |
Vanier CGS Admin Expenditure As % of Total Expenditures | 8.7% | 5.0% | 3.3% | 2.2% | 2.2% |
CIHR Vanier Admin Expenditure As % of Total Expenditures | 9.6% | 6.2% | 2.8% | 2.8% | 2.7% |
NSERC Vanier Admin Expenditure As % of Total Expenditures | 5.9% | 3.8% | 3.0% | 2.0% | 2.1% |
SSHRC Vanier Admin Expenditure As % of Total Expenditures | 10.6% | 5.1% | 3.9% | 1.8% | 1.9% |
Source: Federal Granting Agency Administrative Data
2009-10 to 2012-13 | |||
---|---|---|---|
Total Operations | Total Expenses | Total Operations / Total Expenses | |
CIHR | $252,000,000 | $4,058,400,000 | 6.2% |
NSERC | $222,554,000 | $4,306,478,000 | 5.2% |
SSHRC | $121,591,000 | $2,773,616,000 | 4.4% |
Vanier CGS Total | $3,000,515 | $76,287,253 | 3.9% |
Source: Federal Granting Agency Financial Data
2009-10 | 2010-11 | 2011-12 | 2012-13 | 2013-14 | |
---|---|---|---|---|---|
CIHR - Vanier CGS Program Expenditures | |||||
Total Admin Costs | $282,259 | $365,933 | $226,168 | $240,231 | $230,348 |
Direct Salary | $188,478 | $237,101 | $150,864 | $162,296 | $154,010 |
Direct Operations and Maintenance | $31,584 | $50,589 | $25,519 | $24,378 | $25,514 |
Employee Benefit Plan (20%) | $37,696 | $47,420 | $30,173 | $32,459 | $30,802 |
Accommodation (13%) | $24,502 | $30,823 | $19,612 | $21,098 | $20,021 |
Total Awards Expenditures | $2,670,833 | $5,500,000 | $7,722,401 | $8,286,264 | $8,245,833 |
Total Expenditures | $2,953,092 | $5,865,933 | $7,948,568 | $8,526,495 | $8,476,182 |
Admin Expenditure as % of Total Expenditures | 9.6% | 6.2% | 2.8% | 2.8% | 2.7% |
NSERC - Vanier CGS Program Expenditures | |||||
Total Admin Costs | $169,494 | $216,886 | $250,297 | $165,462 | $176,916 |
Direct Salary | $126,270 | $148,073 | $168,752 | $105,421 | $109,661 |
Direct Operations and Maintenance | $1,555 | $19,949 | $25,857 | $25,252 | $31,067 |
Employee Benefit Plan (20%) | $25,254 | $29,615 | $33,750 | $21,084 | $21,932 |
Accommodation (13%) | $16,415 | $19,249 | $21,938 | $13,705 | $14,256 |
Total Awards Expenditures | $2,700,000 | $5,533,336 | $8,047,076 | $8,225,092 | $8,275,000 |
Total Expenditures | $2,869,494 | $5,750,222 | $8,297,373 | $8,390,554 | $8,451,916 |
Admin Expenditure as % of Total Expenditures | 5.9% | 3.8% | 3.0% | 2.0% | 2.1% |
SSHRC - Vanier CGS Program Expenditures | |||||
Total Admin Costs | $306,723 | $295,816 | $332,281 | $148,965 | $156,906 |
Direct Salary | $85,500 | $101,594 | $219,806 | $111,493 | $115,328 |
Direct Operations and Maintenance | $193,008 | $160,696 | $39,939 | $679 | $3,520 |
Employee Benefit Plan (20%) | $17,100 | $20,319 | $43,961 | $22,299 | $23,066 |
Accommodation (13%) | $11,115 | $13,207 | $28,575 | $14,494 | $14,993 |
Total Awards Expenditures | $2,600,000 | $5,491,667 | $8,218,601 | $8,291,468 | $8,286,666 |
Total Expenditures | $2,906,723 | $5,787,483 | $8,550,882 | $8,440,433 | $8,443,572 |
Admin Expenditure as % of Total Expenditures | 10.6% | 5.1% | 3.9% | 1.8% | 1.9% |
Source: Federal Granting Agency Financial and Administrative Data
Conclusions and Recommendations
Conclusions
Evaluation findings indicate that the program is supporting world-class doctoral students – the majority of whom are Canadian, with most Vanier scholars demonstrating exceptional leadership in the area of research. The Vanier CGS award is having the strongest positive impact on students’ financial situation, their need for obtaining income during their studies and the time they were able to devote to their education.
The evaluation found that the Vanier CGS program is considered to be a highly prestigious award in Canada and the presence of Vanier scholars enhances the reputation of a faculty, program or laboratory but has little to no impact on the reputation of the institution. The majority of Vanier scholars are satisfied with their training and skill development and most report that their training has been useful in preparing them for their career. Of the Vanier scholars who have completed their studies, the majority are employed in the university sector and are living in Canada.
Findings from the evaluation regarding the communication preferences of Vanier scholars can help inform the development of an alumni network of Vanier scholars, which is an expected output of the program that has not yet been implemented.
The evaluation identified key barriers that negatively impact its ability to attract and recruit students from outside of Canada: the inability of institutions to guarantee receipt of the award, the high competitiveness of the program in relation to the relatively small number of scholarships awarded and the reluctance of doctoral supervisors to nominate students with whom they have not yet worked with and who are not already enrolled in the institution in which they are seeking Vanier support.
Opinions on the design and delivery of the program were positive overall, with the majority of Vanier scholars reporting satisfaction with the award amount and duration, however the evaluation found that the three year nomination allocation system is contributing to the reluctance of institutions to nominate new students (including foreign candidates) which limits the ability of the program to attract students from outside the country and has had a negative impact on the quality of candidates put forward for nomination. As a result, many universities suggested that the three year allocation cycle be changed to annual nominations.
As the objective of the Vanier CGS program is to be an internationally competitive scholarship program that enables Canada to attract world-class doctoral students (a key feature which distinguishes it from other federal scholarship programs) and given the reluctance of institutions to nominate foreign candidates, it is important that a portion of allocations be targeted to international students outside of the country in order to facilitate their nomination and better enable the program to meet its objective in terms of attraction. As well, the program’s competition deadlines were viewed by many as being problematic in terms of attracting new (including foreign) students to Canadian institutions and, as such, should be reconsidered.
The interpretation and assessment of the leadership criterion was identified as a challenge and, in light of the program’s expected long-term outcome that Vanier scholars become leaders in Canada and abroad, there is a need to further develop and clarify the definition and assessment of leadership given the important role it plays in the selection process at the university and federal granting agencies.
While the award amount is meeting the educational-related financial needs of almost all recipients, questions remain as to the incremental outcomes that result from the additional financial value of the Vanier award in comparison to other doctoral scholarship awards given the absence of comparator data at the time of this study. There is a need therefore to pursue an additional analysis using comparator data obtained through the evaluations of other doctoral award programs to identify the incremental outcomes that result from the additional financial support offered through the Vanier scholarship.
Recommendations
- To enable the Vanier CGS program to better meet its objective of attracting and recruiting world-class doctoral students to Canadian universities, the following changes to the allocation and application processes should be considered:
- The three year allocation cycle should be changed to annual allocations with restrictions placed on carrying forward unused quotas.
- A portion of nomination allocations should be targeted for foreign students not already enrolled in the institution in which they are seeking Vanier support, with the amount calculated, at least in part, on an institution’s international student enrollment rate. The Vanier CGS program should monitor the extent to which the target of foreign student nominees is being met, per institution and in total, after each competition and use this information as part of the calculation to determine targets and allocations for the next competition year.
- The timing of the application deadlines for the Vanier CGS program should be re-considered in light of findings from the evaluation.
- To help improve the assessment of the leadership criterion in the University and Federal Granting Agency selection processes and increase the extent to which it is interpreted consistently, Vanier CGS program management should establish a clearer definition of what leadership is and how it should be evaluated – especially for foreign students. Suggestions for improvement obtained through the evaluation should be taken into consideration and any changes made should be communicated to universities and Federal Granting Agency selection committee members.
- To help inform the implementation of a Vanier Alumni Network, data obtained through the evaluation on the communication preferences of Vanier scholars for interaction with other award holders and program staff should be considered to ensure that Vanier CGS graduates are connected to the program and to other scholars after completion of their studies.
- Evaluation findings demonstrate that the Vanier program is meeting the educational-related financial needs of almost all of its recipients and that it is viewed as a highly prestigious award in Canada. However, the evaluation was unable to assess what incremental outcomes are associated with the higher value of the scholarship in comparison to the CGS and Federal Granting Agency doctoral awards as evaluations of those programs were in progress at the time of this study and comparison data was not yet available. The surveys used in these scholarship evaluations were designed to enable a comparative analysis of data across programs. As such, it will be important to undertake a further analysis to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes. Issues to be examined would include incentives to enroll in studies, financial situation of students, training (including skill acquisition), research productivity, retention rates and employment. Based on this, it is recommended that Vanier CGS program management consider the results of this analysis in the future planning and design of the program.
Methodology
Consistent with Treasury Board guidance and recognized best practice in evaluation, a range of methods was used to triangulate evaluation findings. The approach of using multiple methodologies involving both quantitative and qualitative evidence is designed to ensure that the evaluation findings are robust and credible and that valid conclusions can be drawn about the performance of the program.
Limitations
Due to issues related to timing and resource requirements, several planned lines of evidence or analysis as detailed in the evaluation matrix of questions and indicators (see Appendix section) were not pursued as part of this study including a survey of Vanier applicants who were unsuccessful in receiving the award. These individuals will be surveyed as part of the ongoing evaluation of the CGS program. Furthermore, comparison data captured through evaluations of the CGS and Federal Granting Agency scholarship programs was not available for use in this study at the time of reporting as data collection for those evaluations had yet to commence. As part of the evaluation recommendations, it is suggested that this line of analysis be pursued in order to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes.
Administrative Data Analysis
A review of Vanier CGS program records including application and grant files and federal granting agency administrative data provided information that described and contextualized the program, its target audience and environment as well as aided in the assessment of program performance and relevance. Where appropriate, data was validated with Vanier CGS program staff to ensure accuracy and reliability.
Document Review
A document and data review was conducted to provide evidence that helped address several evaluation questions pertaining to program performance and relevance. Documentation reviewed included Vanier CGS program literature and reports, federal granting agency publications including previous evaluation reports, Government of Canada publications and documents as well as research articles - particularly those related to PhD education and doctoral students. A google keyword search as well as queries through online journal databases such as JSTOR was performed in order to identify and obtain documents focusing on identified best practices of research training. The search was conducted using key words such as “graduate student” and “funding.” All data collected through the document review was entered into Nvivo for analysis. In the case where electronic versions of information were unavailable, detailed notes were created and imported into an Nvivo file. Imported data was then coded according to relevant evaluation questions and indicators as detailed in the matrix of evaluation questions and indicators. Findings were then synthesized into a technical report following the same structure.
Expert Panel Review
SSHRC and NSERC evaluation staff, in consultation with Vanier-Banting Secretariat staff, convened a panel of current and former Vanier selection committee members to examine a purposively selected sample (N=75) of Vanier CGS nominees and recipients for years 2008–2012, with a view to assessing its overall quality (i.e., world-class). The Expert Panel consisted of three sub-committees who were representative of the disciplines and fields of study - health, social sciences and humanities and natural sciences and engineering. Vanier application files were selected for review by SSHRC/NSERC evaluation staff and divided equally among panel members (15 files per member), taking into account the following selection criteria: (i) gender; (ii) citizenship status (i.e., Canadian; foreign); (iii) funding status (recipient; non-recipient); (iv) language (French; English); and (v) competition year (within the years 2008 and 2012). Findings from the expert panel assessment were used to help address performance in terms of the program’s expected immediate outcome of attracting and recruiting world-class students by assessing the quality of Vanier nominees and recipients and determining whether the quality of applications has increased over time as well as examine issues related to program design and delivery.
Key Informant Interviews – University Representatives, Vanier CGS Program Staff and Federal Granting Agency and Government Representatives
A total of 31 interviews with N=36 key informants were conducted by Circum Network Inc. during the period of March 1st to April 8th 2013 with university representatives (Deans of Graduate Education and scholarship liaison officers), government officials, agency representatives, and Vanier CGS Program personnel. Qualitative data obtained from university representatives was used to help address several evaluation issues and questions as well as provide insight into the university review process including the criteria used to assess applications.
Whereas all relevant government officials, agency representatives and Vanier CGS Program personnel were interviewed, universities were selected using a stratified random sampling approach based on several criteria including (i) Vanier scholarship allocation and application data by agency; (ii) location in Canada; (iii) university enrolment; (iv) number of graduate programs; and (v) the existence of a medical school. The Evaluation Working Group, working in conjunction with the Vanier CGS secretariat, developed a list of potential interviewees from the Vanier eligible universities, the granting agencies and government departments. A total of 23 Canadian universities were included as part of the interviews, representing approximately 44% of all Vanier eligible institutions under the 2010–2012 allocation period. New universities under the new 2013–2015 allocation period were not considered as potential interviewees, given their limited contact with the program.
Interview Target | Number of Interviewees | Number of Interviews |
---|---|---|
Vanier Program Representatives | 3 | 3 |
DFATD | 3 | 1 |
Federal Granting Agencies | 4 | 4 |
Canadian Universities | 26 | 23 |
Total | 36 | 31 |
Source: Vanier CGS Key Informant Interviews Technical Report
Organizational Scan
Findings from an organizational scan of N=17 national and international doctoral and postdoctoral scholarship programs carried out by Vanier CGS program staff was used to compare the design and implementation (e.g., program objectives, award value and duration, selection process, funding competition statistics) of the Vanier CGS program with similar national and international scholarship programs as well as identify best practices. In addition, CIHR Evaluation Unit staff undertook an organizational scan of scholarship programs similar to the Vanier CGS award to address issues and questions related to relevance including the identification of best practices and the degree of overlap of the objectives of Vanier CGS with other federal granting agency doctoral student programs. An internet search was conducted to identify similar scholarship awards worldwide to the Vanier program. A program was included as part of the scan if it had at least one similarity to Vanier in terms of target audience, prestige, award value, award duration, program objectives and/or selection process. A total of N=15 scholarship programs worldwide were included as part of the CIHR Evaluation Unit organizational scan. Invitations for follow-up interviews were sent to representatives from seven organizations whose programs were included. A total of three organizations were interviewed and provided additional data.
Online Survey – Vanier Scholars
The CIHR Evaluation Unit launched an online survey of Vanier scholarship recipients (N=830) over the period of February to March 2014 that provided quantitative and qualitative data that helped address a range of evaluation issues and questions. A census approach was undertaken to survey recipients awarded during the program’s first five years (2008–2012). Where appropriate, common survey questions were developed for use in the current evaluations of the Canada Graduate Scholarship (CGS) and agency specific Master’s and Doctoral scholarships to enable comparison of results across programs. Where appropriate, survey questions were also matched to those adopted in the first evaluation of the CGS program to allow for a comparison of results over time with doctoral students surveyed as part of that study. A survey of unsuccessful Vanier applicants (N=3,441) was originally planned as part of this evaluation. However, due to the close timing of other evaluations of federal granting agency scholarship programs, Vanier non-recipients who did not receive any type of federal scholarship support will be surveyed as unsuccessful scholarship applicants through the second evaluation of the CGS program. Vanier unsuccessful applicants who received other granting agency scholarships will be surveyed through the evaluation of CGS as a scholarship recipient for their most recent award held.
Population Size | Invitations Sent | Completes | Incompletes | # Terminated | Response Rate | Margin of Error (95% CL) | |
---|---|---|---|---|---|---|---|
CIHR Vanier Scholars | 280 | 280 | 175 | 39 | 2 | 63% | 4.5% |
NSERC Vanier Scholars | 277 | 277 | 172 | 48 | 0 | 62% | 4.6% |
SSHRC Vanier Scholars | 273 | 273 | 174 | 35 | 0 | 64% | 4.5% |
Vanier CGS Scholars Total | 830 | 830 | 521 | 122 | 2 | 63% | 2.6% |
Source: ResearchNet; Survey of Vanier Scholars
Online Survey – Vanier CGS Scholar Supervisors
The CIHR Evaluation Unit launched an online survey of Vanier scholar supervisors over the period of February to April 2014 that provided quantitative data to help address a range of evaluation issues and questions. As SSHRC and NSERC do not keep track of the identities and contact information for Vanier supervisors, SSHRC and NSERC Vanier scholars who were in their doctoral program at time of application and who listed a primary doctoral supervisor as a referee in their Vanier application were asked to confirm the identity of their supervisor as part of the survey of scholars. For NSERC and SSHRC Vanier recipient survey respondents whose supervisor information was incorrect or for Vanier scholars whose supervisor information as missing, individuals were asked to consent to receiving and forwarding a future email message from CIHR to their supervisor for the doctoral studies in which they received Vanier CGS support. The email contained a link to a one page questionnaire that enabled supervisors to provide CIHR with their name and contact information in order to receive invitations to participate in the survey.
A post-stratification weighting scheme was developed using a sampling weight to correct for disproportionate cases within each strata as compared to the population (i.e., federal granting agencies/area of research) (Table 4.4).
Population Size | Invitations Sent | Completes | Incompletes | # Terminated | Response Rate | Margin of Error (95% CL) |
|
---|---|---|---|---|---|---|---|
CIHR Vanier Supervisors | 280 | 252 | 118 | 12 | 0 | 47% | 6.9% |
NSERC Vanier Supervisors | 277* | 88 | 69 | 3 | 0 | 78% | 10.2% |
SSHRC Vanier Supervisors | 273* | 84 | 48 | 7 | 1 | 57% | 12.9% |
Vanier Scholar Supervisors Total | 830 | 424 | 235 | 22 | 1 | 55% | 5.4% |
Source: ResearchNet; Survey of Vanier Scholar Supervisors
* Although confirmed identities and contact info of supervisors of NSERC and SSHRC funded Vanier scholars were not available from administrative records, an assumption was held that every Vanier scholar has at least one primary doctoral supervisor whose field of research aligns to some degree to the field of research for the granting agency in which the application was reviewed.
A. Population Size (N) | B. Sample Size (n)* |
C. Sampling Ratio (B/A) | Proportional Weight (A/830) / (B/258) |
|
---|---|---|---|---|
CIHR Vanier Supervisors | 280 | 130 | 0.46 | 0.67 |
NSERC Vanier Supervisors | 277 | 72 | 0.26 | 1.20 |
SSHRC Vanier Supervisors | 273 | 56 | 0.21 | 1.52 |
Vanier CGS Supervisors Total | 830 | 258 | 0.31 | - |
Source: ResearchNet; Survey of Vanier Scholar Supervisors
* includes complete and incomplete responses.
Appendices
Vanier CGS Matrix of Questions and Indicators
Evaluation Question | Indicator | Method |
---|---|---|
Performance - Achievement of Immediate Outcomes: Capacity Development | ||
1. To what extent has the Vanier CGS program achieved its expected immediate outcomes? 1.1. Has the Vanier CGS program attracted and recruited world- class students? |
1.1.1. Analysis of Vanier CGS nominee and recipient profiles, e.g.:
|
|
1.1.2. Perceptions of Vanier CGS recipient supervisors regarding:
|
|
|
1.1.3. Assessment of quality of Vanier nominees (the extent to which students nominated for a Vanier CGS award were world-class) |
|
|
1.1.4. Perceptions of university representatives on the extent to which students applying for university nomination and students submitted for nomination by Canadian universities were world- class |
|
|
1.1.5. Number and type of promotional activities/venues to promote the Vanier program nationally and internationally
|
|
|
1.1.7. Identified ways in which Vanier applicants became aware of the program |
|
|
1.2. Have Vanier CGS recipients received training that has led to their research, leadership and academic development? |
1.2.1. Extent of Vanier CGS recipients and non-recipients involvement and satisfaction with supervisor and research environment:
|
|
1.2.2. Perceptions of, and evidence provided by, university representatives of the impact of Vanier recipients on training/research environment at their university |
|
|
1.2.3. Perceptions of Vanier CGS recipient supervisors on the extent to which award recipients have had an impact on the training/research environment at their department |
|
|
1.2.4. Identified university leadership activities and opportunities provided to Vanier recipients |
|
|
1.2.5. Identified supervisor leadership activities and opportunities provided to Vanier recipients |
|
|
1.2.6. Extent to which Vanier recipients and non-recipients are satisfied with the opportunities available to develop their research, teaching, service and personal/professional skills |
|
|
1.3. Have Vanier CGS recipients received training that meets their career objectives? |
1.3.1. Identified career objectives of Vanier CGS recipients and non-recipients, e.g.,:
1.3.2. Extent to which Vanier recipients and non-recipients indicate that their training prepared them for their career |
|
Performance - Progress Towards Achievement of Intermediate Outcomes | ||
2. To what extent has the Vanier CGS program progressed towards the achievement of its expected intermediate outcomes? | 2.1.1. Perceptions of, and evidence provided by, university representatives regarding the extent to which the Vanier CGS program has enhanced the reputation of Canadian universities |
|
2.1.2. Perceptions of Vanier CGS recipient supervisors regarding the extent to which the program has enhanced the reputation of their university department |
|
|
2.1.3. Perceptions of, and evidence provided by, university representatives regarding the extent to which the Vanier CGS program has enhanced the capacity of their university to attract the best and brightest students |
|
|
2.1.4. % Vanier CGS recipients and non-recipients that have remained in Canada after completion of their studies 2.1.5. % of Vanier CGS recipients and non-recipients that indicate an intention to remain in Canada after the completion of their studies 2.1.6. Identified factors that influenced Vanier CGS recipients and non-recipients decisions to remain in Canada after completion of their studies 2.1.7. % of Vanier CGS recipients and non-recipients that obtained employment in fields/sectors relevant to their career objectives and doctoral training |
|
|
Performance – Progress Towards Achievement of Long-Term Outcomes | ||
3. To what extent has the Vanier CGS program progressed towards the achievement of its expected long-term outcomes? | 3.1.1. Extent to which Vanier recipients, non-recipients and recipient supervisors view the Vanier scholarship as prestigious in Canada and Internationally |
|
Performance - Efficiency and Effectiveness: Program Design and Delivery | ||
4. Has the Vanier CGS program been implemented as intendedFootnote 9? |
4.1.1. Analysis of Vanier CGS university quota allocations, e.g.:
|
|
4.1.2. Identified reasons for any unmet Vanier CGS nomination quotas for Canadian universities 4.1.3. Identified review process used (including assessment criteria applied) by Canadian universities to select Vanier nominees 4.1.4. Perceptions on university review process used (including assessment criteria) including identified strengths, weaknesses and suggested improvements |
|
|
4.1.5. Perceptions on the Vanier CGS university quota system including identified strengths, weaknesses and potential improvements |
|
|
4.1.6. Associations among Vanier CGS recipients and non-recipient profiles (e.g., gender, country of residency, mobility) and their federal granting agency selection committee ratings (overall and by selection criteria), rankings and funding decisions |
|
|
4.1.7. Extent to which Vanier CGS recipients and non-recipients express satisfaction with the application and selection process |
|
|
4.1.8. Comparison of Vanier CGS program design and implementation with similar national and international scholarship in terms of:
4.1.9. Identified best practices among national and international scholarship programs |
|
|
4.1.10. Identified strengths of, and suggested improvements to, the design and implementation of the Vanier CGS program 4.1.11. Identified facilitators and inhibitors to attracting and recruiting foreign world-class doctoral students 4.1.12. Perceptions of comparable or alternative delivery mechanisms to the Vanier CGS program |
|
|
4.1.13. Identified mechanisms to support Vanier CGS recipients in networking with other scholars and receiving program-related updates and communications |
|
|
5. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient manner? | 5.1.1. Ratio of operating expenditures to applications reviewed in total and by federal granting agency 5.1.2. Ratio of operating expenditures to grants funds awarded in total and by federal granting agency 5.1.3. Comparison of existing ratios of operating expenditures to applications reviewed/grant funds awarded with federal granting agency programs (benchmarking) 5.1.4. Proportion of Vanier CGS budget expended in total and per agency |
|
Relevance (Continued need; Alignment with federal government priorities, roles and responsibilities) | ||
6. Is there a continued need for the Vanier CGS program? |
6.1.1. Identified need for support for world-class doctoral students in Canadian universities:
|
|
6.1.2. Extent of duplication/overlap of Vanier CGS program objectives with other federal granting agency doctoral student funding programs |
|
|
7. Does the Vanier CGS program align with government priorities and is it consistent with federal roles and responsibilities? | 7.1.1. Compatibility of Vanier CGS program objectives/expected results with federal granting agencies’ and federal government’s priorities 7.1.2. Identified role for the federal government in supporting world- class doctoral students in Canada |
|
Core Treasury Board Evaluation Issues
Issue #1: Continued Need for program | Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians |
---|---|
Issue #2: Alignment with Government Priorities | Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes |
Issue #3: Alignment with Federal Roles and Responsibilities | Assessment of the role and responsibilities for the federal government in delivering the program |
Performance (effectiveness, efficiency and economy) | |
Issue #4: Achievement of Expected Outcomes | Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes |
Issue #5: Demonstration of Efficiency and Economy | Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes |
Cross-walk of Vanier CGS Evaluation Questions and TBS Core Issues
Vanier CGS Evaluation Question | Core Treasury Board Evaluation Issue Addressed |
---|---|
1. To what extent has the Vanier CGS program achieved its expected immediate outcomes? 1.1. Has the Vanier CGS program attracted and recruited world-class students? 1.2. Have Vanier CGS recipients received training that has led to their research, leadership and academic development? 1.3. Have Vanier CGS recipients received training that meets their research and career objectives? |
Issue #4: Achievement of Expected Outcomes |
2. To what extent has the Vanier CGS program progressed towards the achievement of its expected intermediate outcomes? | Issue #4: Achievement of Expected Outcomes |
3. To what extent has the Vanier CGS program progressed towards the achievement of its expected long-term outcomes? | Issue #4: Achievement of Expected Outcomes |
4. Has the Vanier CGS program been implemented as intended? | Issue #4: Achievement of Expected Outcomes |
5. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient manner? | Issue #5: Demonstration of Efficiency and Economy |
6. Is there a continued need for the Vanier CGS program? | Issue #1: Continued Need for program |
7. Does the Vanier CGS program align with government priorities and is it consistent with federal roles and responsibilities? | Issue #2: Alignment with Government Priorities Issue #3: Alignment with Federal Roles and Responsibilities |
References
CIHR. (2009). Health Research Roadmap: Creating innovative research for better health and health care. CIHR’s Strategic Plan 2009-10 - 2013-14.
Department of Finance Canada. (2008). The Road to Balance: Creating Jobs and Opportunities [ PDF (3 MB) - external link ].
Department of Finance Canada. (2008). The Budget Plan 2008: Responsible Leadership [ PDF (1.54 MB) - external link ].
Employment and Social Development Canada. (2011). Canadian Occupational Projection System (COPS): University Professors and Assistants.
NSERC. (2014). Report on Plans and Priorities.
Science, Technology and Innovation Council. (2011). State of the nation 2010: Canada’s science, technology and innovation system.
SSHRC. (2013) Strengthening Canada’s Cultures of Innovation: Strategic Plan 2013-16 [ PDF (261.79 KB) - external link ].
Statistics Canada (2014). Education Indicators in Canada: An International Perspective: 2013 [ PDF (4.99 MB) - external link ]. (Catalogue no. 81-604-X).
Statistics Canada. (2011). Expectations and Labour Market Outcomes of Doctoral Graduates from Canadian Universities [ PDF (340.98 KB) - external link ]. (Catalogue number 81-595-M No. 089).
Statistics Canada. (2008). Doctoral Education in Canada: Findings from the Survey of Earned Doctorates 2005/2006. (Catalogue number 81-595-M — No. 069).
Tamburri, R. (2013). The PhD is in need of revision. University Affairs.
Footnotes
- Footnote 1
-
Canadian citizens and permanent resident holders of Vanier scholarships may also apply for a CGS Michael Smith Foreign Study Supplement of up to $6,000 to help offset the costs of undertaking research studies outside Canada for a period of three to six months of their degree.
- Footnote 2
-
The Vanier program was harmonized in 2009 which involved adopting a uniform application and selection process across the three federal granting agencies and involved the allocation of nomination quotas across eligible Canadian universities.
- Footnote 3
-
Eligible universities receive three-year allocations based on 600 total nominations per research granting agency (CIHR, NSERC and SSHRC). The distribution of allocations among eligible universities is based on the following method: for Canadian universities that have a doctoral-level program in the subject areas supported by one or more of the granting agencies, the calculation is based on the sum of the three-year rolling funding average used for the last Canada Research Chairs calculations (2007-08, 2008-09 and 2009-10), and the payments made under the respective granting agencies for the Canada Graduate Scholarships doctoral awards for the three most recent fiscal years (2008-09, 2009-10, and 2010-11). Universities have the flexibility to submit as many applications as they wish in any given year, within their maximum three-year allocation per granting agency. Universities cannot exceed their three-year allocations or transfer allocations between agencies.
- Footnote 4
-
See Triennial Nomination Allocations by University for 2013, 2014 and 2015 Vanier Canada Graduate Studies Scholarships Competitions [ PDF (91.65 KB) - external link ] for allocations per university.
- Footnote 5
-
For the purpose of the evaluation, foreign Vanier CGS scholars were defined as residing outside Canada and being neither Canadian citizens or permanent residents of Canada at the time of their application.
- Footnote 6
-
Vanier scholar supervisors who indicated they supervised more than one award holder were asked to respond throughout the survey to their first award holder.
- Footnote 7
-
Vanier scholar supervisors rated leadership on a nine point scale with ‘exceptional’ being the highest positive rating.
- Footnote 8
-
The role of the selection board to oversee processes, policies and results to ensure that the program was meeting its objectives was transitioned to other existing governance bodies. Membership of the board included a distinguished Canadian as Chair, a member of the Diplomatic Corps, the Chairs of the Boards of each Agency, world renowned researchers and the Deputy Minister of Industry Canada as an observer.
- Footnote 9
-
The evaluation will assess whether Vanier CGS has been implemented as intended over the lifecycle of the program, taking into account various changes made over time to design and/or delivery.
- Date modified: