Expert Review Team Report for Institute of Population and Public Health

Submitted by: Professor Sally Macintyre
Chair, Expert Review Team
February 2011

Table of contents

Summary

The creation of the Canadian Institutes of Health Research (CIHR), with the four pillars of biomedical, clinical, health systems and health services, and population health, research was a bold and innovative move. The Institute of Population and Public Health (IPPH) was given an ambitious mandate, covering both its Institute activities and the championing of population and public health (PPH) research across CIHR as a whole. Its mandate also covered knowledge translation (KT) and global health.

This multiple mandate is extremely challenging, and achieving it will not be easy, especially since the resources allocated to IPPH are the same as for institutes without this dual mandate. While at its inception there was already some excellent PPH research and well-regarded research leaders in Canada, there was not an extensive research infrastructure or critical mass.

We were impressed by the enthusiasm and vision of the inaugural and current Scientific Director in dealing with these challenges by initiating efforts to improve research capacity and infrastructure, and to develop strategic initiatives with other institutes and external stakeholders. Excellent investigators and interesting research projects have been funded, and there is support for the Institute among researchers and stakeholders.

Achievements include:

However, the percentage of total CIHR expenditures relating to PPH research from the open grant scheme is low, and has plateaued or even decreased since 2006. Most of the CIHR open grant committees are heavily biomedical or clinical, and there seems to be a problem either with application rates from PPH scientists or with peer review judgements of such applications.

The standard quantitative information provided by CIHR for this review was inadequate for the purposes of reviewing the success and outcomes of the many activities pursued by IPPH. While 10 years might be too soon to expect an impact of its activities on population health and the distribution of health, one would expect to see an impact on the generation of rigorous, innovative, and important research. Unfortunately, the lack of information meant we were unable to judge whether such an impact had occurred.

Recommendations

CIHR should put in place significantly improved systems for robust monitoring and evaluation of its activities, and be in a position to provide far better evidence of impact and outcome for future reviews.

The open grants committee structure and peer review system should be reviewed to ensure it is fit for the purpose of supporting all 4 CIHR pillars.

CIHR should ensure that the resources and infrastructure available to IPPH are commensurate with its dual mandate, and that the mandate to improve the health of populations and promote health equity is suitably qualified to match the overall mission of IPPH.

IPPH should consider how best to fund larger, longer, better resourced, more generalisable and scalable activities, and achieve further focusing of priorities on fewer, innovative large-scale activities.

Section 1 – Institute mandate

The Institute of Population and Public Health was established in 2000 with an integrative mandate: to support research into the complex biological, social, cultural and environmental interactions that determine the health of individuals, communities and global populations; and to apply knowledge to improve the health of individuals and populations through strategic partnerships with population and public health stakeholders and innovative research funding programs. The IPPH mission is to improve the health of populations and promote health equity in Canada and globally by supporting research and encouraging its application to policies, programs and practices in public health and other sectors.

CIHR Institute of Population and Public Health – Internal Assessment for 2011 International Review, pg 1

The Expert Review Team (ERT) noted that this was a challenging and ambitious mandate, given the three remits of working as an Institute to promote and conduct population and public health research; working across CIHR to promote population and public health research as one of CIHR's four pillars; and having a particular responsibility for representing CIHR in promoting global health. The ERT also observed that the mandate was not just to conduct excellent and policy relevant research, but to improve the health of populations and promote health equity. This is a laudable aspiration, but would be difficult for IPPH to achieve in the short to medium term, given the likely time lags between the publication and dissemination even of excellent and relevant research and any impact on health policy and practice, and on the health of the population and its distribution (e.g. in relation to rates of or risk factors for major chronic diseases).

Section 2 - Status of this area of research in Canada

In the ERT's view, in the years leading up to 2000, when IPPH was founded, there were a number of world-leading scholars in the population and public health field researching in Canada. In an international context Canada's population health research was notable for:

As in many other countries, most population health research in Canada focused on description and explanation rather than intervention.

Representatives of IPPH at the review suggested that at its inception there were two main strengths in Canada; epidemiology (covering both clinical and population health) and health promotion. They noted however that there was not much infrastructure on the ground (for example there were no schools of public health in Canada); and that the funding stream for population and public health (the National Health Research and Development Programme) was entirely separate from the Medical Research Council (MRC), and had fewer funds than those available in CIHR for population and public health research.

Overall impression of the Canadian research landscape in this area

The ERT gained the impression of a bewildering array of funding agencies, stakeholders, capacity building schemes and strategic programmes to support the development of public and population health now in Canada. This is, in part, an inevitable consequence of the federal system in Canada, with a mix of local, provincial, Canada-wide and federal schemes and programmes. Little evidence was provided to indicate that there are systems and structures in place that would help to better co-ordinate these activities, building on individual strengths and identifying major gaps in funding and impact.

IPPH has given considerable priority to capacity building and there have been major changes in Canada's PPH landscape; for example a growth in Masters of Public Health programs, the establishment of five schools of public health, the Public Health Agency of Canada, six National Collaborating Centres for Public Health, and other institutions.

The ERT found it difficult on the evidence presented to assess the extent to which these wider activities are at least in part the result of IPPH's leadership.

Directly attributable to IPPH are a number of capacity building activities such as the creation of seven Centres of Research Development, four annual summer schools, the revitalisation and academic strengthening of the Canadian Public Health Association annual meetings, establishment of Applied Public Health Chairs, the launch of the Population Health Intervention Research Initiative Canada, and the funding of 581 trainees involved in 16 different strategic training initiatives.

The evidence presented to us demonstrated considerable activity leading to new types of research infrastructure that were underpinned by IPPH funding. The presence of population and public health research in Canada is now much more prominent, and it may have a higher status than previously given that it is one of the four pillars of CIHR.

The dual mandate of funding population and public health research and providing leadership across CIHR in this area is innovative and forward looking. However, we were provided with very little compelling evidence that the quality, quantity or impact of population and public health research in Canada had improved or that the international standing of any specific areas of research was better than at the inception of IPPH. Indeed an overall impression was that Canada had not built on its legacy and that the international leadership enjoyed by Canada in the five areas listed above might be under threat.

Section 3 - Transformative Impacts of the Institute

There is clear evidence that the IPPH has transformed the funding landscape for population and public health research in Canada, through its capacity building efforts, leverage of additional funds, and championing of population and public health. 124 masters, 266 doctoral, 138 postdoctoral trainees have received support for training and career development, 20 Interdisciplinary Capacity Enhancement grants have been awarded, and 15 applied public health chairs have been awarded. There is evidence of partnership with the Public Health Agency of Canada, for example in sponsorship of Café Scientifiques; dissemination of work in this field in peer-reviewed journal supplements of the Canadian Journal of Public Health: and the revitalisation of the Canadian Public Health Association annual meetings.

These capacity building and dissemination activities hold great promise for creating policy relevant multidisciplinary population and public health research. However, it is not clear from the evidence provided to the ERT whether this funding is continuing to go to those who were already in this field (many of those cited as receiving CIHR funds appear to have already been prominent in this area) or is drawing in people with new or different disciplines, skills and perspectives. It is also hard to assess on the basis of the information available to the ERT whether these activities have yet transformed (or are likely to transform) the quality, quantity or impact of population public health research in Canada, its uptake by the population or policymakers, and its international standing.

We were concerned at the plateauing of the success rates for population and public health researchers in the open grants committees (static or even declining, at below 5% since 2006). We were struck by the number (48) and apparent biomedical dominance of the open grants committees and the lack of reference in their remits to population or public health aspects of the particular topic. For example the three committees on behavioural sciences are subtitled 'behavioural studies in animal models'; 'clinical behavioural sciences'; and 'behavioural studies and neural imaging'. None appears to cover human social behaviour or behavioural interventions.

Both these observations suggest that the inclusion of population and public health research as one of the four pillars of CIHR has not yet transformed that organisation into one encouraging and integrating interdisciplinary collaborations between all clinical, biomedical and population health sciences. Witnesses suggested that there might be a problem with population health peer reviewers being too strict with their peers, but the relative lack of funding for population health research from the open grants committees could be due to either a lack of applications because population health researchers do not perceive the committees as being relevant or receptive to them, or to a lack of success when they do apply. This would benefit from more systematic analysis by CIHR.

Overall impression – to what extent has this Institute been transformative?

The ERT commend the strategic shift to intervention research and to research on social disparities in health. These remain under-researched areas in PPH and have the greatest potential to deliver improvements to the health of populations and to promote health equity. However, if CIHR is serious about improving population health and reducing disparities in health, a paradigm shift may be needed to deal more directly with upstream social determinants of health, which might involve larger, longer, better resourced, more generalisable and scalable activities, and a further focusing of priorities on fewer, innovative large-scale activities. This is in contrast to the apparent plethora of small-scale, shorter-term funding schemes that characterize the current IPPH investments.

Based on the evidence available to the ERT, and the reports of witnesses, it appears that a population and public health approach is not embedded right across CIHR's activities. This may be because of the rigidity of the open grants scheme, the lack of a PPH perspective among the other institutes and grants committees, and/or to the challenges of meeting the Institute's dual mandate.

Section 4 - Outcomes

The ERT was very disappointed with the quality of the information available to it at review. In particular CIHR generally and IPPH specifically appear not to have systems in place for monitoring outcomes and outputs emerging from the funded activities. For example, the list of publications provided to the ERT (after requests for further information) was inadequate in that the publications only covered one sub area of the Institute activities (health disparities); dated from 1997, while no publications can have resulted from the Institute's activities until around 2002; does not state whether the publications are attributable to CIHR funding; and includes publications whose relationship to health disparities or population and public health research was not clear (e.g. 'A randomized controlled trial of surgery for temporal lobe epilepsy').

The bibliometric bubble chart also only covered one sub area, was not restricted to CIHR funded work, and by covering the whole of 2000-2008 did not permit any analysis of trends attributable to CIHR funding.

The report notes the number of papers published since 2004 by some senior and already eminent scholars funded by the Institute but without clarifying whether these were solely attributable to their CIHR funding (which only started in 2004). We understand IPPH has undertaken some internal evaluations of its activities, but these were not made available to the ERT.

Most of the outcomes demonstrated in the report are inputs or activities, and it is hard to assess from these the extent to which this Institute has been successful in achieving outcomes. At a minimum, we would have expected to see lists of directly attributable reports and publications, grants and renewals of grants, disciplines and next destinations of trainees, and evidence of practice or policy changes or guidelines. At a higher level, we would have expected to see a list of major research achievements (i.e. the five or ten most important research findings or changes in policy directly attributable to IPPH funding). The information provided would need to be substantially improved for any future review.

Overall impression – to what extent has this Institute been successful in achieving outcomes?

We were impressed by the enthusiasm and commitment of IPPH staff and the range and number of activities they had set up, but unfortunately robust evidence about outcomes, as opposed to activities, is lacking. The relative lack of success of population health researchers in obtaining funding from the open grant scheme is disappointing.

Section 5 - Achieving the Institute mandate

The Institute mandate is challenging in that it involves not just running one institute but championing population and public health research across CIHR and until recently also championing global health initiatives. The ERT understands that it has had to do this on the same budget (c $8.5 million per annum) as other institutes without this dual or triple mandate. The Institute may only be able to achieve its mandate if CIHR corporately grapples with the barriers to enhancing and embedding population health research across the organization. This may require more financial resources, but more importantly it may require sustained strategic thinking about how best to harness the strengths of population and public health research in order to improve population health and reduce health disparities across all the disease and system areas covered by the CIHR.

The ERT commends the current strategic priorities of the Institute:

To achieve these goals, an important objective should be to champion improved access to social, behavioural and health data about the population and subsets of the population. Canada has good health data at the provincial level, but it was explained to the ERT that there are difficulties in accessing existing data, possibly as a result of privacy concerns and jurisdictional disputes, which impedes Canada's capacity to monitor outcomes of social or health system changes, including using interrupted time-series data to examine the impact of natural experiments. To have any impact on population health or disparities in health (for example in relation to key health risks such as tobacco and alcohol use, obesity, physical activity) large-scale interventions or natural experiments are likely to be required and these can only be evaluated if there are good data on large populations.

Achieving the Institute mandate and these current strategic priorities may also require focusing on fewer, higher-level, activities that are more likely to make a difference than numerous small-scale interventions.

Consideration should be given to size, scalability, and generalisability, which may all be lacking if research funds are spent on a plethora of small, local initiatives. Developing implementation science and the science of change (as opposed to engaging in implementation activities) will also be important.

Overall impression – to what extent has this Institute achieved its mandate?

IPPH has undertaken a large number of innovative and capacity building initiatives, and has, following the 2006 review, narrowed its focus to concentrate on intervention research. However, its mandate is ambitious, and it is too early to tell whether it will achieve it.

Section 6 - ERT Observations & Recommendations

It is a difficult job to achieve the dual mandate of theme champion and running an Institute, especially when this is undertaken by a scientific director seconded for only 50% time. Scientific directors may need more support, and the two institutes with the dual mandate may also need more corporate support.

There is a critical need for better access to large-scale datasets for population public health research in Canada, both to undertake monitoring, descriptive, and explanatory research, and also to evaluate the outcome of any large-scale population health initiatives or natural experiments.

The new priorities are appropriate, but may still be too numerous, and the ERT thinks that it is important to give the priorities long enough time to bed in, before changing them. Intervention research in particular is a long-term enterprise.

It is important to track theme 4 through the open grant committees and other institutes. It may be appropriate to have a population/public health researcher on the Institute Advisory Boards of other institutes, or on the open grant committees, to advocate for and remind colleagues of the importance of a population health approach.

The ERT was impressed by the enthusiasm and commitment of the Institute's Scientific Director and colleagues, and the researchers funded by IPPH. Stakeholders, although hugely supportive, felt that the Institute should now focus on high-quality policy relevant research likely to make a difference.

We noted that IPPH had initially had responsibility for a global perspective in CIHR but that this global perspective was now to be mainstreamed across the organization. If this can be done successfully it is to be applauded, but there is a risk that if it is not anybody's responsibility in particular it will be nobody's. There needs to be a transparent mechanism for championing a global perspective.

The metrics supplied by CIHR for this review were wholly inadequate for the task, and made it difficult for the ERT to assess the performance of the Institute. There need to be improvements in corporate evaluation and monitoring.

Knowledge translation is important, and there is an observable appetite from stakeholders for engagement with research coming out of this Institute. However, there needs to be empirical research into how best to do KT in this field, and CIHR wide systems and support to enable it to happen.

Overall impression of the performance of this Institute

IPPH was allocated a difficult and challenging mandate, given the previous dominance of biomedical and clinical research under the MRC, and the two Scientific Directors to date are to be commended for their innovative, enthusiastic and committed initiatives in capacity building and research funding. It is too early to assess the extent to which these training and research funding activities will create a step change in the quality and impact of population public health research in Canada, let alone a significant change in population health and the distribution of health.

Recommendations

CIHR should put in place significantly improved systems for robust monitoring and evaluation of its activities, and be in a position to provide far better evidence of impact and outcome for future reviews.

The open grants committee structure and peer review system should be reviewed to ensure it is fit for the purpose of supporting all 4 CIHR pillars.

CIHR should ensure that the resources and infrastructure available to IPPH are commensurate with its dual mandate, and that the mandate to improve the health of populations and promote health equity is suitably qualified to match the overall mission of IPPH.

IPPH should consider how best to fund larger, longer, better resourced, more generalisable and scalable activities, and achieve further focusing of priorities on fewer, innovative large-scale activities.

Appendix 1 - Expert Review Team

Chair - Professor Sally Macintyre
Professor, Division of Community Based Sciences, Faculty of Medicine University of Glasgow
Honorary Director MRC/CSO Social & Public Health Sciences Unit, UK

Expert Reviewer – Professor Don Nutbeam
Vice-Chancellor University of Southampton
Professor of Public Health
UK

International Review Panel – Dr. Chris Murray
Director, Institute for Health Metrics and Evaluation
Professor of Global Health, University of Washington
Seattle WA USA

Appendix 2 - Key Informants

Session 1 – Review of Institute

  1. Dr. Nancy Edwards, IPPH Scientific Director
  2. Dr. Gilles Paradis, Chair (2007-2009) – Institute Advisory Board
    Professor, Department of Epidemiology, Biostatistics and Occupational Health
    McGill University
  3. Dr. Roy Cameron
    Executive Director, Centre for Behavioural Research and Program Evaluation
    Professor, Faculty of Applied Health Sciences
    University of Waterloo
  4. Dr. Clyde Hertzman
    Director, Human Early Learning Partnership (HELP)
    Professor, School of Population and Public Health
    University of British Columbia

Session 2 – Consultation with researchers

  1. Dr. Louise Potvin
    Professor, Social and Preventive Medicine, Faculty de Medicine,
    Université de Montréal
  2. Dr. David Hammond
    Assistant Professor, Department of Health Studies and Gerontology
    University of Waterloo
  3. Dr. Patricia O'Campo
    Professor, Division Epidemiology Social and Behavioral Health Sciences
    University of Toronto

Session 3 – Roundtable with stakeholders

  1. Dr. Cory Neudorf
    Chief Medical Health Officer, Saskatoon Health Region
    Chair, Canadian Public Health Association
  2. Dr. Michael Wolfson
    Assistant Chief Statistician, Analysis and Development
    Statistics Canada
  3. Mr. Michael Clarke
    Director, Information and Communication Technologies for Development
    International Development Research Centre (IDRC)
Date modified: