A Guide to Knowledge Synthesis

A Knowledge Synthesis Chapter

Prepared by: Jeremy Grimshaw MBChB, PhD, FCAHS
Canada Research Chair in Health Knowledge Transfer and Uptake
Senior Scientist, Ottawa Hospital Research Institute,
Ottawa Professor, Department of Medicine, University of Ottawa
jgrimshaw@ohri.ca

[ PDF (156 KB) ]

1. Background

Knowledge Synthesis for Knowledge Translation
The Canadian Institutes of Health Research (CIHR) defines knowledge translation as 'a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the healthcare system'. This definition highlights the importance of knowledge synthesis in knowledge translation activities.(1)

CIHR defines synthesis as 'the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. A synthesis must be reproducible and transparent in its methods, using quantitative and/or qualitative methods. It could take the form of a systematic review; follow the methods developed by The Cochrane Collaboration; result from a consensus conference or expert panel and may synthesize qualitative or quantitative results. Realist syntheses, narrative syntheses, meta-analyses, meta-syntheses and practice guidelines are all forms of synthesis.'(1)

CIHR has regular RFAs for knowledge syntheses relevant to the needs of the Canadian health care system. The purpose of this chapter is to discuss the rationale for knowledge syntheses, outline current approaches and methods for syntheses, and highlight available resources to aid potential applicants.

Rationale for Knowledge syntheses
Science is a cumulative process that develops iteratively; few studies by themselves are sufficiently persuasive to change practice or policy. Individual studies may be misleading due to chance or bias. Ioannidis and colleagues have undertaken a landmark series of studies exploring the evolution of basic and applied research that highlights concerns about the reliability and interpretation of individual studies. They observed that the results of the most highly cited basic science and clinical research papers published in the most prestigious journals are frequently overturned or challenged by subsequent less prominent publications.(2;3) Further they observed that the results of early publications in both basic and clinical research were often likely to report more strikingly positive or negative findings than subsequent publications.(4) Together these studies highlight the problems of focusing knowledge translation efforts on individual studies (especially early publications with striking findings) and suggest that the evidence base in any field needs to mature and be synthesized before an observer can reliably understand its implications.

Reviews have always played an important role in health research and knowledge translation. Traditionally reviews were written by acknowledged experts in a field and provided little information with respect to how the expert had conducted the review, what evidence the expert considered when writing the review, or the scientific basis for any of its recommendations. Mulrow assessed the quality of 50 reviews published in 1985-6 in four major medical journals and observed that most reviews did not report the use of scientific methods to identify, assess, and synthesize information.(5) As a result, Mulrow criticized such traditional reviews as 'haphazard, biased and subject to the idiosyncratic impressions of the individual reviewer'.(6) In contrast, Mulrow suggested that knowledge syntheses are an efficient scientific approach to identifying and summarising evidence that allow the generalisability and consistency of research findings to be assessed and data inconsistencies to be explored.(6) Further, the explicit methods used in syntheses limit bias and improve the reliability and accuracy of conclusions.(6)

Knowledge syntheses are important for establishing the key messages from the global evidence in a research field prior to knowledge translation and to inform the design and conduct of new research. It is critical that that is information is produced and conveyed in a timely manner to important end-users (e.g. clinicians and researchers) such that patient care and resources may be optimally managed.

Failures to use formal synthesis methods have led to delays between the generation of evidence on treatments and the time when clinical experts made treatment recommendations in line with new research findings. For example, Antman and colleagues observed a 15 year gap between the time when meta-analysis could have demonstrated the effectiveness of thrombolysis for acute myocardial infarction and widespread recommendations for its use and when meta-analysis could have demonstrated potential harms for routine use of anti-arrhythmics and widespread recommendations against its use.(7)

Failures to use formal synthesis methods have also led to unnecessary repetition of research studies. For example, Fergusson observed that a further 7,674 patients were enrolled in 55 efficacy trials of aprotinin for over 11 years after there was clear evidence of its efficacy;(8) resulting in inefficient use of research funding and delays in the conduct of an effectiveness trial (that subsequently demonstrated the potential harms).(9) Lee and colleagues argued that the lack of treatment benefit (and potential harms) of endothelium receptor blockers in heart failure patients would have been apparent if a formal synthesis of animal model studies had been undertaken prior to human studies; as a result of the failure to undertake a formal synthesis over 1,600 patients were randomized in the ENABLE trial exposing patients in the treatment group to unnecessary risks.(10;11)

Knowledge syntheses have also identified sources of potential bias in primary studies resulting in improved conduct of primary studies. For example, syntheses have highlighted the importance of concealment of allocation in randomized trials of treatments.(12) Knowledge syntheses have also highlighted the ongoing problem of publication bias (failure to publish negative studies)(13;14) that has led to recent initiatives to register intervention studies(15-17) and poor reporting of primary studies that has supported the generation of reporting guidelines.(18)

These (and other) findings highlight the need for formal evidence syntheses using reliable, reproducible and explicit methods to inform policy, practice and future research efforts. Fortunately over the last fifteen years there has been a dramatic increase in the number of syntheses conducted. One recent study suggested that approximately 2,500 English language systematic reviews are published each year in Medline indexed journals.(19) Knowledge syntheses are increasingly recognized as important scientific communications in their own right; for example, syntheses are more likely to be cited than other study designs.(20) Knowledge syntheses are also used as the evidentiary base for other evidence tools. Syntheses are recommended as an integral step by most guideline development agencies (e.g (21). and are three (out of twenty three) criteria in the AGREE instrument focus upon the methods of identifying and synthesising the evidence for clinical practice guidelines.(22). Similarly the International Patient Decision Aids Standards (IPDAS) Collaboration recommends that decision aids should be based on a systematic development process that includes identification of up-to-date scientific evidence.(23) These developments suggest widespread recognition of the importance of knowledge syntheses to underpin knowledge translation activities.

2. Approaches to knowledge synthesis

Purposes of knowledge synthesis
Most syntheses are conducted either for the purpose of 'knowledge support' or for 'decision support'.(24) Syntheses for knowledge support are confined to summarizing the evidence around a specific question or issue and do not undertake additional tasks to support a decision in a particular context. Whereas syntheses for decision support will commonly include some or all of the following steps: engagement of decision making audience in the development of the research question and synthesis protocol; consideration of several related questions using appropriate methods; deliberative process of engaging the decision making audience to interpret and contextualize the results of the synthesis; and development of context specific recommendations.

Stages of knowledge synthesis
Chalmers notes that undertaking a synthesis 'has the same basic components as any other scientific investigation'.(25) There are a growing range of methods appropriate to synthesize different types of evidence for different types of research synthesis. However the majority of these approaches follow a broadly similar approach. In this section, we highlight the common stages in knowledge synthesis based upon the frameworks by Chalmers(25) and of Pope, Mays and Popay.(26) Subsequent sections will discuss methods of different approaches to knowledge synthesis highlighting where the methods differ from these frameworks.

In general syntheses involve the following steps:(25;26)

• Stating the objectives of the research
• Defining eligibility criteria for studies to be included;
• Identifying (all) potentially eligible studies;
• Applying eligibility criteria;
• Assembling the most complete data set feasible, including,
  • data extraction;
  • quality appraisal of included studies;
• Analyzing this data set, using statistical synthesis and sensitivity analyses, if appropriate and possible; and
• Preparing a structured report of the research.

These stages highlight the scientific nature of syntheses and the need for rigorous methods to ensure the validity/trustworthiness of syntheses. In effect, syntheses are scientific studies that derive data from primary studies rather than cell lines, animal models or human subjects. This creates its own methodological challenges including optimal approaches to identify primary studies, methods to appraise individual studies, handling incomplete or missing data, methods to synthesize studies and methods to minimize bias within the synthesis process (e.g., identifying potential publication bias discussed below).

Pope, Mays and Popay(26) also highlight that while some synthesis approaches involve a relatively linear process through the stages, others involve a more iterative approach, such as refining the 'objectives of the research' throughout the synthesis process or using snowball approaches to study identification.

There is increasing recognition of the importance of developing a detailed protocol prior to embarking on a synthesis. Syntheses are in effect retrospective observational studies and there is a danger of bias if researchers do not prospectively describe their scientific methods before looking at the available data. Protocols of relatively linear syntheses should be able to describe the purpose and methods of the synthesis in sufficient detail to allow a third party to replicate the review. Protocols of more iterative syntheses should be able to describe the purpose, methods, principles, and likely decision rules that will guide the conduct of the review. Ideally protocols should be available in the public domain to allow readers of syntheses to assess whether the synthesis teams¹ have followed their research plan. However with the exception of reviews undertaken by The Cochrane Collaboration this is rarely the case, although there is no reason why researchers should not be able to post their synthesis protocols on their own websites². The development and publication of synthesis protocols enhances the transparency and accountability of the synthesis process. In the same spirit, reports of syntheses should be comprehensive and transparent to allow readers to appraise their likely validity/trustworthiness (see below). Opportunities to publish additional information on the web (either linked to the journal of publication or researchers' own websites) increasingly allow for full reporting of syntheses.

3. Synthesis methods

The most common approach to synthesis used in healthcare over the last twenty years has involved systematic reviews of effectiveness (or 'what works') questions. However systematic reviews are a generic approach that can be used to synthesise different types of evidence addressing different types of questions. Recently there has been the development of a broader range of synthesis approaches that expand the types of evidence and questions that can be considered. Given that most of the new approaches build upon the methods of systematic reviews, we describe these methods first, which are then followed by the newer approaches.

Systematic reviews
Systematic reviews are reviews 'of a clearly formulated question that use systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies.'(27)

Systematic reviews are a generic method that can be used to address diverse research questions such as:

• What are the benefits and harms of treatment 'X' in animal models?
• What are the benefits and harms of treatment 'X' in humans?
• What are the benefits and harms of a new service delivery configuration?
• What are the benefits and harms of a quality improvement initiative?
• What is the accuracy of diagnostic test 'X'?
• What is the accuracy of routine coding following hospital discharge?
• What are the experiences of patients undergoing treatment 'X'?
• What is the prevalence of condition 'X'?
• How strong is the association between gene 'A' and disease 'X'?

Systematic reviews of effectiveness questions
In this section we focus on the methods of systematic reviews for questions about the efficacy or effectiveness of healthcare treatments and policies. For further information and practical examples, readers should also consult Higgins and Green,(28) Centre for Reviews and Dissemination,(29) and Petticrew and Roberts.(30)

i. Stating the objectives of the research
As in all research, framing the research question is perhaps the most important foundational step, as it guides the methods and processes of the systematic review. When planning a systematic review it is often helpful to discuss the research objectives widely to ensure that the systematic review is relevant and addresses the needs of the different potential stakeholder audiences. It is recommended that a preliminary search is undertaken in order to ensure that an up-to-date systematic review of the research question of interest does not already exist and to gauge the likely number of studies (and thus, amount of work) that will be included in the review.

In general the more specific the objectives of the research, the more amenable the question will be to systematic review methodology. There are a number of aids to help structure the research question. For effectiveness questions, the PICO mnemonic is often used to frame the research question according to the Participants (e.g., patients with stroke), the Intervention (e.g., admission to a stroke unit), the Comparison (e.g.,compared to standard care) and the Outcome (e.g., mortality) (sometimes Context or Study design are added as a fifth consideration).(31) This challenges synthesis teams to carefully identify the key components of their research question. This is not always straightforward particularly if synthesis teams are interested in the effects of complex interventions. Sometimes synthesis teams need to undertake considerable background work to specify an intervention in detail, especially for complex interventions. It is possible to undertake a comparable process for other types of questions. For example, the Cochrane Collaboration recommends that reviews of diagnostic test accuracy should be framed according to the patients, the target condition and the test or tests being evaluated (including the index test and alternative tests).(32)

A key issue that synthesis teams need to consider when framing the research question is how broadly or narrowly to frame questions. For example, an effectiveness review could focus upon a class of drugs for condition 'X' or a specific drug for condition 'X'. A review of the accuracy of routine coding after hospital discharge could look at studies across all conditions and procedures or only focus upon surgical procedures. Often reviews that address a broad question are called 'lumped' reviews, whereas reviews that address a narrow question are called 'split'.

The 'lumping' rationale is that systematic reviews aim to identify the common generalisable features addressing the research question and that minor differences in study subjects, context and design are not important. 'The 'splitting rationale' is that it is only appropriate to consider studies that have very similar study subjects, context and design to avoid being misled by comparing 'apples and oranges'.(33)

There are good methodological reasons for taking a broad approach.(34) Broad systematic reviews allow the generalisability and consistency of research findings to be assessed across a wider range of different settings, study populations and behaviours. This reduces the risk of chance results. However lumped reviews are more logistically and analytically challenging. They will tend to involve broader search results increasing the number of studies to be screened and included and the gamut of studies captured are often difficult to combine in a meaningful way. Further as a result of allowing a broader range of studies reflecting different study subjects, context and design, heterogeneity is usually expected and interpretation maybe more challenging. In contrast 'split' reviews are relatively easier, quicker and cleaner but may be less generalizable and informative to healthcare decision makers.

Pope, Mayes and Popay note 'Policy makers tend to approach researchers with 'lumping' questions and researchers mostly prefer to answer narrow questions resulting from 'splitting'.(26) The decision about whether to take a lumping or splitting approach involves judgment on the part of synthesis team considering how to maximize the informativeness of the research question within available resources.

ii. Defining eligibility criteria for studies to be included
The next stage is to identify detailed eligibility criteria to determine which studies should be considered for inclusion within the review. These should be driven by the research question. The eligibility criteria should directly inform the development of the search strategies and provide the basis for assessing search results for potentially relevant studies. If they are poorly specified it could lead to the development of search strategies that are insensitive (fail to identify some or all relevant studies) and/or non specific (increase the workload associated with screening searches). A key consideration is what types of study designs are relevant for the specific research questions. For example, as randomized trials are the 'gold standard' for clinical effectiveness questions, a synthesis team would need to carefully weigh the additional benefit that may be gleaned from the inclusion of other designs, against the increased workload, heterogeneity, and potential bias these studies may incur. In some cases however, non-randomized designs may be helpful and/or more informative such as observational studies of treatment harms.(35;36) and for reviews addressing diagnostic test accuracy or gene-disease association questions.

It is also important to specify the eligibility criteria in sufficient detail to guide inclusion/exclusion decisions when screening studies. For example, many complex interventions are poorly described and the eligibility criteria for a systematic review of a complex intervention should provide sufficient detail about the prototypical or core elements of the intervention to allow synthesis teams to assess from the published description of the intervention whether a study should be considered eligible for inclusion.

iii. Identifying potentially eligible studies
The next stage is to develop sensitive search strategies to identify studies that potentially meet the inclusion criteria. This is a highly technical task and should rarely be undertaken without the support of a trained information specialist (librarian). Key issues to consider include which bibliographic databases to search, the development of appropriate search strategies and other strategies to identify potentially relevant studies.

Which bibliographic databases should be searched
Current estimates suggest that around 20,000 medical and health related journals are published each year. MEDLINE is probably the most comprehensive bibliographic database but even this only indexes 5,200 journals in 37 languages.(37) EMBASE indexes 4,800 journals including 1,800 journals that are not indexed in MEDLINE.(38) This suggests that synthesis teams should usually expect to search more than one bibliographic database. Additional databases that synthesis teams might consider include general bibliographic databases such as SCOPUS, CINAHL and CENTRAL and/or regional bibliographic databases and/or specialized databases.(39) Information specialists will often have detailed knowledge about the coverage and overlap of different databases and can advise synthesis teams about which databases to search.

Development of appropriate search strategies
In general, search strategies include both content and methodological terms based upon the controlled vocabulary of bibliographic databases being searched (e.g., MeSH in Medline) and also free text terms. There are an increasing number of highly sensitive methodological term filters that are helpful in developing search strategies. The InterTASC Information Specialists' Sub-Group has developed a web based resource listing various methodological search filters.(40) However it is important to assess the reliability, performance and up-to-datedness of any search filter before using it within a systematic review.(39) It is advisable for synthesis teams to identify examples of potentially relevant studies and consider how these studies have been indexed in bibliographic databases. This will often provide ideas about potential content search terms relevant to the review. It is important to recognize that search strategies developed for one bibliographic database will need to be translated for use in other bibliographic databases. The detailed knowledge of information specialists about both methodological search filters and the controlled vocabulary is invaluable during the development and/or execution of a search strategy.

Other strategies
In addition to searches of major bibliographic databases, synthesis teams might consider other strategies to identify potentially eligible studies. These include:

• Searching 'grey' literature - grey literature include studies that are not formally published in books or journals and can include conference proceedings and abstracts, dissertations and theses, project reports, government documents etc. The benefits of searching grey literature will depend upon the likely yield in relation to the review question. There are a number of grey literature databases available.(39)
• Searching databases of ongoing research - it is increasingly possible to search for ongoing studies given recent initiatives to register clinical trials(15-17) and the availability of research funder databases of ongoing studies. This may identify studies that are nearing completion that might influence the results of the review. It may also identify completed studies that have not been published.
• Tracking citations - it is possible to search for studies that have cited key studies in citation indexes such as the Science Citation Index. In addition, it is possible to check the citations in potentially relevant studies.
• Hand searching journals - hand searching involves a 'manual page-by-page examination of the entire contents of a journal issue' to identify eligible studies.(39) The rationale for hand searching is that the indexing in bibliographic databases is imperfect (e.g. relevant studies may not be indexed in electronic bibliographic databases or if indexed, may be done so in a way that alludes being identified by search strategies) with the result that potentially eligible studies may be missed if synthesis teams only search bibliographic databases.
• Contacting experts in the field - content experts may be aware of research studies that may not be easily identified through any of the above channels. Content experts could be contacted at the beginning of a review to identify key studies or after the searches have been screened to identify any missing studies.

Balancing sensitivity and specificity within available resources
Synthesis teams could commit all their resources to the searching stage of a systematic review, even so it would be impossible to develop the perfect search strategy to identify all relevant studies. When developing search strategies, it is important to consider what trade-offs synthesis teams are prepared to make between sensitivity and specificity given the likely available resources for their review. A highly sensitive search strategy reduces the risk of missing key studies relevant to the review but will increase the total number of records from bibliographic databases that need to be screened. Under some circumstances, synthesis teams should be prepared to accept a less sensitive search strategy if they have limited resources for searching and are prepared to accept potentially missing relevant studies.

Peer review of planned searches
Given the complexity of searches, it is usually advisable to have a second independent information scientist peer review the search strategies before they are formally executed.(41)

iv. Applying eligibility criteria
Higgins and Deeks outline a typical process for screening search results against the eligibility criteria to select studies for inclusion within a systematic review involving:

• Merging results from multiple searches using reference management software and removing duplicate records of the same study;
• Examining titles and abstracts to exclude obviously irrelevant reports;
• Retrieve and examine full text copies of potentially relevant reports against eligibility criteria;
• Contact authors, when appropriate, to seek further information needed to judge the eligibility of potentially relevant studies; and
• Make final decisions on study inclusion.(42)

Synthesis teams need to decide whether screening will be undertaken by more than one screener. Using at least two independent screeners minimizes the risk that relevant studies will be discarded.(42) It is often helpful to develop a practice dataset that screeners can train upon before undertaking screening for the systematic review. This also allows screeners to become comfortable with the screening criteria and provides an initial assessment of agreement between screeners (thus highlighting any need for further training and/or clarification of screening criteria before the main searches are undertaken). When reporting the systematic review, reporting guidelines encourage the use of flow diagrams to represent how studies were identified and selected.(43) This suggests that synthesis teams need to be meticulous in recording the results of their searches, the stage and reason for exclusion of studies and the final list of included studies. Finally, it is helpful for synthesis teams to report a list of excluded studies providing details of 'any studies that a reader might plausibly expect to see among the included studies'.(42)

v.i Assembling the most complete data set feasible
Data extraction involves identification of key data items that need to be collected, development of a data extraction form and coding book, extraction of data from the primary reports, and checking the reliability of data extraction. The key data items should be based upon consideration of what data are needed to address the review question but typically involve information on bibliographic details, methodology, context and setting, participants, interventions, outcomes, results and other data.(42)

Having decided upon the data items to extract, it is helpful to develop a data extraction form to record the key data items and also the source of each data item and where it was found in the study report records. The data extraction form also acts as a historical record of the decisions that occur during the review process. It is helpful to pilot data extraction forms with a diverse sample of included studies as this often identifies problems with the data extraction form which can then be revised prior to large scale data extraction. Petticrew and Roberts provide an example of a data extraction sheet.(30)

Buscemi and colleagues observed more errors when data extraction was undertaken by a single extractor compared with two data extractors.(44) As a result it is good practice to have more than one extractor; furthermore, The Cochrane Collaboration also recommends that data extractors have complementary skills ('for example a methodologist and a topic area specialist').(42)

v.ii Appraising studies
A key element of undertaking a systematic review of an effectiveness question involves an appraisal of the likely validity of the included studies. This recognizes that individual studies may have problems with their design, conduct, and analysis that should reduce the confidence of the synthesis teams about their potential validity.(45) For example, there is empirical evidence that failure to conceal allocation sequence in randomized trials may lead to biased estimates of treatment effects in favor of the intervention group.(46) Appraising studies allows synthesis teams to consider threats to validity during the analysis or interpretation of the review.

There are numerous tools for appraising both randomized(47) and non randomized studies of effects(48) including checklists (that ask specific questions but do not provide an overall summary score) and scales (that combine responses to questions into an overall summary score). Many scales have been poorly validated and include items that are not directly related to internal validity, as a result the use of scales is generally not recommended.(29;45) Recently The Cochrane Collaboration has proposed an approach based on empirical evidence to appraising randomized controlled trials in which assessments are made separately for key study characteristics.(45) Petticrew and Roberts also provide a wide range of examples of tools for assessing different study designs.(30)

vi. Data analysis and interpretation
Deeks and colleagues identify a general framework for synthesis of effectiveness studies focusing on four questions: 1) What is the direction of the effect?; 2) What is the size of the effect?; 3) Is the effect consistent across studies?; and 4) What is the strength of evidence for the effect?(49) These questions can be answered using both narrative synthesis and meta-analytic approaches. Narrative synthesis approaches involve a structured interrogation and summary of the findings of included studies. Meta-analysis involves the statistical combination of results from two or more separate studies.(49) It is always possible to undertake a narrative synthesis within a systematic review; whether is possible and/or wise to undertake a meta-analysis is a judgment that the synthesis team must make. In this section we briefly consider the steps relating to preparing for synthesis, and the methods of narrative synthesis, meta-analysis, sensitivity analyses, and other diagnostics.

Preparing data for synthesis
Synthesis teams may need to transform data extracted from published reports to ensure that the data is in the desired format for analysis, for example synthesis teams may need to compute standard errors from confidence intervals and/or p-values for meta-analysis. Higgins and Deeks provide practical guidance for commonly required transformations.(42) It is also often helpful to tabulate details of study characteristics (including quality) and study results to provide a synthesis team an overview of the included studies prior to synthesis.

Choice of narrative synthesis or meta-analysis
The benefits of meta-analysis include increased power and precision, the ability to answer questions not posed in the individual studies, and to settle controversies arising from apparently conflicting studies.(49) However meta-analysis could be misleading in the presence of: substantial variation (heterogeneity) in study characteristics (leading to the classic criticism of meta-analysis that it combines apples and oranges); inclusion of individual studies with high risk of bias; serious publication and/or reporting bias.(49) As a result synthesis teams need to carefully consider the sensibility of undertaking meta-analyses, attempt to minimize these risks when planning the review (e.g., through use of comprehensive searches to minimize risk of publication bias) and wherever possible explore the risks that a meta-analysis is misleading through the use of sensitivity analysis (e.g., whether observed effects change when studies at high risk of bias are omitted) and other diagnostics (e.g., funnel plots to explore the likelihood of publication bias).

Narrative synthesis approaches
Despite the fact that narrative synthesis is one of the most common approaches to synthesis, there is surprisingly little guidance to their conduct. Petticrew and Roberts identify three stages in narrative synthesis:
i. Organisation of studies into logical categories - This entails organizing studies into discrete categories to guide the analysis. The organizational structure will largely depend upon the review question but could include grouping studies by intervention, population, setting or design. Petticrew and Roberts highlight how ways of organising studies frame the focus of enquiry 'Organising studies primarily by study design ... can illustrate where the strongest evidence ... lies. Grouping the studies by the outcomes ... will facilitate answering studies about how outcomes may best be achieved; and grouping studies by intervention type will help answer questions about the relative effectiveness of those interventions'.(30)
ii. Within study analysis - This involves 'a narrative description of the findings of each study often with a description of study quality'.(30)
iii. Cross study synthesis - The aim of this stage is 'to produce an overall summary of the study findings taking account of variations in study quality and other variations (such as variations in populations, interventions and settings) that may affect the generalizability of studies'.(30) This should result in a description of the number of studies included in the review and range of effects observed across these studies.

Meta-analysis
There are a broad range of meta-analytical techniques addressing different types of study designs and outcomes that reflect different statistical assumptions. This highlights the importance of having an experienced meta-analyst as part of a synthesis team if meta-analysis is anticipated. When undertaking a meta-analysis, synthesis teams need to consider: which comparisons should be made; which study results should be used in each comparison; what is the best summary measure of effect for each comparison; are the results of studies similar within each comparison; and how reliable are those summaries. Most meta-analytical approaches are variations on 'a weighted average of the effect estimates' from the included studies.(49) It is beyond the scope of this module to provide detailed guidance and critique of meta-analytical methods; fortunately there are now a wide range of resources for synthesis teams to consult.(49-52) There also continues to be innovative developments of the methods of meta-analysis (e.g., see(53;54)).

Sensitivity analyses and diagnostics
As mentioned above systematic reviews and meta-analyses can be misleading if their results are substantially influenced by studies with high risk of bias or in the presence of publication and/or other reporting biases. Therefore a key issue for synthesis teams is to explore the robustness of their findings and the potential impact of these biases. The effects of variable quality of the included studies can be explored by undertaking qualitative or quantitative sensitivity analyses that explore whether the observed affects across studies are robust if studies at higher risk of bias are excluded. Similarly synthesis teams should always consider the potential for publication or selective reporting bias and wherever possible use diagnostic tests (for example, the use of funnel plots) to explore these issues empirically.(55;56)

Interpreting the results of systematic reviews
A key step in syntheses relates to the interpretation of the synthesis findings to identify the key messages, the robustness of the body of evidence supporting the key messages and their implications for policy, practice and future research. Frequently the key messages may be clear based upon the findings of the synthesis, however this will not always be the case especially when interpreting the findings of complex syntheses or syntheses whose results are likely to be valued differently by different stakeholder groups. This highlights the importance of having a broad based synthesis team with diverse perspectives and skill sets and the need for discussion of the key messages within the team.

There are a number of systems for assessing the robustness of a body of evidence. For example the GRADE approach provides a method of grading quality of evidence, based upon considerations of study design, within-study risk of bias (methodological quality), directness of evidence, consistency and precision of study results and assessment of likelihood of publication bias.(57-60) Although the primary driving force of GRADE relates to the study design (for example, well conducted randomized trials are considered more robust than well conducted observation studies), the quality of individual randomized and trials and observational studies can be ungraded or downgraded based upon careful appraisal of the individual studies.(57)

When interpreting the results of a synthesis it is also important to assess the likely applicability of the results across diverse populations and healthcare systems to help frame the implications of the review (by for example, highlighting uncertainties of the applicability of evidence to specific population subgroups or contexts). When considering clinical interventions, it is important to consider the extent to which the included studies adequately represent the diversity in biology, context and culture, likely adherence rates and values and preferences of the population of interest. Likewise, when considering policy and management interventions, it is important to consider the extent to which the included studies adequately represent diversity in structural elements of different health systems, the perspectives and influence of stakeholders in different health systems and likely resources in different health systems.(61) It is uncommon for a synthesis to include many studies that adequately represent patient, cultural or health system diversity and so synthesis teams need to consider how well the observed results can be applied to subgroups not directly represented in the included studies.(57)

vii. Preparing a structured report of the research
See Reporting syntheses section below.

Systematic reviews of other types of questions utilizing largely quantitative evidence
The basic approach of systematic reviews can be applied to a broad range of research questions although the conceptual and methodological issues (for example optimal search strategies for different study designs, methods for appraising quality of included studies and methods of synthesizing results) have in general not been worked through to the same extent as systematic reviews of effectiveness questions. Fortunately, there is growing guidance covering different types of research questions. Tetzlaff and colleagues highlight these variations in questions and methods for different types of systematic review.(62) The Cochrane Collaboration is developing guidance on systematic reviews of diagnostic performance.(32) The Cochrane Handbook also includes chapters on special topics such as including non randomized trials, considering adverse effects, incorporating economic evidence, etc.(28) The Centre for Reviews and Dissemination provides guidance on systematic reviews of clinical tests, public health interventions, adverse effects and economic evaluations.(29) The Human Genome Epidemiology Network (HuGeNet) provides guidance on systematic reviews of gene-disease association studies.(63;64) However for many topics, synthesis teams will still need to carefully consider how to operationalise systematic reviews to their specific question.

Syntheses of qualitative evidence
Syntheses have also been used to summarise qualitative research evidence. A number of texts provide conceptual and practical guidance for syntheses of qualitative research.(26;65-67) In general these adopt the same steps as systematic reviews of quantitative evidence. However, systematic reviews of qualitative evidence pose considerable conceptual and methodological challenges, particularly relating to identification of relevant studies, appraisal of included studies, and methods of synthesizing evidence.

Identifying relevant qualitative studies
Noyes and colleagues note that qualitative research 'encompasses a range of philosophies, research designs and specific techniques'.(66) However, indexing of qualitative studies in major bibliographic databases remains poor. Fortunately several groups have developed sensitive search strategies for identifying qualitative research studies in major bibliographic databases.(68-71) Sandelowski and Barroso also identify additional approaches for identifying qualitative research studies including footnote chasing, citation searching, journal runs and hands searching, area scanning and author searching.(65)

Nevertheless there remains debate about the relevance of comprehensive literature searches for some forms of qualitative synthesis (especially interpretative syntheses). Petticrew and Roberts cite a conference presentation by Booth who argued that 'searching for qualitative systematic reviews should demonstrate:

• The identification of major schools of thought in an area while being alert to variants, minority views, and dissent;
• Searching a broad range of disciplines to introduce different disciplinary and stakeholder perspectives; and
• The use of both electronic and manual search techniques to ensure that materials are not missed through indexing or coverage inadequacies'.(30;72)

Greenhalgh and Peacock highlighted the challenges of identifying diverse evidence sources in a meta-narrative review drawing on different disciplinary perspectives of diffusion of service-level innovations in healthcare organizations.(73) They considered 495 evidence sources, however only 30% of sources were obtained from the protocol defined at the outset of the study, 51% were identified by "snowballing" (such as pursuing references of references), and 24% by personal knowledge or personal contacts.

Clearly the approaches to searching should be tailored to the review question, the synthesis approach, and available resources. The development and conduct of searches for evidence from qualitative studies (especially interpretative syntheses) will likely need careful collaboration between experienced qualitative researchers, disciplinary content experts, and information scientists.

Appraising qualitative studies
Noyes and colleagues note that 'Assessment of study quality is a particularly contested issue in relation to qualitative evidence synthesis'. They observed that there are over 100 tools for appraising qualitative research but that there 'is insufficient evidence to inform a judgment on the rigor and added value of various approaches'. Instead they argue that the use of such tools should be considered 'as part of the exploration and interpretation' of qualitative studies. Dixon Woods and colleagues note that many of the approaches to appraising qualitative studies do not recognize the diversity of conceptual and methodological approaches inherent in qualitative research and argue for further development of general criteria that would be relevant to all qualitative research and specific criteria for specific qualitative approaches.(74)

Synthesising evidence from qualitative studies
Many innovative approaches are emerging to synthesise evidence from qualitative studies. For example, Noyes and colleagues identify 'Bayesian meta-analysis, critical interpretative synthesis, meta-ethnography, meta-study, meta-summary, narrative synthesis, qualitative evidence synthesis drawing on grounded theory and the approaches developed by the Evidence for Policy and Practice Information (EPPI) Co-ordinating Centre and the Joanna Briggs Institute (JBI).(66) Dixon Woods and colleagues also identify thematic analysis, cross case technique, realist synthesis, content analysis, case survey, qualitative comparative analysis and case survey.(67)

Pope and colleagues identify two broad classes of approaches to synthesis - quantitative and interpretative approaches.(26) This is conceptually similar to the classification proposed by Sandelowski and Barroso - aggregation and interpretative approaches.

Quantitative approaches involve the 'conversion of (whether qualitative or quantitative) data into quantitative form for simple counts and more sophisticated ... analyses' and include content analysis, quantitative case survey, Bayesian approaches and qualitative comparative analysis.(26) Sandelowski and Barroso provide a detailed description of one quantitative approach - qualitative meta-summary 'a quantitatively oriented aggregation of qualitative findings that are themselves topical or thematic summaries or surveys of data'. The aim is to 'discern the frequency of each finding and to find in higher frequency findings the evidence of replication foundational to validity in qualitative research'. They note that meta-summary could be an endpoint of a synthesis of qualitative research or act as an input into a qualitative meta-synthesis. Briefly meta-summary involves extracting and grouping findings into conceptually coherent categories followed by calculation of the frequency and intensity of categories.

Interpretative approaches entail 'a process of qualitative re-interpretation and re-analysis of text based forms of evidence' with an emphasis on 'generating new conceptual understandings and theoretical explanations'.(26) Pope and colleagues highlight different terms used to describe interpretative approaches (including qualitative meta-analysis, meta-study, meta-synthesis) but note the common approach and purpose across these different approaches 'to bring together, juxtapose, re-analyse and combine findings from several studies into a whole that ideally provides some theoretical or conceptual development that moves beyond the findings of any individual study included in the synthesis.' Pope and colleagues identify two conceptually distinct methods focusing either on comparative or translational approaches.(26) Comparative approaches build upon grounded theory and 'constant comparison as an analytical method and the use of theoretical sampling to develop and test theory'. Translational approaches also utilize a comparative approach but also involve the 're-interpretation and transformation of the analytical and theoretical concepts provided by individual studies into one another'.(26) Sandelowski and Barrosi also discuss interpretative synthesis approaches under the umbrella term qualitative meta-synthesis that includes a diverse range of techniques including taxonomic analysis, constant targeted comparison, imported concepts, reciprocal translation and synthesis of in vivo and imported concepts and event timeline consideration.

Mixed methods syntheses
Increasingly syntheses consider both quantitative and qualitative evidence. Within the context of reviews of effectiveness, Noyes and colleagues identified four contributions of synthesizing qualitative research in a quantitative review:(66)
i. Informing reviews by using evidence from qualitative research to help define and refine the review question.
ii. Enhancing reviews by synthesizing evidence from qualitative research identified whilst looking for evidence of effectiveness.
iii. Extending reviews by undertaking a synthesis specifically of evidence from qualitative studies to address questions related to effectiveness.
iv. Supplementing reviews by synthesizing qualitative evidence to address questions on aspects other than effectiveness.

Noyes and colleagues note two broad approaches to mixed methods syntheses.(66) Multilevel synthesis (also commonly referred to as the EPPI approach) involves the conduct of separate qualitative and quantitative syntheses followed by formal combination of the syntheses.(66) For example, Thomas and colleagues undertook a mixed methods review of the barriers and facilitators to healthy eating in children aged 4-10.(75) The synthesis of quantitative evidence from 21 randomized trials observed that 'interventions described in the trials were able to increase children's fruit and vegetable consumption by about half a portion'; however the results were highly variable across studies. The synthesis of qualitative evidence from eight qualitative studies identified several themes relating to children's perceptions of programs to promote healthy eating of fruit and vegetables. They used these themes to undertake exploratory analyses of the quantitative data that suggested that programs that had little focus on health benefits appeared more effective. They also identified that current randomized trials had not targeted the specific beliefs of children about fruit and vegetables that appeared from the qualitative evidence to be barriers to healthy eating. In contrast parallel synthesis involves the conduct of separate qualitative and quantitative syntheses and where the qualitative synthesis is 'used in parallel and juxtaposed alongside (the quantitative synthesis) to aid interpretation'.(66)

Realist syntheses are another mixed method approach to synthesis.(26) Systematic reviews of effectiveness questions typically establish the benefits and harms of policy options but often provide few insights into the mechanisms of action of the policy options or factors critical to the likely success of the policy options. Realist syntheses specifically attempt to provide 'an explanatory analysis aimed at discerning what works for whom, in what circumstances, in what respects and how'.(76;77) Pope, Mays and colleagues argue that realist syntheses focus 'primarily on 'testing' the causal mechanisms of 'theories of change' that underlie a particular type of intervention or programme'.(26) The stages of a realist synthesis include: surfacing the underlying theories about the causal mechanisms of interventions and programmes of interest (these may be explicitly stated in the original research reports or generated by the synthesis team); iterative purposive searches for both quantitative and qualitative evidence to; synthesis of identified evidence to explore programme theory integrity, to adjudicate between rival programme theories, to consider the same theory in comparative setting and to compare official expectations with actual practice.(26;76;77) Realist synthesis can be undertaken alongside systematic reviews (e.g., Kristjannson and colleagues undertook a realist synthesis alongside a Cochrane systematic review of school feeding programmes(78;79)) or as stand alone syntheses. Pawson and colleagues argue that the strengths of realist synthesis include its theoretical perspective, ability to include diverse types of evidence, engagement with stakeholders and real world experiences, and ability to maximize learning across policy, disciplinary and organizational boundaries.(76) However they also recognize potential limitations. In particular realist reviews are highly based on the judgments and perspectives of the synthesis team and are not 'standardizable or reproducible' and that they 'lead, at best, to tentative recommendations'. To date, there are relatively few published realist syntheses (many of which have been undertaken by the developer of the method).

Synthesis methods for broad and diverse bodies of research evidence
The synthesis methods discussed above largely focus on synthesizing the results of primary studies. In this section we briefly describe synthesis approaches to broad and diverse bodies of research evidence³.

Scoping reviews
Scoping reviews 'aim to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available'.(80) CIHR defines scoping reviews as 'exploratory projects that systematically map the literature available on a topic, identifying key concepts, theories, sources of evidence and gaps in the research' and notes 'They are often preliminary to full syntheses, undertaken when feasibility is a concern - either because the potentially relevant literature is thought to be especially vast and diverse (varying by method, theoretical orientation or discipline) or there is a suspicion that not enough literature exists.'(81)

A scoping review might consider both empirical and conceptual research and often focuses on broader questions than those considered in other syntheses, for example the UK Service Delivery and Organisation Research Programme (SDO) had commissioned scoping reviews on continuity of care, health care workforce issues, and E-health.(82)

Arksey and O'Malley identify four common reasons for scoping reviews including:

• Examining the extent, range and nature of research activity (to provide an overview of the available literature and identify key themes and research foci);
• Determining the value of undertaking a full systematic review (for example by identifying the extent of relevant literature and absence of existing relevant reviews);
• Summarizing and disseminating research findings across a body of research evidence;
• Identifying research gaps in the existing literature to aid planning and commissioning of future research (for example, by identifying whether a research question has likely already been answered by existing studies and by refining the research questions and research methods for new studies to ensure that they are informed by existing studies).(83)

In general the stages of a scoping review are similar to those of a systematic review and involve the 'systematic selection, collection and summarization of existing knowledge in a broad thematic area'.(81) However synthesis teams may reduce the scope of searches depending on the breadth of the scoping review and available resources. Likewise, scoping reviews often do not undertake detailed appraisal of identified evidence sources and detailed synthesis of the results from studies. Instead they often collate the identified evidence using some form of 'analytical framework or thematic construction in order to present a narrative account of the existing literature'.(83)

Overviews of reviews
There are an increasing number of syntheses. Often these address discrete research questions that need to be aggregated to provide decision makers (patients, healthcare professionals, managers and policy makers) with an overview of the available evidence for their specific questions. For example, the Cochrane Effective Practice and Organisation of Care group has undertaken two overviews of systematic reviews of healthcare professional behaviour change interventions.(84;85) The Cochrane Collaboration is currently developing the methods of overviews 'to summarize multiple Cochrane Intervention reviews addressing the effects of two or more potential interventions for a single condition or health problem'.(86) Essentially the steps of conducting overviews of reviews are the same as those for conducting systematic reviews of individual studies except that the unit of analysis is a systematic review rather than a primary research report. There are the search filters to identify systematic reviews(40;87) and validated appraisal instruments to assess the quality of systematic reviews.(88;89) In general, narrative synthesis methods have been used in overviews. Perhaps the major challenge faced when conducting an overview of reviews is handling the variability of methods and quality of the included systematic reviews.

Multiple treatments meta-analysis/Network reviews
In general, systematic reviews focus on direct comparisons of the effects of treatments (against a control group or other treatments). However there are usually gaps in the availability of randomized trials (especially head-to-head comparing of two treatments). This becomes problematic when trying to assess the comparative effectiveness of different treatment options for a health care problem (for example, management of blood pressure). Multiple treatments meta-analysis is a relatively new approach than combines both direct and indirect comparisons of treatment effects.(53;90;91) They make 'similar assumptions to standard meta-analyses but require that they hold over the entire set of trials'.(90) There is increasing use of multiple treatments meta-analysis, but again, experience at this time is limited.

Meta-narrative synthesis
Greenhalgh and colleagues developed an innovative interpretive method for synthesizing conceptual and empirical evidence from heterogenous sources for a synthesis on diffusion of innovations in service organisations.(92;93) Recognising the breadth of relevant research from different diverse research traditions and sectors that could contribute to synthesis question, they used 'the unfolding "storyline" of a research tradition over time' as their unit of analysis. They mapped these storylines with iterative and purposeful searches to identify 'seminal theoretical and empirical work' within a tradition. They identified 13 key 'meta-narratives' from diverse disciplines and sectors noting that 'they told very different over-arching stories of the progress of their research'. By examining these storylines individually and then together they were able to explore the 'complexities and ambiguities of diffusion of innovation in organizational settings' and make sense of apparently contradictory data and storylines. This approach has many of the strengths and limitations of realist syntheses outlined above. Again few meta-narrative syntheses have been published to date.

Reporting knowledge syntheses
Knowledge syntheses use rigorous and transparent methods to make sense of research findings. It is important that they are adequately reported to allow the reader to assess the likely relevance and validity of the synthesis for their purpose. Ideally a reader should be able to replicate the methods of the synthesis based upon the published report.
Some organizations undertaking or commissioning syntheses require them to be reported using a standard structure (for example the Cochrane Collaboration).

There are also a number of reporting guides for knowledge syntheses including PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)(Ref) for systematic reviews and meta-analyses of randomized trials (this is an update to the QUOROM reporting guide(94)) and MOOSE for meta-analyses of observational data.(95) These guidelines were developed by a rigorous international consensus process informed by systematic reviews of empirical evidence about potential biases in syntheses. The international EQUATOR network has developed a library of reporting guidelines that synthesis teams can consult.(18;96) Synthesis authors should be aware of these guidelines and try to plan both the conduct and tracking of their synthesis to ensure that they can meet their requirements. For example, QUOROM encourages the use of flow diagrams to represent how studies were identified and selected - this requires synthesis teams to keep a prospective record of the search results and decisions about eligibility of included and excluded studies.(94) A number of journals require synthesis teams to use these guidelines in manuscripts submitted to them.

It is also helpful for synthesis teams to also be aware of current appraisal instruments for syntheses and design their synthesis to meet current criteria for high quality syntheses (for example (88;89)).

Reflections on synthesis methods
What is clear from the above discussion is that we are currently in a period of rapid conceptual and methodological development of synthesis methods for both quantitative and qualitative research⁴. While many of the methodological issues (but certainly not all) for systematic reviews of effectiveness questions have been worked through, this is not the case for other syntheses addressing different questions and types of evidence. As a result, teams undertaking these syntheses often have to consider and acknowledge the limits of our current methodological understanding of how to best conduct syntheses and plan their synthesis accordingly.

4. Developing a funding proposal for a knowledge synthesis

When developing a funding proposal for a knowledge synthesis, it is important to ensure that: the synthesis team has the necessary skills and perspectives to complete the planned synthesis, there is clear justification for the synthesis (and that there is not an existing synthesis that already addresses the proposed question); the proposed methods are clear, justifiable and achievable.

Synthesis team composition
Undertaking a high quality systematic review requires a combination of technical, content and organizational expertise. In addition CIHR requires a knowledge user as a formal partner on funding applications for syntheses.

Technical expertise is required to develop search strategies for major databases, hand search key journals (when appropriate), screen search results, develop data abstraction forms, appraise the quality of primary studies, and statistically pool data (when appropriate). Synthesis teams should likely include information scientists and experienced synthesis researchers preferably with prior experience of the proposed synthesis method (whether quantitative or qualitative).

Content expertise is necessary to ensure that the review question is sensible and addresses the concerns of key stakeholders and to aid in the interpretation of the data. Content expertise may come from consumers, healthcare professionals, and policy makers. Frequently, content experts may not have adequate technical expertise and require additional support during the conduct of reviews.

Organisational expertise is required to oversee the day-to-day management of the synthesis particularly paying attention to team co-ordination, careful documentation of the process, and quality control. Support is required for retrieval of potentially eligible reports, contact with authors and key stakeholders in the field, arranging team meetings, and meetings with knowledge user partner (if separate from the synthesis team) and preparation of synthesis reports.

Knowledge users are decision makers 'who make decisions about, or influence, health policies or practices'; note that 'Decision makers can be practitioners, educators, health care administrators, elected officials and individuals within the media, health charities, patient user groups or the private sector'.(81) There are a variety of ways that knowledge users can interact with synthesis teams as adjunct input into a researcher led exercise, as co-producer throughout the synthesis process or as the prime drivers of the synthesis process.(97)

Justifying the need for the synthesis
Given the profusion of available syntheses, synthesis teams should consider undertaking exploratory searches to ensure that there is not an existing synthesis that already addresses their proposed question. Clear documentation within a grant application that a search failed to identify an existing review or that there is clear justification for a new or updated synthesis despite the existence of a similar synthesis will reassure peer review panels of the need for the proposed synthesis.

Outlining the proposed methods
Preparing a grant application for a knowledge synthesis is, in general, similar to other types of grants. It is important to be detailed in your description of your synthesis methodology. Potential applicants should consult methodological and reporting guidelines for syntheses to ensure that they address key methodological issues.(98;99) It can be helpful to develop and run a preliminary search strategy to demonstrate necessary information scientist expertise in the synthesis team, and identify the likely size of screening task and potential number of included studies to demonstrate feasibility of proposed synthesis. It is also helpful to have a draft data extraction form when relevant again to demonstrate expertise and understanding in the synthesis team.

5. Sources of information and bibliography

International knowledge synthesis organizations
The Cochrane Collaboration
The Cochrane Collaboration is a worldwide not-for-profit organization that aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of all healthcare interventions. The Cochrane Collaboration involves over 22,000 individuals globally who have produced over 3,500 systematic reviews with a further 1,500 ongoing reviews that are published quarterly in The Cochrane Library.⁵ More recently The Cochrane Collaboration has begun to undertake systematic reviews of diagnostic test ordering. The Cochrane Collaboration provides editorial and methodological support to synthesis teams throughout the process of conducting a systematic review. It has produced a range of resources to support the conduct of systematic reviews including The Cochrane Handbook for Systematic Reviews of Interventions,(28) and Revman software available through The Cochrane Collaboration website.

There are six Cochrane review groups, one Cochrane field and three Cochrane Methods Group located in Canada. In addition, Canadians contribute to a wide range of Cochrane groups based internationally. The Canadian Cochrane Network and Centre provides support and training for Cochrane synthesis teams and users of Cochrane reviews. For further information about The Cochrane Collaboration in Canada, contact the Canadian Cochrane Network and Centre.

The Campbell Collaboration
The Campbell Collaboration undertakes systematic reviews of the effects of education, crime and justice and social welfare policies. It provides editorial and methodological support to synthesis teams throughout the process of producing a systematic review. To date, approximately 40 Campbell reviews have been completed and are available free of charge.

The Joanna Briggs Institute
Established in 1996, the Joanna Briggs Institute (JBI) is an international collaboration involving nursing, medical and allied health researchers, clinicians, academics and quality managers across 40 countries in every continent that undertakes systematic reviews addressing a broad range of questions. It has produced around 60 systematic reviews that are available to members. Its aim is to facilitate evidence based health care practice globally through being a leading international organisation for the Translation, Transfer and Utilisation of evidence of the feasibility, appropriateness, meaningfulness and effectiveness of health care practices.

In Canada, the Queen's Joanna Briggs Collaboration based at Queen's University is the first North American JBI Centre.

The Human Genome Epidemiology Collaboration (HuGENet^TM)
HuGENet^TM is an international collaboration committed to the development and dissemination of population-based human genome epidemiologic information. It aims to:

• Establish an information exchange that promotes global collaboration in developing peer-reviewed information on the relationship between human genomic variation and health and on the quality of genetic tests for screening and prevention.
• Provide training and technical assistance to researchers and practitioners interested in assessing the role of human genomic variation on population health and how such information can be used in practice.
• Develop and promote an updated and accessible knowledge base on the World Wide Web for the use of health care providers, researchers, industry, government, and the public for making decisions involving the use of genetic information for disease prevention and health promotion.

The HuGENet^TM HuGE Review Handbook, version 1.0 is available online, and summarized by Sagoo and colleagues.(64)

Evidence Based Practice Centers
The EPCs review all relevant scientific literature on clinical, behavioral, and organization and financing topics to produce evidence reports and technology assessments. These reports are used for informing and developing coverage decisions, quality measures, educational materials and tools, guidelines, and research agendas. The EPCs also conduct research on methodology of systematic reviews.

The Evidence Based Practice Centres produced an open access special edition of the Annals of Internal Medicine on the Methods used in EPC reports (downloadable online).

Centre for Reviews and Dissemination
CRD undertakes systematic reviews evaluating the research evidence on health and public health questions of national and international importance. They have produced guidance for the conduct of systematic reviews relevant to healthcare.(29).

EPPI Centre
The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) is part of the Social Science Research Unit at the Institute of Education, University of London. It undertakes research synthesis and develops review methods in social science and public policy.

Books and monographs

Key references
Centre for Reviews and Dissemination. Systematic Reviews. CRD's guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination, University of York; 2009.

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley-Blackwell; 2008.

Pope C, Mays N, Popay J. Synthesizing Qualitative and Quantitative Health Research. Open University Press; 2007.

Petticrew M, Roberts H. Systematic reviews in social sciences: a practical guide. Wiley Blackwell; 2005.

Further reading
Borenstein M, Hedges L.V., Higgins JPT, Rothstein HR. Introduction to Meta-Analysis (Statistics in Practice). Chichester: Wiley; 2009.

Cooper H, Hedges LV, Valentine JC. The Handbook of Research Synthesis and Meta-Analysis. Sage; 2009.

Egger M, Smith GD, Altman D. Systematic Reviews in Health Care: Meta-Analysis in Context. BMJ Books; 2001.

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003; 7(1):1-76.

Glasziou P, Irwig LM, Bain C, Colditz G. Systematic Reviews in Health Care: A Practical Guide. Cambridge University Press; 2001.

Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D'Amico R et al. Indirect comparisons of competing interventions. Health Technol Assess 2005; 9(26):1-iv

Greenhalgh T, Robert G, Bate P, Macfarlane F, Kyriakidou O. Diffusions of Innovations in Health Services Organisations: a systematic literature review. Chicester: Wiley Blackwell; 2005.

Khan KS, Kunz R, Kleijnen J, Antes G. Systematic Reviews to Support Evidence-Based Medicine: How to Review and Apply Findings of Healthcare Research. London: Royal Society of Medicine; 2003.

Littell JH., Corcoran J, Pillai V. (2008). Systematic Reviews and Meta-analysis. New York: Oxford University Press

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess 2003; 7(41):1-90.

Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess 1999; 3(12):i-98.

Pawson R. Evidence-Based Policy: A Realist Perspective. London: Sage; 2006.

Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Wiley; 2005.

Sandelowski M, Barroso J. Handbook for Synthesizing Qualitative Research. New York: Springer Publishing Company; 2006.

Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000; 4(10):1-115.

Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Methods for Meta-analysis in Medical Research. Wiley; 2000.

Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Systematic reviews of trials and other studies. Health Technol Assess 1998; 2(19):1-276.

Torgerson C. Systematic reviews. Continuum International Publishing Group; 2003.

Wolf FM. Meta-Analysis: Quantitative Methods for Research Synthesis. Sage; 2008.

Special editions of journals and journal series
Helfand M, Morton SC, Guallar E, Mulrow C. Challenges of Summarizing Better Information for Better Health: The Evidence-based Practice Center Experience. Ann Intern Med 142[12(2)], 1033-1126. 2005.

Black N, Mays N. Synthesizing evidence for management and policy-making. J Health Serv Res Policy 10[Supplement 1], S1:1-S1:56. 2005.

Clarke M. The Cochrane Collaboration: Preparing, Maintaining and Promoting The Accessibility of Systematic Reviews of Health Care Interventions. Evaluation & The Health Professions 25[1], 1-139. 2002.

Warren KS, Mosteller F. Doing More Good than Harm: The Evaluation of Health Care Interventions. Ann. N.Y. Acad.Sci. 703, xi-340. 1993.

6. Reference List

(1) Canadian Institutes of Health Research. Knowledge translation. Canadian Institutes of Health Research 2008.
(2) Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005 Jul 13;294(2):218-28.
(3) Ioannidis JP. Evolution and translation of research findings: from bench to where? PLoS Clin Trials 2006;1(7):e36.
(4) Ioannidis JP, Trikalinos TA. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 2005 Jun;58(6):543-9.
(5) Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987 Mar;106(3):485-8.
(6) Mulrow CD. Rationale for systematic reviews. BMJ 1994 Sep 3;309(6954):597-9.
(7) Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992 Jul 8;268(2):240-8.
(8) Fergusson D, Glass KC, Hutton B, Shapiro S. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials 2005;2(3):218-29.
(9) Fergusson DA, Hebert PC, Mazer CD, Fremes S, Macadams C, Murkin JM, et al. A Comparison of Aprotinin and Lysine Analogues in High-Risk Cardiac Surgery. N Engl J Med 2008 May 14.
(10) Lee DS, Nguyen QT, Lapointe N, Austin PC, Ohlsson A, Tu JV, et al. Meta-analysis of the effects of endothelin receptor blockade on survival in experimental heart failure. J Card Fail 2003 Oct;9(5):368-74.
(11) Kalra PR, Moon JC, Coats AJ. Do results of the ENABLE (Endothelin Antagonist Bosentan for Lowering Cardiac Events in Heart Failure) study spell the end for non-selective endothelin antagonism in heart failure? Int J Cardiol 2002 Oct;85(2-3):195-7.
(12) Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995 Feb 1;273(5):408-12.
(13) Hopewell S, Loudon K, Clarke M, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews 2009.
(14) Dickersin K, Min YI. Publication bias: the problem that won't go away. Ann N Y Acad Sci 1993 Dec 31;703:135-46.
(15) DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Arch Otolaryngol Head Neck Surg 2005 Jun;131(6):479-80.
(16) Laine C, Horton R, DeAngelis CD, Drazen JM, Frizelle FA, Godlee F, et al. Clinical trial registration--looking back and moving ahead. N Engl J Med 2007 Jun 28;356(26):2734-6.
(17) Gulmezoglu AM, Pang T, Horton R, Dickersin K. WHO facilitates international collaboration in setting standards for clinical trial registration. Lancet 2005 May 28;365(9474):1829-31.
(18) Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. Lancet 2008 Apr 5;371(9619):1149-50.
(19) Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007 Mar 27;4(3):e78.
(20) Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact of various study designs in the health sciences. JAMA 2005 May 18;293(19):2362-6.
(21) National Institute for Health and Clinical Excellence. The guidelines manual. London: National Institute for Health and Clinical Excellence; 2006.
(22) Agree Collaboration. Appraisal of Guidelines for Research and Evaluation: The AGREE Instrument. 2007
(23) Elwyn G, O'Connor A, Stacey D, Volk R, Edwards A, Coulter A, et al. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ 2006 Aug 26;333(7565):417.
(24) Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy 2005 Jul;10 Suppl 1:6-20.
(25) Chalmers I. Trying to Do More Good than Harm in Policy and Practice: The Role of Rigorous, Transparent, Up-to-Date Evaluations. The Annals of the American Academy of Political and Social Science 2003;589:22-40.
(26) Pope C, Mays N, Popay J. Synthesizing Qualitative and Quantitative Health Research. Open University Press; 2007.
(27) The Cochrane Collaboration. Glossary of Terms in The Cochrane Collaboration. Oxford: The Cochrane Collaboration; 2005.
(28) Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley-Blackwell; 2008.
(29) Centre for Reviews and Dissemination. Systematic Reviews. CRD's guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination, University of York; 2009.
(30) Petticrew M, Roberts H. Systematic reviews in social sciences: a practical guide. Wiley Blackwell; 2005.
(31) O'Connor D, Green S., Higgins JPT. Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 83-94.
(32) Smidt N, Deeks J, Moore T. Guide to the contents of a Cochrane review and protocol for Diagnostic Test Accuracy. Cochrane Handbook for Diagnostic Test Accuracy Reviews. The Cochrane Collaboration; 2008.
(33) Grimshaw J, McAuley LM, Bero LA, Grilli R, Oxman AD, Ramsay C, et al. Systematic reviews of the effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 2003 Aug;12(4):298-303.
(34) Gotzsche PC. Why we need a broad perspective on meta-analysis. It may be crucially important for patients. BMJ 2000 Sep 9;321(7261):585-6.
(35) Vandenbroucke JP. What is the best evidence for determining harms of medical treatment? CMAJ 2006 Feb 28;174(5):645-6.
(36) Loke YK, Price D, Herxheimer A. Adverse effects. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 433-48.
(37) National Library of Medicine. Medline Fact Sheet. National Library of Medicine 2008
(38) Anonymous. Why do you need EMBASE.com if you are using PubMed (the MEDLINE database)? 2008.
(39) Lefebvre C, Manheimer E, Glanville J. Searching for studies. In: HigginsJPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 95-150.
(40) InterTASC Information Specialists' Sub-Group. Search Filter Resource. InterTASC Information Specialists' Sub-Group 2008 August 2
(41) Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An Evidence-Based Practice Guideline for the Peer Review of Electronic Search Strategies (PRESS). J Clin Epidemiol 2008.
(42) Higgins JPT, Deeks JJ. Selecting studies and collecting data. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 151-86.
(43) Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999 Nov 27;354(9193):1896-900.
(44) Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol 2006 Jul;59(7):697-703.
(45) Higgins JPT, Altman DG. Assessing risk of bias in included studies. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 188-242.
(46) Pildal J, Hrobjartsson A, Jorgensen K, Hilden J, Altman D, Gotzsche P. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36:847-57.
(47) Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995 Feb;16(1):62-73.
(48) Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7(27):iii-173.
(49) Deeks JJ, Higgins JPT, Altman DG. Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 243-96.
(50) Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Methods for Meta-analysis in Medical Research. Wiley; 2000.
(51) Borenstein M, Hedges L.V., Higgins JPT, Rothstein HR. Introduction to Meta-Analysis (Statistics in Practice). Chicester: Wiley; 2009.
(52) Cooper H, Hedges L.V. The Handbook of Research Synthesis. Sage; 1994.
(53) Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat Med 2008 Feb 28;27(5):625-50.
(54) Higgins JPT, Deeks JJ, Altman DG. Special topics in statistics. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chicester: John Wiley & Sons; 2008. p. 481-530.
(55) Sterne JAC, Egger M, Moher D. Addressing reporting biases. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 297-334.
(56) Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Wiley; 2005.
(57) Schunemann HJ, Oxman AD, Vist GE, Higgins JPT, Deeks JJ, Glasziou P, et al. Interpreting results and drawing conclusions. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chichester: Wiley-Blackwell; 2008. p. 359-87.
(58) Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, onso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008 Apr 26;336(7650):924-6.
(59) Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is "quality of evidence" and why is it important to clinicians? BMJ 2008 May 3;336(7651):995-8.
(60) Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, et al. Going from evidence to recommendations. BMJ 2008 May 10;336(7652):1049-51.
(61) Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet 2004 Oct 30;364(9445):1615-21.
(62) Tetzlaff J, Tricco AC, Moher D. Knowledge synthesis: State of the art and science. 2009.
(63) Little J, Higgins JPT. The HuGENet HUGE Review Handbook, version 1.0. HuGENet 2006 [cited 2009 Jan 3].
(64) Sagoo GS, Little J, Higgins JP. Systematic Reviews of Genetic Association Studies. PLoS Med 2009 Mar 3;6(3):e28.
(65) Sandelowski M, Barroso J. Handbook for Synthesizing Qualitative Research. New York: Springer Publishing Company; 2006.
(66) Noyes J, Popay J, Pearson A, Hannes K, Booth A. Incorporating evidence from qualitative research. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.Chicester: Wiley-Blackwell; 2008. p. 571-91.
(67) Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy 2005 Jan;10(1):45-53.
(68) Walters LA, Wilczynski NL, Haynes RB. Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res 2006 Jan;16(1):162-8.
(69) Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res 2007 May;17(5):705-10.
(70) Wong SS, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Stud Health Technol Inform 2004;107(Pt 1):311-6.
(71) Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, et al. Finding qualitative research: an evaluation of search strategies. BMC Med Res Methodol 2004 Mar 16;4:5.
(72) Booth A. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research? 2001.
(73) Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005 Nov 5;331(7524):1064-5.
(74) Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. Qual Saf Health Care 2004 Jun;13(3):223-5.
(75) Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic reviews. BMJ 2004 Apr 24;328(7446):1010-2.
(76) Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005 Jul;10 Suppl 1:21-34.
(77) Pawson R. Evidence-Based Policy: A Realist Perspective. London: Sage; 2006.
(78) Greenhalgh T, Kristjansson E, Robinson V. Realist review to understand the efficacy of school feeding programmes. BMJ 2007 Oct 27;335(7625):858-61.
(79) Kristjansson EA, Robinson V, Petticrew M, MacDonald B, Krasevec J, Janzen L, et al. School feeding for improving the physical and psychosocial health of disadvantaged elementary school children. Cochrane Database Syst Rev 2007;(1):CD004676.
(80) Mays N, Roberts E, Popay J. Synthesising research evidence. In: Fulop N, Allen P, Clarke A, Black N, editors. Studying the organisation and delivery of health services: Research methods.London: Routledge; 2001.
(81) Canadian Institutes of Health Research. Funding Opportunity Details - Knowledge Synthesis Grant 2009-2010. 2009.
(82) Anderson S, Allen P, Peckham S, Goodwin N. Asking the right questions: Scoping studies in the commissioning of research on the organisation and delivery of health services. Health Res Policy Syst 2008;6:7.
(83) Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Social Res Method 2005;8:19-31.
(84) Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ 1998 Aug 15;317(7156):465-8.
(85) Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001 Aug;39(8 Suppl 2):II2-45.
(86) Becker LA, Oxman AD. Overviews of reviews. In: Higgins JPT, Green S., editors. Cochrane Handbook for Systematic Reviews of Interventions.John Wiley & Sons: Chichester; 2008. p. 607-31.
(87) Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005 Jan 8;330(7482):68.
(88) Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2007;2(12):e1350.
(89) Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
(90) Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005 Oct 15;331(7521):897-900.
(91) Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med 2002 Aug 30;21(16):2313-24.
(92) Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82(4):581-629.
(93) Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med 2005 Jul;61(2):417-30.
(94) Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the Quality of Reports of Meta-Analyses of Randomised Controlled Trials: The QUOROM Statement. Onkologie 2000 Dec;23(6):597-602.
(95) Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000 Apr 19;283(15):2008-12.
(96) Anonymous. Equator website, 2007.
(97) Lomas J. Commentary: Whose views count in evidence synthesis? And when do they count? Healthcare Policy 2006;1(2):55-7.
(98) Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche P, Ioannidis JP, et al. The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. PLoS Med 2009;6:e1000100.
(99) Moher D, Liberati A, Tetzlaff J, Altman DG, and the PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Med 2009;6:e1000097.

1. The planning and conduct of syntheses usually require a team with relevant content and methodological expertise. In recognition of this, we use the term ?synthesis team? to indicate the authors of reviews.
2. The above discussion does not assume that post protocol deviations are necessarily inappropriate, as there will be many times during the synthesis process when new information or ideas emerge that result in appropriate changes to the methods of the review. However such post protocol deviations should be identified as such and researchers (and readers) should be cautious about over interpreting them.
3. Realist syntheses could also be included in this section as they can be used to explore theoretical and empirical evidence from difference sectors and disciplines.
4. For a discussion on recent advances in meta-analysis see Sutton and Higgins(53)
5. Between April? December 2009 there is a pilot project allowing all Canadians to access the Cochrane Library freely.

Date modified:

2010-04-08