Section 6.1: Methodologies to Evaluate the Effectiveness of Knowledge Translation Interventions

[ Table of Contents ]

Onil Bhattacharyya, MD, PhD
Elizabeth Estey, MA
Li Ka Shing Knowledge Institute of St. Michael's Hospital, University of Toronto

Merrick Zwarenstein, MBBS, MSc
Sunnybrook Research Institute, University of Toronto

Overarching Framework: The Knowledge to Action Cycle

The Knowledge to Action Cycle
From: Graham et al. Lost in Knowledge Translation: Time for a Map?

Topic Focus: Evaluate Outcomes

  • Context & rationale: the need for evaluation
  • Evaluation study designs
    • Randomized
    • Non-randomized
  • Pragmatic study designs
  • Successes and failures
  • Conclusion
The Knowledge to Action Cycle: Evaluate Outcomes

Context

Challenges of implementation research

  • KT promotes evidence-based medicine (EBM), but methods used to promote EBM are not evidence based
  • Pressure to improve quality of care, but dearth of information on which interventions work
  • 350,000 RCTs in clinical medicine vs. 2,400 experimental trials of interventions to improve health care delivery

Shifting Focus...

  • From developing new treatments to developing approaches to deliver what is already known to work
  • To create and evaluate interventions from evidence-based knowledge

The Need for Evaluation

Evaluation of quality improvement (QI) initiatives is important to help:

  • Determine the effectiveness of their efforts
  • Reduce wasted resources
  • Create knowledge that may benefit others

Evaluation Study Designs

Local vs. Generalizable knowledge

  • Local = managers responsible for QI in an institution
  • Generalizable = knowledge translation researchers studying QI in general

Internal Validity

Defn: relationship between intervention and impact has been accurately measured

Purpose of evaluation is to determine if:

  • There has been an improvement in the outcome of interest
  • This improvement is due to the intervention under study

When an intervention appears to be effective... but is not?

Example: The common cold

A treatment for the common cold may appear to work because a person is cured a few days after taking it. However, the clinical improvement may be due to the effect of the treatment or the natural course of a self-limited disease that lasts a few days.

Study Designs

  1. Randomized → gold standard
    • Randomized controlled trial (RCT)
  2. Non-randomized or quasi-experimental
    • Controlled before-after
    • Interrupted time series
    • Uncontrolled before-after

Randomized Controlled Trials

  • Large sample size enables accurate assessment of intervention effect
  • Increases the chance that groups will have similar distribution of known and unknown confounders

RCT Designs

Number of comparison arms:

Two arm trials most common

Diagram: Two arm Trials

Units of randomization:

Diagram: Units of randomization

Sample size:

  • Large sample size increases ability to determine that there was no impact
  • Important when effect size is small; clustering requires further adjustment

Non-Randomized Designs

  • More subject to bias
  • Require fewer resources
  • Logistics simpler

Types:

  1. Controlled before-after
  2. Interrupted time series

Also:

  1. Uncontrolled before-after

Controlled Before-After Studies

Figure: Controlled Before-After Studies

Interrupted Time Series

Figure: Interrupted Time Series

Generalizability

  • Internal validity = rigorous design, sufficient sample size, blinding of assessors and participants (where possible) to group allocation
  • Perfectly valid study may not allow us to determine the degree to which results are applicable to regular practice conditions
  • Pragmatic trials designed to maximize the relevance of the results for real world decision making

Pragmatic Study Designs

Pragmatic vs. Explanatory trials

  • Pragmatic = designed to help choose options of care
  • Explanatory = designed to test causal research hypotheses
  Explanatory Pragmatic
Purpose To examine efficacy To examine effectiveness
Setting "Ideal" conditions; environment monitored Normal practice
Participant selection Careful selection process and monitoring Clinical indication
Interventions Strict enforcement and monitoring of adherence Flexible application; suited to normal practice
Outcomes Short term surrogates or process measures Outcomes with relevance to participants, funders, healthcare providers, decision makers, and other stakeholders
Relevance to practice Indirect – little effort made to match trial design to needs of decision makers Direct – efforts to link study design to everyday practice

Successes and Failures

  • Randomized and non-randomized studies help us understand the "what", but not the "why"
  • Qualitative studies can fill this gap by answering the "why" questions
  • Despite a significant number of studies investigating KT interventions, we still know very little about what works and what doesn't
    • Rigorous evaluation of quality improvement initiatives needed to increase our knowledge of KT and to improve quality of care

Conclusion

  • Implementation is inherently complex
  • Despite large number of studies, many knowledge gaps remain
  • The choice of evaluation design depends on what you want to know
    • What works in your setting or what works in most settings
    • Consider rigour in study design and pragmatic approaches
  • Using qualitative and quantitative studies help understand if something works and why
  • Given cost of implementation, evaluation is an imperative and need not be difficult

For more information, contact us:
Onil Bhattacharyya: bhattacharyyao@smh.toronto.on.ca
Elizabeth Estey: esteye@smh.toronto.on.ca

Date modified: