Section 3.8 : Case Examples

[ Table of Contents ]

Sumit R. Majumdar
Department of Medicine, University of Alberta, Edmonton, AB, Canada

• Presentation of three case examples of rigorous and successful knowledge-to-action studies, that:
  • Address a common and clinically important problem
  • Evaluate well-designed interventions
  • Have adequate sample sizes
  • Use reasonable and robust analytic plans (without any unit-of-analysis errors
  • Deliver valid results
  • Were published in high-impact mainstream general medical journals

Presentation
Each case example consists of:

• A summary of the main features and results of the trial
• A brief discussion of the main strengths and limitations of the work
• Some take-home messages
• A web link to the original article

Case #1
Using Physician-Specific Performance Feedback and Achievable Benchmarks to Improve Quality Of Primary Care

Background

• Patients receive about half the recommended evidence-based primary care for which they are eligible
• Physicians currently receive a great deal of audit and feedback on how they are doing
• Whether such efforts are warranted is poorly understood

Question

• Over and above traditional quality improvement interventions that included physician-specific feedback, would provision of "achievable benchmark" data lead to better patient outcomes?

Intervention

• Most benchmark data provided as mean or median performance achieved or as an attainable value determined by consensus.
• Achievable benchmarks represent the average performance for the top 10% of physicians being evaluated within the same local context.

Study Design

• Cluster randomized controlled trial with blinded ascertainment of outcomes comparing standard quality improvement efforts plus traditional audit and feedback (48 physicians, 965 patients), versus
• Receipt of achievable benchmark data (49 physicians, 966 patients) in one state in the United States.

Outcomes

• Changes in the process of care before and after the intervention among eligible patients with diabetes, specifically changes in:
  • Influenza vaccination
  • Foot examination
  • Laboratory measurements of A1c
  • Cholesterol
  • Triglycerides

Results

• Achievable benchmarks were:
  • 82% for influenza vaccination
  • 86% for foot examination
  • 97% for A1c
  • 99% for cholesterol
  • 98% for triglycerides
• Compared with controls, achievable benchmark data led to an additional:
  • 12% increase in influenza vaccination (p < 0.001)
  • 2% increase in foot examination (p = 0.02)
  • 4% additional increase in A1c measurement (p = 0.02)
  • No improvements in the two lipid measurements

Discussion

• As a proof of concept, this rigorous trial of KT was able to demonstrate that providing individual physicians with their own data and a comparison with what other top performers in the same environment are able to achieve can lead to improvements in the quality of primary care.
• A major strength of the study was that it examined multiple processes of care.
• It should be acknowledged that improvements in two measures were small (2-4%) and that two measures did not improve at all.

Limitations
There are two important limitations to this work:

• Improvements were only related to processes-of-care –
  • whether clinical outcomes improved or whether there were unintended (unmeasured) consequences of this intervention are not known.
• Investigators did not (a priori) define a clinically important or clinically worthwhile improvement (although a 12% increase in influenza vaccination may be clinically worthwhile, is a 2% increase in foot examination also worthwhile?)

Take Home Messages

• Achievable benchmarks may be an important addition to standard audit and feedback.
• Replication studies and a deeper understanding of why this was not more (and not more uniformly) effective are warranted.

Reference: Improving Quality Improvement Using Achievable Benchmarks For Physician Feedback (Journal of the American Medical Association) 

Case #2
Using Clinical Decision Support with Electronic Prompts to Increase Thromboprophylaxis and Decrease Venous Thromboembolism in Hospitalized Patients

Background

• Despite the fact that deep venous thrombosis and pulmonary embolism (DVT/PE) are a common but largely preventable complication of hospitalization, inexpensive and evidence-based thromboprophylactic measures are universally underused.
• Many attempts to increase DVT/PE prophylaxis have been made.

Question
Will an automated clinical decision support system with physician prompts improve quality of care and reduce rates of DVT/PE?

Intervention

• A computerized decision support system using various predefined algorithms to identify high-risk patients based on clinical risk factors determined whether they received some form of thromboprophylaxis and prompted providers in real time to order thromboprophylaxis if it was not ordered.
• The prompts were not "passive" – they were delivered to the physician order entry screen and acknowledged, and then explicit orders to continue withholding prophylaxis or to deliver some form of prophylaxis had to be entered, that is, a forcing function was present.

Study Design

• Quasi-randomized (allocation based on even or odd patient medical record numbers) controlled trial comparing usual care controls (1251 patients) versus computerized decision support with real-time prompts (1255 patients) at a single U.S. hospital.

Outcomes

• Rates of DVT/PE prophylaxis among potentially eligible patients.
• Rates of clinically diagnosed DVT/PE within 90 days of hospitalization (primary study endpoint).

Results

• Baseline results of prophylaxis were about 85%.
• Of all patients deemed eligible for DVT/PE prophylaxis who were not receiving it, 15% of controls versus 34% of intervention patients were appropriately treated (19% improvement, p < 0.001).
• Overall, 8% of control patients versus 5% of intervention patients had clinically diagnosed DVT or PE within 90 days of hospitalization (3% reduction in clinical events, p < 0.001).
• As a measure of safety, rates of death, rehospitalization, and bleeding were similar between study groups.

Discussion

• This study was designed to detect differences in clinically important outcomes rather than restricting examination to processes-of-care.
• For this particular clinical area, this is the first study to demonstrate that improvements in processes-of-care are tightly linked with outcomes, suggesting that the former are a reasonable surrogate measure of quality.
• Although the investigators defined all patients in the study as at sufficient risk to warrant prophylaxis, two-thirds of intervention patients (and 85% of controls) still did not receive guideline-concordant care.
• This result implies that either there is much greater uncertainty in the thromboprophylaxis literature than acknowledged by the investigators or that the decision support tool itself needs more refinement.

Limitations

• The algorithm and scoring system to define "high-risk" had not been previously validated and was not commonly used.
• Contrary to the authors' statements, this was not a randomized trial – it was a quasi-randomized study, and it is possible that outcomes assessors could have broken the allocation code and been influenced in their ascertainment.
• Only one prompt was studied and worked, but at what point will providers start to experience "reminder-fatigue" as numerous well-intended pop-ups prevent them from quickly and efficiently caring for their patients?

Take Home Messages

• One of the few clear demonstrations that a knowledge translation intervention can improve both process ("surrogate") measures of care and lead to important changes in clinical events that directly reflect how patients feel, function, or survive.
• Replication studies with different and multiple concurrent reminders are warranted.
• A better explanation of why this particular intervention (conducted at what is considered one of four U.S. benchmark institutions for the implementation and study of health information technology) was not far more effective than observed is needed.

Reference: Electronic Alerts to Prevent Venous Thromboembolism among Hospitalized Patients (New England Journal  of Medicine)

Case #3
Using a Multifaceted Intervention Directed at Patients and Physicians to Decrease Antibiotic use for Acute Bronchitis

Background

• Almost all cases of acute bronchitis treated on an outpatient basis are caused by viruses.
• Despite the widespread dissemination of evidence-based guidelines, the majority of patients still receive antibiotics leading to adverse events, increased antibiotic resistance in the community, and excess costs.
• Antibiotic use in this setting needs to be safely curtailed, but most attempts have not been able to change practice.

Question
Will interventions directed at patients and/or their physicians reduce the rate of antibiotic use in patients with viral illness such as acute bronchitis?

Interventions

• High-intensity (household and office-based educational materials for patients and education, audit and feedback, and academic detailing for physicians).
• Low-intensity (office-based educational materials only) interventions versus usual care in a group-model health maintenance organization.

Study Design

• Non-randomized before-after study with concurrent controls:
  • High-intensity site (34,978 patients and 28 providers);
  • Low-intensity site (36,404 patients and 31 providers);
  • Two usual care sites (46,767 patients and 34 providers).

Outcomes
Primary end-point:

• Rate of antibiotic prescriptions for acute bronchitis

Secondary end-points:

• Rates of antibiotic prescription for control conditions (other upper respiratory tract infections and acute sinusitis)
• Unintended consequences (use of nonantibiotic treatments and return visits)

Results

• Before the interventions, rates of antibiotic prescription for acute bronchitis were about 80%.
• Over and above changes in practice at the usual care control sites, the high-intensity intervention led to a 24% absolute reduction in antibiotic use (p = 0.003) while the low-intensity intervention led to a 3% reduction (p = 0.68).
• Rates of antibiotic use for control conditions, use of nonantibiotic treatments, and visit rates were similar across all three arms, suggesting that the interventions were safe and did not lead to unintended consequences.

Discussion

• Acknowledging that the study design is valid albeit nonrandomized, the effect size reported is among the largest ever documented for antibiotic reduction in primary care.
• The investigators clearly demonstrated that all study sites were comparable before intervention and that there were no unintended consequences (e.g. the providers diagnosing patients with upper respiratory tract infection or pneumonia and then prescribing antibiotics to "game" the system).
• There did not appear to be any difference in downstream health resource consumption across sites.

Limitations

• Although two intensities of intervention were tested, this was not a factorial trial:
  • The high-intensity intervention was effective, but whether it was the physician component, the patient component, or their combination that mediated study effect is not know.
  • This has resource implications with respect to continuing the intervention or using similar interventions for other conditions where antibiotics may be overused.
• This was a one-off intervention:
  • Whether patients, providers, and the system "learn" and continue to improve or simply lapse back to usual patterns of practice after the study is complete is an important question that cannot be answered from this study.

Take Home Messages

• Multifaceted interventions directed at patients and their physicians can decrease the unnecessary use of antibiotics.
• The methods can probably be extended to other conditions associated with overuse that are sensitive to patient demands.
• Studies that examine the relative importance (and cost-effectiveness) of the components of the intervention are warranted.

Reference: Decreasing Antibiotic Use in Ambulatory Practice (Journal of the American Medical Association)

Future Research
Case examples illustrate three common problems in KT research that need to be addressed:

1. Investigators often test multifaceted (or multiple component) interventions:
   • If intervention works, those wanting to apply the work in their own settings must apply all components of the interventions.
   • KT researchers need to conduct more three-to-four armed trials or formal factorial trials to determine what works and what does not.
   • Given how many interventions do not work and how often secular improvements in quality occur, some form of "usual care" should be considered the most appropriate control group for most trials (so we can know which components are "mandatory" and which are "optional").
   • Quantitative data (e.g., end-user surveys) and qualitative studies can help us better understand how elements of the intervention package work.
2. Investigators rarely provide enough information for others to replicate their work:
   • Replicability is an important facet of the science of KT.
   • Details are often absent or missing because of journal word count limits, etc. One overview of systematic reviews published in ACP Journal Club and Evidence-Based Medicine Journal, Glasziou and Shepperd found that less than 15% of reports have sufficient information about the intervention to allow clinicians or policymakers to implement it.
   • Open-access publication of detailed methods in papers in journals like Implementation Science is now easier than ever, and many journals permit Web links and unfettered electronic appendices with their articles.
3. Investigators rarely describe what would be considered a clinically worthwhile difference to adopt the intervention if it were found to work:
   • As studies get larger it will become easier to detect small and clinically unimportant but statistically significant improvements in quality of care.
   • It is important to define (preferably before the study starts) how much of an improvement is worthwhile (more important) versus how much of an effect is statistically detectable (less important).
   • Should define how much a practice or organization would be willing to pay for a certain amount of improvement.
   • Unfortunately, formal health-economic analyses are rarely undertaken alongside most KT interventions.

Summary

• Each of these cases in successful KT is an important contributor to the literature.
• Each case demonstrates how disparate various clinical problems may be, how complex interventions may need to be, and how difficult implementation and evaluation will be.
• Collectively, these investigators overcame many problems endemic to the field.
• These cases are state-of-the-art examples of rigorous KT research.
• They illustrate how far the field has come over the last 2 decades and also how much more work needs to be done in this relatively young scientific field.

Date modified:

2011-04-13