Section 3.5.4: Audit and Feedback Interventions

[ Table of Contents ]

Robbie Foy, University of Leeds
Martin Eccles, Newcastle University

Overview

  • Chart audit
    • Rationale and what it is
    • Types of audit criteria
    • Target setting
    • Practical issues
  • Audit and feedback
    • Rationale and what it is
    • Theory
    • Evidence
  • Summary and future research directions

Key learning points

  • Measuring adherence to clinical practice recommendations can highlight important implementation gaps and inform subsequent priorities for knowledge transfer
  • Audit and feedback can be effective in improving professional practice although the effects on clinical practice are generally small to moderate
  • More research is needed on the effects of audit and feedback compared to other interventions, and the mechanisms by and contexts in which it works best

Chart audit

  • Rationale
    • Recognised gaps and delays in the implementation of evidence based practice
    • Can be identified or confirmed by chart audits
  • What it is
    • Documented clinical care is measured against a review criterion: "a systematically developed statement that can be used to assess the appropriateness of specific healthcare decisions, services and outcomes."

Field MJ, Lohr KN. Guidelines for clinical practice. From development to use. Institute of Medicine. Washington: National Academy Press, 1992

Types of review criteria

  • Explicit criteria
    • Aim to maximise the reliability and objectivity of measurement
    • E.g. patients newly diagnosed with essential hypertension who either have persistent high blood pressure (BP) of 160/100 mmHg or more, or are at raised cardiovascular risk with persistent BP of more than 140/90 mmHg have a record of being offered drug therapy
  • Implicit criteria
    • Involve peer or expert clinicians making judgements about the desired care
    • Tend to be more subjective and less reliable than explicit criteria
    • Mainly used to assess complex processes of care or adverse outcomes, e.g. maternal deaths related to childbirth

Naylor CD, Guyatt GH, for the Evidence-based Medicine Working Group. How to use an Article About a Clinical Utilization Review. JAMA 1996;27518:1435-9

  • Structure of healthcare delivery
    • E.g. the presence of calibrated devices for measuring blood pressure
  • Healthcare processes
    • E.g. the prescription of anti-hypertensive medication
    • Structural and process criteria must be valid, i.e. strong evidence exists that their improvement is associated with improvement in outcomes of care
  • Patient outcomes
    • Short term or surrogate outcomes, e.g. blood pressure levels
    • Long term outcomes, e.g. stroke
    • Tend to be less sensitive at detecting changes in practice because many factors may influence patient outcomes
    • Generally require more resources, larger sample sizes, and longer follow up to detect important changes

Target setting

  • Target levels of performance can be set to guide subsequent decisions on whether implementation activities are worthwhile
  • Attempts to improve already high levels of performance may not be as productive as switching attention to alternative priorities
  • Why not to expect 100% adherence
    • For many clinical actions, there is a "ceiling" beyond which healthcare systems' and clinicians' abilities to improve performance are limited
    • E.g. if eligible patients prefer to avoid drug treatment or experience unacceptable side effects

Practical issues

  • Planning and conducting reviews
    • Sampling procedures
    • Sample size
    • How to collect the data
  • Under-documentation
    • Not all clinical actions are documented…
    • But good documentation has become integral to good medical practice

Audit and feedback

  • Rationale
    • Demonstrating the gap between actual and desired performance will motivate clinicians or healthcare systems to take action addressing that gap
  • What it is
    • "Any summary of clinical performance of health care over a specified period of time" given in a written, electronic or verbal format

Jamtvedt G, Young JM, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2006(2)

Theory

  • "Self-regulation"
    • A process of determining goals and then using these as reference values to bring existing states into line with those goals
    • The success of any desired change also depends upon individuals being able to change their behaviour (e.g. clinical practice skills) or external influences on behaviour (e.g. organisational factors)

Carver CS, Scheier M. Attention and self-regulation : a control-theory approach to human behavior. New York: Springer-Verlag, 1981

Evidence

  • Cochrane Review of 118 randomised trials
    • "Audit and feedback can be effective in improving professional practice and that the effects on clinical practice are generally small to moderate."
    • E.g. when percentage adherence with desired practice was measured, the effects ranged from a 16% decrease to a 70% increase, with a median improvement of 5%

Jamtvedt G, Young JM, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2006(2)

Why this variation in effect size?

  • Type of format, i.e. verbal, paper or electronic;
  • Frequency and duration, e.g. as a one-off step or continuously and often over a period of time
  • Source, e.g. whether from a supervisor or professional body
  • Content, e.g. information on healthcare processes or patient outcomes, use of identifiers to permit comparisons between individual professionals, teams or facilities
  • Use of various sources to deliver feedback, such as supervisors or professional bodies.

Uncertainties

  • Higher intensity of feedback
    • Associated with greater effects but …
    • Higher costs of intensive strategies may outweigh benefits
  • Combining audit and feedback with other strategies
    • E.g. educational meetings
    • Uncertainty as to whether increases effectiveness
  • Lack of head to head comparisons
    • Of different methods of providing feedback and of comparisons of audit and feedback versus other interventions
    • Therefore difficult to recommend the use of one intervention strategy over another on empirical grounds

The importance of context

  • Contextual factors
    • Relative effects of audit and feedback greater when baseline adherence with recommended practice is low.
  • Nature of targeted behaviours
    • Clinicians' motivation to change practice and their level of engagement with the feedback intervention (whether they are active or passive recipients but…
    • Effects of audit and feedback are greater for recommendations perceived by clinicians to be less compatible with current norms and for tasks associated with lower motivation

Foy R, MacLennan G, Grimshaw J, et al. Attributes of clinical recommendations that influence change in practice following audit and feedback. J Clin Epidemiol 2002;55:717-722

Palmer RH, Louis TA, Hsu LN et al. A randomized controlled trial of quality assurance in sixteen ambulatory care practices. Med Care 1985;23:751-70

Summary

  • Only limited insights into how and when audit and feedback can be made to work more effectively
  • Its selection as a KT intervention remains a matter of judgement based upon
    • Current evidence base
    • The working 'diagnosis' of the causes of an implementation gap
    • Availability of supporting resources and skills
  • In principle, getting the diagnosis right offers a rational basis for choosing an approach to delivering feedback
    • E.g. if perceived peer pressure was identified as a key determinant of clinicians' practice or motivation to change for a given context, feedback could reasonably incorporate peer comparison

Research questions

  1. By what mechanism or mechanisms does audit and feedback exerts its effects?
  2. Which contextual features (e.g. setting, characteristics of healthcare professionals) and attributes of targeted clinical behaviours negate or enhance the effects of audit and feedback?
  3. How does audit and feedback, by itself or in combination with other intervention, compare against other interventions to change clinical behaviour?

Foy R, Eccles M, Jamtvedt G, et al. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res 2005;5:50

Date modified: