Quality improvement in New Zealand healthcare Part 4: achieving effective care through clinical indicators By John Buchanan, Allan Pelkowitz and Mary Seddon Summarised for Brilliant New Zealand Ltd by Malcolm Macpherson Abstract Clinical indicators can improve the effectiveness of patient care. This fourth article in the NZMJ Series identifies key attributes of clinical indicators and provides a scheme for their appraisal. Clinical indicators are objective measures of the process or outcome of patient care. They can be used to monitor care, to flag potential opportunities to improve care, and to provide evidence that a change in practice has resulted in improvement. Clinical involvement helps to ensure that indicators are used in a formative way with a focus on quality improvement, rather than as a summative mechanism for top-down external accountability which attempts to assure quality. In some cases, such external quality assurance can actually harm quality improvement efforts. Article Performance indicators, key performance indicators (KPIs), and clinical indicators - what are they, how do they differ, and how are they used? Performance indicators were placed firmly on the healthcare agenda in 1986 when the Joint Commission on the Accreditation of Health Care Organisations in the United States launched an Agenda for Change to modernise accreditation. Performance data incorporated into accreditation was to be used to satisfy the demand by the payers of healthcare, for objective evidence on the quality of that care. At the same time, healthcare organisations were progressively embracing the concept of continuous quality improvement and exploring the role of performance indicators in the quest to improve the effectiveness of care. Data generated through the use of reliable and valid performance measures were recognised as central to the CQI process. Thus from the outset there have been two principal uses of performance indicators: As summative mechanisms for external accountability and seeking assurance of the quality of heath care, and As formative mechanisms for internal quality improvement. The distinction is very important because the use of performance indicators in assurance and performance management systems - summative indicators - has the potential to undermine the conditions required for continuous quality improvement in the clinical setting. Summative performance indicators (e.g. accreditation, Pay for Performance) may increase compliance costs, meaning that there is less money available for CQI. If they are used to punish behaviour they may also drive down innovation and trust, leading to gaming of data. Clinical indicators are a subset of performance indicators. They are an objective measure of either the process or outcome of patient care, usually rate based with a numerator and denominator, both of which must be clearly defined. They do not measure quality directly, but flag potential problems and possible opportunity to improve care. The benefits to be gained from the use of clinical indicators do not lie in the collection of the data, but in how those data are used; that is, in the data analysis and the actions taken to achieve sustained improvements in clinical practice. Clinical indicators do not work unless used effectively by clinicians and managers to bring about improvements. There are different objectives for clinical indicators, depending on who is using the indicators and whether the assessment is intended to be summative or formative. They can be used by the manager to control clinical behaviour, usually with the aim of decreasing costs. They can also be used as international benchmarks, and as a means to direct funding. Clinicians use clinical indicators to improve patient care, by measuring an aspect of care over time, as flags to problems and to identify areas for improvement. Clinical indicators can also be used to provide evidence that changes have resulted in improvements in care. Clinical involvement helps to ensure that indicators are used as a formative mechanism for quality improvement in patient care, rather than as summative mechanisms for external accountability with a focus on assurance rather than improvement. Most of New Zealand's clinical indicators are from the Australian Council on Healthcare Standards indicator sets, developed with Australian and New Zealand Medical Colleges, Associations, and Societies. The aims of the ACHS indicator program are laudable (to increase the involvement of clinicians in evaluation and quality improvement activities, and to facilitate the collection of national data on the processes and outcomes of patient care), but there are problems with reliance on the ACHS indicator set: 1 they are mostly not evidence-based, and do not adequately represent subspecialties 2 external clinical indicators remove clinical ownership 3 benchmarking against a standard can encourage complacency once the benchmark is reached. How to choose an indicator? A brief understanding of Donabedian's model for quality improvement will help guide decisions: Structure: Available resources and policies (e.g. staffing ratios, availability of diagnostic equipment) Process: The interaction between clinicians and patients (e.g. diagnostic tests, management, and treatments) Outcome: The result for the patient. (e.g. survival rates, years of healthy life lost, disability, pain) Outcomes are of prime interest, but there are problems with measuring these directly: it may take too long to observe outcomes (therefore need high volumes and/or early endpoints); they can be confounded by problems outside the healthcare sphere of influence (e.g. poor housing, poor incomes); and they are expensive to collect. There is a consensus that measuring process indicators is preferable if there is good evidence that the process being measured is related to outcomes of interest. For example, there is evidence that giving aspirin and beta-blockers (process measures) to patients suffering an acute myocardial infarction improves their survival (the outcome of interest). Key attributes of clinical indicators Attribute Definable Description Can the indicator be clearly defined? (numerator and denominator) Attribute Clear Intent Description Is the intent of the indicator easily understood and interpretable by all users? Attribute Valid Description Does the indicator measure what is intended and point to issues of quality? Attribute Reliable Description Is there demonstrated reliability (reproducibility) of data? Reliability will largely depend on standardised definitions and rigour of data collection mechanisms. It depends on the precision of the definition (numerator/denominator) and the accuracy of, or variation in, data reporting. Clinical indicators may be valid before being reliable, reliability can be improved Attribute Accessible Description Are data easily accessible? Are data routinely collected or will the data have to be extracted from the patient record. Who will do this and how much will it cost? Attribute Useful/utility Description Does the indicator provide useful information to inform quality programs and stakeholders? Attribute Practical benefit Description Does the indicator have a strong cost/utility ratio? Attribute Responsive Description Is the indicator responsive with a potential for action and quality improvement? Attribute Relevance Description Does the indicator measure aspects of care which are relevant and significant? For clinical indicators to be useful in improving patient care (and therefore outcomes) it is important to identify which clinical indicators are likely to be useful. In this way, only those indicators that the clinical team identifies with are chosen, and there is likely to be better ownership and association with improvement efforts. Appraisal of clinical indicators Level 1: Is this indicator technically correct? Are we measuring what we are setting out to measure? Example: Cervical screening: The denominator should exclude all women who do not qualify for screening (e.g. those who have had hysterectomy with removal of cervix) Level 2: Is there evidence to support this indicator? How strong is the evidence that the process leads to the desired outcome? Example: Reduced length of stay may indicate better performance but may also indicate that patients are discharged too quickly. This indicator needs to be backed up by monitoring unexpected readmissions to hospital. Level 3: Is this the correct indicator? Why pick this one? Will it help the patient? What is the purpose and for whom? Example: Influenza vaccination rates are used at international level by governments as a comparator. This may lead to distortion in the expectation on clinicians and managers compared to another possibly more clinically important indicator that is not reported in the political sphere. Level 4: Consequences and opportunity cost (the cost in all aspects of giving up the next best choice when one decides on a certain course of action). What will stop happening when we focus on this indicator? Are we flexible to respond to unexpected outcomes of using this indicator, both positive or negative? Is the system prepared for the changes in workflow that might result from this indicator—e.g. increased procedures, tests, etc. Example: Funding to increase retinal screening in New Zealand did not include funding for treatment of diabetic eye disease that was discovered as a result of that programme. Level 5: Funding. Are payment mechanisms and incentives aligned across the system? Is this financially viable? (cost/benefit across the system). Is implementation easy? Is information readily available and accessible Level 6: Is this important? Are there some areas where the indicator system simply does not work? How does one measure care where the endpoint is not recovery? Example: Although domestic violence has a high societal and health cost, it is not easy to determine indicators that quickly point to improved outcomes. This does not mean that domestic violence should be ignored (indeed, one should strongly guard against this in an environment that demands data) but it should be recognized that it can never compete with cardiovascular disease for a suite of indicators The analysis and interpretation of indicator data by clinicians (who are familiar with the clinical process) is important for quality improvement. Clinical indicators generate data, but data needs to be analysed and presented as useable information if it is going to be used to improve care. Clinicians need to understand the basic principles and limitations of data analysis and presentation to use the information appropriately. Clinical indicator data is collected over time, and the most effective presentation is either through a run chart or a control chart. Both use a set of statistical rules to determine whether the pattern revealed by the data represents the normal fluctuations about a median that is observed in any process (common-cause variation) or whether there is something that needs further investigation (special cause variation). It is important to use these rules to avoid the common problem of seeing trends where none exist, or of over-reacting to common-cause variation (and thereby making the system of care more unstable). Summary Clinical indicators can be a powerful means of effecting change if used correctly. It is important to understand who has defined the indicators and for what purpose. It is also vital that the indicators are adequately assessed in terms of the changes they will make on the whole system. Clinicians and managers will be surprised when the unexpected occurs, and should be in a position to react appropriately - a lot easier with access to accurate and timely data.