Quality improvement in New Zealand healthcare Part 5: measurement for monitoring and controlling performance - the quest for external accountability By Rod Perkins and Mary Seddon Summarised for Brilliant New Zealand Ltd by Malcolm Macpherson Abstract This fifth article in the NZMJ Series examined organisational performance indicators, the consequences of their use, and how to do to minimise adverse effects. The Ministry of Health and district health boards use performance indicators to measure activity and to incentivise performance. Two different approaches are used: report cards (comparing performance) and pay-for-performance (financial payments for providers who do what the funder wants). Given the UK and the US experience with both approaches, it is prudent to proceed cautiously. The integrity of the system depends on clinicians taking an interest, and those using them must learn about system variation and how this affects performance. Article In an attempt to arrive at the truth, I have applied everywhere for information but in scarcely any instance have I been able to obtain hospital records fit for any purpose of comparison. If they could be obtained they would enable us to answer many questions. They would show subscribers how their money was being spent, what amount of good was really being done with it or whether the money was not doing mischief rather than good. Florence Nightingale, 1863 A useful distinction can be made between indicators to improve healthcare quality from within (formative clinical indicators) and those used in external performance monitoring (summative performance indicators). This article focusses on summative measures, examining the role of performance indicators and the two main ways that they are used in an attempt to improve quality of care: report card (or league tables) comparing performance across organisations, and the use of financial incentives attached to improved quality performance indicators (pay-for-performance). What are performance indicators and why use them? Performance indicators measure how an organisation is performing, usually in comparison with other like organisations. What is measured varies. It may be financial performance, market share, or productivity. Information from performance indicators can be used in a number of ways - to verify activity, impose a policy agenda, or to stimulate interest in quality improvement or the quality of care. Another purpose, particularly in publicly-funded institutions, is to ensure that money is well spent. Performance indicators can also be used to satisfy the desire to hold someone accountable. Strategies to modify provider behaviour: report cards Both the US and the UK have invested in public reporting of comparative performance indicators. The evidence suggests that patients and the public at large, are strongly in favour of publicly reported performance in principle. However, most studies conclude that they make little use of such reports. Purchasers and funders of healthcare also seem to be in favour in principle but also make little direct use of report cards. The key audience for public reporting appears to be the provider organisations themselves. Public reporting of performance indicators, or their use in league tables, demands very good data. If an organisation's or an individual clinician's reputations are at stake, then indicators should comparing like with like. The enthusiasm for public reporting is well ahead of the science and even the best risk adjustment may not be able to accurately disentangle the key quality differences between organisations from those due to case-mix differences. Organisational performance indicators, like clinical indicators, should be technically sound - derived from data that can be reliably obtained, be valid measures of what they are intended to measure, and focus on an area of importance. The utility of an indicator is only as good as its ability to be measured accurately. As those in management and governance do not have knowledge about what goes on at the sharp end of service delivery, they can (and do) assume that data is being correctly obtained when in fact it often isn't. Data entry may be delegated to ward clerks who, when faced with compulsory fields, may enter nonsense data so that they can complete the process. The validity of a performance indicator may be called into question if there is not a close relationship between what is being measured and the performance of interest. For example, the MOH requires DHBs to report on the rates of readmission to hospital within 28 days. It is unclear to clinicians why 28 days was chosen. If this indicator was measuring the performance of a hospital’s discharge planning, then re-admission within 7 or 14 days would be a more useful indicator. By 28 days, many patients with chronic illnesses have developed a further exacerbation and perfect discharge planning would not prevent their readmission. Those in charge of funding allocation sometimes aggregate performance indicators for report cards into simple metrics such as ticks/crosses or even smiley/frowning faces. There are two major problems with this: (1) Reducing performance data into a series of smiley/frowning faces masks complexity, can hide problems of validity and reliability (2) Variation in the data may be an intrinsic property of the data (common cause variation) and not a reflection of any variation of the quality of care being measured. Reacting to common-cause variation can damage the system of care by wasting people's time and by making the system less stable. Strategies to modify provider behaviour: financial incentives - pay-for-performance The conceptual and technical problems with performance indicators are compounded when they are linked to financial rewards or sanctions, a situation that has been described as an ever expanding collection of carrots and sticks [in] the hope of influencing quality and cost control. The use of financial incentives linked to performance is most evident in the UK, where hospitals with a 3-star rating have until very recently been eligible for a £1 million bonus (rescinded as too many hospitals achieved 3-star status). The performance indicators used included key targets; patient focus; clinical focus; and capacity & capability focus performance indicators. Examples of some key targets: A&E emergency admission waits (<4 hours) All cancers: maximum 2-week wait Breast cancer: 1 month diagnosis to treatment. Breaches in performance indicators were aggressively managed by many National Health Service (NHS) trusts to ensure that their 3-star rating was protected, leading to some perverse behaviour. For example in response to the arbitrary key emergency care performance indicator: no patient should stay more than four hours in the ED, with a target of 98% compliance, some trusts aggressively managed any breaches. Some of the dysfunctional behaviour included: 1 Starting and stopping the clock at different stages in the patient journey Opening a short-stay ward next to the ED and moving patients into this if they threatened to 'breach 2 Moving patients to an inpatient ward—even of their evaluation was not complete 3 When the indicator was measured quarterly, it was not unknown for trusts to hire more staff at the end of the audit period to inflate the percentages 4 Managers cajoling and sometimes bullying staff to meet the target. Such unintended behaviour avoided the breach but did not necessarily address patient's needs or improve the quality of care. The NHS has also introduced a primary care scheme with the introduction of 146 performance indicators which if satisfied, provides approximately 30% of the general practitioner's salary (~£27,000 per GP). The first evaluation of this policy has shown that targets were met for 83% of patients and primary care practices earned nearly 97% of the possible performance points. The initiative contributed to NHS deficits. Some bonuses were achieved by excluding large numbers of patients. Exceptions included factors such as: the patient had just joined the practice, had refused treatment, or despite three attempts had not attended for care. Since 2001, the New Zealand Government has had a national primary care strategy and is now primed to bring in performance measures and financial incentives. The exact level of this funding is still to be determined, but the lessons from the UK experience are important. Our primary care performance measurement programme has identified a number of performance indicators including: 1 Children fully vaccinated by their 2nd birthday. 2 Influenza vaccinations in the elderly (over 65s). 3 Cervical smears recorded in the last 3 years. 4 Breast screening recorded in the last 2 years. In addition, there are performance indicators which are intended to gauge the potential of Primary Healthcare Organisations (PHOs) to operate effectively and improve performance, for example: Percentage of valid National Health Index (NHI) numbers on PHO patient registers and access for high needs enrolees. Possible dysfunctional consequences All performance indicators give rise to perverse incentives and unintended consequences, and these are likely to be exaggerated when there are financial rewards or losses at stake . Both report cards and financial incentives are blunt instruments designed to change provider behaviour - in the hope that the change will be positive for the quality of care. Possible dysfunctional consequences of public reporting of performance indicators. Organisations or individuals may alter their behaviour by: 1 Concentrating on short-term goals (getting their rating higher) and neglecting the long-term strategy 2 Concentrating on those areas being measured to the detriment of other important areas 3 Placing great emphasis and energy on not being exposed as an outlier, rather than on a desire to be outstanding 4 Eschewing innovation for fear of failure 5 Altering their behaviour to gain strategic advantage (gaming) 6 Entering false or corrupt data 7 Avoiding the treatment of high-risk patients if this is going to reflect badly in a public report 8 Disengaging from quality improvement initiatives if the performance indicators do not seem relevant and are externally imposed. Performance indicators focus on one aspect of care - this may encourage organisations to concentrate on just those areas, and things which are not easily measured may be ignored. Some aspects of healthcare quality lend themselves to measurement: waiting times in the emergency department and delays in surgery. Other important activities (accurate diagnosis and proficiency in discussing end-of-life issues with patients) are much harder to measure, and risk being ignored. Furthermore, unlike clinical indicators, performance indicators cannot drill down to provide information on what actions are needed to improve performance. Organisations that gear themselves to do well in report cards, or to increase their financial gains may be focusing on short-term goals, and in doing so, neglect the long term strategic vision and investment. Performance indicators attempt to externally impose quality assurance, however if clinicians do not have confidence in their validity, or if indicators do not align with professional values and assess an important clinical area, they may disengage from the process or worse still, game the results. Gaming, the alteration of behaviour to look good rather than the implementation of substantive improvements, can be damaging not only to the quality improvement effort but also to the involvement of clinicians in quality improvement work. For example, problems exist with accreditation in New Zealand, where clinicians know that there are deep and serious problems with care provision, yet their organisation gets accredited without these problems being exposed. This leads to cynicism about the value of being involved in the process and to the questioning of the managerial drivers. An even more important perverse behaviour is that high risk patients are avoided, if such high-risk patients are going to make the figures look bad in a league table, there is a danger that there will be pressure to only operate on low risk cases, or concentrate on the easy to reach patients. Even when performance indicators are good measures, they may in fact threaten the trust necessary for clinicians to engage in quality improvement work. By elevating the status of external inspection they decrease that of the internal informal work that most organisations have. Indeed, it has been said that the indicator industry has begun to suffer from the regulators’ delusion that central systems of oversight are the sole guarantors of quality. What is the evidence that performance indicators improve quality of care? Despite the enthusiasm for performance indicators and the millions of dollars spent on their development, relatively little is known about the actual impact on improving quality of care. Performance indicators cannot capture the range and complexity of health service activity and are blunt and dangerous tools when used in pursuit of quality, if they have any impact at all. Healthcare is complex and less deterministic than traditional industries. The link between actions and outcomes in not necessarily strong or direct, and it is modified by non-healthcare factors (such as employment, deprivation) and patient-mix. There is little evidence of a positive impact from performance indicators on health service delivery or health outcomes. Even the oft-quoted example, the New York State league table published in a New York newspaper, where individual cardiac surgeons were ranked by the mortality rates of their coronary artery bypass patients, has been questioned. Yes, the mortality rates improved, but they also improved in other states that did not have public reporting. New England achieved similar benefits in mortality rates through confidential reporting and sharing best practices. Furthermore in New York there was evidence that low volume surgeons stopped operating but also that high-risk cases went elsewhere. The blunt instrument that is public reporting resulted in unintended consequences. What can New Zealand learn from all this? 1 be cautious about performance management systems (maybe de-politicise public reporting. Both the US and UK have agencies that have some political independence) 2 to get clinical buy-in, and command credibility, performance indicators need to address clinically important areas, for both clinicians and patients 3 focus on the technical issues of performance indicators, validity, reliability, and case-mix 4 anticipate and monitor unintended consequences, ensure that performance indicators relate to processes of care over which clinicians have control, and that are not unduly affected by non-healthcare factors 5 there is an opportunity cost involved in developing performance indicators as well as collecting and analysing data, while seeking to risk adjust so that comparisons are useful. This is money that is not being spent on actually improving quality of care 6 financial incentives need to be large enough to influence performance, but not so large that they encourage distortions of the clinical processes and tenets of professionalism. Summary Performance indicators, used in report cards or linked to financial incentives, are on the agenda. If the NHS in the UK is any guide, expect to see increasing surveillance of performance and quality. There are pitfalls. To be successful, they require funders and managers educated in the subtleties of performance management and indicator use, as well as clinicians who can challenge the indicator set. Indicators of health care quality are not axiomatically good. Two things can be guaranteed; performance indicator use by those funding health systems is on the increase, and indicator use is far from an exact science.