Article Text
PDF
Abstract
Background For the first 5 years of the UK primary care pay for performance scheme, the Quality and Outcomes Framework (QOF), quality indicators were introduced without piloting. However, in 2009, potential new indicators were piloted in a nationally representative sample of practices. This paper describes an in-depth exploration of family physician, nurse and other primary-care practice staff views of the value of piloting with a particular focus on unintended consequences of 13 potential new QOF indicators.
Method Fifty-seven family-practice professionals were interviewed in 24 representative practices across England.
Results Almost all interviewees emphasised the value of piloting in terms of an opportunity to identify unintended consequences of potential QOF indicators in ‘real world’ settings with staff who deliver day-to-day care to patients. Four particular types of unintended consequences were identified: measure fixation, tunnel vision, misinterpretation and potential gaming. ‘Measure fixation,’ an inappropriate attention on isolated aspects of care, appeared to be the key unintended consequence. In particular, if the palliative care indicator had been introduced without piloting, this might have incentivised poorer care in a minority of practices with potential harm to vulnerable patients.
Conclusions It is important to identify concerns and experiences about unintended consequences of indicators at an early stage when there is time to remove or adapt problem indicators. Since the UK government currently spends over £1 billion each year on QOF, the £150 000 spent on each piloting cohort (0.0005% of the total QOF budget) appears to be good value for money.
- Primary care
- quality
- indicators
- primary care
- qualitative research
- quality of care
Statistics from Altmetric.com
Introduction
The use of pay-for-performance incentive schemes, which attach monetary payments dependent on levels of quality achievement for quality indicators, is increasingly common worldwide.1 They are prominent in health maintenance organisations in the USA2 and in the UK as part of the family-practice Quality and Outcomes Framework (QOF).3 The QOF was introduced in 2004 as a pay-for-performance scheme for family practices, composed of clinical and organisational quality indicators.4 QOF is now in its seventh year. There have been consistently high, and equitable, levels of achievement for clinical indicators.5–7 The intended consequences of QOF included improving data capture and care processes, and improved patient outcomes. Some potential unintended consequences of QOF were identified a priori, such as fragmentation of care and reduced quality for non-incentivised conditions.8 However, it is important to be continuously mindful of the possibility of ongoing unintended consequences.
For the first 5 years of QOF, indicators were introduced without any form of piloting. However, in 2009, the process for developing QOF indicators changed9 with all new potential indicators piloted in a nationally representative sample of practices.3
This paper describes an in-depth exploration of family physician, nurse and other primary care practice staff ideas, views, concerns and experiences of piloting QOF with a particular focus on unintended consequences of 13 potential new QOF indicators (see table 1).
October 2009–March 2010 Pilot indicators (n=13)
Methods
Participants
Family practices in England were invited to take part in the QOF pilot. Each cohort was recruited to be nationally representative in terms of practice size, QOF achievement and Index of Multiple Deprivation. The Indices of Multiple Deprivation identify the most deprived areas across the country. They combine a number of indicators, chosen to cover a range of economic, social and housing issues, into a single deprivation score for each small area in England. The Indices are used widely to analyse patterns of deprivation and identify areas that would benefit from special initiatives or programmes, and as a tool to determine eligibility for specific funding streams. Approximately half of those approached agreed to participate. The 28 practices in this cohort were drawn from 12 of 152 Primary Care Trusts in geographically disparate areas of England. The authors are non-clinical (SMC, KLH) and clinical (HEL) health-services researchers with a research interest in the quality of care within family medicine. Semistructured interviews were conducted by SMC and KLH in the participants' own general practice workplace from April to May 2010. The topic guide was developed from the team's a priori questions and focused on the intended and unintended consequences of implementing the indicators.
Data analyses
All interviews were digitally recorded and fully transcribed. The study was explorative, not theoretical, so we used open, not axial, or selective coding.10 Each transcript was read and coded separately by two of the authors (SMC and HEL). Transcripts were initially coded inductively without knowing what we would find when we began the data analysis. Upon completion of coding, the codes that had common elements were merged to form categories, to ensure that the analysis of the interview transcripts reflected the recurring and representative themes found throughout the interview transcripts.11 Validity was increased by actively seeking disconfirming evidence and counting the frequency of ideas and themes.12 Our findings are based on a synthesis of all the interviews in this study. All statements are based on the reports of multiple interviewees. The quotations were chosen on grounds of representativeness and are intended only for illustrative purposes.
The study had full ethical committee approval.
Results
Sixty practices were approached in 2009 for this pilot, and 28 were recruited to be nationally representative (see table 2).
Representativeness of Quality and Outcomes Framework National Institute for Health and Clinical Excellence practices—Pilot 1 (October 2009–March 2010)
Fifty-seven family-practice professionals in 24 practices were interviewed: 21 family doctors, 16 practice managers and 12 nurses. Eight other staff were also interviewed, mostly information technology/computer staff or notes summarisers. Six of the family doctors were female, and 15 were male. Three-quarters of the family doctors worked full time. All nurses were female, and two-thirds worked part time.
In this paper, we report practice staff views of the value and problems of piloting and then focus on the four key unintended consequences of the piloted indicators and, where relevant, of the wider framework.
Views on piloting to identify unintended consequences
While the workload associated with piloting was mentioned by a minority of practices, almost all interviewees emphasised the value of piloting in terms of an opportunity to identify unintended consequences of potential QOF indicators in ‘real world’ settings with staff who deliver day-to-day care to patients. This enabled potential problems arising from implementation to be identified and addressed prior to the indicator being used.For people that are actually dealing with it on a day-to-day basis, I think it's very beneficial to pilot it first. So that you can see the pros and cons of the area that you're piloting and not everybody then has to reinvent the wheel and make the same mistakes. And hopefully once it's out to the vast majority, everything will run smoothly and effectively in an organised manner, for the benefit of the patients. (PM, practice 6)
Implementation: unintended consequences
Measure fixation
The unintended consequence cited most frequently involved the palliative-care indicator, where patients on the palliative-care register were asked about their preferred place to receive end-of-life care. The majority of family doctors (the staff most likely to implement the indicator) felt that addressing this single indicator risked actual patient harm owing to the lack of sensitivity of asking about this issue in isolation and timing issues in relation to when the question should be raised.So I think if this is going to become a tick box exercise it might be that the question will be pushed at an inappropriate time, the wrong moment for the sake of obtaining some points. (GP, practice 6)
One GP described how she had asked this question on four separate occasions in order to gain an answer and ‘tick the box’ without reflecting on the possible distress this approach may have caused the patient.
This indicator created the potential to miss the underlying objective of gold standard care, which is to improve the general end-of-life care of people who were receiving palliative treatment. Feedback about possible unintended consequences and patient harm relating to the piloted palliative care indicator led to the indicator not being recommended for inclusion in QOF.
Tunnel vision
Tunnel vision was referred to by a minority of participants in general terms as a response to the incentivisation of some aspects of care over others. They described how time pressures meant they prioritised the financially incentivised issues over other aspects of care.I think because there is a limited time, and if you have to focus on something in order to get the money, obviously if you don't have to focus on it and you don't have the time, then it's going to be ignored automatically. (GP, practice 27)
However, tunnel vision was referred to more frequently in relation to asthma indicator in the pilot which necessitated ticking a box for each of the three separate questions relating to asthma control in terms of the presence/frequency or absence of symptoms during the day and overnight, and that are activity-limiting. This was usually done as part of a computerised asthma template. However, in some practices, these data were entered as free text with additional contextual information about the patient and/or their symptoms. A significant minority of interviewees (mainly practice nurses) noted that there was a danger that in ticking the boxes, which was all that was required to meet the indicator, this extra contextual information might not be entered.So the more we think we've got to click the more likely we are to miss information. (GP, practice 11)
Importance of context
The pilot practices were deliberately sampled from areas of differing deprivation since deprivation affects both health and healthcare. Differing deprivation of practice areas led to diverse interpretations of the indicators, particularly in the context of the physical health-check indicators for people with serious mental illness. Practices in particularly deprived areas were likely to have more patients with serious mental illness registered than those in less deprived areas. Such practices were far more likely to comment about the difficulty of engaging with patients with psychosis to carry out the annual check-ups.But mental health patients are particularly difficult because they're not the sort of patients you can say come back next week. But that's the way it is. (GP, practice 19)
This meant that such practices might have to work harder to achieve that indicator.
Potential for gaming
Within QOF, family-practice professionals are permitted to use their clinical judgement to remove inappropriate patients from achievement calculations (the denominator) for clinical indicators, a process known as ‘exception reporting.’ Permitted reasons for exception reporting include logistical considerations (eg, recent registration of the patient with the practice), clinical reasons (eg, the presence of a supervening condition or terminal illness) and patient-informed dissent (ie, not agreeing to the investigation or treatment). Recently diagnosed or registered patients are automatically excluded by clinical computing systems, whereas practices must actively identify patients who meet other exclusion criteria.
Exception reporting has been and remains a source of contention in the UK, viewed by some as a gaming mechanism.13 However, we found that a few participants in our study alluded to the fact that exception reporting was used inappropriately in other practices they knew, but only two participants, in different practices, admitted to very occasional inappropriate exception reporting but still maintained that their practices had low exception reporting rates overall.14We try and stick to the rules, I think occasionally people get exception reported for reasons that, perhaps, they shouldn't be but we have very low rates of exception reporting. (GP, practice 12)
Within the context of the pilot, many participants identified indicators for which there might be particularly high exception reporting and therefore the possibility of gaming; for example, in relation to patients who are seldom seen in the practice such as older adults with dementia in nursing homes.We imagine quite a bit of exception coding, some of these are frail and elderly people in nursing homes and it wouldn't be appropriate. (GP, practice 29)
Discussion
QOF was introduced in 2004 in over 8600 practices without piloting. Since that time, concerns have been raised both about the dysfunctional behaviours of family-practice staff arising from the introduction of financial incentives and their likely consequences for patient care15 16 and about predicted unintended consequences such as less attention on non-incentivised areas of care.8 This study, however, suggests that the introduction of piloting within the QOF system provides a valuable opportunity to identify concerns and experiences about unintended consequences of indicators at an early stage when there is time to remove or adapt problem indicators.
Strengths and weaknesses of the study
We are not aware of any previous qualitative studies that address staff views of possible unintended consequences of quality indicators. Interviewees were from a nationally representative sample of practices and were all involved in the day-to-day management of QOF within their practice. However, the clinical basis for professionals' statements was not corroborated with data from patients' medical records, and no patients were interviewed.
Comparisons with previous work
The term ‘unintended consequences’ has been used widely in relation to economic or social actions or government policies.17 The concept echoes with the political economy distinction between ‘seen’ and ‘unseen’ effects.18 19 Smith's framework of the unintended consequences of the publication of performance data in the public sector was particularly helpful in understanding these data and helped shape our thinking during the analysis.20 Smith identifies eight unintended consequences of publishing performance data including measure fixation, tunnel vision, misinterpretation, and gaming. In the context of piloting QOF indicators, we found that the palliative care indicator measured a specific process but missed the larger objective of improving patient centred palliative care. The asthma indicator demonstrated how too much focus on a single issue can lead to poorer care overall with context lost to the possible detriment of patient care. Tunnel vision can also lead to less holistic care. Smith highlighted the importance of context by suggesting that even if data are the perfect representation of reality, differences in the external environment mean that misinterpretation is highly likely. In our pilot, differing levels of deprivation appeared to lead to differing views of the indicators, particularly in the context of people with serious mental illness. Finally, introducing indicators where exception reporting is likely to be high from the start suggests the clinical issue being incentivised may be better encouraged through other quality-improvement mechanisms.
Implications for policy and practice
This study has a number of implications for practitioners and policy makers. Piloting to identify unintended consequences was viewed by all practices as a sensible and much-needed element for developing QOF. Since the UK government currently spends over £1 billion pounds each year on QOF, the £150 000 spent on each piloting cohort (0.0005% of the total QOF budget) appears to be good value for money, even in a time of financial constraints.
This study suggests that both practitioners and policy makers need to be mindful of the unintended consequences that can arise from implementing quality indicators in different contexts, of thinking through and seeking out the consequences of tunnel vision and of being constantly mindful of the possibility of gaming.
Perhaps above all, the presence of ‘measure fixation’ appears to be the key unintended consequence. Inappropriate attention on isolated aspects of care conflicts with the wider agenda of patient centred care.21 In particular, if the palliative care indicator had been introduced, this might have incentivised poorer care in a minority of practices with potential harm to vulnerable patients. Since there are currently over 53 000 people on the palliative care register,22 if only 10% of practices had ‘missed the point,’ over 5000 vulnerable people may have been distressed in an unnecessary manner.
Acknowledgments
The authors wish to thank A Barber (administrator), E Kontopantelis and the staff in all participating pilot practices.
References
Footnotes
Funding This study was supported by a grant from the Department of Health.
Competing interests SMC and HEL are contracted to the National Institute for Health and Clinical Excellence to provide advice on removal of indicators and pilot new indicators for the Quality and Outcomes Framework.
Ethics approval Ethics approval was provided by the North West Liverpool Central Committee 09/H1001/74.
Provenance and peer review Not commissioned; externally peer reviewed.