A few years ago I worked as a junior doctor in a medium-sized district general hospital. It was at the time of the Mid Staffs scandal, which revealed inadequate health care and high death rates at Stafford Hospital. I was working in busy, often understaffed, sometimes chaotic, general medical wards, and I wondered whether our trust might be another Mid Staffs in the making. The answer to this question, a more senior colleague informed me, was actually quite difficult to ascertain. This seemed strange: all deaths in the United Kingdom are certified, as is their location; mortality rates therefore appeared to be a particularly robust health metric.
However, as Nigel Hawkes pointed out in a report for the BMJ, in the same year that it was initially investigated by the Care Quality Commission, Mid Staffs had been rated by Dr. Foster, an independent health analytical company, as the ninth safest hospital in England. How was it possible for two analyses to give such different answers? And, following on from this, why was my senior colleague unable to tell me how safe my hospital was relative to other ones in the country?
The answer to these question lies in part in how hospitals report their outcomes in the form of healthcare metrics. At Mid Staffs, for example, the SMR (Standardised Mortality Ratio) after a fractured neck of femur (broken hip) was reported in 2009 as 19.87, compared to a national average of 100. Questioning whether someone could really be five times less likely to die if they broke their hip in Staffordshire compared to the average elsewhere in the country, Nigel Hawkes wrote to the trust asking them to verify this figure. The trust replied stating: ‘We have not always had such a low SMR [standardised mortality ratio] for fractured neck of femur. Our Clinical Coding department advise that the change is due to substantially improved coding procedures.’
Every trust in the United Kingdom has a Clinical Coding department, which assigns ICD (International Classification of Disease) diagnoses to all patients admitted to hospital. The mortality rates provided to the government are then standardised by the degree of co-morbidity that patients had when they died: currently this metric is called the Summary Hospital-level Mortality Indicator. That is, the death of a previously well 50-year-old with no associated health conditions counts for more that than of a 95-year-old with associate heart failure, renal impairment, prostate cancer and so forth. This seems reasonable: it means that trusts are not penalised for working in populations were the average age is higher, for example, or where the average patient is sicker.
However, it also means that it can be hard to interpret metrics from hospitals as a marker of how good the care provided is. Hawkes points out that the crude death rates in England for 2004-05 were 225,439 compared to 222,738 in 2008-09, yet figures for deaths rates adjusted by severity provided by CHKS, another healthcare analytics company, showed a decrease of 50 per cent. Tellingly, he points out that the response of Basildon and Thurrock Healthcare Trust, who was shown to have the highest HSMR in the country in 2009, was that it ‘needed to employ more [clinical] coders’.
Hence why my senior colleague was unable to tell me how safe our wards were compared to other ones round the countries. And, as a doctor, I continue to hear whisperings of pressure to meet government targets leading to gaming of the system – inappropriate last minute movements of patients to avoid Accident and Emergency breaches; manipulation of patients on waiting lists; or wide regional variation in classifications of patients receiving palliative care codes (these deaths do not contribute to adjusted mortality rates).
Image Credit: (Rosmarie Voegtli CC BY 2.0)
Metrics: What Counts in Global Health is a volume of papers by anthropologists looking mainly at the Global South. They are similarly preoccupied by how an emphasis on health metrics is distorting the way that healthcare is funded and delivered. However, the collection is not just a critique of metrics in themselves, but a more wide-ranging exploration of how a small number of big-spending non-governmental organisations are increasingly shaping the health services provided to citizens of the Global South.
So what exactly do they think is wrong with too many metrics? As Vincanne Adams points out in the Introduction, one issue – analogous to the situation in the United Kingdom – is whether the metrics in themselves are as reliable and objective as they seem. Bill Gates in his 2013 annual letter stated ‘how important measurement is to improving the human condition’, and in recent years has invested heavily in an organisation based in Seattle called the Institute of Health Metrics and Evaluation (IHME). Headed by the physician and epidemiologist Chris Murray, the IHME produce metrics that they call the ‘Global Burden of Disease’: in their own words, ‘a systematic effort to quantify the comparative magnitude of health loss due to diseases, injuries, and risk factors by age, sex, and geography over time’. However, I know from first-hand experience that such a large-scale effort to quantify disease and death at a global level relies both on extrapolating from small (and often unreliable) data sources, and a certain degree of complex statistical arithmetic.
In Chapter Two of Metrics, Claire Wendland looks at maternal mortality rates and how complicated the formulae used to calculate and standardise these rates are. She teases apart the model used by the World Health Organisation to calculate them, pointing out that out of six of the seven variables inputted, only one is the actual estimate of maternal deaths. As an example of how these factors can influence metrics, she gives Equatorial Guinea, which despite having no national measure or count in maternal deaths for twenty years from 1990 to 2010, showed a dramatic decline in maternal mortality rates attributable to a rise in GDP as a result of rising oil prices (72). In addition to these methodological issues, Wendland also examines how political pressure in Malawi may be leading to an under-reporting of maternal death rates, where local chiefs have started to impose large fines for anyone reporting such a death (74).
Similarly, as outlined in Chapter Three by Adeola Oni-Orisan, in Nigeria health funding is shown to be contingent on positive metrics: there is pressure to manipulate numbers and construct clinical care in a manner that is not necessarily in the best interests of patients. The death of a mother after the birth of her fourth child en route from a police clinic to the local hospital is not included in either the official health facility record or state government reports (83). Another woman, whose Caesarian section wound burst open a week after delivery, had her care delayed by the need to acquire the correct documentation for her in order to start a consultation (95). Oni-Orisan speculates that money is not spent negotiating solutions with traditional birth attendants and church-based facilities (where most women in Nigeria give birth), because these births cannot be easily counted and monitored. She therefore wonders whether ‘numerical standards for effectiveness can sometimes create incentives […] to “perform good data’’’ (100), rather than improve health care per se.
Other ideas explored in the volume are the way that a ‘data-driven’ approach to health and an emphasis on short-term outcomes limit the possibility of long-term, community-driven health initiatives: for example, amongst the Yup’ik of Alaska (Molly Hales, Chapter Five) or government hospitals in Haiti (Pierre Minn, Chapter Eight). In Chapter Four, Marlee Tichenor investigates striking Senegalese health workers who for three years refused to provide metrics alongside their routine clinical work. The Senegalese government is so dependent on foreign agencies for health funding (who set the production of this data as a precondition for the release of funds) that the two unions involved felt this was the most effective way to hold them to account. Susan Erikson in Chapter Six charts a shift in global health from accountability to ‘value’ metrics, where the language of finance and economics moves into health, and multinational corporations start to see market opportunities in people’s suffering.
None of the authors in this collection dispute that a certain degree of documentation, both of health needs and health interventions, is a necessary prerequisite to providing care that is essential and cost effective. However, as Adams argues, an excessive focus on what is quantifiable and objectifiable can push to the periphery health problems that are not easily rendered numerical, and can devalue what patients often believe to be most important in those who care for them: compassion, a sense of humanity, time. This is by no means a problem limited to the Global South. In a recent editorial in the New England Journal of Medicine, two senior physicians from Yale and Stanford link the escalating volume of work in hospitals carried out away from patients in offices and on computers to increasing dissatisfaction amongst doctors. The reliance of the US healthcare system on what they called ‘the digital representation of the patient, the iPatient’ means that despite advances in technology, doctors are ironically able to spend less time than ever with their patients and colleagues.
So what is the solution? Clearly any metrics-focused system that leads to the distracting degree of gaming described and distorts health systems cannot be good for patients anywhere, whether in the NHS or the Global South. Adams deals with this in her ‘Epilogue’. She argues that what should come first is a healthy scepticism about the value of metrics: ‘we need to […] undo their claims on certainty, on standardization and truth’ (227). Secondly, she supports the pursuit of new models for assessing health needs and evaluating interventions. However, these concluding four pages feel slight in comparison with the rest of the volume. It isn’t clear whether what Adams thinks would be best is some sort of ‘metrics beta’ (i.e. metrics with a little humanity thrown in), or whether a radical overhaul of the system is necessary. In the meantime, this volume will hopefully help stimulate policymakers and researchers to think seriously about whether playing the numbers game is sufficient, either for patients or their clinicians.
However, as Nigel Hawkes pointed out in a report for the BMJ, in the same year that it was initially investigated by the Care Quality Commission, Mid Staffs had been rated by Dr. Foster, an independent health analytical company, as the ninth safest hospital in England. How was it possible for two analyses to give such different answers? And, following on from this, why was my senior colleague unable to tell me how safe my hospital was relative to other ones in the country?
The answer to these question lies in part in how hospitals report their outcomes in the form of healthcare metrics. At Mid Staffs, for example, the SMR (Standardised Mortality Ratio) after a fractured neck of femur (broken hip) was reported in 2009 as 19.87, compared to a national average of 100. Questioning whether someone could really be five times less likely to die if they broke their hip in Staffordshire compared to the average elsewhere in the country, Nigel Hawkes wrote to the trust asking them to verify this figure. The trust replied stating: ‘We have not always had such a low SMR [standardised mortality ratio] for fractured neck of femur. Our Clinical Coding department advise that the change is due to substantially improved coding procedures.’
Every trust in the United Kingdom has a Clinical Coding department, which assigns ICD (International Classification of Disease) diagnoses to all patients admitted to hospital. The mortality rates provided to the government are then standardised by the degree of co-morbidity that patients had when they died: currently this metric is called the Summary Hospital-level Mortality Indicator. That is, the death of a previously well 50-year-old with no associated health conditions counts for more that than of a 95-year-old with associate heart failure, renal impairment, prostate cancer and so forth. This seems reasonable: it means that trusts are not penalised for working in populations were the average age is higher, for example, or where the average patient is sicker.
However, it also means that it can be hard to interpret metrics from hospitals as a marker of how good the care provided is. Hawkes points out that the crude death rates in England for 2004-05 were 225,439 compared to 222,738 in 2008-09, yet figures for deaths rates adjusted by severity provided by CHKS, another healthcare analytics company, showed a decrease of 50 per cent. Tellingly, he points out that the response of Basildon and Thurrock Healthcare Trust, who was shown to have the highest HSMR in the country in 2009, was that it ‘needed to employ more [clinical] coders’.
Hence why my senior colleague was unable to tell me how safe our wards were compared to other ones round the countries. And, as a doctor, I continue to hear whisperings of pressure to meet government targets leading to gaming of the system – inappropriate last minute movements of patients to avoid Accident and Emergency breaches; manipulation of patients on waiting lists; or wide regional variation in classifications of patients receiving palliative care codes (these deaths do not contribute to adjusted mortality rates).
Image Credit: (Rosmarie Voegtli CC BY 2.0)Metrics: What Counts in Global Health is a volume of papers by anthropologists looking mainly at the Global South. They are similarly preoccupied by how an emphasis on health metrics is distorting the way that healthcare is funded and delivered. However, the collection is not just a critique of metrics in themselves, but a more wide-ranging exploration of how a small number of big-spending non-governmental organisations are increasingly shaping the health services provided to citizens of the Global South.
So what exactly do they think is wrong with too many metrics? As Vincanne Adams points out in the Introduction, one issue – analogous to the situation in the United Kingdom – is whether the metrics in themselves are as reliable and objective as they seem. Bill Gates in his 2013 annual letter stated ‘how important measurement is to improving the human condition’, and in recent years has invested heavily in an organisation based in Seattle called the Institute of Health Metrics and Evaluation (IHME). Headed by the physician and epidemiologist Chris Murray, the IHME produce metrics that they call the ‘Global Burden of Disease’: in their own words, ‘a systematic effort to quantify the comparative magnitude of health loss due to diseases, injuries, and risk factors by age, sex, and geography over time’. However, I know from first-hand experience that such a large-scale effort to quantify disease and death at a global level relies both on extrapolating from small (and often unreliable) data sources, and a certain degree of complex statistical arithmetic.
In Chapter Two of Metrics, Claire Wendland looks at maternal mortality rates and how complicated the formulae used to calculate and standardise these rates are. She teases apart the model used by the World Health Organisation to calculate them, pointing out that out of six of the seven variables inputted, only one is the actual estimate of maternal deaths. As an example of how these factors can influence metrics, she gives Equatorial Guinea, which despite having no national measure or count in maternal deaths for twenty years from 1990 to 2010, showed a dramatic decline in maternal mortality rates attributable to a rise in GDP as a result of rising oil prices (72). In addition to these methodological issues, Wendland also examines how political pressure in Malawi may be leading to an under-reporting of maternal death rates, where local chiefs have started to impose large fines for anyone reporting such a death (74).
Similarly, as outlined in Chapter Three by Adeola Oni-Orisan, in Nigeria health funding is shown to be contingent on positive metrics: there is pressure to manipulate numbers and construct clinical care in a manner that is not necessarily in the best interests of patients. The death of a mother after the birth of her fourth child en route from a police clinic to the local hospital is not included in either the official health facility record or state government reports (83). Another woman, whose Caesarian section wound burst open a week after delivery, had her care delayed by the need to acquire the correct documentation for her in order to start a consultation (95). Oni-Orisan speculates that money is not spent negotiating solutions with traditional birth attendants and church-based facilities (where most women in Nigeria give birth), because these births cannot be easily counted and monitored. She therefore wonders whether ‘numerical standards for effectiveness can sometimes create incentives […] to “perform good data’’’ (100), rather than improve health care per se.
Other ideas explored in the volume are the way that a ‘data-driven’ approach to health and an emphasis on short-term outcomes limit the possibility of long-term, community-driven health initiatives: for example, amongst the Yup’ik of Alaska (Molly Hales, Chapter Five) or government hospitals in Haiti (Pierre Minn, Chapter Eight). In Chapter Four, Marlee Tichenor investigates striking Senegalese health workers who for three years refused to provide metrics alongside their routine clinical work. The Senegalese government is so dependent on foreign agencies for health funding (who set the production of this data as a precondition for the release of funds) that the two unions involved felt this was the most effective way to hold them to account. Susan Erikson in Chapter Six charts a shift in global health from accountability to ‘value’ metrics, where the language of finance and economics moves into health, and multinational corporations start to see market opportunities in people’s suffering.
None of the authors in this collection dispute that a certain degree of documentation, both of health needs and health interventions, is a necessary prerequisite to providing care that is essential and cost effective. However, as Adams argues, an excessive focus on what is quantifiable and objectifiable can push to the periphery health problems that are not easily rendered numerical, and can devalue what patients often believe to be most important in those who care for them: compassion, a sense of humanity, time. This is by no means a problem limited to the Global South. In a recent editorial in the New England Journal of Medicine, two senior physicians from Yale and Stanford link the escalating volume of work in hospitals carried out away from patients in offices and on computers to increasing dissatisfaction amongst doctors. The reliance of the US healthcare system on what they called ‘the digital representation of the patient, the iPatient’ means that despite advances in technology, doctors are ironically able to spend less time than ever with their patients and colleagues.
So what is the solution? Clearly any metrics-focused system that leads to the distracting degree of gaming described and distorts health systems cannot be good for patients anywhere, whether in the NHS or the Global South. Adams deals with this in her ‘Epilogue’. She argues that what should come first is a healthy scepticism about the value of metrics: ‘we need to […] undo their claims on certainty, on standardization and truth’ (227). Secondly, she supports the pursuit of new models for assessing health needs and evaluating interventions. However, these concluding four pages feel slight in comparison with the rest of the volume. It isn’t clear whether what Adams thinks would be best is some sort of ‘metrics beta’ (i.e. metrics with a little humanity thrown in), or whether a radical overhaul of the system is necessary. In the meantime, this volume will hopefully help stimulate policymakers and researchers to think seriously about whether playing the numbers game is sufficient, either for patients or their clinicians.
No comments:
Post a Comment