Commonwealth Ranking: Are We Really 19th Out of 19?

If you think that the quality of your health care depends on whether your doctor keeps your medical records on an interoperable computer system or on whether the country has national health insurance – even it if it means that sick people will die while waiting for care – then Why Not the Best?  Results from the National Scorecard on U.S. Health System Performance, 2008 from the Commonwealth Fund is the scorecard for you.¹

Choosing Nonmedical Benchmarks.  In its latest health system scorecard, the Commonwealth Fund updates about 90 percent of the indicators that were used in its 2006 scorecard.  On roughly a third of the measures, U.S. health care improved.  These indicators are generally the ones that measured some quantity associated with actual medical care.  The rest, which measured a hodge-podge of aggregate economic indicators, opinions of unknown quality and compliance with various ideologically-based policy prescriptions, stayed the same or got worse.  The Fund concludes that the United States as a whole “failed to keep pace with levels of performance attained by leading nations, delivery systems, states, and regions,” that access to health care in the United States “significantly declined,” and that health system efficiency remained low.²

Cherry-Picking the Benchmarks.  The score for each indicator is calculated as the ratio of the U.S. average to the Commonwealth benchmark.  Cherry-picking benchmarks helps Commonwealth obscure good results and highlight bad ones.  Many of the benchmarks are culled from the best results in the top 10 percent of states, regions, health plans (those that pay to be evaluated by the National Committee for Quality Assurance) or hospitals.  Since the U.S. average will always be below the best performance in the states or regions that make up the U.S. average, the design of the scorecard ensures that Commonwealth will always be able to report unacceptable performance – even if both the average performance, and the 90th percentile benchmark performance, continue to rise.

Using Questionable Benchmarks.  The indicators themselves have strong ideological components.  Some, like percent of children with a medical home,³ adults with chronic conditions given self-management plans, and the percent of physicians using electronic medical records, have not been shown to decisively improve health.  Some measures, like the OECD measure of health insurance administration, almost certainly contain measurement error, especially in countries with publicly controlled systems.

Other measures, like U.S. infant mortality rates, are well known to be heavily correlated with demographic characteristics that have little to do with the performance of the health care system.  These correlations are not taken into account in the scorecard rankings.

Equating Low Spending with Efficiency.  The scorecard also indulges in the dangerous practice of equating low spending with efficiency.  This ignores the fact that it costs more to treat patients than to let them suffer.  In many cases, additional spending on new techniques and advanced pharmaceutical treatments correlate with better health.  As results from Canada and the United Kingdom make clear, lower spending, which invariably improves scorecard results, is not necessarily an indicator of improvement.

Using Questionable Measurements.  The scorecard often derives blanket measures that sound reasonable from collections of measures that are not.  In the context of evaluating a health care system, the normal assumption about “unsafe drug use” would be that it measures medication errors, inappropriate prescribing, or unsafe drugs.

In the Commonwealth report, “unsafe drug use” measures three things: (1) An estimate of ambulatory care visits for treating adverse drug effects in 2004 (which leaves out hospital reports of adverse drug reactions and fails to distinguish between health system causes such as inappropriately prescribed drugs, unexpected individual reactions to appropriately prescribed medicines, or problems caused when patients fail to follow instructions); (2) a measure of whether children were prescribed antibiotics for a sore throat without a strep test (something that might be a perfectly reasonable cost saving measure in a family of 5 young children all of whom have sore throats and two of whom test positive for strep); and (3) a 2004 ranking of whether the elderly used 1 of 33 inappropriate drugs.

Confusing “Access” with “Third-Party Insurance.”  The 2008 scorecard summary makes much of the supposed fact that “access to health care significantly declined, while health system efficiency remained low.”  This would clearly be cause for concern if the indicators Commonwealth uses to measure access to care actually measure whether people get the medical care they seek.

As the long waits for health care in Europe, Canada, and the United Kingdom show, having the government promise to pay for health care does not necessarily guarantee that any particular individual will actually receive the medical care he needs.  Commonwealth ignores this fact.  Two of its five access indicators measure aspects of third-party payer coverage.  They are “Adults under 65 insured all year, not underinsured” and families spending less than 10 percent of income (or less than 5 percent of income, if low income) of out-of-pocket medical costs and premiums.

A third access measure asks whether premiums for employer sponsored coverage are less than 15 percent of household median income.  As this access measure is affected by state regulation, employer benefits, unionization, and various demographic variables, it is a poor measure of health system performance.  Its effects overlap with another access measure: the percent of “Adults under 65 with no medical bill problems or medical debt,” a measure that would logically be affected by individual income, individual savings behavior, state health insurance regulation, state and federal assistance programs, and the definition of medical debt.  None of these are aspects of a health care system as conventionally described.

The out-of-pocket spending measures, which make up roughly 40 percent of the access score, are crucially dependent on how one defines underinsured.  In the Commonwealth world, underinsured means a health insurance deductible that is no more than 5 percent of income, regardless of assets, or health care expenses that exceed 10 percent of family income.  (The figure is 5 percent for those with incomes below 200 percent of the poverty level which is $20,800 for a single person in 2008.)  Since the United States spends roughly 16 percent of its national income on health care, meeting this goal is unlikely without extensive substitution of government controlled tax funded spending for the individual purchase of health care and/or mandates requiring higher employer health insurance spending. Employer mandates will inevitably be funded by lower wages.

Ignoring Self-Insurance.  By choosing to evaluate the U.S. system using this measure, Commonwealth biases its evaluation in favor of third party payment.  There is a large body of evidence suggesting that third party payment for small or routine expenses does little to improve health or control health care costs.  In fact, once demographic variables are taken into account, there is considerable uncertainty about whether expanding third party coverage in the United States would translate into significant health benefits.⁴

The 6.1 million people who have purchased health savings account (HSA) plans (which allow more individual self-insurance and less third-party insurance) clearly do not agree with the Commonwealth bias in favor of third party payment.  A family HSA plan must have a minimum deductible of $2,200 in 2008.  If the people buying these plans shared Commonwealth’s view that lower deductibles equaled better health care, they would presumably purchase policies with the lowest possible deductibles.  Instead, the best-selling plans had deductibles of $4,846 in the individual market, $4,356 in the small group market, and $3,998 in the large group market.⁵  For ease of calculation, assume that it is now $28,000.  In 2008, Medicare Part B premiums are about $2,300 a year for two people. Part D premiums are about $720 a year for two people.  This means that the median married couple Medicare household is paying $3,020 a year – more than 10 percent of income – in Medicare premiums. Supplemental insurance, long-term care insurance, dental expenses, Medicare deductibles, Medicare copays, and uncovered medical visits are extra.  According to Federal poverty level guidelines, 200 percent of the poverty level is $28,000 for a two person household in 2008.  By Commonwealth’s standards, half of households with an over 65 head are underinsured if they spend more than 5 percent of income, or $1,400, on health care.  Medicare premiums exceed this.  Simple extrapolation suggests that a large fraction of the Medicare population falls into Commonwealth’s underinsured category.

Ignoring Rationing by Waiting.  The next access measure, “Percent of adults with no access problems due to costs” is also biased in favor of a national health system.  People do not worry about costs in national health systems because they usually have minimal out-of-pocket payments.  What they do worry about is access – waits for care are long and advanced care may be denied, especially if one is elderly, disabled or a severely ill infant.

The Commonwealth international access benchmark does not include a single measure designed to capture either rationing by waiting or rationing by denial of specific types of care.  It compares U.S. health care to that in other countries using the percentage of adults who said that they had no access problem due to costs in the Commonwealth IHP telephone surveys.⁶ ,⁷

Lacking basic demographic data on those surveyed poses a problem for those who wish to use the survey results to compare access and quality in the U.S. health care system with that in other countries.  The problem is that access and quality may vary significantly between U.S. private sector health care and government run programs like Medicare, Medicaid, Tricare, and the Veteran’s Administration.  The survey gives no indication whether the people responding to its questions are receiving government run or private sector care, or on how their ages, incomes, and reported illnesses compare across countries.

Blendon et al. reported on the original Commonwealth IHP surveys in the May/June 2003 issue of Health Affairs.⁸  The Blendon paper makes it clear that delays of scheduled medical procedures and long waits for specialist care are considered much bigger problems in the benchmark countries than in the United States – 5 percent of those surveyed reported canceled medical procedures were a big problem in the United States, compared with 9 to 16 percent in other countries.  These data are not included in the Commonwealth scorecard.

Misusing Statistics.  Nine of Commonwealth’s 37 indicators compare U.S. results with those in other countries.  Six of those nine indicators are derived from the authors’ analysis of data obtained from the Commonwealth Fund IHP Survey.  The rankings assigned the various countries creates the impression of difference when, in some cases, a z-test for equal proportions suggests that the results from the survey provide no basis for assuming that a difference exists.

As the table shows, Commonwealth assigned a ranking from 1 to 6 to each country based on the proportion of people in the IHP survey of sicker people who reported that they had been subject to a medical error in the last two years.  Rankings were averaged in the case of a point estimate tie.  Aside from the question of the accuracy of individual impressions of error, the statistical problem is that those proportions are estimated with error from different sample sizes. Using the sample sizes given in the paper, a simple two-tailed z-test for the equality of percentages from the different countries suggests that there is no statistically significant difference at the .05 level between the proportion of medical errors reported for the United States and New Zealand, the United States and Canada, the United States and Australia, or the United Kingdom and Germany.⁹

The simple test of proportions and revised rankings derived from its results suggests that the rankings assigned by Commonwealth create an artificial impression of lower performance in the United States.  It does this by ignoring statistical error and ranking countries from 1 to 6 based on point estimates without considering statistical error.  The scorecard calculation propagates the false impression of difference by using the improperly applied rankings in its calculations.  Unsurprisingly, Commonwealth then concludes that “the U.S. continues to rank last on safe care.”

Where the United States Improved.  For the record, here is the list of Commonwealth Fund indicators that showed improvement between 2006 and 2008.  The improvements may have been judged insufficient by the scorecard depending on the benchmark used for comparison:

1. Deaths per 100,000 population fell.
2. Infant mortality fell.
3. Higher percentages of adults and children received recommended preventive care.
4. Higher percentages of diabetics and hypertensive adults had better blood sugar and blood pressure control.
5. A higher percentage of hospital patients received recommended care.
6. A higher percentage of heart failure patients received written discharge instructions. (This is a process measure, whether it improves outcomes is unknown.)
7. A slightly lower percentage of patients reported medical, medication, or lab test errors.
8. A lower percentage of children were prescribed antibiotics for a sore throat without first getting a strep test.
9. A smaller percentage of short-stay nursing home residents developed pressure sores.
10. A higher percentage of patients reported that doctor-patient communication was adequate.
11. A higher percentage of adults reported that they had no cost barriers to health care access.
12. Actual to expected deaths for hospital mortality ratios fell from 101 to 82.

Notes

Print