Modeling Wonderland: Errors of 26 Percent to Start

ObamaCare implementation is heavily dependent on central planning informed by economic modeling for risk adjustment, pricing, individual consumption behavior, and estimates of actuarial equivalence. People seem to forget that economic models frequently do a lousy job of predicting real outcomes, and that bad things happen when people who make policy confuse model results with reality.

Here’s what economist Jonathan Gruber wrote about one aspect of this problem in 2011:

It is possible that survey estimates of Medicaid enrollment may differ from the administrative totals in that survey respondents tend to under-report Medicaid coverage. This phenomenon is well-documented and occurs in both state and federal surveys of health insurance coverage.

Administrative totals measure the number of people enrolled and receiving benefits from a program. Surveys sample people and ask them if they are receiving benefits. How different are survey estimates from administrative totals? A lot.

In 2011, survey-based estimates for Colorado Medicaid enrollment were used in planning for the Colorado Health Benefits Exchange. They were 26 percent lower than the number of people on the state’s Medicaid rolls. As Colorado budgeted slightly more than a billion dollars in spending for the Medicaid groups that the estimate covered, a 26 percent underestimate equals about $250 million. For FY 2010-11, that is about 4 percent of the state’s general fund spending, an amount equal to roughly 70 percent of all of the state’s corporate income taxes collections.

Health insurers have historically used past claims experience to predict future costs. But in 2011, the Colorado Health Benefits Exchange Board chose to forecast using the Gruber Microsimulation Model (GMSIM). As explained in the January 2012 report delivered to the Board, the GMSIM model does not rely on extrapolations based on past administrative data. Instead, it depends upon extrapolations from survey data developed from a “number of excellent data sources available for Colorado.” Elizabeth Lukanen, Senior Research Fellow at SHADAC, described GMSIM as the model “used by the executive branch during the development of federal legislation.” According to the report, GMSIM has also been used to inform policy in Massachusetts, California, Connecticut, Delaware, Kansas, Michigan, Minnesota, Oregon, Vermont, Wisconsin and Wyoming.

The surveys used to generate the GMSIM forecasts included a 2008-09 survey of 10,000 randomly selected Colorado households, the Current Population Survey, the Medical Expenditure Panel Survey, and projections from the state and the Congressional Budget Office for growth in Medicaid and “demographic and economic information.” “Imputation methodologies” were developed to fill in data collected by the Current Population Survey but not available in the Colorado Household Survey.

While the quality of the GMSIM forecasts will not be known for a few years, its baseline estimates can now be checked against administrative data. For Colorado in 2011, the surveys generated a GMSIM starting point of 465,000 people in enrolled in “public insurance through the Medicaid or SCHIP programs.” The definition of “public insurance was peculiar.” It excluded the elderly and those who “are disabled covered by Medicaid and Medicare,” generally the most expensive Medicaid enrollees on a per capita basis.

Even when all of the elderly and disabled are subtracted, the GMSIM starting point bears little resemblance to actual enrollments. By December of 2011, Colorado’s official data on Medicaid enrollment reported 620,799 people enrolled. Subtracting official subtotals of adults 65 and older and disabled people of all ages and eligibility leaves 513,271 officially enrolled people. Adding the 74,030 people enrolled in the state’s SCHIP program, Child Health Plan Plus (CHP+), results in 587,301 people on “public insurance” as defined in the GMSIM report. A rough check of the model’s accuracy could have been done using Colorado’s March 2010 official enrollment data, which were certainly available at the time the report was written. At 481,584, 2010 enrollment was already higher than the 2011 GMSIM starting point.

The GMSIM forecasts that ObamaCare will raise premiums in the Colorado individual insurance market by 19 percent in 2016. This increase supposedly accounts for the impacts of ObamaCare’s expansion of benefits, its minimum actuarial benefit requirement, its minimum loss ratio, its guaranteed issue requirement, and the 3 to 1 limit on the premiums charged older versus younger people.

Unfortunately, the model also assumes that “premium reductions due to the managed competition efficiencies introduced by the exchange” will moderate the increase. Facing predicted premium increases of at least 19 percent, people in Colorado’s individual insurance market can only hope that the GMSIM managed competition efficiency theories do a better job of predicting premiums than its survey estimates did in predicting Medicaid enrollments.

Comments (11)

Trackback URL | Comments RSS Feed

  1. Devon Herrick says:

    Gruber is correct that there is an established literature that discusses the so-called Medicaid undercount. People cycle in and out of Medicaid eligibility quickly and do always remember past Medicaid coverage when they fill out Census Bureau surveys. Also, people don’t necessarily consider Medicaid to be insurance and don’t always indicate Medicaid coverage when asked about whether or not they were insured, say, two years ago.

  2. Cindy says:

    Self-report is notoriously unreliable. It’s interesting to consider this is a limitation on current central planning necessary for the implementation of health care.

  3. Jordan says:

    Projected savings because of structural efficiency always makes me giggle.

  4. Jennifer Stance says:

    This reminds me of a previous post regarding surveys and their room for error. I’ve never understood why so many of these studies and programs make so much emphasis on survey-based estimates. Generally speaking, people do not give surveys the time and dedication they should to make them as accurate as possible. Thus, most results obtained on the majority of these surveys are simply not reliable. If they keep basing their programs and decisions on the stats and numbers they get from interviewing randomly chosen individuals, then they will continue to make poor predictions on costs and everything else just like they have been doing so far.

  5. Jackson says:

    Very good post Linda!

  6. Smith says:

    its really very nice post to read as i like the way you wrote it thnaks for posting …
    as am doing home business and earning more income to fulfill my needs and to secure my future i just joined to GDI Global Domains International and started earning without any investment for more info go to and use username webincome5 as access code to see the presentation.

  7. Alex says:

    “The GMSIM forecasts that ObamaCare will raise premiums in the Colorado individual insurance market by 19 percent in 2016.”

    Therefore, Obamacare is bad for these people.

  8. Paul says:

    Great post, thanks Linda!

  9. seyyed says:

    on paper looks great but will likely fail in practice.

  10. Ashley says:

    Forecasts are difficult, but policymakers should error on the side of caution and over budget.

  11. Robert says:

    Great work, Linda! Now I’ve just got to go through all the supporting links. Keep it up!