Why Pilot Projects Don’t Work

Remember, the entire ObamaCare approach to controlling costs and improving quality is based on running pilot projects. As the president has said on more than one occasion, “Let’s find out what works and then go do it.” So I want to make sure you are aware of an impressive post by Megan McArdle before Christmas, explaining why this approach is doomed to fail. In a word, she says, “promising pilot projects often don’t scale.”

They don’t scale even when you put super smart people with expert credentials in charge of them. They don’t scale even when you make sure to provide ample budget resources. Rolling something out across an existing system is substantially different from even a well-run test, and often, it simply doesn’t translate.

Here are five reasons:

  • Sometimes the “success” of the earlier project was simply a result of random chance, or what researchers call the Hawthorne Effect, named after a study in which workers became more productive no matter what the stimulus — a change from the ordinary, or the mere act of being studied.
  • Sometimes the success was due to what you might call a “hidden parameter,” something that researchers don’t realize is affecting their test.
  • Sometimes the success was due to the high quality, fully committed staff, whose work won’t be duplicated by folks who are just looking for a job and don’t see the pilot project goal as their life’s mission.
  • Sometimes the program becomes unmanageable as it gets larger.
  • Sometimes the results are survivor bias – the subjects who stay in the project are very different from the random population.

On this last point she explains:

This is an especially big problem with studying health care, and the poor. Health care, because compliance rates are quite low (by one estimate I heard, something like 3/4 of the blood pressure medication prescribed is not being taken 9 months in) and the poor, because their lives are chaotic and they tend to move around a lot, so they may have to drop out, or may not be easy to find and re-enroll if they stop coming. In the end, you’ve got a study of unusually compliant and stable people (who may be different in all sorts of ways) and oops! that’s not what the general population looks like.

Comments (6)

Trackback URL | Comments RSS Feed

  1. Davie says:

    These are all very important considerations. I agree that this article lays out the case very well. The questions of scale and human error are some of the most important ones when determining policy, and ones policy-makers would be wise to consider.

  2. Joe S. says:

    That was a great post by Megan McArdle. One of her best ever.

  3. Nancy says:

    Great post. Really interesting.

  4. Devon Herrick says:

    I would vote for the Hawthorne Effect, where participants work to make the program a success because they are being observed. Having a few dedicated individuals at the top who want to participate in the pilot project undoubtedly enhances the effect.

    Where pilot projects do not work are those circumstances where individuals are not motivated; and success of the pilot project is not in participants best interest.

  5. Tom H. says:

    Fascinating. I guess the Obama administration is wasting millions of dollars on these things.

  6. Brian says:

    I would say that he first and third items strike me the most as being big problems for researchers in many areas.