Why Pilot Projects Don’t Work

Remember, the entire ObamaCare approach to controlling costs and improving quality is based on running pilot projects. As the president has said on more than one occasion, “Let’s find out what works and then go do it.” So I want to make sure you are aware of an impressive post by Megan McArdle before Christmas, explaining why this approach is doomed to fail. In a word, she says, “promising pilot projects often don’t scale.”

They don’t scale even when you put super smart people with expert credentials in charge of them. They don’t scale even when you make sure to provide ample budget resources. Rolling something out across an existing system is substantially different from even a well-run test, and often, it simply doesn’t translate.

Here are five reasons:

• Sometimes the “success” of the earlier project was simply a result of random chance, or what researchers call the Hawthorne Effect, named after a study in which workers became more productive no matter what the stimulus — a change from the ordinary, or the mere act of being studied.
• Sometimes the success was due to what you might call a “hidden parameter,” something that researchers don’t realize is affecting their test.
• Sometimes the success was due to the high quality, fully committed staff, whose work won’t be duplicated by folks who are just looking for a job and don’t see the pilot project goal as their life’s mission.
• Sometimes the program becomes unmanageable as it gets larger.
• Sometimes the results are survivor bias – the subjects who stay in the project are very different from the random population.

On this last point she explains:

This is an especially big problem with studying health care, and the poor. Health care, because compliance rates are quite low (by one estimate I heard, something like 3/4 of the blood pressure medication prescribed is not being taken 9 months in) and the poor, because their lives are chaotic and they tend to move around a lot, so they may have to drop out, or may not be easy to find and re-enroll if they stop coming. In the end, you’ve got a study of unusually compliant and stable people (who may be different in all sorts of ways) and oops! that’s not what the general population looks like.

Print