How Good Are Guidelines If Most Published Research is False?
John Ioannidis has a new paper in JAMA that reviews the effectiveness of biomarkers as determinants of disease risk, prognosis, or treatment response. Choosing only articles that had more than 400 citations, he and Orestis A. Panagiotou evaluated 35 associations and found that meta-analyses often ended up downgrading the importance of the original finding. Only 15 of the markers were statistically significant in the meta-analyses.
Those who believe that Obama Care performance guidelines will improve patient care are invited to review Boyd et al. (JAMA, 2005) on the applicability of clinical practice guidelines for older patients with multiple comorbid diseases. Derek Lowe (In the Pipeline blog) discusses the new paper and reviews Ioannidis’ rules for evaluating scientific research, which are listed below the fold.
The rules, from Ioannidis’ famous “Why Most Published Research Findings Are False,” are as follows:
The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
In the class Research Methods in my graduate program, we were taught there is no such thing as a smoking gun. Rather, to know with certainty that a given finding is accurate, there needs to be a convergence of numerous studies pointing more or less on one direction.
This is why much disseminated research does not translate into changes in behavior, we need to see if stands the test of time and is reproducible. Most is not, so I tell my patients that while the research findings are interesting wait and they may well change.
Answer to the titular question: The guidelines are not very good.
This is a GREAT post. I wish more people could understand what Linda is describing here.
Agree with Bruce: They are not good. Period.