Health Research Validity

As frequent readers know, this commentary largely focuses on health research of various types.  There is a tendency to believe that just because something is published as a research finding, particularly in a prestigious journal, it must be an accurate representation of the truth.  A health researcher has published an article with the provocative title of “Why Most Published Research Findings Are False”.   (Ioannidis Article)   He is quite serious and given the number of allegations of outright research fraud in the last few years, it would not be surprising that the universe of suspect studies is much larger.  His basic premise is that the probability that a research finding is indeed accurate is a function of a) what it represents is the “real” reality; b) the statistical power of the study; c) the level of statistical significance of the findings.

Areas he pinpoints for concern are the fact that many studies are relatively small, they are seldom repeated so we are relying on one trial, and there are ample pathways for bias, intentional or unintentional to creep into results.  Bias can relate to either the trial design and analysis or even to the reporting of results.  It is fairly well-established, for example, that positive results are more likely to be published than negative ones.  Some of his principles, or alerts if you will, include:  the smaller the studies, the less likely the research findings are to be true; the smaller the effect size, the less likely the results are accurate; the greater the number and the lesser the selection of tested relationships in the study, the less likely the findings are true; the greater the range of designs, definitions, outcomes and analytical statistical methods in a given scientific area, the less likely the results are accurate; the greater the financial and other interests and prejudices among the researchers in a field, the less likely the findings represent truth; and the “hotter” a scientific field, the less likely research in it is accurate.  Of particular relevance for what does and doesn’t work in improving health care are the need for large, repeated studies of consistent interventions.  We rarely see that.  And the public, even the informed public, needs to be skeptical of any reported research.