Research in any field is critical to developing credible knowledge on which to base decisions. And the quality of a research study is extremely important in evaluating how credible the information developed is. Unfortunately, science in general, and medical science in particular, often do a poor job in experimental design and execution and in the use of statistics. We frequently see research which is later shown to be erroneous. In the meantime, decisions are made which may jeopardize people’s health. Changes in care recommendations for chronic diseases are a recent example of this problem. An article in the Journal of the American Medical Association did a survey of articles in medical journals which reported a reanalysis of previously published randomized clinical trials, which are theoretically the gold-standard for research. (JAMA Article) The authors identified 37 reanalyses. In these they found that 46 differences in methods were reported in the reanalysis compared to the original published analysis. These differences included such basic errors as the original analysis including patients who should have excluded, mishandling of data and lack of a sample validation. Thirteen of the reanalyses reported a change in findings that implied a different interpretation about what patients should be treated, in some cases finding that fewer should be, but in most cases finding that more should be.
Reanalyses are often difficult to perform because the full set of data and the statistical methods used to analyze the data are not often published, for a variety of weak reasons, including risk to patient confidentiality, release of commercially sensitive information and reanalysis by people with “nefarious” reasons for wanting to do the reanalysis. This is all basically BS. It is inexcusable in any scientific field, and especially in medicine, that the conductors of research that may be relied upon by clinicians, patients, regulators and policymakers is not required to reveal the full experimental or study design, including an explanation of the strengths and weaknesses of that design compared to other possible ones and an explanation of why the specific design was chosen and what the effect of alternative reasonable designs might have been; all the raw data; any adjustments made to the raw data and why those adjustments were made and what the effect of the adjustments was in the analysis of the data; the statistical methods used in the analysis of the data, why those statistical methods were chosen and what the effect of using other reasonable methods of analysis might have been. This level of transparency is critical to allow users to understand the strength of the finding and to allow other researchers to validate or critique the findings.