It is a lot easier to criticize research than it is to do it well. There are so many potential confounders and so many statistical and experimental design pitfalls. This article describes one relating to randomized clinical trials, the typical gold standard in health research. The authors analyzed results in trials designed to prove that an intervention was superior to another therapy or intervention versus trials with an outcome of “non-inferiority” or that the new intervention was at least not worse than existing ones. Researchers may state their hypothesis in a positive or negative form–that the intervention is better, or that it isn’t better. The level of “blinding”, or which patients were in which arm of the trial was also considered.
A large number of randomized trials was examined. For well-designed trials, with minimal risk of inadequate blinding, there did not appear to be a difference in the measured effect between positive hypothesis superiority and non-inferior designs. If the blinding was judged to be poor, there was a substantial difference with higher outcomes shown in the superiority trials. When a negative hypothesis was used, there was no difference in the results of superiority and non-inferiority trials. The message is use a negative hypothesis and be sure you blinding is air-tight. But that is unlikely when your sponsor is a drug or medical device company or someone else who intends to make money from the intervention. (Annals Article)
Kevin writes in the first paragraph – “There are so many potential confounders and so many statistical and experimental design pitfalls. ”
That is especially true. The pro Covid mask studies are a prime example. The single most effective method of reducing transmission of a respiratory virus is to reduce the time of interaction with other individuals. Reducing time of interaction with others is a major cofounding variable. Mask wearers tended to far more paroniod of catching the covid virus and thus they avoided interaction with others much more so than non mask wearers. Yet not a single study accounted for the biggest single cofounding variable.
I previously the Bell McDermott study of increased premature mortality in 96 US Cities when there is an increase in ground level ozone. Increase in ground level ozone is highly correlated to increase in temperatures (hot summer time). That study did a very poor job of accounting for the increased heat as a co-founding variable.
One item to review when evaluating any study is a making an effort to understand what data that is missing from the study that should be in the study and why important data is missing. Is the data overlooked or are the study authors trying to hide something?