Medical research reflections – scandals and a little Statistics
March 24, 2020 | General | No Comments
At this time in history, it’s worth reflecting on medical research and how it’s done. The following two opinion pieces made it into my awareness through recent Twitter posts, and I decided to share them here. I also spent last Thursday writing a letter to the editor at New England Journal of Medicine, and got the rejection email today. So, figured I should just share what I wrote in this post instead.
Scandal of Poor Medical Research
Editorial The Scandal of Poor Medical Research by D. G. Altman in the British Journal of Medicine (BJM) written in 1994 and a follow up opinion titled Medical research — still a scandal written by Richard Smith 20 years later.
- BMJ 1994; 308 doi: https://doi.org/10.1136/bmj.308.6924.283 (Published 29 January 1994) Cite this as: BMJ 1994;308:283
- https://blogs.bmj.com/bmj/2014/01/31/richard-smith-medical-research-still-a-scandal/
My unpublished editorial
In reading my less-than-400-word-editorial again, I’m not in love with it, but so it goes. I was trying to contribute something practical and relevant to the situation we are in — as we try to learn about the virus, the disease, and potential treatments while never having enough time or information.
*********************************
After decades of habitually using statistical significance as if it measures clinical relevance, maybe the Corona crisis will motivate change. Scientists have a responsibility to use all information available to them and to justify their methods and conclusions. We cannot risk ignoring or misinterpreting information from any study that fails to reach an arbitrary threshold in terms of precision in estimates. We cannot afford to fail to pursue potentially valuable treatments simply because of large p-values or intervals including default null values.
In this crisis, we are forced to deal with uncertainty due to lack of information. In response, we must work harder to interpret the information we do have relative to a useful clinical yardstick, rather than an unjustified statistical yardstick. It is time to reach outside the comfort zone of statistical tests to embrace more practically meaningful inference. Statistical methods can quantify uncertainty — under a lot of assumptions — but cannot rid a problem of uncertainty. They do not logically lead to the decisions we would like them to. Common misinterpretations can and will have serious consequences.
- Do not imply a binary decision about clinical relevance based only on whether a default null value is included (or not) in an uncertainty interval. This is equivalent to claiming statistical significance (or not) by comparing a p-value to a cut-off.
- Prepare a practical backdrop for inference. Picture a number line distinguishing (as much as possible) values deemed clinically relevant from those not believed to indicate clinical relevance. This should be based on clinical and scientific knowledge, not previous statistical estimates or tests. Start collaborating, debating, and preparing the backdrop now, even before we have the data!
- Use all the information presented in an uncertainty interval. Does the interval contain values considered clinically relevant? Does the interval contain values not deemed clinically relevant?
- The answer can be ‘yes’ to both.
- An answer of ‘yes’ to the second does not imply an answer of ‘no’ to the first. This is a common mistake in interpretation.
- An answer of ‘yes’ to both does not imply the study should be ignored or that we should conclude a potential treatment is not effective. There is danger in prematurely deciding not to pursue a treatment based on ignoring information.
- Don’t take the ends of an interval as hard boundaries. There’s always more uncertainty than accounted for mathematically.
Additional articles (will continue to add to this)
- Br Med J 1977; 1 doi: https://doi.org/10.1136/bmj.1.6053.85 (Published 08 January 1977)Cite this as: Br Med J 1977;1:85
