Fact detector? It is not.

I continue to search for more effective, and simpler, ways to convey my views on misunderstandings and mis-uses of Statistics to others — including scientists. At the heart of my discomfort are not the limitations of statistical inference (I still find it fascinating and useful), but that we use it as if it provides something it was never intended to provide, and simply can’t provide. I have said in the past that it is not “answer finder” and should not be used as one — but I don’t have a lot of evidence from facial expressions and subsequent work of students and researchers that the idea hits home. I want to try again here, with a little different wording spurred from reading Ignorance: How it Drives Science by Stuart Firestein (2012 Oxford Press) – which is a quick read and a great way for anyone interested in science to start off 2020! My opinions come mainly from my experiences in science over the last 25 or so years, and I want to fully recognize that I understand they do not apply broadly across all scientists or scientific disciplines.

Let’s assume that most people see science as the process of collecting more and more facts (where facts are taken as evidence of knowledge). I find this a realistic assumption because of how science is typically presented, taught, and discussed. I wholeheartedly believe it is more about understanding ignorance than collecting facts, but even then have found myself accidentally reinforcing this view with my own kids at times. Firestein’s excellent discussion of the role of ignorance in science seems, at least to me, to often imply that scientists themselves understand the complexities and creative nature of the scientific process, but just haven’t done a good job conveying that to others. Based on my experiences, I think this is true to some degree, but I also see too much emphasis from scientists on fact-finding and fact-reporting and adherence to expectations for this in dissemination of work. This is where my views on use of Statistics come in. I see statistical methods often used in a way that reinforces, and even further contributes to, a fact-centered way of operating in science. The common (and wrong) explanation for what some statistical methods provide is that of a litmus test for whether observed effects “are real or not.” What does “real” mean? I can’t help but interpreting it as a reflection of the view of Statistics as a convenient fact-finding machine.

Take this quote from Firestein’s book:

How do scientists even know for sure when they know something? When is something known to their satisfaction? When is the fact final?

Page 21, Stuart Firestein, Ignorance: How it Drives Science

Firestein writes to help non-scientists or future scientists understand the reality of the scientific process, but it is worth thinking more about how current scientists can benefit from reflecting on these ideas. And, of course, I want to think about the role of common uses of statistical methods in the process. (Note – I purposefully choose between the phrases statistical methods and statistical inference, because I believe statistical methods can be (and are) used with a disregard for the inference part of the process).

I think the three questions posed by Firestein are uncomfortable to scientists, and they should be because they are actually impossible to answer. But, we’ve created a scientific culture and incentive system that expects scientists to pretend as if they have contributed a new fact and that it is close enough to “real” or “final” to be published in a scientific journal. In this system, who is to judge this? How much space in a presentation or paper should be dedicated to this onerous task? This is why and where I believe we have come to rely on statistical methods to provide a cheap shortcut — a shortcut they aren’t designed for.

Statistical methods are not fact detectors.

Counter to foundations based on concepts like variation, uncertainty and probability — statistical methods have been dressed up as meters to help scientists pretend to answer the impossible-to-answer questions. Instead of leaving them to struggle with the questions or admit they are unanswerable (or the wrong questions to begin with), much effort goes into creation of cheap tests and criteria as a shortcut to the wrong destination.

Statistical inferences are, and should be, complex and uncomfortable, not simple and comforting. Statistical inference is about inferring based on combining data and probability models, not about judging whether an experimental or observational result should be taken as a fact. There is no “determining” and no “answers” and no distinguishing “real” from “not real” — even though this language is common in scientific reporting. It is not helping science to keep pretending as if Statistics can detect facts.

You may think I am exaggerating the problem, but I encourage you to read reports of research with this in mind and judge for yourself. How common is it for scientists and other reporters of scientific work to talk and write about statistical methods as if they are fact detectors? The problem gets more complicated if we start to consider whether the authors actually believe statistical methods are capable of detecting facts, or if they are just following conventions and expectations for their own survival. This complication is beyond this post, because either way, awareness that it is a problem has to be the starting point.

What can we do? Well, here’s an easy to ask question when reviewing your work or the work of others: Are statistical methods being used or presented as fact detectors? If the answer at all leans toward ‘yes,’ then it’s time to back up and think about the shortcut being taken, as well as the presumed destination. What can be added to more honestly acknowledge uncertainty and assumptions in any reported inferences? Let us try hard to avoid using statistical inference as if it presents a shortcut to facts.