Argument for p-values: “Because I understand what they mean”

This is another post inspired by an entry on Andrew Gelman’s blog and the first comment: https://statmodeling.stat.columbia.edu/2019/11/14/is-abandon-statistical-significance-like-organically-fed-free-range-chicken/. I hesitate to continue to give so much attention to p-values, but it’s clearly a topic not going away anytime soon and one we need to be discussing, even if it often feels superficial. Hopefully it is an easily accessible door leading to other conversations.

I am intrigued by this commonly given reason for wanting to continue to rely on p-values: “because I understand what they mean.”

In my experience, it is rare that a researcher/reviewer really understands and can articulate what a p-value means, particularly in the context of observational studies. When we venture outside the simple intro-stats contexts that can be easily programmed with pseudo-random number generators — like one-factor experiments with random assignment or random selection from a well-defined population — things get sticky. I’m all for using simulation to initially teach the concept of a p-value (if needed), but we also need to push students to think about how those ideas do, and do not, extend to more complicated situations and studies without any random mechanism linked to the design. To rely on a p-value as if it represents useful information about the results of a study, the researcher should be able to explain what information is contained in that number in the context of their study — in an interpretation sort of way, not reciting a generic definition. This is hard, even for formally trained statisticians. We combine information from the data with assumptions to arrive at a neatly packaged and attractive number — but what are those assumptions and when do they make enough sense to appeal to? For observational studies, it involves a lot of “as if” hypothetical thinking. If it’s not making your brain hurt, you’re probably not going deep enough.

So…my interpretation of the statement “I understand what they mean” is that it typically represents the person’s belief in their ability to successfully use p-values according to common practices — the same practices that some statisticians have been arguing against for decades. The ability to use p-values in a socially accepted way is not synonymous with understanding what they mean. In fact, being comfortable using p-values in ways not recommended by the people who have thought deeply about them is probably evidence that you don’t really understand what a p-value means. I’m not sure I generally find p-values useful enough to enthusiatically promote spending the time to deeply understand them — though of course if you are someone who would like to keep using them (in your own research or to judge the research of others), then I think you should. I also think judging their degree of usefulness should be context dependent — there is plenty of room for disagreement and assessing reasonableness of embedded assumptions situation by situation.

If we don’t have a deep understanding of a concept, are we qualified to self-assess whether we understand what it means? I suspect most will answer ‘no.’ Unfortunately, if we don’t have a deep enough understanding, then we don’t have the knowledge to self-regulate our self-assessments. Somehow I keep ending up at the “we don’t know enough to know what we don’t know” conundrum. [And, can’t help but wonder where I am inevitably making such mistakes today.]