Attitudes toward models vs. attitudes toward methods

This post began as a comment on Andrew Gelman’s blog in response to Conditioning on a statistical method as a “meta” version of conditioning on a statistical model (March 1, 2020). Not surprisingly, my plan for a brief and concise comment ended up not so brief. It feels more like a post of its own, so putting it here as well.

Andrew’s entire post:

When I do applied statistics, I follow Bayesian workflow: Construct a model, ride it hard, assess its implications, add more information, and so on. I have lots of doubt in my models, but when I’m fitting any particular model, I condition on it. The idea is we take our models seriously as that’s the best way to learn from them.

When I talk about statistical methods, though, I’m much more tentative or pluralistic: I use Bayesian inference but I’m wary of its pitfalls (for example, here, here, and here) and I’m always looking over my shoulder.

I was thinking about this because I recently heard a talk by a Bayesian fundamentalist—one of those people (in this case, a physicist) who was selling the entire Bayesian approach, all the way down to the use of Bayes factors for comparing models. OK, I don’t like Bayes factors, but the larger point is that I was a little bit put off by what seemed to be evangelism, the proffered idea that Bayes is dominant.

But then, awhile afterward, I reflected that this presenter has an attitude about statistical methods that I have about statistical models. His attitude is to take the method—Bayes, all the way thru Bayes factors—as given, and push it as far as possible. Which is what I do with models. The only difference is that my thinking is at the scale of months—learning from fitted models—and he’s thinking at the scale of decades—his entire career. I guess both perspectives are legitimate.

Andrew Gelman, March 1 2020 post on Statistical Modeling, Causal Inference, and Social Science Blog https://statmodeling.stat.columbia.edu/2020/03/01/conditioning-on-a-statistical-method-as-a-meta-version-of-conditioning-on-a-statistical-model/

My thoughts motivated by the post

I find myself thinking a lot about differences (both real and those just implied through language) among “statistical models”, “statistical methods”, and “statistical inference,” but had never explicitly thought about attitudes toward them in the way you describe. I do think it’s a constructive way to frame some fundamental issues related to lack of questioning (or over trusting) of statistical methods, and their models, in practice. The feeling you expressed toward the evangelical Bayesian basically describes how I feel on a daily basis toward anyone using statistical methods with an unexamined, and probably unjustified, degree of faith. I have tried to avoid the religion analogy, but it is all too fitting. It captures my opinion and explains a lot about why I can no longer handle being a typical statistician in practice. In my experiences, researchers in many disciplines form groups of like-minded believers with little desire to carefully examine the beliefs — that are then proselytized through teaching, dissemination, and reviewing. I have trouble reconciling this attitude with being a scientist — but I also understand the very real fear and panic that comes with turning a spotlight on limitations of accepted methods (and therefore previous results), particularly if one’s career is based on using them.

Layers

Your reflection on attitudes toward models vs. attitudes toward methods points to the importance of recognizing and naming the layers of assumptions involved in reaching statistical-based conclusions or predictions. I see particular model assumptions as the most superficial layer (e.g., Gaussian errors, constant variance, linearity, etc.) — they are tangible, easy to describe in words, we present clear strategies for attempting to justify them (although rarely effectively), and we may even explicitly say our results are conditional on them. It’s the layer of assumptions we are the most aware of – through check-list and fact-like textbook presentation. It’s the layer that gets nearly all the attention that goes toward justification of methods — that is, we naively treat justification of model assumptions as justification of a more general method. We teach “checking of assumptions” as generally limited to a model (or maybe a few models) — but it has become largely an exercise carried out on autopilot with ridiculous conclusions such as “the assumptions are met,” demonstrating the lack of thought put into the exercise or even into the understanding of the limitations of a model in general. It’s also very easy to say “results are conditional on the model” without ever trying to think through the broader implications of the conditioning.

There are problems with not taking even the simplest layer seriously enough, and the problem gets much worse when we dig into the deeper layers. I’ll call “methods” the next layer in a way that is consistent with what you communicated. In this layer, we’re not thinking about the specific model chosen, but the more general methods within which the model is used (e.g., Bayesian data analysis). A common, or even default, method often depends on discipline and need not have a strict definition or easy to attach label (like “Bayesian”). Methods are often accepted and trusted (at least within a discipline) with very little explicit acknowledgement of their assumptions or limitations. In terms of conditioning — there is little awareness or discussion of what we are conditioning on (or even that we are conditioning on many things). Methods are described by phrases like “the way we do things” or “the way I am expected to do the analysis.” I cringe at the number of times I have had researchers use such phrases with me — in the context of trying to have conversations about why they are adamant about using a particular method (even despite serious problems from a statistical perspective).

Questioning our methods

I see little questioning of “the way we do things.” Thomas Basboll recently described the distinction between methods (what we do) and methodology (why we do what we do) and I think this distinction is easy to forget and good to keep in mind. In my couple decades as a scientist and statistician, I haven’t sensed much worry or interest in digging into the why we’re doing what we’re doing among practicing researchers. There seems to be an attitude that someone before us put in the hard work to decide how things should be done, and so we just need to follow through with that — and no need to even ask deeper questions like why or how. We act as if we are operating within paradigms that are fully justified, strong, and worthy of blind acceptance. This is dangerous, and wrong. Where are the attitudes of healthy skepticism that lead us to value work that questions the ways we currently do things? Questions results conditional on the current way of doing things doesn’t go far enough, though is definitely easier. Questioning deeper layers isn’t currently part of the “workflow,” but it should be.

They will laugh at us

In my experience, there seems to be an underlying belief among researchers in many disciplines that there are no other reasonable options beyond their view of the accepted methods. This is operating under blind acceptance of a paradigm — and so blind that it’s not even recognized as a paradigm that could be moved away from. Perhaps I shouldn’t bring in the word “paradigm” with its historical and philosophical contexts, but I’m simply using it to describe “the accepted way to do things” at this point in time. It’s so hard to envision the time when we will look back and think how naive we were to be doing the things we were doing (and not questioning them) — because we are trying to do the best we can. Why is it so hard to accept that of course we will look back at some point and realize there were serious problems with how we were doing things — that’s what happens in science and what should happen in science. Even if we don’t know exactly what we should be doing, we can know that what we’re doing is probably wrong, or at least can be dramatically improved. We can’t know what the future will look like, but we can know that we will laugh at ourselves (or other future humans will laugh at us).

If we’re going to ignore, then let’s willfully ignore

Probably the main reason I like Herbert Weisberg’s book Willful Ignorance: The Mismeasure of Uncertainty so much is that it delves into the deeper layers, and in an articulate and accessible way. He hits methods and deeper — to “statistical inference” and the decision to rely on probability in general. In order to proceed with statistical inference, there is a lot we need to willfully ignore. However, the problem in practice is that we are not willfully ignoring — we are unknowingly ignoring. There is a lack of awareness and knowledge about what must be ignored (or what we are conditioning on) to proceed with statistical methods and their models to make statistical inferences. Moving toward actual willful ignorance would be a healthy step.

Look over your shoulder

Finally — You point out that you have a lot of doubts about your models and are always looking over your shoulder relative to methods. I think you are an exception, at least in the world of practicing researchers who rely heavily on statistical methods and associated models. I haven’t seen many people looking over their shoulders, at least with anything more than an obligatory and ineffective glance. We should be looking over our shoulders so much that our necks cramp and we purchase rearview mirrors – but instead we often operate as if we are unaware of any threat, or maybe we just don’t know what a threat looks like. We have to know something about what we’re looking for to be able to recognize it, talk about it, and gain the motivation to keep looking. And looking over our shoulders, or in rear-view mirrors, does not mean we are overly paranoid. It can reflect healthy awareness of our surroundings (what we are conditioning on) — not only of the things straight in front of us, but of the things outside our usual field of vision. There’s a lot there.