Giving too much power to power

In many scientific disciplines, power analysis has become a prerequisite for grant funding. And grant funding has become prerequisite for survival of a scientist.

I strongly believe that effort spent in design phase of any study is the most important part of the research process. But… I have always felt very uneasy about power analysis. I suppose my uneasiness is less about power analysis itself, and more about the extreme and automatic reliance on it, coupled with a surprising lack of accountability for justifying its use and its results. Why do presumably skeptical scientists seem to give so much power over to power analysis?

If you’re reading this and don’t really understand what I mean by “power analysis,” I’m referring to statistical power and its common use in justifying the number of subjects (or other units) for a study or experiment. “Sample size calculations” don’t have to be based on statistical power, but often are. Statistical power is directly related to the concept of Type II error rate — and I have more blog posts coming about Type I and Type II error rates (and what they might really mean, or not mean, to you). There is plenty of intro-level information out there on these concepts — just read it with skepticism. For this post, there’s one really important bit of background information needed, and I don’t think it’s controversial. I’ll state it one long sentence with three parts. Power analysis relies on a set of assumptions; the results (the seemingly satisfying number(s) spit out from the analysis) are conditional on those assumptions; and the results are only as justified as the assumptions are justified in the context of the problem (e.g., “garbage in, garbage out”).

Now, back to the question. Why is there such a tendency to over-rely on power analysis, particularly without adequate justification for the underlying assumptions? I am fully aware I can’t answer this huge question adequately in one blog post, but I would like to throw a few thoughts out there. I have to start somewhere.

The reasons for the often blind trust in power are less about Statistics and more about human nature and current scientific culture and paradigms. I keep coming back to two things that help me understand this phenomenon (and others like it). First, relying on it simplifies life for a lot of people and seems logical if you have a superficial understanding of power analysis. It serves the gate-keepers, allowing them to do their job quickly by simply checking tickets without the knowledge to assess if they might be forged. Second, though related, it provides comfort because it spits out numbers and makes an incredibly challenging study design decision seem very easy and as if it has a correct answer. It provides a false sense that we have taken a challenging problem, full of uncertainty, and dramatically reduced the uncertainty associated with it. The inherent uncertainty does not disappear, but is effectively swept under a rug. As conveyed by Herbert Weisberg, we can proceed with the calculations after willfully ignoring many conceptual sticking points. If one is willing to ignore the very tenuous underlying assumptions, then power analysis appears a very useful construct.

But, I think there’s another layer to the willful ignorance part of the story. The term includes the word “willful,” implying the person appealing to it has enough knowledge to be aware of what they are ignoring. That is, they understand the possible problems and unresolved issues, but they willfully make a decision to ignore them — presumably out of weighing risks associated with doing so. Appealing to willful ignorance brings some sense of comfort at being able to move forward with the problem, but it should also carry a healthy dose of discomfort from an understanding of what is being ignored. If one is not aware of the underlying issues and problems, then the decision to go forward is very comforting because there is nothing to invoke the balancing discomfort. Unwillful ignorance brings far more comfort than willful ignorance — naively proceeding without the ugly knowledge of what is being swept away. It gives a greater sense of trust in the method and its results. This envisions yet another continuum based on depth of knowledge about a topic. In order to willfully ignore something, we have to have awareness of it. To gain awareness, we have to be open to listening to the views of people who have spent time thinking hard about a problem — which is often (hopefully!) the case with statisticians and power analysis.

For nearly 20 years, I have been having conversations with researchers about my views on a healthier approaches when power analysis is desired. I have tried many different strategies, tones, etc. But, it nearly always felt like I was arguing against a tidal wave of pressure pushing researchers to do it “the usual way.” My advice as a PhD statistician usually could not compete with the culture and system the researchers were trying to swim in. I walked a fine and uncomfortable line in trying to help as part of my paid job as a statistician — by trying to help justify assumptions and push out of default settings to critically think through the logic for each problem. I am no longer in such a job and hope never have to be in that position again — the realization that I have escaped it still elicits an overwhelming sense of relief (might even qualify as joy). That said, there is still plenty I want discuss about power analysis. It is a tangible context for researchers and has the potential to be a door into deeper conversation we need to be having about the use of Statistics in science in general.

Here’s an email exchange I had with a very successful researcher. I quotes are direct from emails — I only left out minor detail with the ellipsis to help with anonymity.

“Megan, I don’t know what to tell you. Let’s stop here and if the reviewers want a power analysis I’ll find someone else to help me.”

Me: “Just wanted to reach out again and say I am happy to work on this with you if we can collaborate to think through and justify (as best as possible) the choice of numbers going into the power analysis.”

“Thanks for reaching out, Megan, and thanks also for your kind offer. I know you disagree, but I’m going to stick with my bad science power analysis for this proposal — it’s what the NIH program officer I’ve been talking with told me to do. I will appreciate your help with a real power analysis … once we have some pilot data to inform good decision making. But thank you.”

This scenario is not at all uncommon and spans many disciplines. I share this one not to pick on the person, but because I have it in an easily quotable email. I always talked openly about it with my students, but have found it difficult to motivate and engage in productive discussion with researchers who had already had plenty of success navigating through the gate. It is a sensitive topic that is uncomfortable for many to talk about frankly. It can feel embarrassing for everyone involved. It is not productive to blame any one individual who is trying to survive and thrive in their profession with the current gate keepers. Honest and open conversations are needed without fear of it impacting a career.

My hope is that one day my work to educate and help people think through the foundations and underlying logic of things like power analysis will be valued more than my ability (that I refuse to use) to thoughtlessly punch unjustified numbers into an unjustified formula to appease a gate keeper who probably isn’t aware the tickets they are punching are forged. And, refusal of a statistician to participate in forging a power analysis ticket should be professionally respected.