Quantitative backdrop to facilitate context dependent quantitative research

For years, I have been proposing a relatively simple strategy for better tying quantitative analysis to specific research and decision contexts — with the goal of helping researchers give less authority to statistical methods detached from context (e.g., move away from reliance on arbitrary and default statistical criteria). Through work on different projects, I have developed more explicit steps to help researchers/analysts work through the process of developing a quantitative contextual “backdrop” early in the research process. The ideas started in the context of working through sample size investigations with alternatives to power (https://critical-inference.com/sample-size-without-power-yes-its-possible/), but have evolved into what I see as an important undertaking in any project depending on a quantitative scale and some form of estimation (whether sample size justification is needed or not). The process can also be particularly important if practical decisions based on the results are high-stakes (politically, financially, etc.), as it lays out a framework for decision making in the face of uncertainty before results are in; ideally it can motivate early conversations among stakeholders with potentially opposing values or viewpoints and lead to less conflict around interpretation after data analysis.

The term “backdrop” is borrowed from theater. As part of a play, the backdrop is a picture that hangs behind the action to provide meaningful contextual reference. For quantitative research, the action is study design, analysis, and interpretation — and the picture created in the planning stages of research hangs behind and provides structure and justification along the way. The actual backdrop is very simple (a colored number-line), but the process of developing it is surprisingly challenging — partly because researchers just haven’t been asked to so something like this before.

The following document outlines, and attempts to justify, steps in the process of creating a backdrop. It is a work in progress that I have decided to share publicly before it is “done” — and ultimately is meant to accompany a longer paper (that has been in progress for a few years) and helpful examples.