Thieving from the Assumption Market

This post grew unexpectedly from other drafts. I found myself vaguely referencing an analogy around the cost and purchase of assumptions. I decided it was worth exploring a little more on its own, and for reference in other posts — it ended up being more fun than I expected!

The heart of what I’m after is more awareness and discussion about how we justify our inferences — and any models (in a broad sense) propping them up. This post relates to thoughts I recently shared on the cost of assumptions here.

Envisioning the Assumption Market

Let’s suppose we have a huge store, filled with shelves of neatly stacked and organized assumptions needed to support scientific inferences. Some of the shelves are neatly stacked with shiny mathematical assumptions (the ones we’re most used to talking about — normality, linearity, independence, etc.) and some shelves are messy and dusty from lack of turnover (the ones we don’t talk about much — like assumptions about what is possible to measure, that averages are the most meaningful quantity, what we must ignore to rely on probability models in general, etc.).

The shop owners fill the shelves and aisle ends at the front of the store with the shiny, accessible mathematical assumptions — they disappear quickly and there’s constant demand. The messier and less accessible assumptions are in the very back aisles, and maybe just downstairs in overflow where few customers actually wander. Customers typically run in in a hurry, grab their shiny assumptions from the shelf (usually the popular ones on sale), and head toward the door (hopefully stopping to pay on the way out!)

The pricing challenge

Now, the hard part. What is an assumption worth? How is it priced? What currency should we even use to buy it?

As I sort-of communicated here, assumptions add information into a statistical analysis, and therefore into inferences. There is a tendency to maintain laser focus on the information in the data — while ignoring how much information is added through assumptions. It would be wonderful if we could easily quantify the information in a particular assumption using a data-like currency, but we don’t currently have the ability to do that for most assumptions and I’m not optimistic we ever will, particularly for the assumptions in the back of the store collecting dust. And, even if we were able to come up with a reasonable method to quantify an assumption’s information content relative to data (through some sort of sensitivity analysis) — and thus price them for the Market — that method would have its own assumptions. So … where would it ever end?

I accept that defining currency through such quantification will not be a practical solution, at least one that is broad enough to cover the range of assumptions we should be considering. But, this doesn’t mean the concept isn’t still useful in a more qualitative way. Yeah, it’s messy and difficult and won’t be satisfying to those used to shopping from just the front of the market, but it may be far better than how we’re currently operating.

What am I suggesting practically? Well, I’m still working on this, but here are some thoughts. We can create a list of all the assumptions we are aware that we are making that have some influence on the scientific inference or decision. The list will be long and making it will be tedious and uncomfortable. Then, we need to think through how each one might affect our inferences and make judgements about relative size of the influence. How much information might it be adding to inferences beyond (and relative to) that contained the data? Reorder the list in terms of increasing perceived impact on inferences (or sensitivity of inferences to the assumption). This is then the list of assumptions from cheapest to most expensive — even if a specific price can’t be quantified. Using the information about relative cost, how can we purchase them?

Taking out a loan

How do we actually make a purchase if we don’t really have a currency? Here’s how I envision it. We are all cash poor and need to take out a loan to buy any assumption. The Assumption Market is also a lender — by necessity. In the real world, to secure a loan we need to justify to the lender that we will be able to pay it back. The lender has no way of knowing for sure if we will be good for it — so their decision to lend depends on how well we can convince them that we will be able to pay it back. Ultimately, the transaction proceeds on good faith — but the better the justification that the loan will be repaid in a timely manner, the better the chances of getting that loan. So, the hidden currency is the strength of the justification. The bigger the loan, the more documentation and support is needed — the more convincing the justification must be.

I see purchasing of assumptions as completely analogous to this. The more costly the assumption, the more justification should be required to get the loan to purchase that assumption. Lacking the cash currency to buy assumptions, the default currency to get loans should be justification. The more expensive the assumption, the more extensive and convincing the justification should have to be to get the loan. Unfortunately, in real life, the most expensive assumptions are often the ones with the least expected justification — those at the back of the store that are rarely even purchased. That’s a door to through at a later time, but it does lead me to the next section…

A culture of thieves

In current practice, the amount of justification required to buy assumptions is minimal and often non-existent. We treat assumptions as if they are readily available in the free bin — but they are never free. There is always a cost. If assumptions are relied upon with no justification, they are stolen from the market (gasp!). We have created a culture based on stealing from the Assumption Market. Common statistical (and more generally scientific) practice makes us a bunch of thieves. [Note — listing the shiny assumptions and stating that they are “met” does not count as justification (more here in an old post).]

I don’t suppose anyone is trying to steal from the market or wants to be thief. It’s just the accepted way of doing business. We keep teaching how to sneak into the market, grab what we need, and sneak out — no stopping at the check out, no checking the cost, no applying for the loan — or even teaching students how to apply for a loan. It’s like there’s someone passing out invisibility cloaks to customers as they come in the door.

How to raise a down payment

There is one other important piece to add to the analogy. Loans should be easier to get if one has a down payment and the same should go for buying assumptions. For scientific inferences, down payments can be raised through efforts put into study design. The same assumption requires a much smaller loan for the customer who brings in a large down payment — earned through care in the design. For example, use of random assignment in the design provides enough down payment to easily purchase a probability model based on a randomization distribution. It also puts the buyer well on their way to purchasing the assumptions needed for causal inferences. Random sampling is the other obvious design decision leading to a large down payment.

Let’s come by our assumptions honestly

We could improve scientific inferences (and associated decisions) by coming by our assumptions more honestly. The thieving, and culture of handing out invisibility cloaks, should stop. It will not make life easier, but so it goes. It’s time to acknowledge that assumptions are not free and time to build a system of accountability around their purchase. This might go a long way toward increasing humility around our methods, our models, and most importantly … our inferences.