The Pandora Principle is that once you’ve considered a possible interaction or bias or confounder, you can’t un-think it. The malign converse is when people realize this and then design their studies to avoid putting themselves in a position where they have to consider some potentially important factor. For example, suppose you’re considering some policy intervention that can be done in several different ways, or conducted in several different contexts. The recommended approach is, if possible, to try out different realistic versions of the treatments in various realistic scenarios; you can then estimate an average treatment effect and also do your best to estimate variation in the effect (recognizing the difficulties inherent in that famous 1/16 efficiency ratio). An alternative, which one might call the reverse-Pandora approach, is to do a large study with just a single precise version of the treatment. This can give a cleaner estimate of the effect in that particular scenario, but to extend it to the real world will require some modeling or assumption about how the effect might vary. Going full ostrich here, one could simply carry over the estimated treatment effect from the simple experiment and not consider any variation at all. The idea would be that if you’d considered two or more flavors of treatment, you’re really have to consider the possibility of variation in effect, and propagate that into your decision making. But if you only consider one possibility, you could ostrich it and keep Pandora at bay. The ostrich approach might get you a publication and even some policy inference but it’s bad science and, I think, bad policy.
That said, there’s no easy answer, as there will always be additional possible confounding factors that you will not have be able to explore. That is, among all the scary contents of Pandora’s box, one thing that flies out is another box, and really you should open that one too . . . that’s the Cantor principle, which we encounter in so many places in statistics.
tl;dr: You can’t put Pandora back in the box. But really she shouldn’t’ve been trapped in there in the first place.
The post The Pandora Principle in statistics — and its malign converse, the ostrich appeared first on Statistical Modeling, Causal Inference, and Social Science.