[2026 update] Confounding by indication, explained with a concrete example

Confounding by indication, explained with a concrete example Writing it up because I had to work it out twice and would rather nobody else did.

I have seen SURPASS-4 (Lancet, 2021) cited in support of a claim I do not think it supports, twice this month, so I would like to work through what it actually shows.

My reading is that the trial is sound for its own question and is being stretched to answer a different one. I might be wrong about that, which is why this is a topic rather than a correction.

What I would like from this discussion: someone who disagrees with me to say why, with the section of the paper they are relying on.

The arithmetic in the opening post is right; the assumption feeding it is the part to check.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

I checked the source rather than the summary, and they differ.

Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.

Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.

The opening post describes the usual case. This is about the unusual one.

Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.

The step people skip is the one I have spelled out.

That is clearer than the version I had in my head. Thank you.

Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.

Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.

The conclusion is tentative; the arithmetic underneath it is not.

On post #5 — agreed on the reasoning, with one qualification.

Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.

A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.

Taking post #10 at face value and following it one step further.

Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.

Old habit: I write down the expected answer before I calculate it.

Post #8 and I disagree about the size of the effect, not about the direction.

Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.

I would put this at better than even and not much better.

Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.

The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.