Confounding by indication, explained with a concrete example

On the subject in the title: Confounding by indication, explained with a concrete example Working notes rather than a conclusion.

Comparing SURMOUNT-1 (N Engl J Med, 2022) with SURPASS-2 (N Engl J Med, 2021) and finding the comparison harder than it looks.

Different populations, different durations, different endpoints defined slightly differently, and in one case a different estimand. People compare the headline percentages anyway, including me until recently.

Is there a defensible way to put these side by side, or is the honest answer that there is not and we should stop?

Noted, and I have changed what I was going to do on the strength of it.

The arithmetic in the opening post is right; the assumption feeding it is the part to check.

The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.

Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.

Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.

If the premise is wrong, everything after it is decoration.

Where I part company with post #4, and it is a narrow parting.

Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.

The claim is narrower than it sounds, and deliberately so.

Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.

Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.

For what it is worth, the same held on the two occasions I checked.

Confirming post #6 from a second method, which matters more than confirming it from a second person.

Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.

Written quickly, so the reasoning may be tighter than the wording.

I had written a reply contradicting post #8 and deleted it. Here is what survived.

Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.

Coming back to post #7, because the follow-up matters more than the original answer.

Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.

That much is documented. The rest is how I have interpreted it.

I will take the caveat as seriously as the claim, which is the point of putting it there.

Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.

The conclusion is tentative; the arithmetic underneath it is not.

A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.

The step people skip is the one I have spelled out.