Critiquing an observational claim about a class effect — one year on

Critiquing an observational claim about a class effect — one year on — setting out what I have, and where I think it stops being reliable.

I was wrong about Critiquing an observational claim in a thread last spring and I would like to correct it publicly rather than quietly.

The error was in the units, which changed the conclusion by an order of magnitude. Setting out the corrected version, and the way I now check for that class of mistake.

Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.

Not disagreeing with anyone above, just adding the bit I keep having to look up.

Sample size calculated on an optimistic effect estimate produces a trial that is underpowered for the effect that actually exists. That is visible in the protocol.

The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.

Everything in post #3 holds. The case it does not cover is the one I have.

Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.

It is one reading of the data and not the only reasonable one.

Narrowing post #3, because the general version has more than one answer.

Placebo-controlled versus active-controlled changes what a result means entirely, and comparing across the two is one of the most common errors in this subcategory.

That is a description of practice, not a recommendation of it.

Defending a design that is being criticised unfairly is as useful as criticising one that deserves it, and it happens far less often.

I have kept the units in throughout, for the obvious reason.

Reporting bias within a paper — a result in a figure that is absent from the abstract — is checkable and is worth checking when the claim matters.

Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.

Adding a data point of agreement rather than a data point.