Reading a supplementary appendix and finding the interesting part

Reading a supplementary appendix and finding the interesting part — setting out what I have, and where I think it stops being reliable.

Collecting what is known about reading a supplementary appendix in one place, because it is currently spread across a category, two tag pages and a thread that is hard to find.

This is a summary rather than new work, and I have attributed each part to where I found it.

Two claims get bundled together under reading a supplementary appendix and they need separating. The descriptive one — this is what was observed — is usually well supported. The causal one — this is why — usually is not.

Almost every disagreement in threads like this one dissolves once you say which of the two you are making.

The arithmetic in post #34 is right; the assumption feeding it is the part to check.

Risk of bias: structured appraisal of internal validity. Key things to assess: randomisation method (was it truly random or could someone predict the next assignment), concealment (could randomisation be subverted), blinding (who was blinded and why or why not), completeness of outcome reporting.

Worth one more sentence than it usually gets.

A treatment-policy estimand asks what happens to people assigned to a strategy, including those who abandon it. A hypothetical estimand asks what would have happened had everyone continued. Both are legitimate and they give different numbers.

Adding the measurement that post #59 says would settle it.

Worth stating the null on reading a supplementary appendix before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one.

Something worth flagging about reading a supplementary appendix: the strongest-sounding claims in this thread are the ones with no source attached, which is the usual pattern and not a coincidence.

A treatment-policy estimand asks what happens to people assigned to a strategy, including those who abandon it. A hypothetical estimand asks what would have happened had everyone continued. Both are legitimate and they give different numbers.

That is the practical version. The rigorous version is longer and says the same thing.

Discontinuation handling is the methodological detail that most changes a result and gets the least attention. Read how missing data was imputed before reading the effect size.

The reasoning is more useful than the number, which is why I have shown it.

I read post #90 twice before replying, because I had assumed the opposite.

I changed my mind about reading a supplementary appendix after someone here asked me for the source and I could not produce one. That is worth saying out loud because it is the ordinary way it happens.