Journal club: STEP 1 and the treatment-policy estimand — does this still hold? posts 121–133

I had written a reply contradicting post #117 and deleted it. Here is what survived.

Small correction to my own earlier position on STEP 1. I had the units the wrong way round, which changes the conclusion by an order of magnitude and therefore changes it entirely.

Confirming post #121 from a second method, which matters more than confirming it from a second person.

Where I have landed on STEP 1, having got it wrong once in public: the direction is clear, the magnitude is not, and anyone quoting a precise magnitude has borrowed it from somewhere that did not measure it.

Adding the boring version of STEP 1, because the interesting version keeps getting posted and the boring one is usually right.

Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does.

Coming back to post #121, because the follow-up matters more than the original answer.

On STEP 1: the maintained page in the documentation commons covers the general case with citations and a review date, which is more reliable than any reply here including this one.

The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.

Worth saying I have only my own numbers here, and n is small.

The question underneath STEP 1 is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it.

Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help.

Saving this. It is the version I will quote when the question comes round again.

If someone has run STEP 1 properly I would rather read that than my own reconstruction of it. Posting mine only because the thread has gone quiet.

STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.

For what it is worth, the same held on the two occasions I checked.

STEP 1 would be much easier to settle if anyone reported the denominator. Almost nobody reports the denominator.

Post #130 put the caveat in the right place and I want to underline it.

I read the earlier replies on STEP 1 twice before writing this, because I had assumed the opposite and wanted to be sure I was disagreeing with what was said rather than what I expected.

Session format: read the paper before posting. The discussion is much better when everyone has. Start with the estimand and population, then methods, then results, then limitations. That order makes critique coherent.

Not a conclusion. A place to stand while looking for one.