Journal club: STEP 4 and what a withdrawal design can prove — one year on

Journal club: STEP 4 and what a withdrawal design can prove — one year on Writing it up because I had to work it out twice and would rather nobody else did.

Session topic: STEP 4 (JAMA, 2021). Please read it before posting; the discussion is much better when everyone has.

The question I would like us to start with is what the trial set out to estimate, rather than what it found. Once that is on the table we can talk about whether the design could have answered it, and only then about the numbers.

Specific things I would like covered: the population and how far it generalises, how discontinuation was handled, whether the comparator was a fair one, and what the absolute rather than relative effect looks like.

I will summarise at the end and the summary will feed the relevant digest page.

The arithmetic in the opening post is right; the assumption feeding it is the part to check.

The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.

A conflict-of-interest statement is worth reading aloud and is worth very little as a predictor. Design quality is what to spend the session on.

I have kept the units in throughout, for the obvious reason.

Appreciated. The plain phrasing does more work here than a longer post would.

Adding a small correction to the STEP 4 summary above rather than a disagreement with it. The substance holds; one of the figures is out by a factor that matters.

STEP 4 is a good example of a question where the honest answer is boring and the interesting answers are unsupported. I would go with boring.

Post #5 and I disagree about the size of the effect, not about the direction.

Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows.

STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.

Somebody will have a better source than mine, and I hope they post it.

I read post #5 twice before replying, because I had assumed the opposite.

A definition problem is doing most of the work in this STEP 4 discussion. Once the term is pinned down I suspect the disagreement mostly goes away and what is left is small.

I would be cautious about generalising from the STEP 4 example above. It is a good example. It is one example.

Critical appraisal template: (1) What did the trial set out to estimate? (2) Could the design answer that question? (3) Was the population sufficiently similar to your population to apply the results? (4) What was the absolute effect, not just the relative one? (5) What are the two strongest criticisms available?

Second-hand, so weight it accordingly.

STEP 4 is well covered in the tag pages, and the older discussions are better than the recent ones because they were argued out properly. Worth twenty minutes before adding to this one.

Post #10 is the version of this I will quote in future. One addition.

The reason STEP 4 keeps being re-asked is that the answer is conditional and people quote it without the condition. It is not that the answer is unknown.

Same experience here, different supplier, so it is at least not unique to one of them.