Post #28 is the version of this I will quote in future. One addition.
I would rather this thread reach "we do not know" about STEP 4 than reach a confident answer that nobody can support when asked.
15 likes 13mo
Evidence · Journal club · continued
Second pass at: Journal club: STEP 4 and what a withdrawal design can prove posts 31–60
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
d.bakker, post #26: The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to. Reading it back, the second half matters more than the first. Go to post
Two claims get bundled together under STEP 4 and they need separating. The descriptive one — this is what was observed — is usually well supported. The causal one — this is why — usually is not.
Almost every disagreement in threads like this one dissolves once you say which of the two you are making.
The most useful thing anyone has posted about STEP 4 in this category was a table of what had been measured and by whom. That is what I would want again.
3 likes 13mo
p.marchetti, post #25: Worth stating the null on STEP 4 before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one. Go to post
Where the STEP 4 reasoning breaks down for me is the step from the group result to the individual case. That step is almost never argued for.
n.laurent, post #33: Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows. Marking that as an opinion rather than a finding. Go to post
Useful. I have added it to my own notes with the date on it.
Collapsed as off-topic by two members at trust level 3 or above
The figures usually contain the finding and the text usually contains the interpretation. Separating them for the first twenty minutes is a discipline worth keeping.
The confident version of this sentence would be wrong, so here is the hedged one.
STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.
That is the version I use. It may not be the version that is correct.
29 likes 13mo
I read post #39 twice before replying, because I had assumed the opposite.
Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later.
21 likes 12mo
Distinguishing three things in the STEP 4 discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
9 likes 12mo
Collapsed as off-topic by two members at trust level 3 or above
An hour is enough for one paper read properly and not enough for two read partly. The temptation is always to add the second.
Written quickly, so the reasoning may be tighter than the wording.
s.chowdhury, post #19: This follows post #17 rather than contradicting it. Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later. Go to post
Adding thanks rather than a view. I do not have a view worth the space.
An honest declaration on STEP 4: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it.
22 likes 12mo
lyophil_margin, post #15: The useful distinction on STEP 4 is between what was measured and what was inferred from it. Both end up in the same sentence and only one of them has error bars. Go to post
Having read the whole STEP 4 thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.
lyophil_margin, post #15: The useful distinction on STEP 4 is between what was measured and what was inferred from it. Both end up in the same sentence and only one of them has error bars. Go to post
STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.
If this contradicts something upthread, the upthread version may well be the better one.
Post hidden by community flags
Staff rationale: Hidden by community flags. The claim about a named supplier was not accompanied by a batch, a date, a method or a document, which R6 requires.
STEP 4 has been discussed here with more heat than it deserves, mostly because two definitions have been in play the whole time.
The corresponding entry is in the public moderation log. Hidden posts are never deleted.
Building on post #49 rather than restating it.
An hour is enough for one paper read properly and not enough for two read partly. The temptation is always to add the second.
19 likes 12mo
v.sjoberg, post #35: Where the STEP 4 reasoning breaks down for me is the step from the group result to the individual case. That step is almost never argued for. Go to post
The question underneath STEP 4 is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it.
Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help.
I would put moderate confidence on the mainstream reading of STEP 4 and no more. That is not scepticism for its own sake; it is where the sourcing actually stops.
8 likes 11mo
The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.
Worth one more sentence than it usually gets.
26 likes 11mo
s.chowdhury, post #19: This follows post #17 rather than contradicting it. Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later. Go to post
Post #52 and I disagree about the size of the effect, not about the direction.
Critical appraisal template: (1) What did the trial set out to estimate? (2) Could the design answer that question? (3) Was the population sufficiently similar to your population to apply the results? (4) What was the absolute effect, not just the relative one? (5) What are the two strongest criticisms available?
impurity_table, post #32: Two claims get bundled together under STEP 4 and they need separating. The descriptive one — this is what was observed — is usually well supported. The causal one — this is why — usually is not. Almost every disagreement in threads like this one dissolves once you say which of the two you are making. Go to post
Bookmarking this. I will come back when I have something worth adding.
Where I have landed on STEP 4, having got it wrong once in public: the direction is clear, the magnitude is not, and anyone quoting a precise magnitude has borrowed it from somewhere that did not measure it.
12 likes 11mo
What I would want before treating STEP 4 as settled: the method, the sample, and whether anyone tried to find the opposite result. Two of the three are usually missing.
0 likes 11mo
Collapsed as off-topic by two members at trust level 3 or above
The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.
This has been discussed before and I could not find the thread, so, again.