Second pass at: Journal club: STEP 4 and what a withdrawal design can prove Writing it up because I had to work it out twice and would rather nobody else did.
The question about STEP 4 that I actually want answered is the second one below. The first is context and I have kept it short.
Both are stated with units, and I have said what I already checked so that nobody repeats it.
Confirming the opening post from a second method, which matters more than confirming it from a second person.
Two sentences on STEP 4 and then I will stop, because the rest is speculation and the thread is better without mine.
What is documented is narrow. What is inferred from it is broad. The gap between them is where every argument here lives.
A definition problem is doing most of the work in this STEP 4 discussion. Once the term is pinned down I suspect the disagreement mostly goes away and what is left is small.
Filing a mild objection to the consensus on STEP 4. Mild because I might be wrong; an objection because nobody has addressed the case that does not fit.
This settles it for me, at least until somebody posts a reason it should not.
Critical appraisal template: (1) What did the trial set out to estimate? (2) Could the design answer that question? (3) Was the population sufficiently similar to your population to apply the results? (4) What was the absolute effect, not just the relative one? (5) What are the two strongest criticisms available?
That is a description of practice, not a recommendation of it.
The documentation on STEP 4 is better than this thread and I say that as someone who has posted in the thread.
On STEP 4, the part that usually goes wrong is that the question is asked as though it has one answer. It has a range, and the width of the range is the interesting bit.
If you can post the two or three numbers you are working from, several people here will check the arithmetic rather than argue about the conclusion.
This is the first time the answer has come with its own limits attached. Appreciated.
Narrowing post #10, because the general version has more than one answer.
The figures usually contain the finding and the text usually contains the interpretation. Separating them for the first twenty minutes is a discipline worth keeping.
What would change my mind on STEP 4 is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more.
The useful distinction on STEP 4 is between what was measured and what was inferred from it. Both end up in the same sentence and only one of them has error bars.
A request rather than an answer: could whoever has the primary source for STEP 4 post it? I have seen the claim three times this month and each version had lost a qualifier.
Source for the STEP 4 figure, since it was asked for. It is in the discussion rather than the abstract, which is why the version circulating is stronger than the paper is.
Reading the surrounding paragraph is worth the two minutes. The authors are more careful than their summarisers.
This follows post #17 rather than contradicting it.
Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later.
Worth separating two things that post #16 runs together.
Adding a null result on STEP 4. I looked, carefully, and found nothing, and null results deserve posting precisely because they never are.
Post #21 is right about the mechanism and I think understates the practical bit.
The figures usually contain the finding and the text usually contains the interpretation. Separating them for the first twenty minutes is a discipline worth keeping.
Helpful, and easy to find again, which is half of what a good reply is.
I read the earlier replies on STEP 4 twice before writing this, because I had assumed the opposite and wanted to be sure I was disagreeing with what was said rather than what I expected.
Worth stating the null on STEP 4 before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one.
Post #25 describes the usual case. This is about the unusual one.
Sessions on negative or null results are consistently the most instructive and the hardest to get anyone to attend.
Everything in post #25 holds. The case it does not cover is the one I have.
If you are new and reading this thread for the answer to STEP 4: the answer is conditional, the conditions are in the third reply, and the rest of the thread is worth skipping.