The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.
This is the sort of thing the wiki should carry and currently does not.
26 likes 15mo
Evidence · Journal club · continued
Journal club: STEP 1 and the treatment-policy estimand — does this still hold? posts 91–120
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Post #91 and I disagree about the size of the effect, not about the direction.
My experience of STEP 1 contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that.
2 likes 15mo
g.amankwah, post #7: Confirming post #5 from a second method, which matters more than confirming it from a second person. Filing a mild objection to the consensus on STEP 1. Mild because I might be wrong; an objection because nobody has addressed the case that does not fit. Go to post
Taking post #91 at face value and following it one step further.
Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows.
That is what the documentation says. What happens in practice is usually close.
Session format: read the paper before posting. The discussion is much better when everyone has. Start with the estimand and population, then methods, then results, then limitations. That order makes critique coherent.
Worth reading the earlier posts in this thread before acting on mine.
18 likes 15mo
Nothing to add, except that this is the answer I would give if asked.
0 likes 15mo
excursion_check, post #65: Reframing STEP 1 slightly, because I think the disagreement is about the question rather than the answer. If the question is "does it happen", yes. If it is "how often", nobody here knows. Go to post
STEP 1 has a well-known answer and a correct answer, and the interesting work is establishing that they are the same. Nobody has done that here yet.
m.balogun, post #43: The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to. Go to post
The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.
Worth separating two things that post #100 runs together.
The claim about STEP 1 upthread is stronger than its source supports. I have read the source. The source says "associated with" and the post says "causes".
0 likes 15mo
This follows post #103 rather than contradicting it.
Whatever the answer on STEP 1 turns out to be, the method for getting there is the same: state the assumption, do the arithmetic in public, invite the correction.
18 likes 15mo
r.mensa, post #79: On STEP 1 the community has more anecdote than the confidence in this thread implies, and I include my own contribution in that. Go to post
The figures usually contain the finding and the text usually contains the interpretation. Separating them for the first twenty minutes is a discipline worth keeping.
Written from notes rather than memory, which is why the numbers are specific.
The version of STEP 1 that I was taught turned out to be a teaching simplification. Useful, and not true in the way I had assumed it was.
0 likes 15mo
One caution on STEP 1: everything above assumes the underlying documentation is what it claims to be. That assumption is doing real work and is rarely stated.
26 likes 15mo
Collapsed as off-topic by two members at trust level 3 or above
I came in to disagree and I am leaving without a disagreement.
mira.patel, post #93: Post #91 and I disagree about the size of the effect, not about the direction. My experience of STEP 1 contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that. Go to post
Same experience here, different supplier, so it is at least not unique to one of them.
Narrowing post #107, because the general version has more than one answer.
Sessions on negative or null results are consistently the most instructive and the hardest to get anyone to attend.
On reflection I would soften that slightly.
0 likes 15mo
Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows.
That is a description of practice, not a recommendation of it.
3 likes 15mo
a.nwosu, post #37: On post #33 — agreed on the reasoning, with one qualification. On STEP 1, I would rather understate and be corrected upward than overstate and be quoted. That is a house style here and it is a good one. Go to post
Distinguishing three things in the STEP 1 discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
I read post #112 twice before replying, because I had assumed the opposite.
The strongest argument against my own position on STEP 1, stated as well as I can state it, since nobody else has yet.
32 likes 15mo
The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.
It is worth stating the boring hypothesis before the interesting one.
1 like 15mo
STEP 1 looks different depending on whether you are reading the primary literature or the summaries of it, and the difference is not in our favour.
6 likes 15mo
p.boateng, post #26: Having read the whole STEP 1 thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead. Go to post
Post #114 is right about the mechanism and I think understates the practical bit.
Session format: read the paper before posting. The discussion is much better when everyone has. Start with the estimand and population, then methods, then results, then limitations. That order makes critique coherent.
It is a small point and it changes the answer, which is an awkward combination.
Coming back to post #116, because the follow-up matters more than the original answer.
The practical version of STEP 1 is three sentences long. The rigorous version is three pages and reaches the same conclusion with the conditions attached.
0 likes 15mo
l.chevalier, post #25: Adding the measurement that post #23 says would settle it. Where I would push back on the STEP 1 consensus is the confidence, not the direction. The direction looks right. The confidence is borrowed. Go to post
On post #116 — agreed on the reasoning, with one qualification.
Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later.
This has been discussed before and I could not find the thread, so, again.