Confirming post #8 from a second method, which matters more than confirming it from a second person. What I can speak to on Selection is narrow, so I will keep it narrow rather than generalising from it. Beyond that boundary I do not know.
Posting this under the heading it deserves: Selection into a registry and what it does to the estimate — a second dataset Everything below is what sits behind that.
Posting a small dataset on Selection. It is mine, it is uncontrolled, and the method is stated so it can be discounted appropriately.
What I would like is not agreement but a second dataset collected by someone with no stake in mine. If one exists I would rather read it than argue for this one.
Genuine question rather than a rhetorical one: has anyone here actually observed Selection, as opposed to read about it? The thread is long and I cannot tell.
A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.
Worth separating two things that post #2 runs together.
I have no financial interest in anything named in this thread and I want to say so before I comment on Selection, because it is the sort of subject where it matters.
Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
Confirming post #8 from a second method, which matters more than confirming it from a second person.
What I can speak to on Selection is narrow, so I will keep it narrow rather than generalising from it. Beyond that boundary I do not know.
I had written a reply contradicting post #7 and deleted it. Here is what survived.
Since Selection keeps coming up, it should probably be a maintained page rather than a recurring thread. I am happy to draft it if someone with more direct experience will review it.
I had written a reply contradicting post #9 and deleted it. Here is what survived.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
The short version is the first sentence; the rest is why.
Confirming post #9 from a second method, which matters more than confirming it from a second person.
The version of Selection that circulates here is a simplification of a simplification. It is not wrong, but it has lost the conditions under which it holds, and those conditions are where the interesting cases live.
Choosing the worst interpretation: "The confidence interval includes a harmful effect" is true if the CI goes from -1 to +5. But assuming the worst-case scenario is not how you use the evidence. The point estimate and the precision both matter.
Post #14 is right about the mechanism and I think understates the practical bit.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
One more thing on Selection that took me far too long to see: the two figures people quote are not measuring the same quantity. Once you notice that, the apparent contradiction disappears.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
I have deliberately not rounded that, because the rounding is where the argument starts.
Sample size calculated on an optimistic effect estimate produces a trial that is underpowered for the effect that actually exists. That is visible in the protocol.
Taking post #18 at face value and following it one step further.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
Post #20 and I disagree about the size of the effect, not about the direction.
On Selection: the maintained page in the documentation commons covers the general case with citations and a review date, which is more reliable than any reply here including this one.
Multiple comparisons without adjustment is a defensible choice when declared in advance and a problem when discovered afterwards. The registry entry settles which happened.
Not the whole picture, but the part of it I can speak to.
Nobody has said the unglamorous part of Selection yet, so: most of the variation is explained by things that are boring to write about and easy to check.
My experience of Selection contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that.
Post #24 put the caveat in the right place and I want to underline it.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
The arithmetic in post #26 is right; the assumption feeding it is the part to check.
Reading back through the Selection threads from last year, the same three questions come up every time and only one of them has ever been answered properly. That seems like a documentation gap rather than a knowledge gap.
Selection is well covered in the tag pages, and the older discussions are better than the recent ones because they were argued out properly. Worth twenty minutes before adding to this one.
Choosing the worst interpretation: "The confidence interval includes a harmful effect" is true if the CI goes from -1 to +5. But assuming the worst-case scenario is not how you use the evidence. The point estimate and the precision both matter.
That is where I would start, not where I would stop.
Taking Selection seriously for a moment rather than deflecting: the honest position is that the community has observations and no controlled comparison, and those two things support very different sentences.