Post #4 put the caveat in the right place and I want to underline it. Checked the Criticising the method without criticising claim against the primary source this morning. It survives, with a narrower scope than the version quoted here. Posting the narrower scope.
On the subject in the title: Criticising the method without criticising the authors — one year on Working notes rather than a conclusion.
A methods question rather than a substantive one, about Criticising the method without criticising.
Everyone quotes the same figure and I cannot find anyone who says how it was arrived at. That is not an accusation; it usually means the derivation is somewhere obvious and I have missed it.
The opening post and I disagree about the size of the effect, not about the direction.
Multiple comparisons without adjustment is a defensible choice when declared in advance and a problem when discovered afterwards. The registry entry settles which happened.
I am confident about the direction and much less about the magnitude.
A critique should end with what would change your mind. Without that it is a position rather than an argument.
If that is already documented somewhere, ignore me and link it.
Post #4 put the caveat in the right place and I want to underline it.
Checked the Criticising the method without criticising claim against the primary source this morning. It survives, with a narrower scope than the version quoted here. Posting the narrower scope.
Defending a design that is being criticised unfairly is as useful as criticising one that deserves it, and it happens far less often.
I had written a reply contradicting post #8 and deleted it. Here is what survived.
Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.
I read post #7 twice before replying, because I had assumed the opposite.
Placebo-controlled versus active-controlled changes what a result means entirely, and comparing across the two is one of the most common errors in this subcategory.
I am not the right person to answer the follow-up to this.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
That is the version I use. It may not be the version that is correct.
Same experience here, different supplier, so it is at least not unique to one of them.
Narrowing post #14, because the general version has more than one answer.
Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.
That has held every time I have looked, which is not the same as always.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
I would be interested in a counterexample if anyone has one.
What I want from this Criticising the method without criticising thread is the list of things that would need to be true for the claim to hold. If we can write that list, we can check it.
Adding a small correction to the Criticising the method without criticising summary above rather than a disagreement with it. The substance holds; one of the figures is out by a factor that matters.
Post #20 is the version of this I will quote in future. One addition.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Selection into a registry and what it does to the estimate — a second dataset
Posting this under the heading it deserves: Selection into a registry and what it does to the estimate — a second dataset Everything below is what sits behind that. Posting a small dataset on Selection. It is…
|
+28 | 32 | 1.3k | 1d |
|
Coming back to: A critique that turned out to be unfair, retracted by its author
A critique that turned out to be unfair, retracted by its author — setting out what I have, and where I think it stops being reliable. Reading back through what has been written here about critique, three…
|
2 | 1.3k | 14mo | |
|
Confounding by indication, explained with a concrete example
On the subject in the title: Confounding by indication, explained with a concrete example Working notes rather than a conclusion. Comparing SURMOUNT-1 ( N Engl J Med , 2022) with SURPASS-2 ( N Engl J Med ,…
|
+21 | 25 | 21k | 11mo |
|
Immortal time bias in a claims-database study — one year on
Posting this under the heading it deserves: Immortal time bias in a claims-database study — one year on Everything below is what sits behind that. I would like to disagree carefully with the settled view on…
|
+41 | 46 | 3.7k | 22h |
|
Measurement error in a self-reported exposure
Posting this under the heading it deserves: Measurement error in a self-reported exposure Everything below is what sits behind that. The question about measurement error that I actually want answered is the…
|
+117 | 129 | 45k | 5mo |
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
The retatrutide phase 2 obesity paper, read closely
Posting this under the heading it deserves: The retatrutide phase 2 obesity paper, read closely Everything below is what sits behind that. Retatrutide phase 2 obesity paper, from the point of view of someone…
|
2 | 65k | 10mo | |
|
[2026 update] A report I filed that turned out to be wrong, and the correction
On the subject in the title: A report I filed that turned out to be wrong, and the correction Working notes rather than a conclusion. Structured report rather than an opinion, following the format the…
|
+15 | 19 | 18k | 2d |
|
Amylin analogue mechanism: satiety signalling separate from GLP-1
Amylin analogue mechanism: satiety signalling separate from GLP-1 — setting out what I have, and where I think it stops being reliable. Trying to work out what would count as evidence on amylin analogue…
|
+128 | 142 | 38k | 13mo |
|
The "under review" status pill and when a moderator applies it
Posting this under the heading it deserves: The "under review" status pill and when a moderator applies it Everything below is what sits behind that. A question about how this site should cite something that…
|
+96 | 119 | 17k | 3mo |
|
Selection into a registry and what it does to the estimate — a second dataset
Posting this under the heading it deserves: Selection into a registry and what it does to the estimate — a second dataset Everything below is what sits behind that. Posting a small dataset on Selection. It is…
|
+28 | 32 | 1.3k | 1d |