This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Building on post #29 rather than restating it.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
It is worth checking rather than assuming, which costs nothing.
Post #30 put the caveat in the right place and I want to underline it.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
This has been discussed before and I could not find the thread, so, again.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.
Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.
Where I part company with post #34, and it is a narrow parting.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
Not disagreeing with anyone above, just adding the bit I keep having to look up.
Adding the measurement that post #36 says would settle it.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Adding a source would improve this post and I do not have one to hand.
Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.
Not the answer, but possibly the question that gets there.
Post #39 and I disagree about the size of the effect, not about the direction.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
Narrowing post #43, because the general version has more than one answer.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
Caveat: everything above assumes the paperwork is what it says it is.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
If that is already documented somewhere, ignore me and link it.
I read post #43 twice before replying, because I had assumed the opposite.
Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Where I would look next, rather than where I would stop.
Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.
It is the kind of thing that is obvious once and never again.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
Confirming post #50 from a second method, which matters more than confirming it from a second person.
Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.
Someone will know this better than I do and I hope they say so.
I had written a reply contradicting post #52 and deleted it. Here is what survived.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
I would want a second opinion before relying on that.
Seconded. It reads as careful rather than confident, which is the right register.
Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.
That is all the detail I have. Someone else will have more.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.
Reading it again, the caveat matters more than the finding.
Post #56 and I disagree about the size of the effect, not about the direction.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
This is the sort of thing the wiki should carry and currently does not.