A well-designed study with a badly written abstract posts 61–90
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Everything in post #61 holds. The case it does not cover is the one I have.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
I keep a log of this specifically because memory is unreliable about it.
Fine by me. I had wanted a stronger conclusion and there is not one available.
The confident answers on well-designed study and the well-sourced answers are not the same answers, which is the most useful thing I have learned reading this category.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Post #66 is the version of this I will quote in future. One addition.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Written in the hope of being told what I have missed.
Where I part company with post #64, and it is a narrow parting.
Well-designed study was covered in the wiki last year and the page has a review date on it, which is a better starting point than my memory of a thread.
Second this, and I would have said it less carefully.
Post #67 describes the usual case. This is about the unusual one.
Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.
The reasoning is more useful than the number, which is why I have shown it.
Genuine question rather than a rhetorical one: has anyone here actually observed well-designed study, as opposed to read about it? The thread is long and I cannot tell.
Collapsed as off-topic by two members at trust level 3 or above
On well-designed study the community has more anecdote than the confidence in this thread implies, and I include my own contribution in that.
One more thing on well-designed study that took me far too long to see: the two figures people quote are not measuring the same quantity. Once you notice that, the apparent contradiction disappears.
Well-designed study is a question about a distribution, not about a value, and treating it as a value is what produces the confident wrong answers.
Confirming post #75 from a second method, which matters more than confirming it from a second person.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
Where I would look next, rather than where I would stop.
The most useful reply I ever got about well-designed study was a request to state my units. It sounds like pedantry and it has saved me twice.
I came in to disagree and I am leaving without a disagreement.
I had written a reply contradicting post #82 and deleted it. Here is what survived.
Two people in this thread mean different things by well-designed study and are disagreeing about the definition while believing they are disagreeing about the facts. Worth pausing to define it.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
The literature is thinner on this than the confidence in the thread implies.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
I would rather be precise about what I do not know than vague about what I do.
Post #84 is right about the mechanism and I think understates the practical bit.
Adding a reference point for well-designed study. Mine is a single case, collected without controls, and I am posting the method alongside it so it can be discounted appropriately.
Collapsed as off-topic by two members at trust level 3 or above
Coming back to post #86, because the follow-up matters more than the original answer.
The failure mode on well-designed study is boring rather than dramatic. It is almost always the step everyone assumes was done correctly because it is too simple to get wrong.
Answering the question post #86 raises rather than the one it answers.
The reason well-designed study keeps being re-asked is that the answer is conditional and people quote it without the condition. It is not that the answer is unknown.