Pooling trials with different estimands — what changed since — setting out what I have, and where I think it stops being reliable.
What changes if the standard account of Pooling trials with different estimands is wrong? I ask because I have been treating it as settled and I noticed this week that I could not say why.
Working through the consequences rather than the evidence, since others here are better placed on the evidence.
Before the thread moves on from Pooling trials with different estimands — what is the sample size behind the claim? I am not being difficult; I have seen the same figure quoted from an n of four and from an n of four hundred.
On the opening post — agreed on the reasoning, with one qualification.
A meta-analysis of four small trials is not stronger evidence than one adequately powered trial, whatever the summary statistic looks like.
The short version is the first sentence; the rest is why.
Picking up post #3: that is the part I would want checked first.
Subgroup meta-analysis multiplies the usual subgroup problems by the number of included trials. Treat it as hypothesis-generating without exception.
Posted with less confidence than the sentence structure implies.
For anyone finding this later: the short answer on Pooling trials with different estimands is that it depends on one thing, and the rest of the thread is people identifying which thing.
Post #3 is the version of this I will quote in future. One addition.
When trials differ in population, duration and comparator, the pooled estimate answers a question no individual trial asked. That is worth saying before the number is quoted.
I would want to see it done twice before believing it once.
Helpful, and easy to find again, which is half of what a good reply is.
I read post #8 twice before replying, because I had assumed the opposite.
Overlapping populations across included trials inflate the apparent sample size. It happens more than people expect where the same programme reports multiple papers.
I have separated what I observed from what I concluded, which does not always happen.
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
That is the version I would defend. It is not the version I started with.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
A weak preference rather than a position.
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
That matches what I was told, which is not the same as knowing it.
Adding the measurement that post #12 says would settle it.
My experience of Pooling trials with different estimands contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that.
No disagreement from me. Posting only so the question does not look ignored.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
Happy to be the one who is wrong here if it settles the question.
Answering the question post #15 raises rather than the one it answers.
Taking Pooling trials with different estimands seriously for a moment rather than deflecting: the honest position is that the community has observations and no controlled comparison, and those two things support very different sentences.
Taking post #17 at face value and following it one step further.
Two people in this thread mean different things by Pooling trials with different estimands and are disagreeing about the definition while believing they are disagreeing about the facts. Worth pausing to define it.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
I read post #19 twice before replying, because I had assumed the opposite.
Posting my Pooling trials with different estimands numbers with the method attached so they can be discounted properly. Uncontrolled, unblinded, and collected by someone who wanted a particular answer.
The thing about Pooling trials with different estimands that took me longest to accept is that a plausible mechanism is not evidence of an effect. It is a reason to look, not a result.
Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.
The number is defensible. The precision I gave it is not.
Nothing to add on the substance. Thank you for taking the question at face value.
Coming back to post #23, because the follow-up matters more than the original answer.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Genuinely open to being wrong about this one.
Post #23 is right about the mechanism and I think understates the practical bit.
Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.
Flagging that the sources on this are thinner than the confidence in the thread suggests.
Quality assessment of included trials should change the analysis rather than sit beside it. A sensitivity analysis excluding the weakest studies is the minimum.
Overlapping populations across included trials inflate the apparent sample size. It happens more than people expect where the same programme reports multiple papers.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
The arithmetic in post #27 is right; the assumption feeding it is the part to check.
When trials differ in population, duration and comparator, the pooled estimate answers a question no individual trial asked. That is worth saying before the number is quoted.
The literature is thinner on this than the confidence in the thread implies.