Where the Pooling trials with different estimands discussion usually stalls is that nobody wants to say "I do not know" and everyone is willing to say "it varies". Those are the same sentence with different clothes on.
16 likes 10mo
Evidence · Meta-analyses · continued
Pooling trials with different estimands — what changed since posts 31–60
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
I had written a reply contradicting post #28 and deleted it. Here is what survived.
If you are new and reading this thread for the answer to Pooling trials with different estimands: the answer is conditional, the conditions are in the third reply, and the rest of the thread is worth skipping.
32 likes 10mo
s.lindqvist, post #9: I read post #8 twice before replying, because I had assumed the opposite. Overlapping populations across included trials inflate the apparent sample size. It happens more than people expect where the same programme reports multiple papers. I have separated what I observed from what I concluded, which does not always happen. Go to post
Fixed-effect and random-effects models answer different questions. The first assumes one true effect; the second assumes a distribution of them. Choosing between them is an assumption, not a technicality.
That is what I would do. It may not be what is correct.
Fair, and the limits you put on it are the part I will remember.
23 likes 10mo
Worth stating the null on Pooling trials with different estimands before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one.
0 likes 10mo
j.delacroix, post #7: Helpful, and easy to find again, which is half of what a good reply is. Go to post
Picking up post #34: that is the part I would want checked first.
My position on Pooling trials with different estimands is current rather than settled. I have revised it once already and I expect to again, so treat it accordingly.
j.petrov, post #15: Fixed-effect and random-effects models answer different questions. The first assumes one true effect; the second assumes a distribution of them. Choosing between them is an assumption, not a technicality. That is where I would start, not where I would stop. Go to post
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
I have changed my mind on this once already, so take it as current rather than settled.
Post #38 answers the question as asked. The question underneath it is different.
Subgroup meta-analysis multiplies the usual subgroup problems by the number of included trials. Treat it as hypothesis-generating without exception.
The evidence for this is thinner than the way I have phrased it suggests.
7 likes 9mo
Post #37 describes the usual case. This is about the unusual one.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
24 likes 9mo
Adding the measurement that post #41 says would settle it.
Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.
11 likes 9mo
f.kimani, post #17: Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified. Adding the caveat now so it does not have to be extracted later. Go to post
Two questions I would want answered before drawing anything from the Pooling trials with different estimands data above: how were the cases selected, and what happened to the ones that dropped out.
Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.
Posting it because the silence on this was starting to look like agreement.
18 likes 9mo
Adding a note of thanks rather than an opinion. I did not know most of that.
7 likes 9mo
c.dahlberg, post #13: Adding the measurement that post #12 says would settle it. My experience of Pooling trials with different estimands contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that. Go to post
I had written a reply contradicting post #45 and deleted it. Here is what survived.
Overlapping populations across included trials inflate the apparent sample size. It happens more than people expect where the same programme reports multiple papers.
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
0 likes 9mo
endo_fellow_rk, post #25: Coming back to post #23, because the follow-up matters more than the original answer. Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified. Genuinely open to being wrong about this one. Go to post
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
The short answer was in the first line; everything after is the working.
Narrowing post #48, because the general version has more than one answer.
Practical note on Pooling trials with different estimands: write down what you expect before you look. The number of times I have found what I went looking for is higher than chance would allow.
0 likes 8mo
This is the answer, and the reason it is the answer is the more useful part.
0 likes 8mo
j.delacroix, post #7: Helpful, and easy to find again, which is half of what a good reply is. Go to post
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Correct me on the arithmetic if it is wrong; I would rather know.
I disagree with the framing of Pooling trials with different estimands above, and I think it is a substantive disagreement rather than a terminological one. Setting out why, so it can be checked.
The reasoning depends on an assumption that is doing a lot of work and is never stated. If the assumption holds, the conclusion follows. I do not think it holds generally.
12 likes 8mo
The arithmetic in post #53 is right; the assumption feeding it is the part to check.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
The claim is narrower than it sounds, and deliberately so.
26 likes 8mo
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
Adding it because I spent an afternoon working it out and nobody should have to twice.
0 likes 8mo
j.baptista, post #14: No disagreement from me. Posting only so the question does not look ignored. Go to post
What would change my mind on Pooling trials with different estimands is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more.