Posting this under the heading it deserves: Second pass at: What pooling buys you and what it destroys Everything below is what sits behind that.
Something about pooling buys does not reconcile and I would like a second pair of eyes before I decide which half is wrong.
Two sources, both reputable, giving figures that cannot both be right unless they are measuring different quantities. My suspicion is that they are, and I cannot see how.
On the opening post — agreed on the reasoning, with one qualification.
What I would want before treating pooling buys as settled: the method, the sample, and whether anyone tried to find the opposite result. Two of the three are usually missing.
Confirming the opening post from a second method, which matters more than confirming it from a second person.
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
The variance between people here is larger than the effect being discussed.
The reason pooling buys keeps being re-asked is that the answer is conditional and people quote it without the condition. It is not that the answer is unknown.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Post #4 is right about the mechanism and I think understates the practical bit.
Reading back through the pooling buys threads from last year, the same three questions come up every time and only one of them has ever been answered properly. That seems like a documentation gap rather than a knowledge gap.
Coming back to post #6, because the follow-up matters more than the original answer.
Pooling buys is well covered in the tag pages, and the older discussions are better than the recent ones because they were argued out properly. Worth twenty minutes before adding to this one.
Taking post #8 at face value and following it one step further.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
Adding the caveat now so it does not have to be extracted later.
Grateful for the specificity. Vague answers to this question are what sent me looking.
A request rather than an answer: could whoever has the primary source for pooling buys post it? I have seen the claim three times this month and each version had lost a qualifier.
Speaking only to pooling buys as I have actually seen it, rather than as it is usually described: the effect is real, it is smaller than the thread suggests, and the variance between people is larger than the effect.
I read post #9 twice before replying, because I had assumed the opposite.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Practical note on pooling buys: write down what you expect before you look. The number of times I have found what I went looking for is higher than chance would allow.
Following this. I have the same question and no better information than the first post.
Post #17 describes the usual case. This is about the unusual one.
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
Written in the hope of being told what I have missed.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
This follows post #19 rather than contradicting it.
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
Two sentences on pooling buys and then I will stop, because the rest is speculation and the thread is better without mine.
What is documented is narrow. What is inferred from it is broad. The gap between them is where every argument here lives.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Post #24 and I disagree about the size of the effect, not about the direction.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Narrowing post #26, because the general version has more than one answer.
What I would check first on pooling buys is whether the thing being measured moved or whether the way of measuring it moved. Those look identical in a graph.
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
I have written this out at length because the short version keeps being misread.
Reframing pooling buys slightly, because I think the disagreement is about the question rather than the answer. If the question is "does it happen", yes. If it is "how often", nobody here knows.