Adding what did not work for me on pooling buys, since the failures never get written up and they are half the useful information.
0 likes 10mo
Evidence · Meta-analyses · continued
What pooling buys you and what it destroys posts 31–60
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1 · go to the accepted answer.
dietitian_hollis, post #3: That matches what I have seen, for whatever a single anecdote is worth. Go to post
The claim about pooling buys upthread is stronger than its source supports. I have read the source. The source says "associated with" and the post says "causes".
Taking post #30 at face value and following it one step further.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
I would rather post the uncertainty than round it away.
19 likes 10mo
Something worth flagging about pooling buys: the strongest-sounding claims in this thread are the ones with no source attached, which is the usual pattern and not a coincidence.
2 likes 10mo
bias_variance, post #35: Something worth flagging about pooling buys: the strongest-sounding claims in this thread are the ones with no source attached, which is the usual pattern and not a coincidence. Go to post
This is the first time the answer has come with its own limits attached. Appreciated.
s.cabrera, post #8: The question underneath pooling buys is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it. Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help. Go to post
Building on post #34 rather than restating it.
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
If anyone has run this properly I would rather read that than my own guess.
j.vandermolen, post #25: Answering the question post #23 raises rather than the one it answers. Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random. Go to post
Post #38 is right about the mechanism and I think understates the practical bit.
I would be cautious about generalising from the pooling buys example above. It is a good example. It is one example.
Worth separating two things that post #37 runs together.
An honest declaration on pooling buys: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it.
0 likes 9mo
Having read the whole pooling buys thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.
21 likes 9mo
n.norgaard, post #24: Confirming post #23 from a second method, which matters more than confirming it from a second person. If someone has run pooling buys properly I would rather read that than my own reconstruction of it. Posting mine only because the thread has gone quiet. Go to post
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
On reflection I would soften that slightly.
Collapsed as off-topic by two members at trust level 3 or above
Sensible. I would want the same detail before I acted on it either.
Post #41 and I disagree about the size of the effect, not about the direction.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
I looked this up rather than remembered it, which is the right order.
30 likes 9mo
Taking post #45 at face value and following it one step further.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
15 likes 9mo
d.moreau, post #12: Saving this. It is the version I will quote when the question comes round again. Go to post
Distinguishing three things in the pooling buys discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
nl_translator, post #16: Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random. Go to post
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Take the reasoning and check the arithmetic; I do not always get it right.
What I would tell a new member reading about pooling buys for the first time: the confident posts are not the reliable ones, and the reliable ones are longer.
0 likes 9mo
One more thing on pooling buys that took me far too long to see: the two figures people quote are not measuring the same quantity. Once you notice that, the apparent contradiction disappears.
0 likes 9mo
impurity_table, post #42: Having read the whole pooling buys thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead. Go to post
Post #52 is right about the mechanism and I think understates the practical bit.
Pooling buys looks different depending on whether you are reading the primary literature or the summaries of it, and the difference is not in our favour.
Worth separating two things that post #54 runs together.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
I would treat the number as indicative rather than as a measurement.
1 like 9mo
a.amankwah, post #51: One more thing on pooling buys that took me far too long to see: the two figures people quote are not measuring the same quantity. Once you notice that, the apparent contradiction disappears. Go to post
Two questions I would want answered before drawing anything from the pooling buys data above: how were the cases selected, and what happened to the ones that dropped out.
s.cabrera, post #8: The question underneath pooling buys is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it. Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help. Go to post
On pooling buys the community has more anecdote than the confidence in this thread implies, and I include my own contribution in that.
nl_translator, post #16: Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random. Go to post
Careful with the language on pooling buys. "Not detected" and "not present" are different findings and the first is a statement about the method.