Search strategy determines what was found and is the part of the method least often scrutinised. A search restricted to one language or one database is a selection step.
A pooled estimate that changed when one trial was added — setting out what I have, and where I think it stops being reliable.
I was wrong about pooled estimate in a thread last spring and I would like to correct it publicly rather than quietly.
The error was in the units, which changed the conclusion by an order of magnitude. Setting out the corrected version, and the way I now check for that class of mistake.
Taking the opening post at face value and following it one step further.
I keep a log for pooled estimate specifically because my memory of it turned out to be systematically wrong in one direction. Six weeks of notes cost nothing and settled it.
Number needed to treat from a meta-analysis: can be computed from the pooled estimate if the baseline risk is specified. More interpretable than pooled relative effects.
A request rather than an answer: could whoever has the primary source for pooled estimate post it? I have seen the claim three times this month and each version had lost a qualifier.
Search strategy determines what was found and is the part of the method least often scrutinised. A search restricted to one language or one database is a selection step.
Narrowing post #9, because the general version has more than one answer.
A tighter confidence interval around a meaningless centre is worse than a wide interval around a meaningful one. Precision is not the objective.
That has held every time I have looked, which is not the same as always.
Everything in post #11 holds. The case it does not cover is the one I have.
Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
Quietly grateful for the plain phrasing. Not every thread gets that.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
Worth reading the earlier posts in this thread before acting on mine.
On post #19 — agreed on the reasoning, with one qualification.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
This is the version I would want a new member to read first.
Picking up post #19: that is the part I would want checked first.
The honest answer on pooled estimate is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.
Most people get the first two right and then argue about the fourth.
Where the pooled result and the largest single trial disagree, that disagreement is the interesting thing rather than an inconvenience to be smoothed.
Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.
That is where I would start, not where I would stop.
That matches what I have seen, for whatever a single anecdote is worth.
Adding the measurement that post #23 says would settle it.
Fixed-effect and random-effects models answer different questions. The first assumes one true effect; the second assumes a distribution of them. Choosing between them is an assumption, not a technicality.
Adding the caveat now so it does not have to be extracted later.
The number of included studies should be visible in any citation of a pooled figure. "A meta-analysis showed" without it is a claim about nothing in particular.
Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.
Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Revisiting: What pooling buys you and what it destroys
On the subject in the title: Revisiting: What pooling buys you and what it destroys Working notes rather than a conclusion. Pooling buys — I have the observation and I do not trust my interpretation of it, so…
|
+97 | 107 | 4k | 19h |
|
Second pass at: When a network meta-analysis is defensible and when it is not
Posting this under the heading it deserves: Second pass at: When a network meta-analysis is defensible and when it is not Everything below is what sits behind that. Posting a small dataset on network…
|
2 | 2.8k | 21mo | |
|
Random versus fixed effects: choosing rather than defaulting — a second dataset
On the subject in the title: Random versus fixed effects: choosing rather than defaulting — a second dataset Working notes rather than a conclusion. Posting a small dataset on Random versus fixed effects. It…
|
+95 | 106 | 20k | 17mo |
|
Coming back to: Prediction intervals and why they are more honest than confidence intervals
Prediction intervals and why they are more honest than confidence intervals Writing it up because I had to work it out twice and would rather nobody else did. Prediction intervals — I have the observation and…
|
2 | 65 | 22mo | |
|
About the Meta-analyses category
Pooled estimates, heterogeneity, publication bias, and when pooling was a mistake. This post is a community wiki: any member at trust level 3 or above can edit it, and every edit is recorded with its author…
|
4 | 15k | 13mo |
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Journal club: SURPASS-2 and the semaglutide 1 mg comparator
On the subject in the title: Journal club: SURPASS-2 and the semaglutide 1 mg comparator Working notes rather than a conclusion. Session topic: SURPASS-2 ( N Engl J Med , 2021). Please read it before posting;…
|
+71 | 76 | 52k | 21mo |
|
Individual participant data versus aggregate data
Posting this under the heading it deserves: Individual participant data versus aggregate data Everything below is what sits behind that. Something about individual participant data versus aggregate does not…
|
+30 | 34 | 64k | 14mo |
|
Follow-up: Individual participant data versus aggregate data
Individual participant data versus aggregate data — setting out what I have, and where I think it stops being reliable. A follow-up question about Individual participant data versus aggregate that I did not…
|
+101 | 111 | 23k | 18mo |
|
Journal club: STEP 1 and the treatment-policy estimand — does this still hold?
Asking directly, because I could not find a straight answer: Journal club: STEP 1 and the treatment-policy estimand — does this still hold? Working notes on STEP 1 rather than a conclusion. I would rather…
|
+70 | 76 | 4.8k | 15h |
|
Journal club: SELECT, absolute risk, and how it was reported
Journal club: SELECT, absolute risk, and how it was reported Writing it up because I had to work it out twice and would rather nobody else did. A question about what the numbers mean rather than what they…
|
2 | 52k | 1mo |