Reading a meta-analysis you disagree with, fairly Writing it up because I had to work it out twice and would rather nobody else did.
Reading a meta-analysis — I have the observation and I do not trust my interpretation of it, so I am posting the observation and holding the interpretation back.
Numbers, method and the conditions under which they were collected are below. Interpret them however they warrant.
Reading rather than answering, but this is the post I would point somebody at.
The opening post and I disagree about the size of the effect, not about the direction.
My experience of Reading a meta-analysis contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
This is where my knowledge stops and I would rather mark the edge than blur it.
The arithmetic in post #5 is right; the assumption feeding it is the part to check.
Worth separating Reading a meta-analysis as a question about the compound from Reading a meta-analysis as a question about the documentation. They get answered by different people and only one of them is answerable here.
Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.
Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.
I am describing what is, rather than arguing for what should be.
Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.
Worth separating two things that post #12 runs together.
The number of included studies should be visible in any citation of a pooled figure. "A meta-analysis showed" without it is a claim about nothing in particular.
It is the kind of thing that is obvious once and never again.
Fair, and the limits you put on it are the part I will remember.
The practical version of Reading a meta-analysis is three sentences long. The rigorous version is three pages and reaches the same conclusion with the conditions attached.
On post #16 — agreed on the reasoning, with one qualification.
I would keep Reading a meta-analysis and the decision it usually gets used for separate in this thread. They are related and they are not the same question, and merging them is why the last one went badly.
Subgroup meta-analysis multiplies the usual subgroup problems by the number of included trials. Treat it as hypothesis-generating without exception.
That distinction has done more work for me than anything else in this category.
I had written a reply contradicting post #16 and deleted it. Here is what survived.
Reading the pooled figure without reading the inclusion criteria is the equivalent of quoting a trial without its population. Both are common and both are avoidable.
Caveat: everything above assumes the paperwork is what it says it is.
A meta-analysis of four small trials is not stronger evidence than one adequately powered trial, whatever the summary statistic looks like.
The uncertainty is in the assumption, not in the calculation.
Adding the measurement that post #21 says would settle it.
Where the pooled result and the largest single trial disagree, that disagreement is the interesting thing rather than an inconvenience to be smoothed.
The short version is the first sentence; the rest is why.
Where I part company with post #21, and it is a narrow parting.
Reframing Reading a meta-analysis slightly, because I think the disagreement is about the question rather than the answer. If the question is "does it happen", yes. If it is "how often", nobody here knows.
Picking up post #25: that is the part I would want checked first.
Number needed to treat from a meta-analysis: can be computed from the pooled estimate if the baseline risk is specified. More interpretable than pooled relative effects.
Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.
Bookmarking this. I will come back when I have something worth adding.
Everything in post #25 holds. The case it does not cover is the one I have.
Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.