Adding the measurement that post #58 says would settle it.
Small methodological point on heterogeneity: repeating a measurement is cheap and resolves most of what is being argued about here at no cost to anyone.
0 likes 6mo
Evidence · Meta-analyses · continued
Heterogeneity as information rather than as a nuisance posts 61–90
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Post #60 describes the usual case. This is about the unusual one.
Where I would push back on the heterogeneity consensus is the confidence, not the direction. The direction looks right. The confidence is borrowed.
2 likes 6mo
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
The right answer here may simply be that it has not been measured.
13 likes 6mo
g.tamm, post #13: Where I part company with post #9, and it is a narrow parting. Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified. Go to post
An honest declaration on heterogeneity: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it.
Heterogeneity is worth one more sentence than it usually gets, and the sentence is the one about how the number was arrived at.
0 likes 6mo
On post #64 — agreed on the reasoning, with one qualification.
Quality assessment of included trials should change the analysis rather than sit beside it. A sensitivity analysis excluding the weakest studies is the minimum.
0 likes 6mo
Nicolaides, post #58: Confirming post #56 from a second method, which matters more than confirming it from a second person. The claim about heterogeneity upthread is stronger than its source supports. I have read the source. The source says "associated with" and the post says "causes". Go to post
Post #66 is right about the mechanism and I think understates the practical bit.
Heterogeneity sits at the boundary between what this community can usefully discuss and what it cannot, and I think it falls on the discussable side, narrowly.
Collapsed as off-topic by two members at trust level 3 or above
That matches what I have seen, for whatever a single anecdote is worth.
Post #67 describes the usual case. This is about the unusual one.
Adding the boring version of heterogeneity, because the interesting version keeps getting posted and the boring one is usually right.
Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does.
0 likes 6mo
a.nyberg, post #5: Appreciated. The plain phrasing does more work here than a longer post would. Go to post
That is a fair summary of where the discussion has got to.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
The step people skip is the one I have spelled out.
0 likes 6mo
On post #71 — agreed on the reasoning, with one qualification.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
That is all I can say without guessing.
0 likes 6mo
LJankowiak, post #64: An honest declaration on heterogeneity: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it. Go to post
Small correction to my own earlier position on heterogeneity. I had the units the wrong way round, which changes the conclusion by an order of magnitude and therefore changes it entirely.
Building on post #80 rather than restating it.
Second-hand on heterogeneity, so weight it accordingly — someone whose method I trust told me this and I have not verified it myself.
31 likes 6mo
Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.
0 likes 6mo
ni.stanescu, post #39: Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random. I would put this at better than even and not much better. Go to post
If you are new and reading this thread for the answer to heterogeneity: the answer is conditional, the conditions are in the third reply, and the rest of the thread is worth skipping.
Nicolaides, post #58: Confirming post #56 from a second method, which matters more than confirming it from a second person. The claim about heterogeneity upthread is stronger than its source supports. I have read the source. The source says "associated with" and the post says "causes". Go to post
Everything in post #82 holds. The case it does not cover is the one I have.
Offering a way to settle heterogeneity rather than another opinion about it. Two measurements, taken the same way, a fortnight apart. If the difference is within the noise, the question was not answerable at this precision.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
That is what I would do. It may not be what is correct.
1 like 6mo
ro.frisk, post #50: Distinguishing three things in the heterogeneity discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both. Go to post
The arithmetic in post #84 is right; the assumption feeding it is the part to check.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
Same conclusion as the reply above, reached differently, which is mildly reassuring.
Worth stating the null on heterogeneity before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one.
0 likes 6mo