Evidence · Meta-analyses

Risk-of-bias tools and the judgement they conceal

Evidence › Meta-analyses meta-analysis heterogeneity publication bias effect size

Wiki

k.brandl_deTL3Translator · DE16 Apr 2025#1

Community wiki post. Any member at trust level 3 or above can edit this post; every edit is recorded. Last edited by policy_reader on 27 Nov 2025.

23 May 2025 — titration_diary: Replaced an unsourced figure with the published one and cited it.
20 May 2025 — g.pemberton_uk: Removed a claim that the cited source did not support.
27 Nov 2025 — policy_reader: Clarified the distinction that was causing repeat questions below.

Editors: titration_diary, g.pemberton_uk, policy_reader

Risk-of-bias tools and the judgement they conceal Writing it up because I had to work it out twice and would rather nobody else did.

I have spent a fortnight trying to pin risk-of-bias tools down and I want to set out where I have got to, because I suspect the honest answer is duller than the thread this will produce.

What I have: a consistent observation across a small number of cases, collected the same way each time. What I do not have: any controlled comparison, or any reason to think my cases are representative.

The specific question is whether the pattern survives once the obvious confounder is removed. I cannot remove it with what I have.

0 likes 15mo

h.amankwahTL26 May 2025#2

Speaking only to risk-of-bias tools as I have actually seen it, rather than as it is usually described: the effect is real, it is smaller than the thread suggests, and the variance between people is larger than the effect.

18 likes 15mo

l.piresTL221 May 2025#3

The opening post describes the usual case. This is about the unusual one.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

4 likes 14mo

m.ekstromTL23 Jun 2025#4

h.amankwah, post #2: Speaking only to risk-of-bias tools as I have actually seen it, rather than as it is usually described: the effect is real, it is smaller than the thread suggests, and the variance between people is larger than the effect. Go to post

Source for the risk-of-bias tools figure, since it was asked for. It is in the discussion rather than the abstract, which is why the version circulating is stronger than the paper is.

Reading the surrounding paragraph is worth the two minutes. The authors are more careful than their summarisers.

0 likes in reply to #2 14mo

c.inglethorpeTL3Regular15 Jun 2025 · edited#5

k.brandl_de, post #1: Risk-of-bias tools and the judgement they conceal Writing it up because I had to work it out twice and would rather nobody else did. I have spent a fortnight trying to pin risk-of-bias tools down and I want to set out where I have got to, because I suspect the honest answer is duller than the thread this will produce. What I have: a… Go to post

The most useful thing a review can publish is the excluded-studies table with reasons. It is where the judgement is and it is almost always in a supplement.

Posted with less confidence than the sentence structure implies.

0 likes in reply to #1 13mo

r.molnarTL227 Jun 2025#6

Post #5 is right about the mechanism and I think understates the practical bit.

Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.

25 likes 13mo

ThibodeauTL38 Jul 2025#7

Collapsed as off-topic by two members at trust level 3 or above

Noted, and thank you for writing it out rather than summarising it.

f.chowdhuryTL218 Jul 2025#8

What would change my mind on risk-of-bias tools is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more.

1 like 12mo

m.onwukaTL228 Jul 2025#9

Post #5 put the caveat in the right place and I want to underline it.

The most useful thing anyone has posted about risk-of-bias tools in this category was a table of what had been measured and by whom. That is what I would want again.

19 likes in reply to #1 12mo

mira.patelTL4 Admin7 Aug 2025#10

f.chowdhury, post #8: What would change my mind on risk-of-bias tools is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more. Go to post

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

8 likes in reply to #8 12mo

s.bruunTL217 Aug 2025#11

Where the risk-of-bias tools discussion usually stalls is that nobody wants to say "I do not know" and everyone is willing to say "it varies". Those are the same sentence with different clothes on.

0 likes 11mo

g.rasmussenTL227 Aug 2025#12

If you are new and reading this thread for the answer to risk-of-bias tools: the answer is conditional, the conditions are in the third reply, and the rest of the thread is worth skipping.

1 like 11mo

a.reyesTL4 Admin5 Sep 2025 · edited#13

Post #10 answers the question as asked. The question underneath it is different.

Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.

9 likes 11mo

This topic was referenced in

Pooling trials with different estimandsEvidence › Meta-analyses · 132 replies

Topic	Participants	Replies	Views	Activity
What pooling buys you and what it destroys Evidence › Meta-analyses publication bias risk of bias meta-analysis heterogeneity Solved The question in the title: What pooling buys you and what it destroys I will give what I have already checked below so nobody repeats it. The question about pooling buys that I actually want answered is the…	+74	83	6.6k	8mo
Coming back to: Prediction intervals and why they are more honest than confidence intervals Evidence › Meta-analyses publication bias heterogeneity effect size risk of bias Prediction intervals and why they are more honest than confidence intervals Writing it up because I had to work it out twice and would rather nobody else did. Prediction intervals — I have the observation and…		2	65	22mo
When a network meta-analysis is defensible and when it is not Evidence › Meta-analyses effect size heterogeneity publication bias Solved When a network meta-analysis is defensible and when it is not I have a specific reason for asking rather than idle curiosity, and the context is below. Network meta-analysis, from the point of view of someone…		2	27k	4h
Follow-up: Individual participant data versus aggregate data Evidence › Meta-analyses meta-analysis effect size publication bias heterogeneity Individual participant data versus aggregate data — setting out what I have, and where I think it stops being reliable. A follow-up question about Individual participant data versus aggregate that I did not…	+101	111	23k	18mo
Random versus fixed effects: choosing rather than defaulting — a second dataset Evidence › Meta-analyses heterogeneity risk of bias publication bias On the subject in the title: Random versus fixed effects: choosing rather than defaulting — a second dataset Working notes rather than a conclusion. Posting a small dataset on Random versus fixed effects. It…	+95	106	20k	17mo

Related topics — sharing the tags meta-analysis, heterogeneity, publication bias

Topic	Participants	Replies	Views	Activity
When a network meta-analysis is defensible and when it is not Evidence › Meta-analyses effect size heterogeneity publication bias Solved When a network meta-analysis is defensible and when it is not I have a specific reason for asking rather than idle curiosity, and the context is below. Network meta-analysis, from the point of view of someone…		2	27k	4h
[2026 update] Journal club: is percentage weight change the right endpoint? Evidence › Journal club effect size risk of bias randomised trial Journal club: is percentage weight change the right endpoint? — that is the question, and I have not found it answered plainly anywhere I have looked. A question about what the numbers mean rather than what…	+105	118	18k	11mo
Second pass at: Journal club: STEP 4 and what a withdrawal design can prove Evidence › Journal club number needed to treat risk of bias journal club effect size Second pass at: Journal club: STEP 4 and what a withdrawal design can prove Writing it up because I had to work it out twice and would rather nobody else did. The question about STEP 4 that I actually want…	+81	90	54k	9mo
Measurement error in home scales, with a worked standard deviation Research Methods › Statistics effect size heterogeneity number needed to treat confounding Solved Wiki Measurement error in home scales, with a worked standard deviation Writing it up because I had to work it out twice and would rather nobody else did. What changes if the standard account of measurement error…	+26	31	4.3k	4mo
Journal club: STEP 8 and the fairness of the comparator dose — a second dataset Evidence › Journal club risk of bias journal club effect size estimand Posting this under the heading it deserves: Journal club: STEP 8 and the fairness of the comparator dose — a second dataset Everything below is what sits behind that. Posting a small dataset on STEP 8. It is…	+18	22	656	13h

Sponsored PeptideMeter — peptide analytics and supplier verificationPurity analysis and structured supplier verification, published as dated records rather than scores. Referenced throughout our sourcing documentation.View analytics →

This topic was referenced in

Suggested topics

Related topics — sharing the tags meta-analysis, heterogeneity, publication bias