Post #3 describes the usual case. This is about the unusual one. The practical version of Publication bias detection is three sentences long. The rigorous version is three pages and reaches the same conclusion with the conditions attached.
The question in the title: Publication bias detection and its low power — does this still hold? I will give what I have already checked below so nobody repeats it.
Collecting what is known about Publication bias detection in one place, because it is currently spread across a category, two tag pages and a thread that is hard to find.
This is a summary rather than new work, and I have attributed each part to where I found it.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
That is clearer than the version I had in my head. Thank you.
Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies.
Reading it back, the second half matters more than the first.
Two questions I would want answered before drawing anything from the Publication bias detection data above: how were the cases selected, and what happened to the ones that dropped out.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Narrowing post #7, because the general version has more than one answer.
Small methodological point on Publication bias detection: repeating a measurement is cheap and resolves most of what is being argued about here at no cost to anyone.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
I would be interested in a counterexample if anyone has one.
The version of Publication bias detection that circulates here is a simplification of a simplification. It is not wrong, but it has lost the conditions under which it holds, and those conditions are where the interesting cases live.
The arithmetic in post #12 is right; the assumption feeding it is the part to check.
Agreed on Publication bias detection, with one qualification that I think matters. The reasoning holds for the case as described. Change the starting assumption and it does not, and the starting assumption is the part nobody states.
Answering the question post #10 raises rather than the one it answers.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Marking my uncertainty on Publication bias detection explicitly. I am confident about the direction, much less confident about the size, and not confident at all that it generalises past the case in the first post.
Where I part company with post #15, and it is a narrow parting.
Reading the pooled figure without reading the inclusion criteria is the equivalent of quoting a trial without its population. Both are common and both are avoidable.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Happy to be the one who is wrong here if it settles the question.
Post #20 is right about the mechanism and I think understates the practical bit.
Publication bias detection has a well-known answer and a correct answer, and the interesting work is establishing that they are the same. Nobody has done that here yet.
Seconded. It reads as careful rather than confident, which is the right register.
I read post #23 twice before replying, because I had assumed the opposite.
Protocol registration for a systematic review matters for the same reason it does for a trial. An unregistered review can select its inclusion criteria after seeing the results.
Post #23 answers the question as asked. The question underneath it is different.
An observation about Publication bias detection that I cannot explain and am posting anyway, on the principle that unexplained observations are more useful public than private.
Saving this. It is the version I will quote when the question comes round again.
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
What pooling buys you and what it destroys
The question in the title: What pooling buys you and what it destroys I will give what I have already checked below so nobody repeats it. The question about pooling buys that I actually want answered is the…
|
+74 | 83 | 6.6k | 8mo |
|
Heterogeneity as information rather than as a nuisance
Posting this under the heading it deserves: Heterogeneity as information rather than as a nuisance Everything below is what sits behind that. An honest uncertainty about heterogeneity rather than a disguised…
|
+88 | 95 | 1.2k | 6mo |
|
When a network meta-analysis is defensible and when it is not
When a network meta-analysis is defensible and when it is not I have a specific reason for asking rather than idle curiosity, and the context is below. Network meta-analysis, from the point of view of someone…
|
2 | 27k | 4h | |
|
Prediction intervals and why they are more honest than confidence intervals
Posting this under the heading it deserves: Prediction intervals and why they are more honest than confidence intervals Everything below is what sits behind that. I have spent a fortnight trying to pin…
|
+21 | 25 | 46k | 4mo |
|
Publication bias detection and its low power
On the subject in the title: Publication bias detection and its low power Working notes rather than a conclusion. Working notes on publication bias detection rather than a conclusion. I would rather post the…
|
2 | 3.2k | 1d |
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Tracking a preprint to publication: a worked example — the long version
Posting this under the heading it deserves: Tracking a preprint to publication: a worked example — the long version Everything below is what sits behind that. A preprint question rather than a question about…
|
+38 | 47 | 560 | 22mo |
|
Why a preprint's supplementary material is often the best part
Why a preprint's supplementary material is often the best part I have a specific reason for asking rather than idle curiosity, and the context is below. A question about how this site should cite something…
|
+38 | 42 | 863 | 22mo |
|
Measurement error in home scales, with a worked standard deviation
Measurement error in home scales, with a worked standard deviation Writing it up because I had to work it out twice and would rather nobody else did. What changes if the standard account of measurement error…
|
+26 | 31 | 4.3k | 4mo |
|
How this community labels and discusses unrefereed work
How this community labels and discusses unrefereed work — that is the question, and I have not found it answered plainly anywhere I have looked. Reading an unrefereed manuscript properly, and asking whether I…
|
+112 | 119 | 1.3k | 2y |
|
The "under review" status pill and when a moderator applies it
Posting this under the heading it deserves: The "under review" status pill and when a moderator applies it Everything below is what sits behind that. A question about how this site should cite something that…
|
+96 | 119 | 17k | 3mo |