Prediction intervals and why they are more honest than confidence intervals

Posting this under the heading it deserves: Prediction intervals and why they are more honest than confidence intervals Everything below is what sits behind that.

I have spent a fortnight trying to pin prediction intervals down and I want to set out where I have got to, because I suspect the honest answer is duller than the thread this will produce.

What I have: a consistent observation across a small number of cases, collected the same way each time. What I do not have: any controlled comparison, or any reason to think my cases are representative.

The specific question is whether the pattern survives once the obvious confounder is removed. I cannot remove it with what I have.

Coming back to the opening post, because the follow-up matters more than the original answer.

Checked the prediction intervals claim against the primary source this morning. It survives, with a narrower scope than the version quoted here. Posting the narrower scope.

Picking up the opening post: that is the part I would want checked first.

Counterpoint on prediction intervals, offered without confidence: the same observation is consistent with a much duller explanation, and nobody has ruled the dull one out.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

On balance I think that is right, and I would not bet much on it.

A note on how prediction intervals gets discussed rather than on prediction intervals itself: the confident posts get the replies and the careful ones get ignored, and the careful ones have been right more often.

This follows post #4 rather than contradicting it.

The honest answer on prediction intervals is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.

Most people get the first two right and then argue about the fourth.

Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

That is the shape of it. The detail is where I would expect to be corrected.

What I want from this prediction intervals thread is the list of things that would need to be true for the claim to hold. If we can write that list, we can check it.

Understood. Thank you for being specific about the limits of it.

On post #9 — agreed on the reasoning, with one qualification.

A tighter confidence interval around a meaningless centre is worse than a wide interval around a meaningful one. Precision is not the objective.

I would put this at better than even and not much better.

The thing about prediction intervals that took me longest to accept is that a plausible mechanism is not evidence of an effect. It is a reason to look, not a result.