On the subject in the title: Second pass at: Absence of evidence and evidence of absence Working notes rather than a conclusion.
Two things I would like separated before anyone answers on Absence of evidence, because they get bundled and then argued about as one thing.
The first is descriptive: what has actually been observed, by whom, and how. The second is causal: why. I am asking about the first only.
Coming back to the opening post, because the follow-up matters more than the original answer.
P-values and significance: p<0.05 means the data would be surprising if the null hypothesis were true, not that the null hypothesis is false. A non-significant p-value does not mean "no effect".
Adding this to the thread rather than to the wiki, because I am not confident enough for the wiki.
Adding the measurement that post #2 says would settle it.
One more thing on Absence of evidence that took me far too long to see: the two figures people quote are not measuring the same quantity. Once you notice that, the apparent contradiction disappears.
Post #4 describes the usual case. This is about the unusual one.
Percentages of small denominators should be reported with the denominator. Two out of three is not sixty-seven per cent in any useful sense.
Posted with less confidence than the sentence structure implies.
Appreciated. The plain phrasing does more work here than a longer post would.
Since Absence of evidence keeps coming up, it should probably be a maintained page rather than a recurring thread. I am happy to draft it if someone with more direct experience will review it.
A note on how Absence of evidence gets discussed rather than on Absence of evidence itself: the confident posts get the replies and the careful ones get ignored, and the careful ones have been right more often.
Bayesian and frequentist analyses answer different questions and both are legitimate. What matters is that the reader knows which is on offer.
I would treat the number as indicative rather than as a measurement.
Right, and stated more narrowly than I would have dared to state it.
Answering the question post #11 raises rather than the one it answers.
My position on Absence of evidence is current rather than settled. I have revised it once already and I expect to again, so treat it accordingly.
Confounding: a third variable explains an apparent association. In randomised data, randomisation balances confounders. In observational data, confounders can be adjusted for but unknown ones cannot.
Anyone with a larger sample, please post it.
Building on post #15 rather than restating it.
Absence of evidence came up in a thread eighteen months ago and was answered well. I cannot find it, which is itself the problem, so here is the reconstruction.
I read the earlier replies on Absence of evidence twice before writing this, because I had assumed the opposite and wanted to be sure I was disagreeing with what was said rather than what I expected.
Absence of evidence and evidence of absence: if a study is small and finds no effect, that is absence of evidence, not evidence of absence. A larger study might find an effect that a small study missed.
Helpful, and easy to find again, which is half of what a good reply is.
Checked the Absence of evidence claim against the primary source this morning. It survives, with a narrower scope than the version quoted here. Posting the narrower scope.
Narrowing post #20, because the general version has more than one answer.
The honest answer on Absence of evidence is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.
Most people get the first two right and then argue about the fourth.
A distribution shown is worth ten summary statistics. Where a paper shows individual data points, read those first.
Fair, and the limits you put on it are the part I will remember.
Answering the question post #26 raises rather than the one it answers.
The arithmetic on Absence of evidence is the easy part and it is where the errors are, which is an uncomfortable combination. Show your working and someone will catch it.
Confirming post #28 from a second method, which matters more than confirming it from a second person.
Time-to-event analysis handles differing follow-up in a way a simple proportion cannot, which is why event rates and Kaplan-Meier estimates can differ.
Nothing above should be read as advice about what anyone else should do.
I had written a reply contradicting post #26 and deleted it. Here is what survived.
I changed my mind about Absence of evidence after someone here asked me for the source and I could not produce one. That is worth saying out loud because it is the ordinary way it happens.