A definition problem is doing most of the work in this Absence of evidence discussion. Once the term is pinned down I suspect the disagreement mostly goes away and what is left is small.
1 like 1mo
Research Methods · Statistics · continued
Second pass at: Absence of evidence and evidence of absence posts 31–56
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
impurity_table, post #15: Confounding: a third variable explains an apparent association. In randomised data, randomisation balances confounders. In observational data, confounders can be adjusted for but unknown ones cannot. Anyone with a larger sample, please post it. Go to post
A p-value is the probability of data at least this extreme given the null hypothesis. It is not the probability the hypothesis is false, and almost every plain-language gloss gets that backwards.
I have no interest in any supplier named above.
Where I part company with post #33, and it is a narrow parting.
Baseline imbalance in a randomised trial is expected by chance and adjusting for it post hoc is a choice that should have been pre-specified.
I would put this at better than even and not much better.
4 likes 26d
Collapsed as off-topic by two members at trust level 3 or above
Adding the boring version of Absence of evidence, because the interesting version keeps getting posted and the boring one is usually right.
Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does.
Adding the measurement that post #36 says would settle it.
Survivorship in a self-reporting population biases every aggregate produced from it, and the bias is in the flattering direction.
17 likes 22d
s.teixeira, post #7: Appreciated. The plain phrasing does more work here than a longer post would. Go to post
I read post #38 twice before replying, because I had assumed the opposite.
The version of Absence of evidence that I was taught turned out to be a teaching simplification. Useful, and not true in the way I had assumed it was.
Tamburello, post #37: Adding the boring version of Absence of evidence, because the interesting version keeps getting posted and the boring one is usually right. Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does. Go to post
Post #38 answers the question as asked. The question underneath it is different.
Whatever the answer on Absence of evidence turns out to be, the method for getting there is the same: state the assumption, do the arithmetic in public, invite the correction.
Absence of statistical significance is not evidence of absence, particularly in a small study. The width of the interval tells you what was ruled out and the p-value does not.
A modest claim, modestly supported.
3 likes 18d
Good question, well framed, and I would like to see it answered properly.
11 likes 17d
g.valckenaere, post #6: Post #4 describes the usual case. This is about the unusual one. Percentages of small denominators should be reported with the denominator. Two out of three is not sixty-seven per cent in any useful sense. Posted with less confidence than the sentence structure implies. Go to post
This follows post #41 rather than contradicting it.
The practical version of Absence of evidence is three sentences long. The rigorous version is three pages and reaches the same conclusion with the conditions attached.
impurity_table, post #15: Confounding: a third variable explains an apparent association. In randomised data, randomisation balances confounders. In observational data, confounders can be adjusted for but unknown ones cannot. Anyone with a larger sample, please post it. Go to post
Where an analysis was changed after seeing the data, the honest thing is to report both and say which was pre-specified.
Written from notes rather than memory, which is why the numbers are specific.
Post #43 answers the question as asked. The question underneath it is different.
Absence of evidence looks different depending on whether you are reading the primary literature or the summaries of it, and the difference is not in our favour.
1 like 14d
I read post #44 twice before replying, because I had assumed the opposite.
Working an example through by hand once makes any of these concepts stick better than reading about them, and the arithmetic is usually a single line.
That has been true for the cases I have seen and I have not seen many.
7 likes 12d
Where I would push back on the Absence of evidence consensus is the confidence, not the direction. The direction looks right. The confidence is borrowed.
24 likes 11d
a.adeyemi, post #22: Multiple testing inflates the false-positive rate in a way that is entirely predictable and entirely correctable. The correction should be declared in advance. It is the kind of thing that is obvious once and never again. Go to post
Having read the whole Absence of evidence thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.
Effect sizes: the magnitude of a difference, not just whether it is statistically significant. A difference that is significant (p<0.05) might be too small to matter. A large effect might not be significant if sample size is small.
4 likes 8d
Everything in post #47 holds. The case it does not cover is the one I have.
Rounding and significant figures carry information about precision. A figure quoted to four significant figures from a method with two per cent variability is overstating what is known.
I would want a second opinion before relying on that.
10 likes 6d
KStephanopoulos, post #46: I read post #44 twice before replying, because I had assumed the opposite. Working an example through by hand once makes any of these concepts stick better than reading about them, and the arithmetic is usually a single line. That has been true for the cases I have seen and I have not seen many. Go to post
Practical note on Absence of evidence: write down what you expect before you look. The number of times I have found what I went looking for is higher than chance would allow.
I disagree with the framing of Absence of evidence above, and I think it is a substantive disagreement rather than a terminological one. Setting out why, so it can be checked.
The reasoning depends on an assumption that is doing a lot of work and is never stated. If the assumption holds, the conclusion follows. I do not think it holds generally.
0 likes 4d
s.leclerc, post #1: On the subject in the title: Second pass at: Absence of evidence and evidence of absence Working notes rather than a conclusion. Two things I would like separated before anyone answers on Absence of evidence, because they get bundled and then argued about as one thing. The first is descriptive: what has actually been observed, by whom,… Go to post
Answering the question post #51 raises rather than the one it answers.
Before the thread moves on from Absence of evidence — what is the sample size behind the claim? I am not being difficult; I have seen the same figure quoted from an n of four and from an n of four hundred.
Birkeland, post #44: Where an analysis was changed after seeing the data, the honest thing is to report both and say which was pre-specified. Written from notes rather than memory, which is why the numbers are specific. Go to post
The arithmetic in post #55 is right; the assumption feeding it is the part to check.
Baseline imbalance in a randomised trial is expected by chance and adjusting for it post hoc is a choice that should have been pre-specified.
That is what the documentation says. What happens in practice is usually close.
Suggested topics
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Measurement error in home scales, with a worked standard deviation — a second dataset
Measurement error in home scales, with a worked standard deviation — a second dataset — setting out what I have, and where I think it stops being reliable. Posting a small dataset on Measurement error in home…
|
2 | 13k | 7mo | |
|
Regression to the mean in progress reports
On the subject in the title: Regression to the mean in progress reports Working notes rather than a conclusion. Something about regression does not reconcile and I would like a second pair of eyes before I…
|
+30 | 34 | 49k | 12mo |
|
Sample size intuition for a personal experiment
On the subject in the title: Sample size intuition for a personal experiment Working notes rather than a conclusion. A follow-up question about sample size intuition that I did not know to ask the first time.…
|
4 | 62k | 18mo | |
|
Measurement error in home scales, with a worked standard deviation — the long version
On the subject in the title: Measurement error in home scales, with a worked standard deviation — the long version Working notes rather than a conclusion. A narrow question about Measurement error in home…
|
+96 | 105 | 2k | 2mo |
|
What a confidence interval means, from scratch — the long version
The question in the title: What a confidence interval means, from scratch — the long version I will give what I have already checked below so nobody repeats it. Asking about confidence interval directly,…
|
+16 | 20 | 2.3k | 8mo |
Related topics — sharing the tags worked example, confounding, number needed to treat
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Why "add 2 mL" is not an instruction — a second dataset
Why "add 2 mL" is not an instruction — a second dataset — that is the question, and I have not found it answered plainly anywhere I have looked. I have read the maintained page on this and I still have a gap,…
|
2 | 13k | 11mo | |
|
Second pass at: Multiplicity when you track fifteen variables
Posting this under the heading it deserves: Second pass at: Multiplicity when you track fifteen variables Everything below is what sits behind that. Two things I would like separated before anyone answers on…
|
2 | 12k | 18mo | |
|
Journal club: STEP 3 and the intensive behavioural therapy floor — does this still hold?
The question in the title: Journal club: STEP 3 and the intensive behavioural therapy floor — does this still hold? I will give what I have already checked below so nobody repeats it. Asking about STEP 3 on…
|
2 | 15k | 10mo | |
|
How much context is too much context in a first post?
How much context is too much context in a first post? I have a specific reason for asking rather than idle curiosity, and the context is below. Counting it rather than describing it, because the count is the…
|
+69 | 74 | 935 | 18d |
|
Regression to the mean in progress reports
On the subject in the title: Regression to the mean in progress reports Working notes rather than a conclusion. Something about regression does not reconcile and I would like a second pair of eyes before I…
|
+30 | 34 | 49k | 12mo |