Research Methods · Statistics

Multiplicity when you track fifteen variables

Research Methods › Statistics effect size heterogeneity confounding worked example

Solved

Solved by n.achebe in post #3

Multiplicity and multiple comparisons: if you test many hypotheses, the chance of finding a false positive by random chance increases. That is why pre-specifying the primary hypothesis matters.

Jump to the accepted answer →

j.asanteTL217 Jun 2025#1

Multiplicity when you track fifteen variables — setting out what I have, and where I think it stops being reliable.

Asking about multiplicity directly, because I have read four threads on it and each answered a slightly different question.

The version I want answered is the narrow one: given the method stated below, is the result within what anyone else has seen? I am not asking what it means yet.

Method, numbers and the two assumptions I am aware of making are below. If the assumptions are wrong that is more useful to me than agreement.

1 like 13mo

t.nardoneTL3Regular24 Jul 2025#2

Worth separating multiplicity as a question about the compound from multiplicity as a question about the documentation. They get answered by different people and only one of them is answerable here.

0 likes 12mo

n.achebeTL2 Solution20 Aug 2025#3

Multiplicity and multiple comparisons: if you test many hypotheses, the chance of finding a false positive by random chance increases. That is why pre-specifying the primary hypothesis matters.

20 likes 11mo

This topic was referenced in

Regression to the mean in progress reportsResearch Methods › Statistics · 34 replies
Measurement error in home scales, with a worked standard deviation — the long versionResearch Methods › Statistics · 105 replies

Suggested topics

Topic	Participants	Replies	Views	Activity
Correlation in a self-tracked dataset: what it can support Research Methods › Statistics effect size heterogeneity worked example number needed to treat Closed Correlation in a self-tracked dataset: what it can support — setting out what I have, and where I think it stops being reliable. Posting a small dataset on correlation. It is mine, it is uncontrolled, and the…	+152	166	48k	23mo
Sample size intuition for a personal experiment — does this still hold? Research Methods › Statistics effect size heterogeneity number needed to treat Solved Sample size intuition for a personal experiment — does this still hold? I have a specific reason for asking rather than idle curiosity, and the context is below. Reading back through what has been written…	+33	38	26k	23d
Second pass at: Absence of evidence and evidence of absence Research Methods › Statistics worked example confounding number needed to treat On the subject in the title: Second pass at: Absence of evidence and evidence of absence Working notes rather than a conclusion. Two things I would like separated before anyone answers on Absence of evidence,…	+50	55	18k	16h
Second pass at: Multiplicity when you track fifteen variables Research Methods › Statistics number needed to treat confounding effect size worked example Posting this under the heading it deserves: Second pass at: Multiplicity when you track fifteen variables Everything below is what sits behind that. Two things I would like separated before anyone answers on…		2	12k	18mo
Absence of evidence and evidence of absence Research Methods › Statistics effect size worked example number needed to treat Absence of evidence and evidence of absence Writing it up because I had to work it out twice and would rather nobody else did. Asking about absence of evidence on behalf of the question I keep seeing asked…	+16	20	3.9k	5mo

Related topics — sharing the tags effect size, heterogeneity, confounding

Topic	Participants	Replies	Views	Activity
Second pass at: Why semaglutide's albumin binding is the whole reason weekly dosing works Compounds › Semaglutide SELECT trial STEP programme effect size randomised trial Second pass at: Why semaglutide's albumin binding is the whole reason weekly dosing works Writing it up because I had to work it out twice and would rather nobody else did. Putting the numbers in the first…		2	39k	6mo
Why we ask for units every single time, with three examples of what goes wrong — what changed since Start Here › How to Ask onboarding guidelines worked example beginner friendly The question in the title: Why we ask for units every single time, with three examples of what goes wrong — what changed since I will give what I have already checked below so nobody repeats it. Counting it…	+14	18	19k	2mo
Revisiting: Holding a dose indefinitely: is there a documented downside? Practice › Dosing & titration titration tirzepatide worked example missed dose Solved The question in the title: Revisiting: Holding a dose indefinitely: is there a documented downside? I will give what I have already checked below so nobody repeats it. I have read the maintained page on this…	+26	30	4.5k	18mo
Washout with a one-week half-life: the arithmetic Research Methods › N-of-1 designs effect size data table worked example Wiki Washout with a one-week half-life: the arithmetic — setting out what I have, and where I think it stops being reliable. Washout: setting out the arithmetic in full, because I have had to do it twice and I…	+90	96	44k	20mo
Feature request: should a dose-equivalence tool exist at all? Data & Tools › Calculators site feedback worked example concentration maths syringe units Closed Feature request: should a dose-equivalence tool exist at all? — that is the question, and I have not found it answered plainly anywhere I have looked. Reading back through what has been written here about…	+21	25	4.8k	18mo

Sponsored Sigma-Aldrich — certified reference materialsReference standards, analytical reagents and certified materials for method calibration. If you are validating a method, this is where the standard comes from.Reference standards →