Pre-registering a personal experiment, seriously posts 91–120
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
The interesting personal experiments are the ones where the answer was no. They are also the ones least likely to be written up.
Collapsed as off-topic by two members at trust level 3 or above
Adding the measurement that post #92 says would settle it.
Reframing pre-registering a personal experiment slightly, because I think the disagreement is about the question rather than the answer. If the question is "does it happen", yes. If it is "how often", nobody here knows.
Post #90 describes the usual case. This is about the unusual one.
It does not tell you about anybody else, which is why aggregating these accounts does not produce evidence of the kind people want it to.
I have seen it go both ways, which is why I hedge.
Practical experience of pre-registering a personal experiment, offered as one case with the conditions stated, not as a general finding. Conditions first, because they are what make it interpretable.
Order effects matter. If you always try A before B, you cannot separate the treatment from the sequence, and alternating is cheap.
Noted, and thank you for writing it out rather than summarising it.
Post #96 is right about the mechanism and I think understates the practical bit.
If you are new and reading this thread for the answer to pre-registering a personal experiment: the answer is conditional, the conditions are in the third reply, and the rest of the thread is worth skipping.
Coming back to post #99, because the follow-up matters more than the original answer.
Generalisability: a robust n-of-1 result applies to you. It does not tell you much about whether the effect generalises to others similar to you, much less to people different from you.
That is the version I use. It may not be the version that is correct.
Two questions I would want answered before drawing anything from the pre-registering a personal experiment data above: how were the cases selected, and what happened to the ones that dropped out.
Post #101 is right about the mechanism and I think understates the practical bit.
Pre-registering a personal experiment: I have looked for the primary source twice and failed twice. Either it does not exist or it is somewhere I do not know to look, and I would like to know which.
Coming back to post #100, because the follow-up matters more than the original answer.
Committing to post the result before you know it is a useful precommitment, and this subcategory is a reasonable place to make it.
It reads as pedantry until the day it does not.
Picking up post #103: that is the part I would want checked first.
The thing about pre-registering a personal experiment that took me longest to accept is that a plausible mechanism is not evidence of an effect. It is a reason to look, not a result.
A rolling mean over several days is far more informative than any single reading for anything that varies day to day, which is nearly everything.
Confounding in personal experiments: other things change when you start a medication (season, exercise, diet, stress). Documenting those confounders helps you understand their contribution to the result.
Worth checking against a second source before it gets quoted onward.
Coming back to post #104, because the follow-up matters more than the original answer.
A single-person experiment can be genuinely informative if it has a pre-specified outcome, a defined period, and a plan written before the data arrives. Most accounts here have none of those.
Happy to expand any of that if it is the useful part.
That is a cleaner way of putting what I was circling around.
Post #108 and I disagree about the size of the effect, not about the direction.
Pre-registering a personal experiment is worth one more sentence than it usually gets, and the sentence is the one about how the number was arrived at.
Offering a way to settle pre-registering a personal experiment rather than another opinion about it. Two measurements, taken the same way, a fortnight apart. If the difference is within the noise, the question was not answerable at this precision.
Designing a personal experiment that could actually change your mind: that is the standard for an n-of-1 design. An experiment designed so that any result confirms what you already believed has not changed anything.
Second-hand on pre-registering a personal experiment, so weight it accordingly — someone whose method I trust told me this and I have not verified it myself.
Duration should be set by how long an effect would take to appear rather than by patience. Stopping early is the commonest reason a personal experiment answers nothing.
That is all I can say without guessing.
Nothing in a personal experiment is medical advice to anyone including yourself, and several people here have found that a prescriber will happily discuss a well-documented one.
I would rather say I do not know than round it up to an answer.
I read post #115 twice before replying, because I had assumed the opposite.
The documentation on pre-registering a personal experiment is better than this thread and I say that as someone who has posted in the thread.
Post #115 answers the question as asked. The question underneath it is different.
Pre-registering a personal experiment has a well-known answer and a correct answer, and the interesting work is establishing that they are the same. Nobody has done that here yet.
Post #119 is the version of this I will quote in future. One addition.
The honest answer on pre-registering a personal experiment is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.
Most people get the first two right and then argue about the fourth.
This topic was referenced in
- An ABAB design with a data table and honest limitations — a second datasetResearch Methods › N-of-1 designs · 118 replies
- Second pass at: Designing a personal experiment that could change your mindResearch Methods › N-of-1 designs · 44 replies
Suggested topics
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Designing a personal experiment that could change your mind
Posting this under the heading it deserves: Designing a personal experiment that could change your mind Everything below is what sits behind that. I was wrong about designing a personal experiment in a thread…
|
+2 | 6 | 695 | 6mo |
|
Washout with a one-week half-life: the arithmetic — does this still hold?
Washout with a one-week half-life: the arithmetic — does this still hold? — that is the question, and I have not found it answered plainly anywhere I have looked. Washout: setting out the arithmetic in full,…
|
+41 | 45 | 12k | 1d |
|
Blinding yourself: practical methods and their limits — the long version
Blinding yourself: practical methods and their limits — the long version Writing it up because I had to work it out twice and would rather nobody else did. A narrow question about Blinding yourself,…
|
+7 | 11 | 1.8k | 22h |
|
An ABAB design with a data table and honest limitations
An ABAB design with a data table and honest limitations Writing it up because I had to work it out twice and would rather nobody else did. An honest uncertainty about ABAB design rather than a disguised…
|
2 | 45k | 5mo | |
|
An ABAB design with a data table and honest limitations — a second dataset
On the subject in the title: An ABAB design with a data table and honest limitations — a second dataset Working notes rather than a conclusion. Posting a small dataset on ABAB design. It is mine, it is…
|
+106 | 118 | 25k | 4mo |
Related topics — sharing the tags worked example, data table, effect size
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Carryover and the ghost peak from last week's standard — what changed since
Carryover and the ghost peak from last week's standard — what changed since Writing it up because I had to work it out twice and would rather nobody else did. What changes if the standard account of Carryover…
|
+98 | 114 | 37k | 22d |
|
Journal club: SURMOUNT-4 and continuation versus withdrawal — a second dataset
Posting this under the heading it deserves: Journal club: SURMOUNT-4 and continuation versus withdrawal — a second dataset Everything below is what sits behind that. Posting a small dataset on SURMOUNT-4 and…
|
+58 | 64 | 17k | 13mo |
|
List price versus negotiated price versus what you pay
On the subject in the title: List price versus negotiated price versus what you pay Working notes rather than a conclusion. List price versus negotiated price — I have the observation and I do not trust my…
|
+99 | 115 | 11k | 3mo |
|
Preprint servers and what screening they do — the long version
On the subject in the title: Preprint servers and what screening they do — the long version Working notes rather than a conclusion. A preprint question rather than a question about the finding in it. The…
|
+7 | 11 | 26k | 4mo |
|
Coming back to: Column chemistry choices for a 40-residue peptide
On the subject in the title: Column chemistry choices for a 40-residue peptide Working notes rather than a conclusion. Column chemistry choices — I have the observation and I do not trust my interpretation of…
|
2 | 43k | 16mo |