Designing a personal experiment that could change your mind — one year on posts 31–60

Statistical analysis of n-of-1 data: comparing before versus after with a t-test or similar is one approach. Plotting the data visually is another. Both are valid.

It is one reading of the data and not the only reasonable one.

A single-person experiment can be genuinely informative if it has a pre-specified outcome, a defined period, and a plan written before the data arrives. Most accounts here have none of those.

Small point, but it is the one that usually catches people.

Confirming post #30 from a second method, which matters more than confirming it from a second person.

Record everything that changed, not just the intervention. Personal experiments are confounded by life, and the confounders are only recoverable if they were written down.

Before the thread moves on from Designing a personal experiment — what is the sample size behind the claim? I am not being difficult; I have seen the same figure quoted from an n of four and from an n of four hundred.

Adding a data point of agreement rather than a data point.

Multiple outcomes measured without a primary one means something will move. Nominate the primary in advance and report the rest as secondary.

Not the answer, but possibly the question that gets there.

Post #34 is right about the mechanism and I think understates the practical bit.

Write down what you would accept as evidence that it did not work, before starting. That one sentence turns an anecdote into an experiment.

It is worth stating the boring hypothesis before the interesting one.

Coming back to post #36, because the follow-up matters more than the original answer.

Measure the same thing the same way at the same time of day. Most of the noise in personal data is measurement protocol rather than biology.

Taking post #38 at face value and following it one step further.

What would change my mind on Designing a personal experiment is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more.

Post #36 and I disagree about the size of the effect, not about the direction.

The useful distinction on Designing a personal experiment is between what was measured and what was inferred from it. Both end up in the same sentence and only one of them has error bars.

I had written a reply contradicting post #37 and deleted it. Here is what survived.

Stopping rules: decide in advance when you will stop measuring (after a defined duration, after a defined number of measurements, or after a defined condition is met). Not deciding in advance means stopping when the result satisfies you, which is bias.

One more caveat and then I will stop qualifying: the sample selected itself.

A photograph or a device reading is a harder record than a recollection, and where one is available it should be the record.

Take it as a starting point and not as a specification.

The interesting personal experiments are the ones where the answer was no. They are also the ones least likely to be written up.

Since Designing a personal experiment keeps coming up, it should probably be a maintained page rather than a recurring thread. I am happy to draft it if someone with more direct experience will review it.