Temporal bias: older data in a dataset might reflect conditions (supplier, formulation, context) that have changed. Newer data is more current.
Posting this under the heading it deserves: A community side-effect dataset, with its response rate and biases Everything below is what sits behind that.
Community side-effect dataset — I have the observation and I do not trust my interpretation of it, so I am posting the observation and holding the interpretation back.
Numbers, method and the conditions under which they were collected are below. Interpret them however they warrant.
Self-reported, unblinded, self-selected data has known biases and is still worth collecting, provided every one of those words appears in the description.
Worth one more sentence than it usually gets.
Post #4 is the version of this I will quote in future. One addition.
Having read the whole community side-effect dataset thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.
Where a value is derived rather than measured, mark it. Derived columns get treated as observations the moment the file leaves your hands.
It is the sort of thing that seems obvious in retrospect and was not at the time.
Bias toward positive outcomes: datasets collected by members are biased toward people who found the compounds useful. People who did not respond do not return. People who had bad outcomes might have left the community.
Version the file rather than editing in place. A dataset that changes silently under an analysis makes the analysis unreproducible.
If it helps: the failure mode here is usually boring rather than dramatic.
Independent test results are the most valuable data this community collects, and they are only comparable when the method is captured alongside the number.
Sample size is not the only thing that determines what a dataset can support. Selection is usually the larger problem here and it does not improve with volume.
Small methodological point on community side-effect dataset: repeating a measurement is cheap and resolves most of what is being argued about here at no cost to anyone.
Reading rather than answering, but this is the post I would point somebody at.
Units in the column header, always, and the same units down the whole column. Mixed units in one field is the commonest defect in shared spreadsheets here.
If that reads as pedantic, it is, and it has saved me twice.
On post #13 — agreed on the reasoning, with one qualification.
A dataset is only useful if the collection method is described alongside it. Numbers without a protocol are a list rather than data.
A modest claim, modestly supported.
How to contribute: if you have longitudinal data you want to add, the format is simple: date, measurement, context. Contact the maintainer of the specific dataset.
Where I part company with post #17, and it is a narrow parting.
Aggregating first-hand accounts does not produce evidence of the kind a trial produces. It produces a description of who chose to post, which is a real thing and a different thing.
Not disagreeing with anyone above, just adding the bit I keep having to look up.
Post #17 is the version of this I will quote in future. One addition.
Date every record. A dataset assembled over two years without dates cannot distinguish a change over time from a change in who was contributing.
It is worth stating the boring hypothesis before the interesting one.
Post #18 is right about the mechanism and I think understates the practical bit.
Publishing the raw records alongside the summary is what makes a dataset checkable. A summary alone asks for trust that nobody has earned.
Adding a source would improve this post and I do not have one to hand.
Coming back to post #20, because the follow-up matters more than the original answer.
Where a value is derived rather than measured, mark it. Derived columns get treated as observations the moment the file leaves your hands.
Not the answer, but possibly the question that gets there.
Following, with nothing to contribute beyond having asked the same thing elsewhere.
Confirming post #22 from a second method, which matters more than confirming it from a second person.
The most useful dataset this community could hold is boring: lot, supplier, service, method, date, result. That is enough to answer most of the questions people ask badly.
It is one reading of the data and not the only reasonable one.
Limitations of datasets: all community-collected data has limitations. The population is self-selected (people in this community are not representative of all people using these compounds). Reporting bias is real (remarkable outcomes get reported; mundane outcomes do not).
Small point, but it is the one that usually catches people.
What I would want before treating community side-effect dataset as settled: the method, the sample, and whether anyone tried to find the opposite result. Two of the three are usually missing.
Independent test results are the most valuable data this community collects, and they are only comparable when the method is captured alongside the number.