Revisiting: How this community labels and discusses unrefereed work

Revisiting: How this community labels and discusses unrefereed work Writing it up because I had to work it out twice and would rather nobody else did.

Reading an unrefereed manuscript properly, and asking whether I am doing it in the right order.

What I do at the moment: check the version and the posting date, read the methods, check whether the analysis matches whatever plan is described, then read the conclusion last. The abstract I read only afterwards, on the theory that reading it first anchors everything.

Is there a step in that sequence which is doing no work, and a step I have left out?

Agreed, and I will stop repeating the version of this I had been repeating.

Adding the measurement that the opening post says would settle it.

Adding the boring version of community labels, because the interesting version keeps getting posted and the boring one is usually right.

Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does.

Where I part company with the opening post, and it is a narrow parting.

Worth separating community labels as a question about the compound from community labels as a question about the documentation. They get answered by different people and only one of them is answerable here.

Post #5 is the version of this I will quote in future. One addition.

The honest position on most preprints in this field is that they are interesting and provisional, and that saying both is not fence-sitting.

The question underneath community labels is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it.

Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help.

Unrefereed work is not worthless. A great deal of it survives review substantially unchanged. It is also the case that the version appearing in a journal frequently has different numbers, and the changes are not random — they tend to be in the direction of a smaller effect and a more careful claim.

Old habit: I write down the expected answer before I calculate it.

A request rather than an answer: could whoever has the primary source for community labels post it? I have seen the claim three times this month and each version had lost a qualifier.

Speaking only to community labels as I have actually seen it, rather than as it is usually described: the effect is real, it is smaller than the thread suggests, and the variance between people is larger than the effect.

Agreed on all of that, and I have nothing to add to it.

Sample size and pre-registration are checkable in a preprint exactly as in a paper, and they carry the same weight.

It is the kind of thing that is obvious once and never again.

Small correction to my own earlier position on community labels. I had the units the wrong way round, which changes the conclusion by an order of magnitude and therefore changes it entirely.

Post #12 put the caveat in the right place and I want to underline it.

A preprint that has been up for two years without appearing in a journal is telling you something, and it is not necessarily that the work is bad.