dgfitz
a month ago
I spent two days trying to get repeatable data from a piece of hardware. Was not able to replicate a dataset once, found out we had a bad high-pressure hose for the brake system and the numbers were all broken.
I have no fucking clue how a soft science study could replicate itself.
stogot
a month ago
If even an author can’t replicate it, shouldn’t that make the results meaningless?
avs733
a month ago
not neccesarily? It depends on what the claims are. If prior art says 'this is impossible' then a study with one result that is hard to replicate is an interesting insight about the margins. If it is commonly accepted wisdom, we may need to address what that is. If it is your experiment, it doesn't mean your result is wrong but it means it should be significantly questioned. If I say its more that is on me...if I say it is what it is then the results are the results.
I think it is a growing, and problematic, misconception that science is about just reporting facts. It is about adding knowledge...soemtimes that is new facts, but more often it is theories, observations, and questions increasingly at the margins of randomness and probability as our species grows and learns.
inglor_cz
25 days ago
Substack psychologist Adam Mastroianni argued that if you need advanced statistics to tease out some effect in the soft sciences, it is actually likely that the effect is bogus.
How many of those non-reproducible studies showed a robust effect, in contrast to the barely there effect?
aeim
25 days ago
so, measuring progress by lines of knowledge added?
speaking openly
that kinda science seems like a bullshit factory (a very lucrative one, of course)
and analogous to an llm hallucination — shallow satisfaction of constraints by enumerating a problem space irrespective of contextual appropriateness, a liability, and forgotten as soon as real value comes along
surely the best science isn't just reproducible by scientists — engineers can get the grubby mits on it, to improve (ok, shape) the circumstances of our lives, and survival
that other stuff is busywork, not "literature". they are very different things...