"they are AWFUL at stats."

SF author Michael Flynn was a process control engineer as his day job; he wrote about how designing statistically valid experiments is incredibly difficult, and the potential for fooling yourself is high, even when you really do know what you are doing and you have nearly perfect control over the measurement setup.

And on top of it you're trying to measure the behavior of people not widgets; and people change their behavior based on the context and what they think you're measuring.

There was a lab set up to do "experimental economics" at Caltech back in the late 80's/early 90's. Trouble is, people make different economic decisions when they are working with play money rather than real money.

I was very surprised at how many statistical methods are taught in undergraduate psychology. Far more statistics than I ever touched in engineering for sure. Yet the undergrads really treated statistics as a cookbook, where they just wanted to be told the recipe and they'd follow it. Honestly they'd have been better off just eyeballing data and collaborating with statisticians for the analysis.

The problem with a lot of the “higher free variable” sciences like psychology, ecology and sociology etc

Is they are the ones who need to be at the bleeding edge of statistics but often aren’t

They absolutely need Bayesian competitive hypothesis testing but are often the least likely to use it

> And it isn't a skill issue as much as a cultural one. They teach it wrong

It's definitely a skill issue then

As someone who's part of a startup (hrpotentials.com) trying to bring truly scientifically valid psychological testing into HR processes .... yeah. We've been at it for almost 7 years, and we're finally at a point where we can say we have something that actually makes scientific sense - and we're not inventing anything new, just commercializing the science! It only took an electrical engineer (not me) with a strong grasp of statistics working for years with a competent professor of psychology to separate the wheat from the chaff. There's some good science there it's just ... not used much.

There’s surely irony here

Um, actually I’d say it is the responsibility of all scientists, both professional and amateur, to point out falsehoods when they’re uttered, and not an act of smugness.

Reminds me of Feynman’s Cargo Cult Science speech:

https://people.cs.uchicago.edu/~ravenben/cargocult.html

I read in a study that it takes 10,000 hours to become proficient in statistics /s

I think one would wish the famous ones to be more often replicable.

Thanks for providing the reference, that's useful context. Those are awful replication rates, worse than a coin flip. Sounds like the OP can add their own introduction to their list. From the introduction:

> Most results in the field do actually replicate and are robust[citation needed]

Sure, dump it on the lowest level employee, who has the least training and the most to lose. Punish them for someone else's bad research. Grad school already takes too long, pays too little, and involves too much risk of not finishing. And it doesn't solve the problem of people having to generate copious quantities of research in order to sustain their careers.

Disclosure: Physics PhD.

How interesting would it be if every PhD thesis had to have a "replication" section, where they tried to replicate some famous paper's results.

If you were to meet a "normal" person, would you interpret that as meaning "perpendicular" or as meaning "the kind of person that doesn't look at everything like it's a mathematical expression"?

Yes, the APA certainly has a lot to answer for in their history. The Guantanamo Prison torture scandal is still fresh in my memory.

https://www.democracynow.org/2007/8/20/apa_members_hold_fier...

I thought you were pointing out some bias by comparing the research to previous research from the same authors. It took me far too long to realize that the experiment was from 100 years ago, and you were pointing out that the names were coincidentally the same.

I'm still amazed that wikipedia doesn't have redirect away from its mobile site

Yeah, the marshmallow one taught me to have patience and look for the long returns on investments of personal effort.

I think there may be something to a few of these, and more may need considering regarding how these are conducted.

Let’s leave open our credulities for the inquest of time.

> in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.

The normal distribution predates the general factor model of IQ by hundreds of years.[0]

You can try other distributions yourself, it's going to be hard to find one that better fits the existing IQ data than the normal (bell curve) distribution.

[0] https://en.wikipedia.org/wiki/Normal_distribution#History

I took an IQ test as a high school student, and one of the subtests involved placing a stack of shuffled pictures in chronological order. I had one series in the incorrect order, because I had no understanding of the typical behavior of snowfall. The test proctor said almost everyone she tested mixed that one up, because it doesn't snow in the area where I live.

I have no doubt that IQ tests reproducibly measure the test takers ability to pass tests, as well as to perform in a society that the tests are based on.

I think it's disingenuous to attribute IQ to intelligence as a whole though, and it is better understood as an indicator of cultural intelligence.

I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.

What exactly are they meant to replicate other than other IQ tests? They don't make a statement about anything that is falsifiable, other than that if you give somebody who scores high on a test carefully designed and tested to match the results of previously given IQ tests when given to the same people, they'll tend to match the results that those people will get on other tests that were calibrated in the exact same way.

If you're trying to say they replicate over the lifetime of the same person, I've had a 15 point swing between tests, out of the few I've taken. What did stay constant for me from age 10 to age 40 was my Myers-Briggs test (my dad was a metrics obsessive), and that's obvious horseshit. Consistency doesn't mean you're measuring what you claim to be measuring.

edit: if it matters, scores were between 137 and 152, so exactly an entire standard deviation. That's like the difference between sub-Saharan and European that racists are always crowing about, but in the same person. IQ doesn't even personally replicate for me.

Survivorship bias. You can easily make someone's IQ test not replicate. (Hit them on the head really hard.)

<On average, african americans are taller than white men and have a higher muscular density.>

Are you comparing direct descendants of Yoruba versus descendants of Celts in America ? or mixed descendants of Bantu and Cherokee versus mixed descendants of Anglo-Saxons and Slavs ? In your study would Barack Obama be a person of color or a person of pallor ?

Or is this data you have gathered observing people at Costco. Just checking on your scientific methodology.

Differences are hidden because (1) differences, even small ones, are used to justify discrimination (2) some feel the need to correct for stereotypes (3) these differences often don't really exist or amount to a small effect size.[0]

In the end, we're talking about distributions of people, and staring at these differences mischaracterizes all but those at the mean.

All that matters is who can pass the test.

[0]: I also encourage you to ask ChatGPT/Grok/Claude "men vs women math performance studies." You'll be shocked to find most studies point to no or small differences.

[1]: Malcom Gladwell wrote a great piece about his experience as a runner that seems appropriate to share https://www.newyorker.com/magazine/1997/05/19/the-sports-tab...

> We know for a fact that sex or ethnicity impacts body yet we seem unable to cope with the idea that there are also differences in how brains work.

Here is your error. You’re assuming that a physical difference in morphology is linked to behavioral or neural correlates. That’s not the case, since observed statistical- or group-level differences need not be driven by biology. You’re assuming biological determinism, and the evidence for direct genetic effects on behavior isn’t there.

"on average, women tend to learn languages easier than men": I'm a linguist (although not an expert on second language learning), and I've never heard that. Citation?

Circular reasoning can be used to "prove" anything, so it's not helpful as a basis for policy making.

https://doi.org/10.1093/socpro/spv028

"Are Smart People Less Racist? Verbal Ability, Anti-Black Prejudice, and the Principle-Policy Paradox"

Simple study that implies that you should expect white people who think black people are less intelligent than white people to have about 8-11 fewer IQ points than other white people. Just a survey about racial attitudes, and a test of verbal ability that correlates well with IQ tests.

> Women have, on average, a higher emotional intelligence which is e.g. tied to higher linguistic proficiency. That helps in many different fields and, on average, women tend to learn languages easier than men.

Has this been experimentally shown to be the case with studies that don‘t fail to replicate?

Between studies that fail to replicate and pure conjecture and pseudo-science I certainly favor the former, at least actual studies that fail to replicate can be disproven, your conjectures are just race/sex science and nothing but pseudo-science. I can either take you at your word, or choose not to believe you. I pick the latter.

The most obvious one is the breakdown of trust in scientific research. A frequent discussion I would have with another statistics friend of mine was that that anti-vax crowd really isn't as off base as they are more popularly portrayed and if anything, the "trust the science!" rhetoric is more clearly incorrect.

Science should never be taught as dogmatic, but the reproducibility crisis has ultimately fostered a culture where one should not question "established" results (Kahneman famously proclaimed that one "must" accept the results of the unbelievable priming results in his famous book), especially if that one is interested in a long academic career.

The trouble is that some trust is necessary in communicating scientific observations and hypothesis to the general public. It's easy to blame the failure of the public to unify around Covid as based around cultural divides, but the truth is that skepticism around high stakes, hastily done science is well warranted. The trouble is that even when you can step through the research and see the conclusions are sound, the skepticism remains.

However, as someone that has spent a long career using data to understand the world, I suspect the harm directly caused by the wrong conclusions being reached is more minimal than one would think. This is largely because, despite lip service to "data driven decision making", science and statistics very rarely are the prime driver of any policy decision.

I imagine it's comparable to the damage done when policies are set that are not based on studies.

Let's be candid: Most policies have no backing in science whatsoever. The fact that some were backed by poor science is not an indictment of much.

We rack up quite a lot of awfulness with eugenics, phrenology, the "science" that influenced Stalin's disastrous agriculture policies in the early USSR, overpopulation scares leading to China's one-child policy, etc. Although one could argue these were back-justifications for the awfulness that people wanted to do anyway.

i'm struggling to imagine many negative effects on society caused by the specific papers in this list

I once did a corporate internal management course that was filled with pseudoscience bullshit. I imagine the impact of that course on the company's productivity was net negative. I'm sure lots of orgs have similar courses.

Learning styles have also been debunked for decades though they continue to be used in education. I saw an amusing line in an article that said 90% of teachers were happy to continue using them even after accepting they're nonsense.

And that's just theories that have been debunked (i.e. proven wrong).

You'd think it's so far in the past that it isn't even considered, but Zimbardo was elected president of the American Psychological Association in 2002, which wasn't all that long ago.

In many cases--longitudinal studies are an example, but not the only one--that's not feasible. And it's often expensive--who would pay for it?

Agreed, and the independent lab should be chosen by the publisher, and be kept secret until results are in.

From working in industry and rubbing shoulders with CS people who prioritize writing papers over writing working software I’m sure that in a high fraction of papers people didn’t implement the algorithm they thought they implemented.

Nowadays everyone publishes their code. There’s typically a project page on github.io, a paper on arxiv.org, and a public repo.

And most medical studies. It's just as bad as social psych, if not worse, because there's real money at stake in churning out new drugs.

>some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.

I don't like Giancotti's claims. He wrote: >This post is a compact reference list of the most (in)famous cognitive science results that failed to replicate and should, for the time being, be considered false.

I don't agree with Giancotti's epistemological claims but today I will not bloviate at length about the epistemology of science. I will try to be brief.

If I understand Marco Giancotti correctly, one particular point is that Giancotti seems to be saying that Hagger et al. have impressively debunked Baumeister et al.

The ego depletion "debunking" is not really what I would call a refutation. It says, "Results from the current multilab registered replication of the ego-depletion effect provide evidence that, if there is any effect, it is close to zero. ... Although the current analysis provides robust evidence that questions the strength of the ego-depletion effect and its replicability, it may be premature to reject the ego-depletion effect altogether based on these data alone."

Maybe Baumeister's protocol was fundamentally flawed, but the counter-argument from Hagger et al. does not convince me. I wasn't thrilled with Baumeister's claims when they came out, but now I am somehow even less thrilled with the claims of Hagger et al., and I absolutely don't trust Giancotti's assessment. I could believe that Hagger executed Baumeister's protocol correctly, but I can't believe Giancotti has a grasp of what scientific claims "should" be "believed."

The idea isn't that it is easier to do things when not tired. It is that you specifically get tired exercising self control.

I think that can be subtly confused by people thinking you can't get better at self control with practice? That is, I would think a deliberate practice of doing more and more self control every day should build up your ability to do more self control. And it would be easy to think that that means you have a stamina for self control that depletes in the same way that aerobic fitness can work. But, those don't necessarily follow each other.