Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability.

The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it:

1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires.

2. Even if some subset of people did write down a definition, nobody is checking anyone is following it.

So people conflate science the Platonic methodological ideal with Science™, the institutions that claim most forcefully to be its avatar. But those institutions have neither a definition nor any enforcement arm, so fraud and abuse runs rampant. There's no reason why anything should self correct in this environment outside of corporate science, where the self-correction mechanism is managerial intervention or, in the absence of that, bankruptcy.

> Science is self-correcting, but sometimes

As long as people mislead or make mistake unknowingly, it's fine to rely to self-correcting mechanism of science. The institutional rot is much deeper. It is not good to leave all this fraud to self-correcting mechanisms of science. We also have "trust" to worry about.

It's a bit like how people believe markets are self-correcting. Yes, only over an infinite time horizon, in theory. Edit: and like chemical equilibrium, you could say it depends on the "rate of wrongness production" compared to the "rate of correction". Edit edit: self-correction of course must fight against the higher virality of work selected for its impact rather than correctness, and incorrect work can then spawn further incorrect work, like an autocatalytic process or epidemic.

I have been having the opposite problem with my papers lately.

Instead of glossing over mistakes, the reviewer report missing information which is clearly stated in the text.

This is quite frustrating and hard to stay polite after the fifth "please just read the text."

I really wish there was some serious rework of the publishing system. Nobody is happy about it but nothing seems to be moving.

Would be nice if you kepy ypur career and created some magazine / journal that focuses on checks and pointing out flaws.

That cheaters would dread being included there (would probably be a thick journal too)

Even as imperfect as it is, it's infinitely better than the alternatives like belief in religion/religious texts/pastors or anybody else who is just making shit up based on whatever they think sounds good. None of that is falsifiable or makes any attempt to iteratively arrive at anything true or real.

There is literally nothing better that we have other than the scientific method. Yes, science is self-correcting, even if it is slow or whatever else. Nothing else that we have can approach even that bar.

>I made a promise to myself that if no real action is taken, I will seriously consider changing careers.

Personally I think you should name and shame the author as well as the publications and the names of the peer "reviewers" who clearly did sweet-fuck-all.

The fraud that Data Colada presents here is unambiguous. It was also no accident in my opinion that they are specifically mentioning that the 3 Excel files were “Last saved by” “Dan Ariely”

I'm sure the statute of limitations has expired for this, but I really wish the Feds would prosecute cases like this as wire fraud instead of leaving insular academic institutions to self-police. This type of fraud is done for personal gain and self-enrichment and causes real, identifiable harms.

Reminds me of Francesca Gino for data fabrication. She was also a best seller of sorts.

https://en.wikipedia.org/wiki/Francesca_Gino

Dan Ariely's fraud is pretty widespread.

He sold his research to an insurance company, who should have known better: https://www.lemonade.com/blog/oh-behave/

We'd find problems with many if not most published studies if we scutinized them the same way. People like him get discovered because they step out of their academic bubble into the light of the world stage.

And mind you this is psychology, the entire discipline is only borderline scientific anyway. But even studies in the hard sciences are very often flawed if not outright fabricated. Adjusting data sets to fit the desired result (who'd ever know as long as you do it properly), changing the experiment and measure until they get what they they want, repeated measuring until the "right" data for the paper appeared randomly - those are all things I personally observed in the lab.

Not to even get started on the house of cards that is basing ones flawed assumptions on other flawed studies without even the slightest attempt at replicating their results first. It's standard practice to simply presume what's been published (especially by friendly authors) can be regarded as reliable.

At this point the surprising part isn't that another Ariely result has problems, it's that some of these datasets apparently went decades without this kind of basic scrutiny

> It revealed that Ariely had a longstanding relationship with Epstein, including having visited him at his New York home on several occasions.[45] One of the documents is an image of an email from Ariely to Epstein, in which he requests the name and email of a "redhead" that Epstein had previously introduced to him

I mean, I've always consigned 100% of NYT self help slop to the bin of books for people who aren't smart but desperately want to seem like they but this is just a little much now, Dan

Ah yes also the author of “The Honest Truth about Dishonesty”. The most amazing thing is he is still a professor at Duke. Wonder if he focused much on academic dishonesty in his book…

"Ariely is named 636 times in the Epstein files".

Yikes.

> In 2006, when he was a professor at the MIT Media Lab, Ariely conducted experiments including administering electric shocks with a research assistant who had no human–subject training. […] Ariely confirmed this and said that he wasn't aware that the research assistant did not have the needed one-hour online human–subject training.

A one-hour online training is all that’s needed to be considered ready to conduct experiments involving things like electric shocks on human subjects?

Forget about this guy, the entire field needs to be burned to the ground and rebuilt from scratch.

Replication is unpaid QA on someone else's product. Nobody gets tenure for finding the bug, hence the field has been meaning to get around to it for 20 years.

> The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now

They were just putting it off.

Replication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science:

https://blog.plan99.net/replication-studies-cant-fix-science...

N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:

https://www.fantasticanachronism.com/p/whats-wrong-with-soci...

> Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.

So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.

The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.

So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.

Why do we even accept anything as true without replication by default?

"easy to replicate (no fancy equipment needed, just ask some people to proofread)"

? This was a human subject study, that would cost $$ and there would be very little upside to confirmation and only a little notoriety with a failure to replicate (which by the way is not dispositive, people might need multiple failed replications to know who to believe, depending on the size of the replication). Maybe you didn't read the post: this was data fraud discovered by statistical irregularities. That is indeed becoming easy to spot with more widespread statistical literacy, at least for unsophisticated jobs like this from decades ago.

> Is there some way to change incentives to get people to try and replicate?

Sure. Pay them to do it.

> I came here for a procrastination joke comment and didn't find it, but I suppose someone will get around to it eventually.

Too soon; they'll all do it later.

Here's one. I always leave my procrastination for later.

I’ll do it later

Not all science has this issue. But to reference Thiel, a lot of the sciences with "science" in the name, seem to.

That rule would make it difficult to report on a failure to replicate a published paper.

In the physical sciences, any sufficiently important paper is independently verified. Sounds like cost/effort of these multi year studies is not faesible to replicate, but then again, I know nothing about social science so w/e

> The replication crisis in science is real, and it won't take long for no one to believe anything anymore.

If you think replication in the sciences is in crises, wait till you see the field of psychology.

It is interesting how all these famous non-replicating psych paper results feel reasonably intuitive to me. Like I can buy that having a more structured and detailed deadline helps you procrastinate less. I can even buy that doing a goofy little "power pose" exercise kind of hypes yourself up to give some self confidence and do better on an interview or whatever.

You would almost expect the opposite and that all the famous studies would be the ones that run counter to intuition. Maybe those get more replication attempts, or people feel less confident faking them.

The direction is intuitive, but the effect sizes are, as you say, unusually large. I'm no statistician, but if you have an effect size that's several multiples of the standard deviation, that would mean that it dominates the natural variation in results right? So you're kind of making an implicit claim that other factors in play (for eg in this proofreading task, how busy someone is or their reading level) are peanuts in comparison.

Further, if there are these relatively simple factors like how a deadline is expressed that make you several times better, you'd expect anybody who stumbles into several of them and stacks them together to be absolutely off the charts. Somebody who listens to Mozart, gets the right priming, doesn't deplete their ego, and receives short deadlines is suddenly like 50X more productive, which doesn't seem to match reality.

Per the post, the original authors have requested that the original paper be retracted. That’s well above the bar of “not doing much”.

There is no alternative to deadlines because time is money, and those downstream of you can ill afford to wait on your deliverables. Part of management's job is to determine how much time it should take so a reasonable timetable that stakeholders can agree to can be set, and then hold you accountable for delivering within that time frame.

I wish we could revive Tim Bryce's mind with AI, because I'd love to hear what he'd have to say in a podcast about this topic:

"Fraud in a procrastination study? You don't say! I don't understand why all this taxpayer money is going to study procrastination in the first place. Procrastination is a well-known phenomenon with a well-understood cause: laziness. Procrastinators didn't have a boot placed to their behinds growing up, so they reached adulthood slacking and coasting on their trust-fund money or couch-surfing at their friends' places, mooching off others. You see this a lot in modern software organizations, especially among the programmers. And now those programmers are all crying because they're about to get replaced by Claude Code. Boo hoo! The military knew how to deal with procrastinators, too: send them to scrub latrines with a toothbrush, and drum them out if they didn't shape up after that! We don't need millions of dollars in government grants going to academic research to study this phenomenon. It's as old as humanity itself—we called it the deadly sin of sloth. Friends, keep the faith!"

In this case even emailing the author didn't work. The data was obtained via some roundabout method.

But it's not the real reason IMHO. You could use LLMs to write takedowns like this one. I gave ChatGPT the paper PDF and some neutral prompts. Although it didn't go straight to 'this is clearly fraud' - it's too nice for that - it did compute and point out the absurdly huge effect sizes, and give a relatively low confidence the paper's central claim would replicate.

The real reason there aren't bots systematically destroying papers is because universities defend fraud, they don't punish it. You'd just be burning tokens to reveal lots of fraud with no outcome.

There is a lot of bad ML research for the same reason.

It's arguably worse. Jason Arday was doing qualitative work in education. Nobody pays any attention to anything UK educationalists write. Dan Ariely was a genuine superstar and writer of popular bestsellers (with academic content).

Dan Ariely was one of the biggest disappointments and betrayals of my intellectual life. I found the "research" he did to be mind-blowing, groundbreaking stuff at the time, and I told many people his work, sharing his papers and podcasts as a true enthusiast.

I am astonished that he holds an academic-adjacent position anywhere, no matter what his regrets might be today. Private redemption shouldn't be out of reach, but it shouldn't be possible for him to hold a position at any institution concerned with uncovering Truth. He should have resigned out of respect, if his institution was too gutless to do so. It's a tragedy.