False claims in a widely-cited paper

222 pointsposted 5 hours ago
by qsi

57 Comments

thayne

10 minutes ago

> their policies allow only authors to request corrections

Say what now?

So the only way to get a correction for a paper is if the author is willing to publicly admit they messed up? Something that an unethical researcher is very unlikely to do.

zx8080

2 hours ago

Hey, don't take kids joy! The paper was cited thousands of times, lots of uni students built their early career using it!

zx8080

2 hours ago

Somewhat unrelated but relevant thought: from software engineering experience in large orgs, correction of any issue rarely worth any effort. AI will drive commiting more and more papers with less and less review. The review takes effort, too much in the age of easy generation.

With this, science will probably lose trust even more in the coming years.

ernsheong

2 hours ago

I'm very confused because there are 2 Andrews, the author in the blog post only states "Andrew", and by the list of Authors the author seems to be Andrew Gelman, but the slug in the first link is "aking", and then there is also Andrew King, lol.

flexagoon

an hour ago

Andrew King seems to be the person who published the original exposé of the paper:

> The above story came from my occasional collaborator Andy King [...]

pjdesno

2 hours ago

Are there any factual allegations on that page? All I could find was "the method described in the paper is not the method the authors actually used", without any elaboration.

I'll add that the reaction of most of academia will be "It's in a management journal - of course it's nonsense."

banana_sandwich

4 hours ago

“Professionals” in traffic engineering still religiously cling to “standards” that are largely based on BS served up by auto companies pre 1940.

Many such cases of this, it seems.

komali2

3 hours ago

At least your traffic engineers set standards. In Taiwan often the standards come straight from the legislative yuan, aka just vibes laws from people who are driven around in private cars their whole lives.

nativeit

3 hours ago

Don’t worry, we’re getting there. They just started dismantling what they refer to as the “administrative state”, but which largely deferred substantial questions requiring skill and non-partisan judgement to their respective experts. It was never perfect, nor free from partisan and/or economic concerns, but the replacement appears to be self-interested narcissists and sycophants and their personal fiefdoms, with precious little space for competence, logic, or integrity.

shrubby

an hour ago

At least now the masks (and Musks) are off.

It was never between the left and the right or any other false dichotomies, but always between the Epstein-class and the actual human beings.

The question now is that do the normal people realize and act on the fact that the elevator to Epstein class was never working. Or even better, they don't want to become the zillionaire class husk of a human.

tovej

13 minutes ago

You say it's not left v. right, but it sure sounds like you think the right is what's causing inequality. Which I agree with.

melagonster

3 hours ago

This is just for reference: last time someone tried to address this issue, they found that no one was willing to vote for them again...

rudderdev

16 minutes ago

Peer reviews need to be more transparent and accountable. Otherwise, we are sure to lose to the misinformation war that is rapidly reaching its peak, thanks but no thanks to AI.

t0lo

5 hours ago

So we're firmly in the era of few people caring about few things now aren't we.

lotsofpulp

5 hours ago

I have always assumed the further away from math and physics a field is, the higher the probability of any given “research” to be false. Even biology, I might give 50% odds at best, but that is due to the difficulty of observing and measuring in that field. Anything past biology might as well be as fiction due to how easy it is to come up with whatever results you want.

chromacity

4 hours ago

I think that theoretical math and physics are special, but probably not in the way you assume. It's just that there isn't a whole lot of grant money, prestige, or influence associated with them (unless you accomplish something truly exceptional).

Computer science is very close to math and should be even easier to verify, but there's plenty of dubious results published every year, simply because it's more profitable to game the system. For example, I'd wager that 50%+ of academic claims related to information security are bogus or useless. Similarly, in the physics-adjacent world of materials science, a lot of announcements related to metamaterials and nanotech are suspect.

pjdesno

2 hours ago

I would point out that most products are useless, and either fail or replace other products which weren't any worse. None of which prevented me from cashing my paychecks for the first half of my career when I worked in private industry.

Most scientific research represents about the same amount of improvement over the state of the art as the shitty web app or whatever that you're working on right now. It's not zero, but very few are going to be groundbreaking. And since the rules are that we all have to publish papers[*], the scientific literature (at least in my field, CS) looks less like a carefully curated library of works by geniuses, and more like an Amazon or Etsy marketplace of ideas, where most are crappy.

[* just like software engineers have to write code, even if the product ends up being shitty or ultimately gets canceled]

Neither of us are going to be changing how the system works, so my advice is to deal with it.

rain-princess

18 minutes ago

Hey I have also called research a marketplace for ideas before! cool.

kmaitreys

3 hours ago

Great take. I have seen the discussion on this often gets turned into a hard vs soft science debate where in actuality it's just simply about money.

stogot

3 hours ago

I track these across all fields. It’s money and prestige and arrogance and ignorance and “keep my job” and more

suzzer99

2 hours ago

> Anything past biology might as well be as fiction due to how easy it is to come up with whatever results you want.

I used to work for the leading statistical expert witness in the country. Whenever I read something like this:

> The empirical strategy in Eccles, Ioannou, and Serafeim (2014) rests on a demanding requirement: the “treated” and “control” firms must be so closely matched that which firm is treated is essentially random. The authors appear to recognize this, reporting that they used very strict matching criteria “to ensure that none of the matched pairs is materially different.”

I just assume they kept trying different "very strict matching criteria" until they got the matches they wanted. Which is basically what we did all day to support our client (usually big auto or big tobacco). We never presented any of the detrimental analyses to our boss, so he couldn't testify about them on the stand even if asked.

Although in this case it sounds like the authors couldn't even do that, and just fudged the data instead.

sigbottle

2 hours ago

Observing, measuring, but also repeatability and ground truth.

Math (and theoretical adjacents like TCS) claim not to make any fundamental claims about the actual world (compared to 17th century philosopher-mathematicians like Leibniz), and physics studies the basest of, well, physical phenomenon.

I don't even know how you would begin actually rigorously studying sociology unless you could start simulating real humans in a vat, or you inject everybody with neuralink. (but that already selects for a type of society, and probably not a good one...)

To be clear, I don't think all sociological observations are bad. However, I tend to heavily disregard "mathematical sociological studies" in favor of just... hearing perspectives. New ones and unconventional ones especially, as in a domain where a lot of theories "seem legit", I want to just hear very specific new ways of thinking that I didn't think about before. I find that to be a pretty good heuristic for finding value, if the verification process itself is broken.

erikerikson

4 hours ago

I appreciate that physics and math are simple, reductive, and first principles enough to be tractable. Solving easier problems always has better optics so long as all problems look equivalent. I'm guilty myself, only rising to neuroscience and relatively superficially at that...

p-e-w

4 hours ago

I fully expect that future programs for formalizing mathematics will reveal that most sufficiently complex proofs are riddled with gaps and errors, and that some of them actually led to false results.

Annals of Mathematics once published a supposed proof (related to intersection bodies IIRC) for a statement that turned out to be false, and it was discovered only by someone else proving the opposite, not by someone finding an error.

Georgelemental

4 hours ago

Quantum physics, due to its own "difficulty of observing and measuring", has its fair share of nonsense too

hnburnsy

3 hours ago

  If it doesn't have "science" in the name, it's a science 
  If it has the suffix "logy", it's a semi-science
  If it has the word "science", it's not

austinjp

4 hours ago

Oh I'm sure the grifters will find ways in. The other disciplines may have provided a "moat" for the past few decades, but it won't last forever.

ls612

4 hours ago

Management Science, how am I not surprised? They have the worst rep of any Econ/Econ adjacent field for good reason.

SanjayMehta

4 hours ago

Management, Political, Economic, Social Sciences are not sciences.

tovej

11 minutes ago

Social sciences definitely are.

Analemma_

3 hours ago

The consequences here don’t seem all that bad, it’s just a silly management fad. By contrast, “Growth in a Time of Debt” from Reinhardt and Rogoff steered multiple national governments into pointless self-destructive and immiserating austerity, despite being equally bunk, and none of the authors ever saw any consequences for that either. You can’t even blame that one on “management science”, it was a straight macroeconomics paper.

There’s no accountability for junk science, especially if it props up the political status quo.

dannyobrien

an hour ago

So, I was interested in this statement, and looked into it barely, and on one side, its conclusions were replicated in a number of other papers[1] (despite the headlines, three years after its publication, of a simple calculation error)[2]. I'll state that neither of these points are a slam-dunk if you're a member of one political side or another. If you're a believer in austerity, you'll look at the corroborating studies; if you think that was a bad policy choice, you can argue that they're all junk science, pushed out by supporters of the status quo.

I suspect what it narrowly shows though is that this isn't the same category of error as what's being discussed here.

[1] https://www.mercatus.org/research/policy-briefs/debt-and-gro...

[2] https://www.bbc.co.uk/news/magazine-22223190

malshe

3 hours ago

That paper wasn’t even peer reviewed. If I remember it correctly it was published in the AER Papers and Proceedings.

qsera

3 hours ago

political? what about business? If junk science can help a huge market to keep selling, I think it will be the biggest blocker.

tdeck

3 hours ago

> pointless

I'm sure it benefitted some people.

paulpauper

5 hours ago

Peer review is a joke still and exists now to please deans (for hiring and promotion) and enrich publishers. Bad papers get published if it reaffirms the biases of editors, and actually good and original stuff gets rejected. Rather than facilitating the exchange of knowledge, it acts as a barrier, especially when it cannot even be relied on for quality control.

Aperocky

4 hours ago

For almost the last two centuries, we have grown accustomed to the fact that theory derive practical and useful results. This made academic system flourish including practices such as peer review, etc.

But for the millenniums preceding that, it was the reverse, practice and observation drove theory, and I wonder if we are going back to that and practice and once again dominate how we discover new things as a civilization.

JoeOfTexas

4 hours ago

Status quo changes at the speed of snail.

Tostino

4 hours ago

Usually when people die and vacate their seats of power in society.

BobbyTables2

4 hours ago

Even in more respected journals, peer review is often done by beleaguered grad students who could be still relatively new to the field. They lack the experience to look at things with a critical eye.

tylerhou

3 hours ago

Graduate students! Hah! ML researchers can only hope their papers at ICLR/ICML/NeurIPs are reviewed by graduate students!

sillysaurusx

5 hours ago

> and actually good and original stuff gets rejected

This seems to be the key part. Are you sure that's true?

In other news, (a) apparently you can now submit URLs with anchors to HN, previously a perennial problem; (b) this submission anchors to a comment that just says "I will try this. Suggestions welcome" with no further context.

Ironically, (b) was exactly why (a) was disallowed for the longest time. Anchors are usually a mistake by the submitter, since whatever's being anchored to usually has a permalink. Except Github. Hello, Github comments.

hansvm

4 hours ago

> good stuff rejected, are you sure that's true

In the academic circles I frequent, it's not true. Any one journal might reject the good stuff, but it doesn't take more than a few applications to find a journal who recognizes it, and the cost of producing the research is so high that with the current career incentives it'd be ridiculous not to continue submitting. That does mean that journal "quality" matters less than you might think, but I don't think anyone's surprised by that notion either.

Errors the other direction are more common. I'll state that as an easily verified fact, but people like fun stories, so here's an example:

One professor I worked with had me write up a bunch of case studies of some math technique, tried to convince me that it was worth a paper, paid somebody else to typeset my work, and told me to compensate him if I wanted my name on the "paper." I didn't really; it was beneath any real mathematician; but there now exists some journal which has a bastardized, plagiarized version of my work with some other unrelated author tacked on available for the world to see [0], and it's worth calling out that nothing about the "paper" is journal-worthy. It's far too easy to find a home for academic slop, and I saw that in every field I spent any serious amount of time in.

[0] https://www.m-hikari.com/ams/ams-2019/ams-9-12-2019/p/jabbar...

paulpauper

13 minutes ago

I mean at a top or middle ranked journal. There are tons of predatory journals that will publish anything

qsi

5 hours ago

Ooops, sorry... I cannot edit the URL in the submission. I should have checked.

sillysaurusx

5 hours ago

No it's fine, it thoroughly amused a HN nerd like me. I've been keeping track of how HN works for well over a decade, and noticing small changes like this is something that's genuinely gratifying. The mods will no doubt be by to clean up the url shortly.

I'm just relieved you can submit anchored URLs now. I once stayed up for a few hours trying to submit some work I made as a github comment only to be disappointed that it would always redirect to the toplevel issue.

cwillu

4 hours ago

You can always send a short polite email to hn@ycombinator.com with corrections you can't make yourself

qsi

3 hours ago

I did, thanks for the suggestion.

zer00eyz

4 hours ago

> and actually good and original stuff gets rejected

This isnt a new thing though.

Cantor: https://en.wikipedia.org/wiki/Controversy_over_Cantor%27s_th... they didnt just reject him, they basically publicly beat him down, and drove him away from math and into depression.

David Bohm: https://en.wikipedia.org/wiki/Quantum_potential spent years on the outside for having his ideas on this.

Geoffrey Hinton: was considered a quack and an outsider for YEARS because of his ideas on AI... the breakthrough he spawned was done on a shoestring of a budget (read: home pc).

Edit: I forgot John Yudkin: Pure White and Deadly, talking about how bad sugar is for you in 1972...

Rejected by the mainstream academics, and in a brutal way, happens a LOT more than we think.

bsder

4 hours ago

Katalin Karikó and her work on mRNA: https://en.wikipedia.org/wiki/Katalin_Karik%C3%B3

Her advisor, Suhadolnik, was a gigantic asshole and paid no price whatsoever for it. University of Pennsylvania demoted her and denied her tenure and nobody involved paid any price for that. etc.

arjie

3 hours ago

There's this 'criterion of embarrassment' / 'cui bono' sort of standard[0] that really helps judge these things. So many people perform science that seems to always confirm the positions they've held. All the "society is terrible today" people like to quote LendingClub's "70% of Americans live paycheck to paycheck" without knowing it's LendingClub content marketing. The snail darter guy happened to find a novel species that is endangered and genetically identical to a non-endangered one just in the place where he was trying to get a dam banned. The sustainability guys find that companies that focus on sustainability do better. The diversity guys find that companies that focus on diversity do better. A scientist who gets a grant from Philip Morris finds that cigarettes aren't bad for you.

It reminds me of something my dad said while watching Generation Kill - a TV show adapted from the written work of an embedded journalist in Iraq. The show, made by Americans, depicts the US armed forces as ramified through with bumbling fools seeking glory with a few competent people in there. So we finish watching the series and my dad says "Only the Americans would make a show like this" and it's somewhat[1] true. I think perhaps that being able to create a machine that tells you the truth is crucial to success and I feel that the US's peak period as unipolar hegemon (Gulf War I to the end of Obama I) this was more the case than it is today, though this is more of a feeling than anything I have verified.

It also reminds me of an old sort of censorship, one which George Orwell talks about in regards to Animal Farm[2] - a book that was criticized because it perhaps harmed the greater cause of communism. There's too much to quote in his essay because I find the whole thing worthy of reading, but here's one bit:

> Both publicly and privately you were warned that it was ʻnot doneʼ. What you said might possibly be true, but it was ʻinopportuneʼ and played into the hands of this or that reactionary interest.

...

> Is every opinion, however unpopular – however foolish, even – entitled to a hearing? Put it in that form and nearly any English intellectual will feel that he ought to say ʻYesʼ. But give it a concrete shape, and ask, ʻHow about an attack on Stalin? Is that entitled to a hearing?ʼ, and the answer more often than not will be ʻNoʼ. In that case the current orthodoxy happens to be challenged, and so the principle of free speech lapses.

There is even today an orthodoxy of sorts and if you were to contradict it, it is considered sinful to say so. I'm Indian so perhaps it is safe for me to use this as a race of choice but what if it were found that Indians actually are less smart than, say, White people. Could such a thing be published if it were true? People often say "what are you going to do with that information?" and somehow I don't share that view that all science must necessarily immediately deliver applied benefit. Knowing is good for its own sake. Truth is good for its own sake. Or at least that's what I believe.

I suppose I'll only know through the period of my own life whether this belief is adaptive. Who knows, a present or future power might be one formed entirely through inaccurate data and information[3], and we might be as Orks and painting things red might make them faster because we believe it so in sufficient numbers.

0: Obviously there are limits. Eli Lilly benefits from GLP-1RA drugs working well but they do in fact work well.

1: Others obviously also make fun of themselves, but something like In The Loop parodies specific people more than the whole machine and its participants. Generation Kill feels much more real a depiction of large organizations and their incentive mechanisms - especially how they grind forward and get the outcomes they want despite everything else. Perhaps my least favourite parts were the emotional-breakdown bits at the end, which I've since found out that the participants themselves said were invented for TV.

2: https://www.marxists.org/archive/orwell/1945/preface.htm

3: Open societies like ours have the problem that external misdirection leaks into internal data but perhaps with sufficient computerization we can keep separate truth and propaganda within the structure of government

bluefirebrand

2 hours ago

> I'm Indian so perhaps it is safe for me to use this as a race of choice but what if it were found that Indians actually are less smart than, say, White people

This is an interesting thought experiment.

I'd like to think that if such a thing was discovered, we would investigate why this is the case. I'd like to believe that we wouldn't just accept this as some kind of de-facto truth, and start treating Indians as lesser

I'm an idealist I guess though