The reproducibility crisis and other problems in science: John Ioannidis [video]

54 pointsposted 7 months ago
by domofutu

17 Comments

dgfitz

7 months ago

I spent two days trying to get repeatable data from a piece of hardware. Was not able to replicate a dataset once, found out we had a bad high-pressure hose for the brake system and the numbers were all broken.

I have no fucking clue how a soft science study could replicate itself.

stogot

7 months ago

If even an author can’t replicate it, shouldn’t that make the results meaningless?

domofutu

7 months ago

There's also all the (potentially relevant) factors no one's considering - https://www.thetransmitter.org/methods/mouse-housing-tempera...

aaron695

7 months ago

A bunch of women going on about giving mice air-conditioning or not?

Probably. We've filled science with Imposters. It seems predominantly women, they are the largest change in science.

The worst non-replicable sciences are also female majority.

Busywork on mice conditions is exactly the false science we are talking about. More excuse why it's not reproducible when mice are mice and are great sub-ins for humans.

"In mice" is the meme create to excuse the garbage that's produced. The studies never originally worked on mice, but now they have an excuse why they don't work on humans.

I wouldn't necessarily blame it 100% on women though. But I see your point.

mapt

7 months ago

This guy doesn't get to say shit to us about literally anything after what he did. 2020's John Ioannidis stood on the shoulders of giants (2005's John Ioannidis) and was one of the first "Experts" to proudly proclaim that in his professional judgement, COVID-19 was no big deal and didn't warrant countermeasures.

https://sciencebasedmedicine.org/mistakes/

If you choose to become a policy entrepreneur and your success ends in a Holocaust-scale outcome, you get to shut up, retire, and thank your lucky stars we live in a society that strongly discourages blood debts. That's the deal. No book tour, do not pass go, do not collect $200, and stay the fuck off social media.

ttoinou

7 months ago

Seems like he was correct though. This article doesn’t debunk him, you can’t use official statistics to debunk a claim that the statistics are not computed correctly

decUser3

7 months ago

> in an effort— well intentioned—to control the coronavirus, we may inflict great damage on ourselves.

And you believe he was wrong and you believe Covid was a "Holocaust-scale outcome" ? He said the rates would be exaggerated and now with the benefit of hindsight, although some would claim common sense after seeing the sensationalism of the media, we can see he was correct. Or do you believe that's not true ?

d0mine

7 months ago

It is worth pointing out that compared to * everything else on the internet, science is the epitome of truth, facts, reproducibility, etc.

It might not appear so at the first glance due to Gell-Mann Amnesia effect.

7thaccount

7 months ago

I don't think this isn't what the reproducibility issue is about.

The problem is there is a broken system in place that rewards publishing in high volumes, only getting positive results, and also doing research that supports those in charge. That doesn't mean new ideas don't win out when clear evidence exists but it is the exception.

All of this leads to economic incentives to lie with the data and then you get to scientific papers that people are making decisions based off (like where to put research dollars for Alzheimers) that are fraudulent. This is not good at all.

vacuity

7 months ago

The practice of the scientific method is the path to empirical truth. This doesn't necessarily mean that a nominal scientific finding is true. It is likely true that, on a given topic, an acredited scientist in that field has fairly correct opinions, but I would not take this too far. Conflicts of interest, personal biases, incentives to obtain results, simple lack of reproduction, corrupt peer review, etc. are clearly issues, and it's all the more unfortunate that we can't even say just how deep these issues run.

Science is timeless and powerful. Scientists are human beings that nominally, preferably, fallibly practice science.

karaterobot

7 months ago

Gell-Mann amnesia is when you notice that somebody who is writing about something you're an expert in has made critical mistakes, and should not be trusted, but then you go on and trust people writing about topics you don't know as much about, failing to recognize they're probably also making critical mistakes about those other topics. I'm not sure the connection here.

Regarding the comparison between science and folk explanations on the internet, I think it's reasonable to hold science to a higher standard than we hold the general population. If most published research findings are false—as this video claims—and most of what idiots on the internet say is also false, that's neither an equivalence nor a victory for science. On the contrary, it erodes the value of science, both literally and figuratively. Right now we need sources of authority.

jMyles

7 months ago

This paper was life-changing for me as an undergrad (and I didn't discover the rest of his body of work until I ran into it later here on HN in 2010 or so).

We are blessed as a species that John stuck to his principles - and his thirst for empiricism - during the COVID-19 panic, and supported / encouraged his colleagues to do likewise.

This video is only the first 12 minutes of the talk. The rest is here (though it is possibly semi-paywalled? It let me watch it, even though it said it was going to make me sign up for a trial):

https://iai.tv/video/why-most-published-research-findings-ar...

matthewdgreen

7 months ago

My understanding was that Ioannidis hugely underestimated the IFR of COVID and he did it mostly by cherry picking a handful of small-sample-size studies that were friendly to his political views. It was very much a “your heroes will absolutely let you down” moment in my scientific life, and to the extent that non-scientists have forgotten the episode, that’s kind of what I expect.

https://www.dailymail.co.uk/news/article-8843927/amp/Just-0-...

MrMcCall

7 months ago

Thanks. This guy (John Ioannidis) is the real deal. You can tell both by the dense, detailed fact sets, presented one after the other in logical order. And hearing the honest, intelligent tone of voice ensures his fidelity.

"The voice never lies." --Blind woman speaking to a friend

And his story about his hearing of Theranos is lowkey hilarious. And topical, because he's a Dunning-Kruger true-expert.

sdenton4

7 months ago

During the early pandemic, he underestimated the rate of fatality from COVID (which, remember, was much higher before vaccines and paxlovid were deployed), and forcefully advocated for policy based on his lower fatality rate estimates. It was a stunning display of hubris: working on very limited information, he was pushing incautious policy responses which could have cost millions of lives.

https://en.m.wikipedia.org/wiki/John_Ioannidis