Idk if we're automating humans out of the publishing loop as much as rapidly automating the production of crap. I had a very similar experience reviewing for EMNLP recently.

We are nowhere near AI being able to judge the quality of research (in fact, one might reasonably state that even most humans can't really judge the quality of research). Most things in society are not like math: we can't automate (via verification) our way out of noise overwhelming the signal.

Folks are willing to entirely abuse the public resource that is faithful, honest reviewing. (This is unsurprising; the abuse of the commons / public resources has been rising for a long time). There isn't a good solution other than something akin to draconian social scoring to limit access to the reviewing system.

It’s important to notice that flagged papers were still accepted; anyone who spends any time around researchers knows the fact that most people are salivating over AI. That’s why they don’t want to punish it, they’re also using it. And they don’t want to create an environment where it is overly punished, at least not while they can take advantage of it. There is only a small amount of serious people complaining, but in my experience the opinion of the vast majority is “stop worrying and learn to love it”. Which brings to the surface a very harsh reality: almost nobody gave a damn about the science to begin with.

We just need journals run by an AI that charges other AI to read them, and then the AI run colleges can promote the AI with the most AI journal entries and citations.

I have been rolling over the idea of “reasoning deserts” in my head, akin to food deserts. Places where economic calculations lead to only a facsimile of the real thing being provided, without the actual components necessary for human health, wellbeing and flourishing.

Most things weren't really reviewed that well as it was. It was always a small fixed set of people that actually played their part in the process in good faith. Those people continue to do to today. As a percentage, they were always a small minority. Just that today, they have become even more of a minority.

We try to iterate regarding efficiency (saying that as a neutral observation). Not sure if that works out. As often the case, sciences that don't serve as a foundation for real world results (can this plane fly faster now?) are in much more danger.

It’s going to go back to the old boys club where personal connections between research groups and institutions will matter more.

> We are very rapidly automating humans out of the academic publication loop

We are rendering it irrelevant. If this is the norm for academia, I’m sympathetic to the folks looking to cut its funding.

Unless someone has discovered some magic sauce for getting AI to write well, I have immense sympathy for anyone trying to wade through these papers.

I may generate slop from time to time, but I do my best to keep it to myself.

> We are very rapidly automating humans out of the academic publication loop here

It's a sad thing. But the monetization and enshitification of journal publications over the decade or so, even prior to AI, certainly has not helped this trend.

What’s the point?

That's correct. And 4–5 is low-balling.

I was forced to review 7 papers.

Now, how are these papers chosen for you?

You get to bid on which to review.

Bid on 30 papers out of 30,000.

I got none of those, and I had to review 7 papers I wasn't really competent enough to review.

Speaking for the *ACL conferences, they are not "random people" in the sense that they must have "(a) at least two papers in main ACL events or Findings, plus (b) at least one more paper in the ACL Anthology or a major ML/AI venue" [1]. So they are published authors.

As for it "being a thing": sadly yes. The number of papers has been steadily increasing over the years and the number of volunteer authors is simply not enough.

[1] https://aclrollingreview.org/incentives2025

I thought that it was. I thought journals were starting to implement full bans upwards of a year+ for those who don't honestly disclose AI usage in their work? If I'm not mistaken, arXiv is doing this as well? granted, disclosure is different from overuse, but it seems like a small jump to just go ahead and just ban not checking ones work! ...disclosed or not disclosed. my own personal view on it, is if you can't be bothered to spot hallucinations and other such errors, its not ai that is the issue, it is incompetence and laziness

So wrist slap?

Since the log trend seemingly started before the age of LLMs papers - cant we just innocently hope that this reflects increasing popularity of prepublishing on arxivx?

I don't think it's academia not taking it seriously, more like these publishers are just coasting on brand name and trying to milk every cent out of it. Peer review is essentially being a Reddit mod or something, you get nothing out of it but karma or a sticker, and the platform/publisher gets all the benefit.

Imagine being an author and paying to publish your novel. Also btw the editor is another author but has to proofread your book for free.

Many academics will publish preprints or their more popular papers on their own websites etc. It's just common sense because to academics they don't get paid a single cent by the publisher (and instead have to pay the publishers instead) and more publicity for them is always better than less.

Validating existence should already be trivial: virtually all journal already have publicly-available indices which contain at least author information and an abstract. Combine that with DOI citations and you're basically done - even with closed-access journals.

Checking the content is of course a lot more difficult, but that doesn't magically become trivial with open-access journals: you still need to read and interpret what is being said in the paper and compare it to the claims being made in the citation. Granted, these days you could use AI for a first pass, but it's still going to be incredibly tedious work.

> This is a side effect of academia never taking open accessibility to papers and journals seriously.

From my perspective, it's a clear manifestation of humanity's most pervasive failing - the one that defines every group, eventually.

    No One Anywhere Wants To Clean Their Own House.

It is trivially easy to validate the existence of a cited paper.

I can accept having a human first draft and let LLM proofreading and/or do some polish on language, but not the reversed order.

Clearly not, if the reason you care is quality.

I think Pangram is de facto measuring the default Claude voice. It doesn't fire for me on technical GPT writing.