It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.

And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.

And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.

When I have asked claude to use proper vocabulary it wrote a memory "do not use blog-speak", so I guess the problem is that it is trained on blog articles much more that on good documentation.

And then it is also trained on it's our claude code sessions, so the more people accept claudish the more it becomes present in training data.

Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.

I'm a writer, a technical writer, and a highly technical person in the executive space. Communication has always been one of my strong points, and the blathering on that AI does drives me to distraction. It's impossible to really tailor the output of LLMs very much. I value information dense, direct writing when it comes to technical things and they're always embellishing and going off on side quests and it's obnoxious.

As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.

Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.

This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.

> As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.

At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.

The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.

The American models for the most part care about engagement - some a lot more than others.

I feel like also all the code optimizations make it treat all language like code and that produces patterns that align with how it generates code. Earlier models that focused on language seem to have done better - 4o seems like language wise it’s pretty good but advancements elsewhere make it difficult to use.

This. The only “verifiable” writing is relatively short and verified through comprehension of that text. The way it writes is successful for that, but it doesn’t extrapolate to longer form, and under frequent use becomes problematic (sloppy).

I'm gonna be controversial, probably? The reason why AI is so tasteless and superficial may be that they contract it out to evaluators who otherwise are unemployed and averaged over the whole pool (this being the load-bearing qualifier XD) less skilled or tasteful than those who retained their job.

I actually think they stopped caring about the prose a long time ago. Now all the resources get poured into making sure the right tool is called, remember the labs dont want humans to be the main users of these in the end. They want them to autonomously work on their own or with each other, and another agent doesn't care how an agent writes.

Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.

I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.

I sort of agree but I think the issue is more subtle.

Languages change over time, and one of the well-documented "laws" of language change (using that term very fuzzily) is that people like to take irregular features of a language and generalize them to make them more regular. And there are many English verbs (particularly verbs expressing a change of state) which can be used transitively or intransitively, with equal validity:

> "I opened the door" -- "the door was opened" -- "the door opened"

> "the heat melted the ice" -- "the ice was melted" -- "the ice melted"

> "the impact cracked the glass" -- "the glass was cracked" -- "the glass cracked"

This generalization has been common, especially in jargon-y contexts, long before the rise of LLMs. A pedant might insist on saying "wait for the compiler to compile the code", and Merriam-Webster thinks "compile" can only be used transitively. But I think most developers would say "wait for the code to compile [i.e. be compiled]" is fine too. (see e.g. this XKCD comic from 2007: https://xkcd.com/303/)

In my view, this usage is completely fine as an occasional thing, especially when the social process of language change has normalized it over time for a particular verb. The issue is that LLMs drastically overuse it and pepper it everywhere. Just like "it's not X, it's Y", which is fine occasionally in human writing, but has become a dreadful cliche through overuse.

Maybe a hot take, but I think that example is fine. Less verbose than “the form is submitted” and there’s no ambiguity, it’s the English analogue to `form.submit()`.

In your other comment, “Talking also costs.”, it’s clarified through context that talking costs tokens.

I think it’s a side-effect of the AI companies trying to reduce verbosity and redundancy, which were my personal main complaints with LLM writing. Though I believe LLMs are still verbose, and I suspect it’s because they don’t know what to remove, and these are cheap gains where the LLM removes verbosity that is obvious but expected because everyone does it.

I’ve noticed this pattern a lot too, it’s awful. No human would ever write like that.

> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language

This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.

Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.

Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.

Most of the effort I put into writing is modeling the reader's mind. I never would have known this if someone hadn't invented a soulless writing machine that faked this process!

This theory doesn't hold up when you use base models (i.e., models with only pretraining, no RLHF or instruction tuning). They sound much more human. I've tested pairs (like Llama 3 base vs. instruct), and the base model can match human style much better than the instruct version can: https://www.pnas.org/doi/10.1073/pnas.2422455122

I think it's not the raw training corpus, but the samples collected specifically for reinforcement learning, direct preference optimization, or the other instruction-tuning steps that get a base model to turn into a chat/reasoning model. (We're working on testing this with the Allen AI models that have open training data.)

The other side of the same coin is that the core audience probably prefers such style, and that's why companies use it and AI now mimics it.

I cannot imagine a typical suburban American consumer tolerating a chatbot as blunt and direct as Linus.

Exactly. It’s been trained on petabytes of clickbait, engagement-farming content.

The slop-in, slop-out theory. I'm very much an advocate.

The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.

But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.

Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)

People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.

The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.

Symbolic AI is coming back, it's just that the mortar is random word association.

Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.

One part of me suspects we'll never see a frontier lab do this because we pay by the token

Also the training punishes clearly wrong text harshly compared to not clearly wrong text.

Which gives a relative incentive to hard to understand text.

That, and one can feel that there is no intention behind the text. There are no introductions and main points and fun asides, just a mechanical, steady stream of words with much too frequent, quasi-randomly distributed fake "main points".

Try making it write the full draft and then correct it, it's the most efficient way and you'll likely need way fewer correction instructions.

I don't think the analogy holds because AI still produces good code.

> its optimized for the most amount of information in the lowest amount of context or tokens as possible

Is it? I mean they get payed more if they return longer answers.

> For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.

Actually LLM generated text is almost impossible to skim. It's too flowery and the actual meaning is spread too thinly across the text, which means that you can't just read the "important" sentences of a paragraph.

In my mother tongue "baseline system state transition latency validation" would have grammatically been one word. I sometimes miss that in English. If you're tired and see "baseline system state transition*s* latency validation", note the hallucinated "s", the meaning changes and you need to read it again because you're unsure if you actually did see an "s" or if it's "one thing".

"Baselinesystemstatetransitionlatencyvalidation" is surprisingly enough less confusing to me because I know it's one word describing one thing, and my eyes registers each joined word with (years of) training, even though what my eyes look at might be the following chunks in order:

1) [ Baseline ] systemstatetransitionlatencyvalidation

2) Baseline [ systemsta ] tetransitionlatencyvalidation

3) Baselinesystems [ tatetransitio ] nlatencyvalidation

4) Baselinesystemstatetransiti [ onlatency ] validation

5) Baselinesystemstatetransitionlaten [ cyvalidation ]

My eyes essentially scan the word, and although it might be a tad bid slower than reading "baseline system state transition latency validation", reading and understanding is faster because I already know it's one thing before I know which thing it is.

Edit: The example would be easier to demonstrate if HN supported bold font, but I don't think it does (or know how).

Well that’s exactly the authors’ claim. Legalese and patents intent is not to facilitate communication but to maintain a claim under the most bad faith, hostile reading.

AI language, it seems, is optimized to support reasoning from the interior monologue.

I thought patents were deliberately bad? Like, the point is to get the protection, despite paying as little lip service as possible to the "it's in society's interest that inventions get published" side of it?

LLM reinforcement learning improves those tasks by having the agents "reason" until they hit a verifiable output state that passes a black or white test. So they learn language which is tuned to the task in a silo, never communicating with anyone or anything.

In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.

The process causes a warped and compressed version of language.

Optimizing for contradicting objectives is always hard, this behavior will always leak this way or another. Even basic instruction training will affect writing produced piece by piece, because the structure of a typical assistant reply will leak into each piece, for example making characters use narrative echo to annoying degrees. Older models (that did indeed write better) still had this particular quirk. This can be alleviated by using assistant prefill, and its removal from frontier models is a disaster.

Of course if you optimize for natural-sounding language it will learn to sound more natural.

But, if I understand this correctly, if we make it speak "human", then we lose the accuracy of the results, and we're back in hallucination land.

I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.

Maybe new models training on the previous generation’s synthetic output is compounding the effect? LLM Ouroboros.

I find Claude’s writing unbearable. That ChatGPT offers personalization options (Base style, warmth, enthusiasm) and Claude doesn’t suggests Anthropic doesn’t recognize this as a problem. Plus, people sharing ChatGPT writing won’t all have the same voice.

Yeah this drives me up a wall. You ask any AI to make a change and then it thinks polluting your README.md with those details is appropriate. No I don't think "this project deliberately does [random detail]" is needed when that detail is irrelevant outside of recent context.

I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.

I think the problem is that LLMs are verbose. They’re like business or self-help books— a lot of filler.

Have you read entirely generated documentation of code? I wouldn't say it's not understandable but it's no where near good quality human documentation.

I think it’s a reading comprehension level kind of thing. If I slow right down, it’s all there and understandable. But the quality of prose is terrible. It doesn’t lend itself to fast skimming. So it feels like I have to slow way down to understand it. To be fair a lot of humans at work write this poorly as well.

A combination of words that are infrequently observed in natural language. For example in the preceding sentence some of the 3-grams are "combination of words", "infrequently observed in", and "words that are". N-grams are just this for all values of n. They are saying that often times the exact sequence of a short fragment of text is not very common in human written text.

Rarely used N length chain of words.

I.e. a high frequency 3-gram is "by the way".

A low frequency 3-gram is "elephants speak french".

Unusual word combinations, basically. They’re word combinations that are rare in the training data, i.e. rare in normal language, but used more frequently by AI.

They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.

Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.

Yes, I tend to find the longer a context window grows, the more the model tends to come up with its own jargon to describe previous findings, which, if you haven't kept up, starts sounding incredible dense; certainly not something I'd share with anyone else, i.e. in a PR comment.

I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)

This is funny to me. For a while, I thought that Google's Gemini could give good insight and guide me better than a Tech Support representative could. It certainly seemed the right choice if I had questions about my Chromebook or Pixel Phone, or Android.

So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.

I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.

So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.

Because people love to add slogans, catch phrases and witty sentence fragments, and AI was trained to mimic people. AI has shone a spotlight on it, and because AI isn’t perfect, it tends to overuse and use it where it shouldn’t. But go read news articles, blogs and corporate press releases from before the AI boom and take notice how often AI “tells” also show up there. The internet made hot take mic drop sentence fragments, “witty” turns of phrase and clickbait “it’s not foo, it’s bar” cliches a core part of its lexicon. It really shouldn’t be surprising that it shows up in AI too.

> Nah I don't have any issue understanding, so the actual question here is why do some people find AI-generated language hard to understand?

As far as I can tell, it's only fluent readers who have trouble.

What's the test for understanding? I find that I know what all the words mean, just like in Markov chain text. In LLM text I also know what the sentences mean. In both cases I don't always know what the document means.

This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.

In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.