BurritoKing
4 days ago
As someone who is aggressively pro AI, I've also noticed how awful everything is to read lately. It's always so hard to put my finger on the exact thing, but it's a mixture of everything sounding identical and everything being a rambling unconcise mess. I find I routinely start reading interesting posts and just .. stop part way, abandoning it. A site I read stories on has been flooded with AI generated content, all of it terrible, and it just makes me retreat to things created before 2023 as a way of filtering it all out.
HarHarVeryFunny
4 days ago
A model trained to minimize statistical prediction errors is going to sound ... predictable (aka boring). A (good) human author tries to keep their writing interesting, while an LLM strives for predictable instead.
It's not just predicting content but also the form/shape of the response, highly shaped by RLHF preferences, which seems to encourage longer form responses which need padding with filler to generate the "preferred" amount of verbiage.
It seems most of these models have been trained to interpret questions as wanting a long-form authoritative/comprehensive-sounding response (even when midwit filler is all it has to offer), rather than initiating a short-turn conversation which would better allow you to steer the conversation. Apparently Muse is much better in this regard, but so far I've resisted signing up for any Meta/FaceBook service, and not inclined to break this winning streak!
ThrustVectoring
4 days ago
Human writers also use specific constructions for definite (though not always consciously known) reasons, and mapping those correlations onto writing models is only getting done on accident. This is a big part of why LLMs doing pointless negations (it's not X, it's Y) is so jarring: people use that construction in order to preempt an expected misconception, whereas LLMs try to use linear algebra over the learning corpus to generate it in appropriate quantities and places according to RLHF/fine-tuning/etc.
In other words, it isn't just verbiage that has a "preferred" quantity, but also higher-level language features, and the appropriate contexts for those features feel like the sort of thing that will be extraordinarily difficult to replicate without a theory of mind.
edit after coming back and re-reading: the "in other words," and rephrase is because I'm trying to explain something complex enough that I'm not even sure I fully understand it, so I try typing out multiple explanations and leave the ones that make the most sense to me. This might be idiosyncratic to my writing and not a significant feature of training corpuses, but even if it wasn't it's a second example of the sort of thing I'd suspect LLMs to be bad at using appropriately.
ahartmetz
4 days ago
Another LLM weirdness is to explain what code is not doing. My theory: such negations were somehow encouraged in training because they are very valuable where used properly (and sparely) by humans. "Not doing the obvious thing due to xyz happening two modules over" is a very useful comment. "Not using bubblesort because quicksort is faster" is garbage.
benjamincburns
4 days ago
I think a lot of the time it's prompt leakage. Forgive the personification, but it's like they don't have a mental model of the reader of the thing that they're authoring being separate from the person who's prompting them to author it, so it often seems like the main conversation kind of spills into the prose.
Not to pick on GenHTTP Lambda, especially because I think it's someone's fun side project, but I was just trying to read though their docs and I was finding it super difficult. I kept getting distracted thinking "why the hell did it write it _that_ way?"
Like, the first sentence of "The control center" section [1] is the exact sort of weird negation that you're describing:
> The editor link opens a control center rather than a text box: most of the code here is written by agents, so the first thing on the screen is how your lambda is doing.
I can't help but think it generated the first round of the page with a text box to paste the code into, got told "no, I want a control center," and then this ended up in the docs.
I also think the phrasing of that sentence (and many others in those docs) is just super weird and unnatural - e.g. "so the first thing on the screen is how your lambda is doing." It also feels like it mirrors a prompt - like, "no, it should be a control center, not a text box, and the first thing the user should see is how their lambda is doing." But in docs I'd never expect it to be written "the status of your lambda" or "the health of your lambda." And what does the code being written by agents have to do with it?
Kaliumhexacyano
4 days ago
Lambda is my side project and I 100% agree. Updating documentation with Claude does not work, unless you spend so much effort that you could have written a perfect version yourself. Opus 5.5 got better at it, but it's still a long way to go. I personally cannot stand the PR descriptions generated by Claude. Three pages of text for just a simple change ...
zenoprax
3 days ago
This is my experience as well. I have spent many hours trying to eliminate this kind of contamination and the only solution I've found is to re-write from scratch every time. If the result isn't to your liking just reset its context and provide a more detailed prompt.
bavell
4 days ago
1000% this. I encounter this all the time (with Claude) and it drives me nuts!
hxtk
4 days ago
I vaguely remember a study that I was able to find when fact-checking this comment before publishing it [1], which found that music needs periods of high predictability to nurture listener expectations for the sake of later subverting those expectations, and the most enjoyable parts of music are those moments where expectations are subverted. This sounds like a similar effect.
1: PDF warning https://www.cell.com/current-biology/pdfExtended/S0960-9822(...
thin_carapace
4 days ago
uh huh. so music is like a joke - the payoff is unexpected, while simultaneously adhering to scenario expectations.
HarHarVeryFunny
4 days ago
Yes, with the "scenario expectations" just being the necessary setup to deliberately create false expectations and make the punchline that much more unexpected/funny.
I think both music and humor are both exploiting the same evolutionary essentials of human nature - to find comfort/safety in predictability, and excitement/danger in unpredictability. Laughter would seem to be a social thing - perhaps it helps develop a theory of mind in signalling that you get the joke - that the punchline is indeed unexpected to you as well ? And/or perhaps it helps draw others attention to an unexpected situation with learning potential.
jacmilts
4 days ago
Gemini models are so, so much better in this regard. Their outputs are actually readable.
api
4 days ago
Gemini tends to glaze you though. I now believe I’m a genius at pretty much everything after trying it a little.
FranzFerdiNaN
4 days ago
Every model has done this. And sometimes the makers go the other way to compensate, and then you get Opus 5 who answered like it was possessed by the ghost of stack overflow.
3371
4 days ago
Not really, Gemini is at a different level, it will naturally burst out compliments and honorable nickname to provide emptional values. My last company was iterally destroyed because the old boss (successful boomer who had no idea how software industry work) believe everything his Gemini told him and gives us all kinds of unrealistic assignments and force us to use whatever tools/services Gemini suggested.
mythrwy
3 days ago
Agreed. In my experience it gets to the heart of information faster rather than so much filler. Also better information and more comprehensive in general.
The main issues seem to be glazing that sibling mentions and also it seems to play a bit loser with facts and reality and makes up more stuff.
blobbers
4 days ago
"A model trained to minimize statistical prediction errors is going to sound ... predictable (aka boring)." -- what if the goal is to be a (good) human author.
In that case, model will in fact strive to keep their writing as close to interesting writing as possible.
HarHarVeryFunny
4 days ago
The goal (literal, not figurative) of an LLM is to reduce training/inference errors - to predict the training material.
The "quality/goodness" (likability) of output is only selected for to extent that it also reflects RLHF preferences originating from low paid contractors who are scoring based on "looks good to me", not based on Pulitzer prize prose potential.
To explore a false hypothetical, suppose an LLM was only trained on the output of a single popular author, then you'd certainly expect it to reproduce that author's style, but it would not be "upredictably interesting" but rather predictably quirky, reproducing the same "surprises" over and over so that they no longer are surprising and therefore interesting.
To continue the hypothetical, if you trained an LLM exclusively on the output of multiple good authors, then rather than get the best of each you'd get a statistical blend (this is how training works) that emphasized what they had in common rather than the individual quirks that made them interesting.
There is a fairly common misperception that "AI" (LLMs) is always trying to do it's best, where "best" has been anthropomorphized to have the human sense rather than the LLM sense of "the most statistically accurate continuation prediction" it can make. For example, if you give an LLM the moves of a partial chess game and ask it to predict next moves for one of the players, it will NOT play to win, but will rather play to predict how the game would have continued, which means continuing at the level (ELO rating) and style of the player who had made those moves.
Of course you can ask an LLM to play to win, just as you can ask it to write in a style that is, say, "unpredictable and interesting", but it can't escape itself - you will necessarily get predictable unpredictability.
blobbers
3 days ago
I think you're missing what I'm saying; the LLM can and will have trained on all interesting prose, and it will evaluate all interesting prose. If you indicate that you are goal setting for amazing prose, and you give enough chain of thought, it will self evaluate.
Your chess game example misses the point. If you give an LLM the moves of a partial chess game and ask it to predict the moves of the players, it will predict how the game would have continued, even if it is low level players. But if you ask it how Magnus Carlsen would win the game, it will likely come up with a sequence that can beat most human players, or possibly just decide to invoke a chess solver, which will most definitely win.
The key for your prompts and thinking loops is you need to indicate proper goals. "Write the a story about 1775 England." is different then setting a goal for it to write the a fantastic storyin the style of Dickens about England in 1775.
Regarding the ML portion; what you've written about a 'statistical blend' is a little bit coarse. That would be what you might have gotten from a 2 layer neural net or a regression in 1999; the neural networks today operating with attention or an incredibly high number of parameters are non-linear and non-interpretable, that's the entire point. This isn't a regression of simple parameters, it's all interactions of all parameters deeply netted. Model interpretability vs. Model Prediction power are generally inversely related.
The model can in fact escape itself in that it has many, many, many more parameters than a simple "predictable unpredictability" and that is exactly the point of it.
HarHarVeryFunny
3 days ago
I'm not sure how you are imagining an LLM "self evaluating". They are feed-forward auto-regressive predictors. The only source for the next word they spit out is what came before and the patterns that contains.
If you ask an LLM to "win like Magnus" it can't because, as above, it is a feed-forward auto-regressive predictor. Do you realize how much different this is to how something like Stockfish (that can play at, and far beyond, Magnus' level) works?
A big LLM is still an LLM - the architecture and operation doesn't change. It just has more capacity to learn, both to learn deep predictive patterns and to memorize factual knowledge.
Does it not occur to you that your beliefs of what LLMs are capable of (generating amazing prose) is totally at odds with what we see them doing? Where is all the amazing prose?
throwaway27448
4 days ago
What if? But this certainly doesn't describe LLMs.
skinfaxi
4 days ago
Interesting, I was thinking the walls of text are better for the watermarks.
benjamincburns
4 days ago
I'd be shocked if it wasn't a shotgun strategy learned during RLHF or RLVR post-training.
A short response that's wrong is going to be punished, whereas a long answer that contains the correct information, and then a whole bunch of other stuff surrounding that info, is more likely to be rewarded. Even if you discount the reward by answer length, when there's ambiguity involved it's still going to be a better strategy on average to hedge by saying too much vs risking saying too little.
robotresearcher
3 days ago
> It's not just predicting content but also the form/shape of the response
Nicely done.
bebimbop
4 days ago
That doesn't follow. Predictability is your brain reacting to the lack of novelty. What an LLM produces is a chain of words following some kind of conditional distribution, whose output could vary significantly from your understanding of novelty depending on its training and priors.
With a single, generalized prompt ("produce something unpredictable"), ChatGPT responded with:
> 00:— / ANOMALOUS RESIDUE > Vekh. > Ìssara. > Nnnn—tq—èl. > > A temporary thickness enters the meat of the signal. > > Not rain. Not a substance with permission to fall. > > UL / 9 / ferrous / ∂ > The measurements are eating each other. > > Someone has left an entire childhood in the wrong unit of pressure.
That is not predictable to me, at least. It could be boring, but I think that is mainly a product of me knowing where the text is from. LLM-generated text is inherently boring, to me, but that has nothing to do with predictability.
NichoPaolucci
4 days ago
Don't these sort of conflict with being "aggressively pro AI"?
> how awful everything is to read lately > flooded with AI generated content, all of it terrible > everything sounding identical and everything being a rambling unconcise mess.
I can see the benefits of AI and enjoy using the technology - but, like many have said, it's wreaking havoc on the "content" that's out there nowadays. Gotta take some bad with the good, I suppose.
soulofmischief
4 days ago
I can like the concept of the engine, and the leverage it provides, but I hate traffic and get anxious about having an accident on the freeway. And the people behind the industrial revolution have destroyed the climate.
I love computers and the internet, and the internet has ruined a lot of people and things around me.
It's possible to be interested in a new technology and enjoy its leverage while still being extremely apprehensive about how it intersects with game theory and politics. The race to the bottom of the barrel keeps coming for the jugular of every cool new thing.
throwaway_7274
4 days ago
I’ve been starting to think that a lot of the variance in how people see the world can be explained by whether their attention is drawn to first-order localized effects, or nth-order distributed effects. This is not the same thing as optimism vs. pessimism, but it sure is correlated!
bluefirebrand
4 days ago
Or just short term gains versus long term pains (or vice versa, depending on how you view it)
I feel like in my life all I ever see is people mortgaging the future for today though. Hey how about that record storm in California?
ctippett
4 days ago
This sounds intriguing. Can you expand on what your thinking is or link to somewhere I could read more?
soulofmischief
4 days ago
It's a multi-mandate to focus on what's best for your yourself and society such as your mind, your body, your chosen family, your friends, your neighborhood/community, your city, state, country, the global population, and more abstract things such as the human condition, when evaluating things such as privilege or fairness or duty.
You often have to make compromises somewhere in the hierarchy, especially in late-stage capitalism where even access to basic needs comes at a moral cost, and as such those compromises sometimes come with a moral duty. I am writing this via a computer built from limited natural resources, some of which have been obtained through what I see as slave labor. I don't eat meat and still have to acknowledge the harm that clearing and maintaining farmland does to wildlife, even if the result could theoretically be a net gain.
When you start viewing situations through the lens of game theory, you realise some compromises are inevitable for the rational individual. But you must acknowledge the individual is not the only concern. As such, I use AI as part of my work, but I opposed the local buildout of a large data center in my community and tried in vain to make the person responsible for the deal understand the issues it could introduce. This isn't NIMBYism, we need a halt on all buildouts until fundamental environmental and accountability concerns are addressed.
And many other aspects of my existence and quality of life, poor and underprivileged as I was growing up, have depended upon US imperialism. I have also produced a ton of waste just existing and owning pets. It's abhorrent.
I continue to ask myself how I can fulfill the moral duty which I have accrued and reduce my dependence on exploitative economic systems, but I also recognize that AI is the new social leverage and we only have so much time to use it to our advantage before it becomes institutionalized and wielded against the people. I also might only find future income stability by evolving with my industry.
Life is complicated.
alskdfqlkj
4 days ago
Well, we have a good understanding of the benefits that the industrial revolution and the computing revolution have given us.
The AI situation is different. The technology is obviously extraordinarily powerful but the benefits to humanity are still nebulous and there are many very obvious harms like the one we're discussing. So to describe yourself as "aggressively pro AI" you must believe that future, as-yet unrealized benefits will outweigh the harms. I don't think that's a crazy thing to believe but it deserves scrutiny and demands elaboration.
DrewADesign
4 days ago
> it deserves scrutiny and demands elaboration.
These challenges never go anywhere because it’s essentially a religious conviction. We’re likely in sort of uncharted territory on a macro scale and I reckon people a) either acknowledge that or refuse, and b), either trust or distrust the tech world’s stewardship of this technology. There’s a hundred bazillion data points, and I’m pretty pessimistic about the tech scene at large right now, but I think reasonable people can disagree on this without being factually or logically wrong. But true zealots in the midst of such uncertainty are driven by faith, which can certainly be deftly swayed over time, but definitely can’t be conquered in a debate.
mday404
4 days ago
Eh, I think many of the harms of AI are currently also nebulous. It's not as though the harms are all concrete and massive and the benefits are vague and "in the future"
alskdfqlkj
3 days ago
I agree but I see a lot of very concrete harms too. It's been catastrophic for schools, the sloppification of the web has been largely a disaster so far, it's been a disaster for cybersecurity, it's wreaked havoc on the consumer electronics market, it's driven a lot of financial and existential insecurity for students and early career people across various roles, and while I'm not as militant about data centers as some I think their proliferation has been a net negative environmentally (including increased reliance on coal).
smcleod
4 days ago
This resonates with me and I suspect a lot of others here. As always life is more nuanced that many would have argue.
iLemming
4 days ago
> Gotta take some bad with the good, I suppose.
Well, it's quite simple if you think about it. Intelligence is all about awareness of the imminence of death and the finite amount of time given to any of us. So, we humans intrinsically evolved to make the most of it. The majority of the tools we create have the ultimate purpose - to save time. Even money is nothing but a way to cut some time. It's not really about time but experiences - we want to have as many and as rich as we can get, and we innately try to cut time somewhere else, so we can use what we saved for better experiences.
Therefore, inventing cars was an easily predictable event. There's not a single hypothetical timeline or dimension where humans don't end up with cars - to transport things and to travel faster. Similarly, it was always simply inevitable - we would have invented something like LLMs, sooner or later.
Cars required significant sacrifices - they are absolutely deadly machines, they kill thousands of people every single day, kill wildlife, endager our own existence, and we're kind of okay with that. I can absolutely assure you, if someone tomorrow invents a gravitational jump travel that allows moving things between continents in minutes, even with a success rate as low as 10%, we would still be using it.
So, yes. We can't un-invent cars and AI, the only way to move forward is to adjust to this new reality and extract the best possible experiences, preferably by not robbing other people of their - i.e., enjoy life while it lasts, it still remains finite.
skinfaxi
4 days ago
> Intelligence is all about awareness of the imminence of death and the finite amount of time given to any of us.
You lost me there. Children are pretty intelligent and have no concept of time. I don't see the relation you find obvious and the rest of your points similarly don't follow.
> Therefore, inventing cars was an easily predictable event.
This feels like a benefit of hindsight unless you can call out other obvious and imminent (but not yet existing) inventions.
iLemming
4 days ago
Yeah, I guess, "intelligence is all about awareness of death" is kind of a bomb of a premise. I admit, not the best thought I spat out through my keyboard. I suppose I mean some narrower sense of intelligence here. On one hand, you're absolutely right - crows, octopuses and toddlers solve problems without any hint of understanding mortality. Yet "children have no concept of time" is incorrect: not only do they have one, but most kids also grasp that death is irreversible and happens to everyone by about age 5-7.
I don't blame you for immediately rejecting the rest of it. But do hear me out: the point about the inevitability of invention does have real support. Historians call it multiple discovery - tons of things have been invented independently - calculus, the telephone, the jet engine, to name a few.
Neural nets, GPUs, and the scale of Internet text made something like LLMs close to inevitable. Of course, we could debate forever whether any of these on its own was ever inevitable. The point is - society tolerates any cost of a useful invention for the benefit of saving time. So it's reasonable to assume this will be the case with AI as well, which is really happening as we speak.
idiotsecant
3 days ago
Sometimes less is more
iLemming
3 days ago
Ironically, this is the case where "nothing is better than more", because I have no idea of what property or context of any of these three comments above you're referring to.
safety1st
4 days ago
I'm kind of pro AI while also being pro LESS AI.
These are ultimately all software systems where updates can be pushed out with the press of a button. We just found an amazing new tool in the LLM and people are trying out all kinds of ways of using it, many of them dumb or stupid, this is discovery that has to be done, we'll keep doing it and pushing updates.
It doesn't take rose colored glasses really to infer that over time we'll cull the dumbest uses and extrapolate on the best ones. There's a huge difference between using a generic chatbot, even one driven by a SOTA model, vs targeting a specific business task with good guardrails, skills, and tool interfaces that call out to real business logic when the model isn't the right thing. The latter will emerge more and more over time.
elxr
4 days ago
> Don't these sort of conflict with being "aggressively pro AI"?
I don't think so. AI is broad, and he's against low quality output enabled by people taking shortcuts using AI.
Using AI doesn't make your output worse, that's almost entirely down to lazyness and a lack of skill. Or people are just choosing to spend less time on the same piece of work instead of doing additional iteration and experimentation (which AI makes a lot more doable).
derefr
4 days ago
You can be pro-"thinking AI is going to get really good and solve many problems" (i.e. be the sort of person who would "go long" on AI), while also being anti-"applying AI right now to domains outside of the few it's been specifically optimized for [e.g. coding]" (because in these other domains, AI is yet to approach, let alone surpass, "authentic quality" equivalent to that of a skilled junior professional.)
tehjoker
4 days ago
Only a programmer or a business guy would think just because it becomes capable we would want to see AI produce "content". The effect is the erasure of human culture.
crooked-v
4 days ago
I think there could be room for something like how holodeck programs are shown to be authored in Star Trek, i.e.: the author does all the primary work, and then the system, if a particular work's design allows it, extrapolates from that to fill in the blanks when somebody decides to make Moriarty super-intelligent or put a hundred androids in a Wild West scenario or take part in a go-karting death race with Frankenstein's monster.
But, that comes with with the same kind of cultural acknowledgement, as seen with the characters in Star Trek, that such algorithmically generated material is an experimental diversion, and inherently not worth the same respect or focus as experiencing something authored with intentionality by a person.
tehjoker
4 days ago
Right, deployed thoughtfully, in a way that improves the human condition (especially aboard a ship far from civilization!) there are uses of course. When everyone is offloading their thinking and writing this is effectively erasing the production of culture.
blackqueeriroh
4 days ago
This is somewhat of a dramatic statement. What, exactly, is the mechanism by which human culture is erased because LLMs write some stories or compose some music?
Vaushite
4 days ago
The flood of LLM-produced stories and music. Spam killed email.
pegasus
4 days ago
> Spam killed email.
Sure, but it didn't kill human culture. As writing didn't either. Nor will AI do it. Technology changes, human culture does the same.
tehjoker
3 days ago
The most powerful people in silicon valley are openly aiming to replace the entire species with robots in space. This is their genocidal aim. The fact that bad things haven't happened yet doesn't mean that directionally, they aren't attempting to replace as much human culture as possible.
It sounds silly! That's because they are completely divorced from the human experience, and they have enough power to really make us all suffer.
derefr
3 days ago
I'm sometimes those things. I'm more often a writer and a futurist. I am enough of a writer specifically, to really not want to live in a world where AI-generated prose substitutes for human stories and essays; and I feel empathy toward other creative professions (most of my friends are visual artists) and don't wish to see that kind of thing happen there either. I do think we have to push back on things like that happening. And I do think that people trying to push toward that end for monetary gain are erasing much of the joy of being human.
But that being said. I don't think that our need to push back on that particular use of AI, has anything to do with the need to push back on AI itself. Because that's not the only, or to me even the main, point of (generative) AI.
(It's just that "generating wholesale slop that people end up seeing" is the usage-profile that the AI companies can most-easily intermediate and make money on; and it's also the usage-profile that makes it most obvious that "AI is happening", and so markets AI to people not already using AI. So this is the usage profile that all the AI companies benefit from everyone focusing the conversaton around AI on!)
I think a lot of people have a failure of imagination when they think about the upside of "good" generative AI. Which to me is, specifically, the places where generative AI can be better put to use as a tool in a workflow for human creative labor to pass through, rather than being something that substitutes for said human labor.
The programming example might be the most obvious to people here: generative AI lets you write e.g. a hairy SQL statement in the form of a very long natural-language description of the query, or an example of how you'd do it in an actual imperative/functional programming language, and have that transformed into SQL grammar. So you don't have to try (and fail) to remember the arcana of SQL (e.g. how to use things like correlated subqueries, or window functions with explicit windows, or CUBE), always eventually resorting to looking it up anyway. (This is really a condemnation of SQL for having specific one-off syntax sugar for every single thing it wants to encode, rather than a uniform syntax that can fit in your head / trigger IDE auto-suggest.)
But consider Photoshop's "generative fill" tool. Yes, if you're selecting entire huge regions of the image and running that tool on them, it's going to be "doing original work" and the results will be awful. But that's not the use-case the tool was designed for. It was designed to be a better version of content-aware fill, i.e. PatchMatch. If you select a dust speck or tear on a scan of an old photo and run the generative-fill tool over it, the result will usually just be a better version of what you'd have got by running the content-aware fill tool over it. (And if it's not, you don't need to try to fight the tool; you just drop back to using the other tools in the toolbox. Not every tool is applicable in every situation, and that's okay.)
Or consider that newer versions of Melodyne use generative-AI models to better separate audio stems into their own tracks. The input here is your own manually-performed track. The output here is also your own track. Despite being a generative model, the AI here is only generating an internal byproduct that gets discarded (hypotheses for timbre- and acoustic-matched reference instrument performances) to aid the classifier and reduce stemming error/residue.
(And apparently, the compressed-sensing models used by MRI scanners are getting approximately the same benefit by approximately the same means. Less familiar with this, but it's probably the most important benefit here since improving the fidelity of MRI reconstruction means doctors can be more sure that an opacity is really "something" rather than an artifact, and so will have more confidence to diagnose real problems earlier.)
Or consider generative AI used for automatic human-language translation (where it captures much more of the speaker's original meaning and nuance than previous translation models, at the expense of giving the result "LLM voice.") As AI improves at this task, presumably there will be less "LLM voice" and the result will just sound like what the person themselves would have said if they were bilingual, because... that's the task.
In none of these cases is generative AI "producing content." In these cases, generative AI is subsuming the role not of human labor, but of tools that were already used: of "by-the-book" approaches, of heuristic algorithms, of weaker ML models relied upon by heuristic algorithms, etc. But all of these cases benefit from generative AI being improved.
And, to restate my premise: there are many tasks just like these, where generative AI currently has no place (because replacing or "enhancing" any tool in the task's workflow with gAI would currently make the results worse), where you can expect to see gAI eventually replace or "enhance" some tool in the workflow, once gAI model companies turn their attention and benchmarks to trying to optimize for performance in those sorts of tasks.
user
4 days ago
ballsac
4 days ago
[dead]
jonshariat
4 days ago
Agree but also want to point out that this started with the SEO boom of the last decade where sites started rambling so they would keep you on the site longer and get more SEO "points", they were articles meant for a robot to catch humans.
AI has made it even worse and lots of content on the web, for both these reasons, is not worth you time (now wonder users don't read most of the text on our apps!)
altairprime
4 days ago
In a prior evolution of this, if you look into the Get Rich Quick marketing copy of the early 2000s (before SEO had taken over content formatting), you can see a parallel of sorts in those marketers doing human sales optimization Impact with bold and underline. It derives from research into the human response to ads from decades of print ads and centuries of printed matter prior, and I think there is a direct lineage from ‘how can we bias humans through restructuring text’ to ‘how can we bias machines through restructured text’. This isn’t to disagree, but to call out how rapidly humanity (merely twenty years!) has switched from marketing to people, to marketing to algorithms, in search of a favorable bias.
psychoslave
4 days ago
Ah yes I remember meeting this end of spontaneous expression to bend toward whatever might please some algorithm.
Soulless Enshitificatoon Oblivion now exercises a huge weight on societies to such an extent, it's ridiculous.
basch
4 days ago
I suspect it’s partially an average / regression to mean.
If you built a model only based on long form journalism and The New Yorker voice, including both the template structure of the article and the grammar and syntax, the model would replicate the patterns it is built upon. You could do the same for high vocabulary rap.
When you stuff all of English or all of human thought into it with somewhat equal weight, including very technical writing the sum is a lack of style and elegance and personality.
The second problem is that writing often doesn’t happen linearly. You write bookends and fill in, edit and revise. You sort of know where you’re going and can then make decisions how to get there. Language models may benefit from being redesigned to perform passes. Select a template of some sort, fill in the mad lib, then run a detail pass, then run a persuasion / emotion pass, then run a style pass. Current model writing very much sounds like I would expect autocomplete to.
notfromhere
4 days ago
It’s heavily RL’d and it’s gotten worse because they’re using LLM judges, that’s why it’s gotten worse. The last good writing model was GPT 4.5.
I suspect they’re leaning heavily on some kind of auto verifier, and that’s why the writing sucks, because good writing is so subjective.
And they’re probably nerfing what it can get from training as they’ve lost lawsuits for scraping copyrighted work, so they’ll purposefully avoid anything that references a specific writers style, etc.
BrokenCogs
4 days ago
Isn't this easy for frontier 'labs' to fix? If the user requests lots of writing and prose, then use an auxiliary model which has been trained on 'award-winning' writing instead of all the crap on the internet?
FeteCommuniste
4 days ago
Award-winning writing has all kinds of different styles. The average of them will probably get you some boring median soup once again, though maybe a bit less bland than the LLMs trained on oceans of LinkedIn and Reddit slop.
ThrustVectoring
4 days ago
Even if it doesn't, the effect of so much writing getting pushed out the door through default LLM settings is going to make it feel same-y and bland in short order.
krige
4 days ago
Oh boy, more IP theft and copyright infringement!
capnrefsmmat
4 days ago
The "regression to the mean" or "it's just the training data" hypothesis doesn't work. Pretrained models do not have these weird writing styles and sound much more human. It's only after instruction tuning -- SFT, DPO, etc. -- that LLMs develop weird writing styles.
That is, the original training on massive text corpora seems to produce an average of human writing, but that average looks pretty average. It doesn't feel like LLM slop. But the instruction tuning on specific tasks (answer this prompt, summarize this text, do this math problem), which turns text predictors into chatbots, seems to create strange style features as a side effect. And because that tuning step depends on company-specific training data (rather than just a crawl of the entire Internet), each company's LLM has a different style.
It's not that difficult to measure this. Linguists have had measures of style variation for a long time, though NLP and AI researchers seem to have forgotten. We've generated parallel human and AI corpora to measure it, and can show specifically which writing features change: https://arxiv.org/abs/2410.16107
Currently we're working on tweaking the instruction tuning process to see if we can pin down which steps and what types of tasks tend to produce the weird style features.
basch
3 days ago
I didn’t really specify whether the regression to the mean was from data or tuning. It’s just happening. They are creating neutral average language instead of varied expressive.
capnrefsmmat
2 days ago
But it isn't neutral average language; it's extremely idiosyncratic. GPT-4o used present participles something like 5 times more often than the average human, for instance.
Though there evidently was an attempt to make GPT-5 sound more neutral, but in a literal sense: the emotional content of its text was flattened out. Suddenly it wasn't nearly as sycophantic or enthusiastic, and wrote much more dry, neutral text.
I think it's more accurate to say the text is not regressing to a mean, but to an artificially constructed style defined by the training process. And the AI companies have only partial control over that style, so weird things emerge from it.
psadauskas
4 days ago
The part I find grating is the attempts to punch everything up. ("It's not X. It's Y"). Like, its a react badge component, it doesn't need punching up. You didn't find the smoking gun in a murder mystery, just a misplaced CSS rule.
triMichael
4 days ago
I've noticed this as well, I've been comparing it to 4Chan Greentext memes. Everything is a short punchy sentence, and every sentence has the same "weight" to it. There's no flow or rhythm, it's like reading off bullet points.
Animats
4 days ago
> sounding identical
In AI-generated entertainment, the names "Marcus", "Chloe", "Sterling", "Meridian", and "Apex" keep appearing. Those aren't common names, but LLMs seem to prefer them.
Vaushite
4 days ago
Elias the lighthouse keeper
skinfaxi
4 days ago
Do you have more info on this?
lelanthran
4 days ago
I am willing to believe it happened the same way goblins crept in.
zby
4 days ago
My model is about gaps - all ideas start with gaps. LLMs can fill in these gaps quickly - but some ideas are just bad - so the filling in does not improve them. Maybe it would be useful to treat that filling in as a kind of hallucination - the LLM trying to meed impossible requirements. But on other cases the ideas are good and the LLMs fill in the gaps in the right way. It is a kind of skill to see when LLMs are hallucinating in the gaps. I use LLMs very extensively and I am adding checks to that should catch when the filling in is weak - but it is still not fully autonomous.
I also try to keep the llm-enhanced (or even fully llm-generated) content separate from my human writing - because it is true that the failure modes are different. But think LLM writing might be useful even if it is not consumed directly - it can enhance LLM answering your questions.
zby
4 days ago
And apparently I am repeating here what I already have in my KB: https://zby.github.io/commonplace/theory/llm-generation-rela... - I borrowed the model from Boretti. The agent says that the note is much more careful and better developed than my comment above. You can judge yourself - but I think it does have many more details and hopefully most of these details are the good kind of filling in - but I guess it might be harder to read.
ashkankiani
4 days ago
Do you have some personal financial or other incentive to be “aggressively pro AI”?
alabut
4 days ago
I’ve got the same bright red line for 2023. E.g. I treat a handful of youtube channels with long electronic jams almost like oldies at this point - keep listening to the same stuff from at least two years ago because it’s the authentically good stuff.
domga
3 days ago
Same for me, nothing is safe unless it is verifiably old. Basically devalues tons of stuff like writing and music jams if recent, I don’t have the energy to verify all this.
e12e
3 days ago
I call it uncanny valley effect for text [Ed: ops use of prose, rather than text is better formulated]. It's a little like with AI images - they started out with six fingers, but now they look "ok" - but slightly off.
Reading some AI text feels like looking at a Michael Murphy sculpture - makes perfect sense, until you start to look closely at the text, and gaps appear - and you start to realize there's no there, there.
pton_xd
4 days ago
LLMs have made the internet unreadable. It used to be just Google search results were SEO-gamed empty prose, but now it's every blog post, news article, product announcement ... everything. I've stopped reading basically everything online except user comments on a few websites. It's the last place remaining to actually connect with other human minds, but even that seems like it won't last forever.
domga
3 days ago
Even user comments are not safe, especially big sites. Generally internet is just awful nowadays and I feel like I keep coming back out of habit.
fxwin
4 days ago
> it's a mixture of everything sounding identical and everything being a rambling unconcise mess.
For me it's mostly the latter, which is a natural consequence of LLM writing. When the (time) cost-per-word goes down, there is less incentive to keep signal-to-noise ratio high, so it ends up verbose and bloated. I also believe that this is (at least partially) upstream from everything sounding identical: take two articles on two different subjects - when SNR is high, there isn't really much room for them to become similar (The ingredient list for a risotto will never read the same as the ingredient list for pizza), but when 80% of them is fluff, they can blend together more easily (To take the recipe comparison, this would be the background/story part above the recipe), especially when written by the same LLM
3RTB297
4 days ago
This is true across every topic. Anything at all that someone can put into an LLM and get out an emoji-laiden wikipedia article sized post to slap on a heavily SEO-ed website is absolutely flooded with it.
My parents live on a rural property, and I'll help them do research on things like livestock practices or occasionally "prepper" related things to give them a sanity check before they spend money. I have to weed through a dozen or more obvious ChatGPT-created pages that are the first page or two of results on every search engine before I get to anything a human has touched. The vast majority of the LLM-created pages are either massively verbose and vague, or often wrong. But they're getting clicks. Very often, when asking an LLM the same question and to give citations, I'll get a combination of useless, irrelevant sites with no connection to the material, maybe one of these LLM-generated sites, and maybe one online forum.
I'm not aggressively pro-AI, I'm in the middle, but stuff like this has me worried that we're looking at the equivalent of the Dark Ages all over again. But instead of simply no one bothering to write anything down as was the case in the early Middle Ages, it's going to be the task of searching for real human work and thought in a sea of aggressively spammed LLM-generated slop pages.
Vaushite
4 days ago
Sounds like it was trained on Reddit. Reddit has this effect where catchy snippets get upvoted training people to write more catchy snippets... While reality has nothing to do with it.
blobbers
4 days ago
Sorry, meant to reply to your comment, but my post above...
"You can be pro AI productivity but anti-AI creativity. Automating creativity is the death of art."
Meaning you reject the creative value of AI but values it's ability to drive a car safely, or write code for your job, or turn off the lights when you leave the room. objective goal oriented task completion great.
But AI draws your a picture? What's the point. AI writes you a song? No feeling. AI creativity isn't interesting.
vips7L
4 days ago
Maybe stop being pro ai then.
antonvs
4 days ago
That’s taking the wrong lesson from this. The fault here lies entirely with people who are unable to recognize when AI output is subpar, or else don’t care. “AI” here is just a scapegoat for people’s intellectual laziness, lack of discernment, and/or inability to reflect on their actions.
There isn’t any doubt that AI can be a useful tool. So what you’re essentially saying is that we should avoid using a useful tool because many people, without any relevant training, use it badly.
By that standard, you should not be pro-car, pro-airplane, pro-boat, etc. etc.
vips7L
3 days ago
This just sounds like an addict making excuses for his addiction.
antonvs
3 days ago
That's a non-responsive response. You're not engaging with anything I wrote, or even describing how you believe your response relates to what I wrote.
What you're doing is engaging in major denial in response to a change that, for whatever reason, you're unable to handle.
Max Planck's famous quote is relevant here: "A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it." Much the same often goes for new technologies.
Of course it's not purely an age issue; plenty of older people are embracing AI. But the kind of people who are completely unwilling to entertain the idea that it could be a useful tool fit Planck's quote perfectly.
vips7L
3 days ago
Yawn. Maybe leave your prompts and look around man. Just because something is useful doesn’t mean it’s good for society.
antonvs
3 days ago
You're apparently a software developer. That means you've spent your career automating things that put people out of work. Why are you suddenly deciding to draw the line now? You think it's ok if you're the one putting people out of work writing code by hand?
This reminds me of the people who think they're having economic trouble because immigrants are taking their jobs.
You'd have to tell me what you see when you "look around", because it's clearly not what I see. All the problems I see come from people doing things that they shouldn't be allowed to do, that should be taxed, regulated, etc. I see monopolies that should be broken up, money being used to drown out opposition with the backing of law, financial scams of all kinds propping up the supposedly most valuable companies in the world. The idea that "AI" is the bogeyman that's somehow causing any of this is drivel.
But you write one-sentence comments that don't actually describe your position, so it's impossible to know what you're actually trying to say. Perhaps you should ask an LLM to help articulate your ideas, if you want people to take you seriously. And then you'd find that AI can actually be a force for good.
vips7L
3 days ago
God what a sad little ai driver you are. Writing more text doesn't mean you're more thoughtful.
antonvs
3 days ago
It helps you communicate your ideas, which you seem incapable of.
I'm not using an LLM to write comments, because I'm capable of expressing my thoughts in writing on my own. Which goes back to what I wrote earlier - the issue is how people use AI. AIs aren't forcing people to abdicate their intellectual abilities. And in a case like yours, where expressing yourself is not your strong suit, they can help.
vips7L
a day ago
Bro you’re not open to my ideas. You assumed my ideas from the first interaction.You told me what I was thinking. Fuck off and go play with Claude. Fucking loser. You’re peak HN who thinks they’re smarter than everyone else.
> And in a case like yours, where expressing yourself is not your strong suit, they can help.
Do you even read the shit you type? Or copy and paste because you’re a sad little ai driver? Of course I’m not interacting with you in this conversation. You’re a fucking pompous retard.
xendo
4 days ago
This applies to code btw, it's so much nicer to read a human made pull requests. It's also much faster as people tend to be lazy and produce denser code.
cellsstudio
4 days ago
100 percent agree. It's like the consensus for tone and vocabulary has been "midlevel digital marketer from 2015" and now everything is starting to sound like a lazy sales pitch. I'm with you on AI but man, this is such an annoying side effect.
jansport123
4 days ago
to AI verbosity and weird use of diction is natural, to humans it makes it hard to parse. The number of time i start reading long winded AI written design docs, and i just can't get into the flow after reading a few sentences and have to keep re reading certain sentences.
kinduff
4 days ago
How is someone "aggressively pro AI"? This is the first time I read such concept.
unified101
4 days ago
Is typically called ai booster. The opposite is ai doomer.
SrslyJosh
4 days ago
Importantly, both boosters and doomers are believers (or perhaps cultists).
A doomer isn't skeptical; they're nearly as credulous as boosters, or maybe more so in some ways.
bigfishrunning
4 days ago
Not necessarily -- I am extremely anti-AI; I think the danger is not that it's good, it's that it's convincing. Turns out most people don't care about the quality of art or prose or code that they interact with, and it's very depressing.
topaz0
4 days ago
That's not exactly what I've heard called a doomer. Doomers are the ones who think we're building the wrong AI god.
(But I agree with you fwiw)
Vaushite
4 days ago
Most gods have never been observed to have any real world impact. Only their worshippers cause real world effects. So god is not a bad analogy even for doomers.
squidmaster
4 days ago
[dead]
ragequittah
4 days ago
My take is that most art, prose, and code was really bad before AI too. Now there's just more of it good and bad. It's like hating that YouTube exists because cable TV was so good. Now anyone with a camera can make and upload whatever nonsense they want but in the end lots of very good content was made on top of the huge pile of rubbish.
drusepth
4 days ago
paulddraper
4 days ago
Leans into AI adoption.
vikramkr
4 days ago
Honestly I think it's a good thing it's so hard to get AI to talk normal and I hope they keep making the voice more and more distinct. Agents should run as background tasks and automations. They should be understandable so they can communicate with you about what they did, and for observability/ai safety. Im perfectly fine reading llm speak when it's an ai telling me what work it did. When in reading a blog post with a human listed as the author though, I do not want to read an AIs thoughts that the human is laundering. Honestly I'd probably tolerate it if it was framed as "I asked my Claude to do a cool thing here is its report on what it did." But in general it's really nice to have an instant filter of - this post is a waste of time - ah this post has an actual human's thoughts
SrslyJosh
4 days ago
> I've also noticed how awful everything is to read lately.
Yeah, thanks dude. We appreciate your efforts to enshittify everything.
Edit: Sure, downvote, but the parent comment is really not very different from "arsonist puzzled and upset that so many buildings have burned down".
giancarlostoro
4 days ago
Personally, for blog posts, documentation (unless I flat out vibe code the entire thing) etc are all written by me by hand. If the model can save me hours of "oops that was really dumb of me" why can't I take time to read and document things by hand? I think everyone's embracing AI in the wrong ways. Maintain your knowledge of things, and keep the documents nice. There's still room for letting your agent manage agent specific docs, but for the humans, just write it yourself, ask the model for feedback and tweak it by hand. Let the machine talk to the machines.
gabriel666smith
4 days ago
My guess is that tokenization is typically not (strictly) syllabic.
OP’s linked article mentions the importance for them of reading their writing aloud. Humans construct sentences with a (mostly unconscious) sonic influence. How things sound really matters.
My theory is that LLM prose sounds slop-like because the base unit for an LLM is not a single sound.
Thus, perhaps, you get the uncanny, meandering, hollow-feeling outputs, which feel vapid to human readers, without the reader really knowing quite why.
I think this is also part of the reason why LLMs are rewarded for and thus more strongly exhibit their faux-punchy constructions, such as the infamous “it’s not [X], it’s [Y].” The transformer model can, with current tokenization, articulate some features of “good writing”, but not quite all.
I think there are a number of fairly simple SOLNs to this (or ways to test this hypothesis) without making tokenizers purely syllabic across the board, which would probably be a mistake.
Whether prose output quality even matters for models which are not explicitly trained for creative writing is highly debatable, though.
Cthulhu_
4 days ago
It's (at least) two things I think; one is that thanks to the internet, the ability to focus on things for long is shot - I struggle to read long form content on the internet, generally speaking. I also did my whole career with Google and skim reading documentation looking for solutions for an immediate problem more than getting a wholesale end-to-end understanding.
(this isn't a new observation / thing, see https://randsinrepose.com/archives/nadd/ originally from somewhere in the 2000s)
But now with AI it's easy to generate a lot of prose. I'm thinking for example of the merge request descriptions my colleague submits. And here's the thing, it's not even bad or incorrect - I read it thinking, if I were to really sit down for it, get into The Zone etc, I could crank this out myself.
But I don't because honestly the MR is relatively trivial and can be described in just a sentence or two. Style and accuracy aside, my current take is that LLMs write things that humans wouldn't bother with, because it's too much effort and there's not enough value.
I mean it's an MR for an enterprise library project, the chances that anyone will ever read that MR description again once it's merged is close to zero.
That said: LLMs themselves don't get that, they will read everything you tell them to with the same amount of attention. Granted, their context windows are limited (and performance declines the more you put in it), so they too will use some tools to skim and pick out the important parts, but I think that's a temporary thing. The huge amount of LLM prose will end up being consumed by other LLMs. I don't know enough about the subject whether that's a good or a bad thing though, right now I think it just adds to the noise until it becomes noise in, noise out.
pj_mukh
4 days ago
Gentle reminder that most AI token spend is used for TPS report type writing not Cutting edge fiction writing. Creative writing, or general human-to-human communication was never really the target. It's not surprising the first majority use of this was code.
Endless headlines like, "aArON SORkIn SAYs hE WiLL Never UsE ai to WRite" doesn't change that base fact. No one asked him to, other than clickbait journalists.
If you find this hard to believe, just think about relative market sizes.
qlte
3 days ago
Why are you sidestepping OP's clearly articulated grievance about LLM writing infiltrating every corner of the internet to bring up some personal Aaron Sorkin hobbyhorse?
Lying with statistics (well, your own guesswork not even hard figures) to say actually LLM-ese is mostly confined to TPS reports where they're not hurting anyone is in conflict with our own experiences being subjected to LLM spam/slop every day.
(Not to mention that many of the same frustrations wading through a massive volume of tediously long, low SNR LLM also apply to the workplace where the unfortunate recipient of those "TPS reports" now has to wade through all that filler to find the occasional important detail. While the sender pats themselves on the back for 10x-ing their productivity, by creating more work for their coworkers)
pj_mukh
3 days ago
AI slop is a misuse of AI. All technologies have misuses. Cars kill 40,000 people a day, but nobody questions them because everybody uses a car everyday.
The fact that AI is quietly automating TPS reports in the background and slowly uplifting productivity metrics making things cheaper for us all is not immediately obvious.
And therefore…gentle reminder.
spencerflem
3 days ago
Cars also suck and we’d be better off if our cities were t designed around them
pj_mukh
3 days ago
Agreed, but trucks will remain the workhorse of our freight industry all the way to the last mile in our cities. AI is in a similar position behind the scenes.
I’m happy to hear of design ideas to reduce slop on the internet that doesn’t involve a direct attack on the underlying technology.
DeusExMachina
4 days ago
> As someone who is aggressively pro AI
But then
> I've also noticed how awful everything is to read lately
> A site I read stories on has been flooded with AI generated content, all of it terrible
I'm not taking either side, but I'm curious about how you reconcile being aggressively pro something that, in your own estimation, is making everything terrible.
rebeccaskinner
4 days ago
I think you can find a technology useful and think that it will continue to improve and solve some problems while simultaneously being frustrated at the technology being used badly and applied inappropriately.
krupan
4 days ago
And yet you remain aggressively pro AI? Why?
allthetime
4 days ago
So, you're still aggressively pro AI even though you recognize how quickly it is destroying the quality of online content?
I've noticed this a lot in conservative spaces especially, when they say "I love Trump, but..." -- why the need to say how much you love AI before criticizing it? Who are you trying to convince?
I use LLMs constantly for work and they have genuinely improved my life. But I absolutely despise generated "content". Text, audio, video, images -- for the reasons you describe.
It's ok to have a nuanced and multi-faceted opinion and approach to this and not have to call yourself a raving fanboy.