Did Claude increase bugs in rsync?

202 pointsposted 9 hours ago
by logicprog
(alexispurslane.github.io)

186 Comments

aesthesia

3 hours ago

I don't have a dog in this fight, but a few points that look a little suspicious:

- The release with the highest number of attributed bugs is the release _right before_ the first release with Claude-coauthored commits, released in January; is there a chance that unattributed LLM-authored commits made it into this release?

- The release attribution methodology is not great, since it will tend to attribute bugs introduced in a minor version update to the longest-lived patch release of that minor version. I doubt that 3.4.1 actually introduced a lot of bugs, but since it was released a day after 3.4.0, bugs that were introduced in that release get attributed to 3.4.1.

- Relatedly, more recent releases have had less time to have bugs filed against them, so there may be a bit of a bias toward evaluating recent releases as less buggy.

OptionOfT

3 hours ago

You can use LLMs in multiple ways, from very hands on to make local changes to completely hands-off.

I've seen plenty of code that was LLM generated but the commit message itself did not have the co-author attached to it. This only seems to happen when someone's interface to the codebase is completely though Claude/Codex/..., and those are usually the most verbose commits, and yet they say the least, because they just summarize the code changes, not the why.

On the other hand I've seen developers using Claude as a tool. They have VSCode open and a terminal window with Claude and go back and forth, ensuring they write correct code, and leave the plumbing to Claude.

So maybe the author of the code started off small and it grew over time?

PunchyHamster

2 hours ago

Let's start with most outright alarming error - the claude statistics are taken out of whole 2 data points

logicprog

2 hours ago

That's sort of the point. There isn't enough data to extrapolate, and yet that's exactly what those outraged about AI were doing, and when you do do the very minimal types of analyses (permutation tests, and looking at distributions, mostly) that are actually valid, safe, standard, and useful to do on such low amounts of date, again, no evidence for the outrage shows up, and the two releases look so normal that it sort of shows no one would've cared if they hadn't known or found out that Claude was involved.

I really think this a much better standard of evidence — limited though it is — to outrage-fueled cherry-picked anecdotes, which is what has been driving this whole thing. If you disagree, and think the outrage should go one when I've shown there's an absence of evidence entirely for it (although of course, that's not evidence of absence; maybe I'll have to eat my words 5 releases down the line, but appealing to that now feels like a Russell's Teapot), would you care to explain why?

ofjcihen

an hour ago

I know you’re defending your work here but this behavior does absolutely nothing to help your point.

logicprog

an hour ago

Fair point. Let me edit (if I still can) to tone it down.

runarberg

an hour ago

The interpretations of the p-value is also alarming. One of the first thing they teach you in statistics class is: “an absence of evidence is not evidence of absence”.

This analysis showed that there is indeed an absence of evidence, but it concludes there is evidence of absence.

Traditional p-hacking is done by oversampling and overtesting. If you do 20 analysis on average one will show p < 0.05 by random chance. This analysis is doing the inverse of that. Under-sampling, and concluding with p > 0.05

xmddmx

22 minutes ago

The concept you need here is "Statistical Power".

The ELI5 version is that there are two mistakes you can make when looking at a P value:

Type I error, where your P value is falsely low. In the experiment being discussed here, it would lead one to conclude that AI code is worse. Otherwise known as a false positive.

Type II error, where your P value is falsely high, leading you to conclude that AI code is no different. Otherwise known as a false negative.

https://en.wikipedia.org/wiki/Power_(statistics)

One can calculate statistical power for a given experimental protocol.

My hunch is that if you did this, you would find this experiment is grossly under-powered.

This means you can't make the "absence of evidence" claim.

logicprog

an hour ago

> This analysis showed that there is indeed an absence of evidence, but it concludes there is evidence of absence.

I tried pretty hard to avoid saying that, can you point me at how to rephrase? The point I'm trying to make is just that there is absolutely no evidence at all for what people are saying with such absolutism and claimed objectivity (that Claude made rsync worse), and thus it doesn't justify the outrage.

> Under-sampling, and concluding with p > 0.05

How would I avoid under-sampling here? And if you're going to say it's because I only have 2 data points, well, the side making the positive claim — that Claude made rsync worse — only had two as well, and unremarkable ones at that, as I've tried very hard to show.

runarberg

an hour ago

You are interpreting the p-values on their own merit rather then using them to test a null-hypothesis. Quotes like:

> With a p-value of 74%, the answer is a decisive no. The odds ratio is 1.06 — essentially 1:1. Claude releases are no more likely to be above the median than any other releases.

are problematic in this context as the correct conclusion here is you just don‘t have enough data conclude whether or not you are more likely to encounter a bug after a Claude commit.

> How would I avoid under-sampling here?

You don‘t. You admit that you don’t have enough data and move on. What you are trying to do here is prove a negative, which is extremely hard to do. In your discussion you claim that the users complaining had no right to, however nothing in your analysis showed they were wrong. We simply don‘t have enough data (yet) to say either way. When we have enough data they may be proven right or wrong, but until then, we cannot conclude either way.

If you insist still, I recommend looking into bayesian analysis. Theoretically at least the posterior distribution from a bayesian analysis can be interpreted directly and analyses on its own merits. However I suspect your posterior will have way too much uncertainty to reach any conclusions.

logicprog

13 minutes ago

Edited that claim, and made several clarifications elsewhere. The whole point of this analysis is that outrage is unjustified on the basis of two totally statistically unremarkable releases that no one would have remarked on pre-AI (my further proof of this is that there was a pre-AI remarkably broken release, and no one did comment!) and zero positive evidence outside cherry-picked anecdotes for any negative impact. We should wait for outrage and version pinning and cancelation until there is evidence, no? I'm just trying to say that these specific releases are unremarkable, and there's no evidence at all of harm currently; I'm not trying to build any kind of predictive model for future Claude releases to say anything grander than "these specific releases are fine, what are we freaking out about?", not some claim about what Claude-exposed releases will look like or trend like in the future or in general.

logicprog

3 hours ago

Your first and second points seem to contradict each other because if all of the bugs for 3.4.1 should be attributed to 3.4.0, that pushes the timetable back even further that unattributed LLM commits would have to have been being committed to the project, which just makes your point even more absurd.

Which brings me to my overall response, which is that there is absolutely no evidence, and nothing even intimating this hypothesis, that LLM commits were secretly being added to earlier releases before they were attributed, and that's why the rate of bugs is higher. There's no reason to think that it's an unreasonable thing to think, and there's no evidence for that whatsoever unless you beg the question and assume that higher bug counts must automatically indicate AI involvement, which is just circular reasoning. You're essentially just making up a hypothesis out of thin air to preserve your point.

Regarding your third point, that one's fair, but I've done the analysis and I can put it up if you want, as to how long it usually takes to find bugs and how far through the release cycle we are for each version.

aesthesia

3 hours ago

Sorry, I should have said this explicitly in the original comment: I think you're likely _correct_ that there isn't a clear increase in the rate of bugs attributable to LLM-authored code in rsync. Your analysis provides evidence in this direction; these are just the things that made me go "hmm". They're not accusations or claims that the conclusion is invalid. But they're definitely things to be curious about.

Regarding unlabeled LLM-authored commits, I don't think it's unreasonable in general to think that an open-source project might have had unlabeled LLM-authored commits at some point before 2026. Looking more closely at rsync's recent commit history, I think it's less likely in this case. There's just a low number of commits in general, _until_ large batches of Claude-authored commits start showing up early this year. But this then raises some questions about the bugs-per-commit metric; it does correct for something like "size of release", but also obscures a significant shift in commit velocity that may be downstream of adding LLM development tools to the workflow.

Like I said, I don't have a dog in this fight, and I try not to approach sorts of questions from a position of explicit advocacy. I do think it's an interesting question, though, and we should try to understand what the data is actually telling us.

jonquark

3 hours ago

Isn't the metric that you've used "bugs per commit ~ per new line of code" going to miss the issue?

All code is technical debt.

If rsync releases used to have 500 lines changed and 5 bugs in and AI-powered rsync releases have 50000 lines and 500 bugs, it's the same bugs/line but much worse experience for the user?

I've not looked into the details of this case and I do use AI assistance coding at work but in my experience, the problem is that it's too easy to write lots of code and therefore hard to review the huge volumes of code and this analysis will ignore that?

edit: actually your table shows there weren't unusually large numbers of commits in this release, so perhaps my initial skepticism shows a bias I have?

thorum

3 hours ago

Unfortunately for the people mad about this, I predict the only thing they will accomplish by pressuring the rsync maintainers, is to discourage everyone else from responsibly disclosing their use of AI. You’re just going to make people disable Claude attribution on their commits to avoid drama.

zzyzxd

3 hours ago

I never care about AI usage disclosure, because I don't believe that human produced code is necessarily better than AI produced code, unless it's someone I personally know.

People need to be responsible for code they commit and push anyways. This has never changed. Whether the code is written by hand, by their cat walking over keyboard, or by AI, is not my concern.

A project's code quality can decline for all kinds of reasons. I don't think it's productive to laser-focus on whether it's produced by AI or not. That's a distraction. If a person just want to find excuse to criticize AI, and another person wants to fight back and defend AI, sure, go for it. But that's not how you would want to assess a project's code quality.

calvinmorrison

2 hours ago

something as simple as requiring sign-offs like the DCO maybe relevant to people who care. I do think the driveby stuff may get smaller. People dont need to get stuff upstream. I have lots of patches I am keeping downmstrea and instead have a trigger system when new packages updates drop into debian and i rebuild the package with my patches on top using quill. Other systems like gentoo basically always supported this flow.

So - why bother forking or going upstream? maybe its selfish. I think publishing the patches are cool but I feel less of a need to force other people into doing what I want or even writing every possible configuration or solution. I just hack it for me

matheusmoreira

3 hours ago

> You’re just going to make people disable Claude attribution on their commits to avoid drama.

People should be doing this regardless of drama. No reason to provide free advertising for trillion dollar corporations. Generated-by trailers are only relevant when contributing to third party projects, in that case disclosure is polite.

Aurornis

2 hours ago

The value of the Claude attribution is that you can tell at a glance who used AI.

I don't care about the advertising angle. We all know Claude by now. I want some indicator that AI was used.

block_dagger

an hour ago

At my employer, if AI is not used, it shows up on your performance report and you’ll be told if you don’t start using it, you will be dismissed. I work at a medium sized successful YC-backed SaaS. So here, the attribution is meaningless - they look at your Bedrock and LLM API calls as well as Claude Code history.

matheusmoreira

2 hours ago

And why do you want to know that? So you can call our projects slop? Ostracize us?

Hammershaft

2 hours ago

Because LLMs are not humans, and the code they produce will have a different distribution of failure modes than human written code, so attribution is useful info while reviewing?

matheusmoreira

2 hours ago

> while reviewing

As I said, disclosure is polite when contributing code to third party projects which will undergo human review.

No need for such things in one's own projects.

Groxx

an hour ago

>which will undergo human review

This can be largely assumed to be true for any open source code. It's kinda the point of open source.

matheusmoreira

an hour ago

Nope. It cannot be assumed at all. Maintainer could just as easily tell Claude to review the hand written code you sent instead of spending any effort on it. Maintainer could sit on the patch for months on end only to swoop in later and rewrite it instead of engaging with you, thereby erasing your contribution and attribution. Maintainer could just ignore you entirely despite the pervasive "patches welcome" attitude.

If there's one thing I learned not to do in open source, it's to assume nonsense like that.

Groxx

an hour ago

I'm referring to the fact that "open source" quite literally means "readable by humans [and machines]", and anything beyond that is a subject of debate. There are more users than readers in nearly all cases, but being able to read the code as a user is a significant benefit at times, and it's one of the reasons it's such a large ecosystem in terms of both users and contributors. (it usually being free is another big reason, of course)

Even with coding agents gaining popularity, many humans still look at the code at some point.

matheusmoreira

an hour ago

I see. That depends on how much I care about the project. My favorite ones get weeks of review and refinement, to the point I still consider them to be more or less hand written. Not all projects get to be that important.

codygman

2 hours ago

So that the AI model that generated code can get proper credit and we'll know to use (or not use it) next time.

matheusmoreira

2 hours ago

That's not at all what someone who wants to "tell at a glance who used AI" actually wants to know.

ezst

an hour ago

Some people prefer organic grown food for all kinds of reasons, does it matter to you they would want the same for code? (Also, I'm not picking a side here)

matheusmoreira

an hour ago

It matters when I'm contributing to their projects. In that case I'll go out of my way to be polite and learn their rules.

eschaton

44 minutes ago

So we can know which commits will be infringing others’ copyright.

Aurornis

2 hours ago

You don't need an AI attribution tag to recognize slop. In my experience reviewing PRs, the slop-pushers are most aggressive about stripping the AI attribution anyway. It's the normal devs who use a little bit of AI who leave it in.

The tag is helpful because AI authorship is different than the human authorship. When you work with a project or team for long enough you start to trust certain people and their intuition, but when they start submitting AI-produced code you have to reset and review it like AI code.

I use these tools a lot, too. But I want to know where the code came from so I can review it accordingly. The source matters.

> Ostracize us?

I don't know why you're so defensive. If AI wrote the code just be honest about it.

If you outsourced the code writing to some guy named Bob on Fiverr, I'd want to know that too.

matheusmoreira

2 hours ago

> I don't know why you're so defensive.

Check it out:

https://lobste.rs/s/29pm2f/llm_generated_submissions_should_...

https://lobste.rs/s/ytim7h/collection_small_low_stakes_low_e...

Aurornis

2 hours ago

I'm not interesting in joining into some argument you're having with someone on lobste.rs

matheusmoreira

2 hours ago

You're not supposed to join. You said you didn't know why I was defensive. I showed you those posts as evidence of the stigma attached to LLMs and their usage. Now you know why.

eschaton

42 minutes ago

It doesn’t help your case that your response is to say “well I’ll just hide my use.” That’s fraud.

julianeon

3 hours ago

If Claude is actually good enough to commit to rsync, of course I'm going to look at that and think "it's good enough for my side project too." And (benefit to companies aside) that is info it is useful to know, if it's true.

amiga386

3 hours ago

Yeah, this is why it's obnoxious and this is why scummy marketers do it. If you don't aggressively turn it off, they leech an implicit endorsement out of you.

- Sent from my iPhone

AnotherGoodName

2 hours ago

Alto hug the iphone sigoff is hilaripus sonce fhe meyboard is so bad it always comes across asa an ask doe forgivebeds

— Sent from my iPhone

AlienRobot

2 hours ago

Indeed. The best endorsement is done explicitly by obnoxious users.

I use Linux, btw.

trwired

3 hours ago

Is that a bad thing? I mean from the perspective of Anthropic's marketing department sure, but if agents are just another type of tool in developer's tool belt - as I see people recently like to claim - attribution feels kinda weird. In the end it is the developer who is responsible for their commits.

eli

3 hours ago

Yeah I think it's a bad thing. It's context about how open source code was written that is lost.

And I guess maybe there's no such thing as bad press but at least in this cases it doesn't seem like effective marketing for Anthropic.

eschaton

an hour ago

“Don’t get mad at people for doing something unethical or immoral, or they’ll do something unethical or immoral!”

Disabling attribution of LLM-generated code is fraud, because you’re saying you wrote the code.

Of course that fits right in with the use of an LLM to generate code in the first place, since what it’s actually doing is regurgitating its inputs stripped of any license and copyright notice.

UebVar

24 minutes ago

I'm very certain that this is not fraud, across multiple legal systems, both roman and common law. In both cases fraud requires a person is deprived of a material good. Neither the defrauded person or their material loss is present in this case. Maybe there is a oddball legal system somewhere in the world where fraud is something entirely different, but i doubt it. "Fraud", just like "Decorator Pattern" is a well established concept and pretty simple concept, even if there are edge cases. This does not fit at all.

In academia this is miss-attribution, outside of academia this does not exist.

This is clearly not not copyright infringement either as LLMs do not claim copyright, nor could they. Just like the photograph taken by the monkey, or pictures drawn by crows. LLM output is not a creative work either.

If this is unethical or immoral is a totaly different question. I really dont think so and I dont think you argue that position well.

overgard

2 hours ago

I mean, I don't think commits are the place for tool attributions. I want to know what the change was, I'm not really interested in your tool selection (put that in the PR if it's relevant). It'd be just as irrelevant to see "written on my macbook in neovim"

hnav

2 hours ago

Depends on what the claude attribution actually means. A lot of people will just get the thing building and then ship. To me that attribution is generally a red flag.

eschaton

41 minutes ago

It means “this contribution likely infringes someone else’s copyright.”

potsandpans

3 hours ago

I think it will be funny to watch people lose their collective minds when open source maintainers start requiring llm use.

This idea that the community can try to pressure an open source maintainers about the tools they use based off of kneejerk political reactions is so offensive.

Let's go the opposite way: "sorry I'm closing this pr because it didn't use an llm."

automatic6131

3 hours ago

"let's go the opposite way"

Do you have any popular open source projects? Or are you just an Internet gremlin?

potsandpans

3 hours ago

I'm a successful distinguished engineer within mag 7, what are your qualifications? Please send me your resume and social security number to verify that you're qualified to speak on the matter.

matheusmoreira

3 hours ago

It makes no sense at all to do that. The only thing that matters is whether the code is good.

eschaton

28 minutes ago

That’s not the only thing that matters. The provenance of the code also matters enormously, specifically whether the person contributing it actually has the right to do so.

If I contributed code to an Open Source project behind my old employer’s back, that would have been bad, because that code was owned by them and not me, even if I wrote it on my own time using my own equipment, because of the contract I signed with them.

If I copied code out of an AGPLv3-licensed codebase and contributed it to a BSD-licensed codebase without telling anyone, that would have been bad, because I did not have the right to change the license on that code to BSD (or change the license on the codebase to which I was contributing to AGPLv3).

If you use an LLM to produce code, you may well be doing the latter since an LLM is actually just regurgitating portions of its inputs. This is not a hypothetical scenario; I’ve personally encountered a case of someone using an LLM attempt to contribute code I recognized from a specific Open Source project under one license to another project under a different license, while claiming they “wrote it themselves.”

Any project that accepts contributions needs to take liability seriously and manage their risk appropriately.

matheusmoreira

22 minutes ago

"LLM produced licensed code and person contributed it" is indistinguishable from "person contributed licensed code". The LLM is irrelevant. Result is the same as if they had copy pasted it.

potsandpans

6 minutes ago

The genie is out of the bottle here. If this were true then all fortune 500 companies would be pearl clutching and limiting their developers access to these tools.

But for better or worst I can assure you (for which you have no reason to believe me, just look at the headlines): nearly all tech companies are setting internal goals to have x% of code generated by llms by y date. And speaking as an insider, that x number is very large and that y date is very soon.

And before everyone continues to downvote me because I'm saying things that you don't want to hear, you have to realize that this is the world we live in now.

So, either you're right and the legal entities attached to some of the most powerful tech corporations have just decided to flaunt the law. Or you are missing something, or the game has changed.

Open source projects that want to hide behind provenance as a gate keeper to introduce llm generated code into their code base are going to get smoked.

There's nothing stopping a company like anthropic from funding an open source division that starts forking projects and accelerating the development. Expect 1000x more Buns.

There's nothing stopping an wealthy individual who wants to do that.

When the dust settles, no one is going to be worried about what you've typed here.

And if somehow the ip lawyers and capitalists won, then China will become the tech hub of the world.

Whether it's right or wrong, that is the reality.

potsandpans

2 hours ago

This is my whole point. The whole thing is ludicrous.

And lo and behold, people are losing their collective minds, bridgading my posts, flagging me and demanding credentials.

mohamedkoubaa

an hour ago

I'd be willing to be that an undisclosed LLM disclosure will follow a developer around for the rest of their career

eschaton

27 minutes ago

That kind of fraud absolutely should. (I suspect you mean “undisclosed LLM use.”)

tiahura

a minute ago

Write with your own voice and then polish with ai.

scsh

9 hours ago

> It does not control for commit complexity, security intensity, or bug severity. It does not distinguish between a one-line typo fix and a CVE patch. It is a blunt instrument. But the critics' accusation is also blunt: "Claude is making things worse." A blunt instrument is the fairest response.

If by fairest you mean to say that this analysis and response is sufficient, then I'm sorry but I have to disagree. We really need to understand if the nature of the bugs are worse from a user's perspective. Even if the rate stayed unchanged, if the result is the perceived quality of the software declined then I would personally consider that worse, especially if I were a project maintainer.

That's not meant to be wholly dismissive either. But in general, I don't think quantitative analysis alone is enough to fully answer this type of question.

skeledrew

8 hours ago

But it is fair. Up to this point I have yet to see anyone say they did an analysis of the code and found X regressions of Y severity. All they say is "there are more bugs because LLM". This analysis, which you can verify yourself if you wish, says "the bugs [number of] are pretty average even with LLM", which is a direct response to that. If you'd like a more nuanced analysis you're welcome to do one and share the result, if you're so inclined.

MostlyStable

2 hours ago

That which is asserted without evidence can be dismissed without evidence. This is more evidence, and of greater rigor, than was used to make the assertions. That's good enough for me. If someone wants to actually do the work to support the original claims with better evidence, great. I'd love to see it. Until then, I'm going to not worry about this issue.

mikaeluman

3 hours ago

Not going to critique this survey. Must have taken a lot of time and required a lot of patience. Great work!

I think it will be up to some group in academia to make a real full blown study across several repositories.

There must be tons to learn on how LLMs have changed software development and perhaps the cleanest separation will simply be going by what repositories declare e.g. "No LLM involved" vs those that proudly do the opposite or are neutral.

Bugs is not the only variable of interest here. I am guessing someone is already doing this as we discuss it here...

AEVL

31 minutes ago

How does the analysis look if we only count the >=90 severity cases—that is, if we downgrade the severity of all <90 cases to 0?

lbrito

an hour ago

Wait, how is any of this relevant if there were only 2 Claude commits? My statistics courses are far behind me, but don't you need at least 30 data points to conclude anything?

logicprog

an hour ago

Depends on the methods you use. If you're trying to fit curves and so on, yes. The methods I use were designed for very low amounts of data, and are generally okay for that, specifically and especially when you're just trying to show a lack of evidence for some non-null hypothesis.

And again, that's kind of the point. There's exactly zero actual evidence, however you slice it, that "Claude broke rsync" except cherry-picked anecdata, and the whole point of my analysis is to demonstrate the total lack of any such trend/evidence at all, and just how in-distribution/normal these releases are, to show that if people hadn't known Claude was involved in them, they wouldn't have remarked on them.

faitswulff

9 hours ago

> The analysis uses a single metric: bugs per 10 commits (bugs/10c).

Bugs per commit as a metric papers over severity, both in terms of security severity as well as the effect on the user. A mislabeled button has the same weight as the entire app crashing in this framework.

germanjoey

3 hours ago

IMO "bugs per commit" is even worse than that, because, in addition to what you say, it also hides the extraordinary spike of commit activity of a project that had previously been stable. [0]

It is the exact metric you'd choose if you wanted to make the current situation of rsync look like not a big deal.

[0] https://github.com/RsyncProject/rsync/graphs/commit-activity

logicprog

3 hours ago

Yes, but we know why there was an "extraordinary spike," and it has nothing to do with rsync being "vibe coded." The maintained has directly addressed this.

floxy

3 hours ago

Seems like this would be a good place to link to that.

logicprog

3 hours ago

I link to it multiple times in TFA and quote the specific thing I'm talking about here in there to explain that possible confounder. I think I've done more than the work I'm obligated to it.do to make all of the relevant information available to you. You are just refusing to use

runarberg

2 hours ago

I am not finding these links in TFA, I see a link to an issue #929 which (as mentioned in TFA) has over 350 replies, and and opinionated summary of what transpired, including some detailed description of specific posts there. However I did not find the maintainers response.

Of interest is this post here: https://github.com/RsyncProject/rsync/issues/929#issuecommen... which echos the same concern which was raised up thread, however, I failed to find the maintainers’ response.

EDIT: Found it! it is in the (untitled) discussion section (after the results).

https://lobste.rs/s/k1b0za/rsync_outrage#c_2iowov

EDIT 2 (and advice on design): The page design changes backgrounds after the results sections, which kind of conveys to the user that they have reached the end of what was is important and can just skim over the rest (usually pages have a radical change in typography like these when you’ve reached the comment section), however this is what is analogous to a discussion in a typical paper, and is arguably the most important part. I had simply assumed that you just left it at the result and skipped the discussion as a stylistic choice.

logicprog

2 hours ago

> EDIT: Found it! it is in the (untitled) discussion section (after the results).

I also paraphrase Tridge himself explicitly saying that this is why commits/releases have increased:

> Essentially, this isn't a "Claude" problem, it's a "more security work" problem, something that Tridge himself confirmed in his response, describing how a flood of AI-generated CVE reports forced rapid, extensive changes to rsync's attack surface.

> The page design changes backgrounds after the results sections, which kind of conveys to the user that they have reached the end of what was is important and can just skim over the rest (usually pages have a radical change in typography like these when you’ve reached the comment section), however this is what is analogous to a discussion in a typical paper, and is arguably the most important part. I had simply assumed that you just left it at the result and skipped the discussion as a stylistic choice.

Good point, I assumed everyone would read till the end, that's on me. I'll give it a heading.

ex-aws-dude

4 hours ago

Why don't you prove the bugs increased then?

Why is it that some unfounded claim is made and the onus is suddenly on the project maintainer to prove it beyond all doubt?

It should be on the person making the claim to prove it

logicprog

2 hours ago

I've now resolved this. The new version, which should be live on GH Pages soon, uses — what I think is — a pretty good methodology for assigning severity to each bug, normalizes it to 0.0-1.0, sums that, and treats that as the total severity weighted bugs, then does the analysis based on that. It did not change the analysis in any material way.

skeledrew

8 hours ago

There was no analysis of severity in all of the rage posting that occurred. The single point being pushed was "use of an LLM led/leads to more bugs". The author specifically states that's what they're addressing (blunt accusation -> blunt response).

atmavatar

5 hours ago

The specific problems mentioned were all reasonably severe. The original post itself described a show-stopping bug:

    So my systems recently updated to rsync 3.4.3, and as soon as that happened my backup system - which does incremental backups using multiple --compare-dest= arguments - started to fail on anything but a full backup.
Incremental backups is perhaps the primary use of rsync, and they were broken for this person. That's pretty severe.

The second reply is similar:

    i wondered why my 3d printers were running like sh*t and at 100% cpu; turns out log2ram uses rsync.
This one I took with a grain of salt, since it read more like a dogpile than an actual bug report. However, if it's genuine, it's also reasonably severe.

Later in the comments, someone attempted to provide a list of issues that had been added: https://github.com/RsyncProject/rsync/issues/929#issuecommen.... The list included several failures to build or run rsync that appear to have resulted from broken backward compatibility. That seems reasonably severe. If intentional, I would have expected mention in the release notes about the removal of backwards compatibility, but none was made.

The issue comments already degraded into a lot of unnecessary vitriol even before the above mentioned comment and only gets worse from there, so I stopped. But, the fact remains that the whole issue started with a severe bug.

I applaud the attempt at dispassionately analyzing whether the recent LLM releases of rsync were normal or outliers as far as bugs are concerned, but I don't think you can do so properly without analyzing severity.

skeledrew

4 hours ago

To keep such an analysis fair and contextually relevant, it would have to be extended to the previous 928 issues as well (of course filtering for bug reports). I don't see anyone doing such an analysis, I think because they don't expect they'd find it useful (at least not as the rage fuel that many are seeking); what they'd be more likely to find is that there is a similar severity-mix going all the way back to v1.0.0, because these things inevitably happen whether coding is done by human or machine.

"A lot of claims in the wider discussion have treated every recent bug report as if it had the same cause. That is not accurate. Some reports were regressions from recent security hardening, some were missing historical test coverage, some were older bugs found because rsync suddenly had more eyes on it (especially by AI that can find issues quickly) and some were packaging or environment-specific failures. A Co-authored-by line is not enough by itself to establish root cause." - https://github.com/RsyncProject/rsync/issues/929#issuecommen...

parliament32

27 minutes ago

Thank you for (re)writing this in your own voice. Despite how much effort might be put into methodology, data collection, etc.. reading slop is unbearable, full stop. It's not intentional, but I have almost a nauseated reaction when the "AI tone" comes though, regardless of how good the data or how accurate the writing is.

Your verbosity and sentence structure are not a problem. I hope that publishing this gives you a bit more confidence in your writing, because it's legitimately good.

geraneum

9 hours ago

> But the critics' accusation is also blunt: "Claude is making things worse." A blunt instrument is the fairest response.

So the criticism was bad, and that somehow makes it ok to use a bad metric?

logicprog

9 hours ago

That's not what I'm saying. What I'm saying is that if the criticism is referring to a broad set of metrics like bugs per release and number of commits that were made by Claude, then it's correct to look at precisely those things because that's what the claim is about.

abirch

3 hours ago

AI + Interest != Expertise

I come to hn because I get very nuanced, informed information and glorious puns.

epolanski

2 hours ago

What would be a better one?

tptacek

2 hours ago

This is a neat post and I'm glad it got written and this is a little bit off-topic but:

Hey, 'logicprog, your writing is fine!

Use LLMs to critique your writing, check its structure, vet your choice of topic sentences, check flow from graf to graf and section to section, look for passive voice and overused words. LLMs are fantastic for that. But don't use a single word an LLM suggests in your actual writing. If it suggests something really fucking good, too bad, those words are disqualified. It's an easy red line to adhere to, easier than it sounds, and it'll keep your writing human.

(You ended up somewhere around here anyways, but that was after you posted something with LLM-written language because you weren't confident enough in your own writing. The things you do "worse" than an LLM are what make you you; be protective of them!)

mmonaghan

an hour ago

I think there's evolution at play here - if you dislike AI enough to opt out of using any ai-generated code, you will likely suffer. I think there's definitely a conversation to be had about whether to disclose AI use or not but that's a separate issue if you assume that everyone is using it in some respect.

PunchyHamster

2 hours ago

The fact last few commits were attributed to claude doesn't mean previous ones didn't use it.

Also if you write a paper where you get statistical conclusions out of whole 2 datapoints you'd be laughed out of the room

vintagedave

an hour ago

Why not? Claude marks its commit messages. That there were none, and then there were, seems a signal.

Especially since if the earlier commits were so clearly AI authored yet without the Claude marker, surely you or anyone would be able to spot them. You could say, X commit does not have the Claude commit marker yet was AI written. But for all the speculation on this thread, I haven’t seen anyone actually doing that. What may be possible is that the rsync maintainers used AI to assist yet reviewed and edited themselves, as many devs do, and if so then the stats in this article are still notable: there are no poor quality outliers that can reliably be attributed to AI and if one specific release (3.4.0) was, the subsequent releases which presumably also had as much AI as this speculative hidden AI release only show improvement and thus act as a pro-AI argument.

The blog has many more datapoints than two. It compares many releases. You’re looking at 2-vs, not 2.

logicprog

an hour ago

> Also if you write a paper where you get statistical conclusions out of whole 2 datapoints you'd be laughed out of the room

I'm using methods appropriate to that low amount of data, first of all. Second of all, since I'm only trying to show there's no evidence for the anti-AI hypothesis (not disprove it, or prove the null hypothesis), that's sufficient in itself. Also, I wonder why nobody said things like you're saying ("there's too little data to tell") in response to all the absolutist claims that AI caused rsync to get worse?

> The fact last few commits were attributed to claude doesn't mean previous ones didn't use it.

At this point, you're just positing Russel's Teapot: you'll keep assuming more and more of the code was "secretly" Claude when there's no evidence for it and no reason to think so, just because you've started with the assumption that Claude makes things worse and you want to find a way to prove it.

logicprog

3 hours ago

Another update: did an automated severity analysis on each bug report (~2000 of them!) using an LLM at temp=0 with a very strict rubric (and I checked to make sure that it rated things in a consistent, stable way using it). The rubric, LLM used, and some example ratings are included in the methodology section. For now, the information was just stored per-bug in the DuckDB and used to filter out non-bug bugs, to get a clearer signal. I'm going to try to use it to see if the post-Claude bugs were more severe in any way next.

Polarity

9 hours ago

so the answer is: no. actaully less bugs. thanks

gjvc

3 hours ago

"fewer"

WesolyKubeczek

an hour ago

The discussions around this have devolved to excrement anyway, I feel tempted to invoke the meme where the goose asking a guy what his jacket is made of, asks “where is your reproducer case!?” instead.

Instead we have a shitstorm over presumably legit issue, for which the only source is some mastodon post.

One command that used to work in 3.4.1 and stopped working in 3.4.3. Just one! We could have already bisected the living shit out of this and go home, but no.

steno132

an hour ago

This is just narrow thinking. Say Claude did increase the bugs in rsync by a negligible factor.

So what? You've saved a significant amount of time for a decent number of humans, and if those humans are working on other projects, the overall net output for the world is net positive compared to without LLMs.

You have to broaden your perspective. It's not just about how rsync was affected.

boxed

an hour ago

Let me translate this comment:

> ok, so I was wrong and badly, but I will double down and say I was right anyway

KronisLV

2 hours ago

Pretty cool site!

> v3.4.3 has been out long enough that its rate (5.00) is already comparable to historical releases. The "wait and see" argument is an appeal to an unknowable future that shifts the burden of proof away from the critics. If more bugs surface, they will enter the distribution like every other release. There is no reason to expect a regime break.

I mean, as someone who uses LLMs, it might be a good idea to consider how one might limit the amount of bugs that will appear in the future at least a little bit: parallel iterative code review loops would probably be the easiest and most applicable to LLMs, though I guess test coverage and other code analysis tools help too.

overgard

3 hours ago

The TLDR seems to be: needs more data.

gadrev

9 hours ago

Ok.

  $ apt-cache policy rsync | grep Installed
    Installed: 3.4.1+ds1-7ubuntu0.2
  $ sudo apt-mark hold rsync     
    rsync set on hold.

imurray

9 hours ago

That version has security fixes from the same day as the latest rsync release: https://ubuntu.com/security/notices/USN-8283-1

As usual, Ubuntu backported fixes and didn't upgrade to a new version. Whether or not they also backported regressions in edge cases that afflict the latest rsync, I don't know. Pinning the Ubuntu package may prevent getting further regressions, but is preventing you getting any future such backported security fixes.

logicprog

9 hours ago

Did you face any actual bugs or regressions? Or are you doing this just because of the bandwagon that's going around right now? Because until you can actually present an argument for why this release is worse than any of the others, which is precisely the subject of my post, then this is not an argument against my post at all. This is just a self-referential appeal to authority.

overfeed

44 minutes ago

> Did you face any actual bugs or regressions?

This is a terrible argument; I didn't need to have had secrets exfiltrated before applying row-hammer mitigations. If rsync is the cornerstone of my backup strategy, and has been for years, I need to trust that on its correctness, and for it to not lose my data. If I wait until I "face any actual bugs or regressions" - that will be far too late.

Stability is another issue not discussed. If the error rate holds steady, but number of significant PRs merged per release goes up from 5 to 200, that would be huge net-negative for my use case.

gadrev

9 hours ago

Nah, I skimmed TFA but then I went into the linked GH issues thread, and that's the one that scared me a bit. I just want to hold it for a while and not run into some of the things I'm reading since I'm on the latest ubuntu. Just a precaution.

I didn't have the time to actually think about any "arguments" at all tbh it's just a knee jerk reaction as I get ready to log off for the weekend. Not actually looking to argument for or against your post at all lol.

themafia

an hour ago

> If anyone complains about my verbosity or sentence structure — as they usually do, which is the reason I originally let the AI write the prose, among other reasons obsoleted by templating — they can go fuck themselves.

You can write for an audience or you can write for yourself. Which is fine either way but you shouldn't pass the blame for bad results on to your audience.

> and recieving almost no substantive input, discussion, or response on the actual content of the article

Well did you write it for that purpose?

> "Just wait, more bugs will surface" -- v3.4.3 has been out long enough

Wait for _more releases_. As your own data shows the bug rate is not consistent between releases. So this is probably not a worthwhile metric. Perhaps systems touched, new features included, or attempted fixes would be a better way to contextualize releases and the goals of the author.

yobid20

2 hours ago

needs a tldr; im not reading all that. maybe claude can summarize it for me.

logicprog

an hour ago

And anti-AI people accuse people who use AI of being intellectually lazy. First of all, it's long because it's expanded to respond to all the criticisms. It seems that either something can be short, and dismissed as incomplete, or it can be complete, and dismissed as being long. Nice Kafka trap. Additionally, there's literally an Executive Summary section right there, for your TLDR.

noAnswer

26 minutes ago

Asked your Clanker what a joke is.

pushcx

7 hours ago

    What followed was extraordinary: 329 comments and counting, ranging from thoughtful concern to outright harassment.
    The thread did not stop at words. One user posted My Little Pony drawings of themselves strangling the "project janitor that pushed vibecoded commits":
    It spread to Hacker News and Lobsters, generating hundreds more comments.
This is false, it did not appear on Lobsters. Here is the function in the codebase that prohibits this kind of brigading: https://github.com/lobsters/lobsters/blob/main/app/models/st...

Please correct your article.

tptacek

3 hours ago

It is neat that Lobsters has this feature (and HN should too), and I'm glad you took a beat to explain it. I think you didn't need the last sentence, though.

logicprog

3 hours ago

I have done so! that was a misremembering on my part. first mention of Lobsters is now here:

> On Lobste.rs, in response to the Medium essay Tridge himself posted in response, finally some users like boramalper begin to actually ask for evidence one way or another:

nairboon

9 hours ago

Is this an analysis made by/with Claude?

overgard

3 hours ago

FWIW, I asked ChatGPT to review the article just for my amusement. It's conclusion was:

"My honest assessment is that this is a competent calculation performed on a badly confounded measurement, followed by conclusions substantially stronger than the calculation warrants. It is useful as a rebuttal to “the Claude releases are obviously unprecedented disasters,” but not as evidence that Claude was harmless."

quentindanjou

9 hours ago

It very obviously is. "The Outlier Nobody Noticed" -_-"

dang

3 hours ago

[stub for offtopicness]

[see https://news.ycombinator.com/item?id=48416020 for how all this happened in the first place]

logicprog

9 hours ago

Some notes on this:

- I used GLM 5.1 to help with the coding and math for this.

- However, I explicitly dictated where the data should be pulled from (GitHub, Bugzilla, mailing list), how it should be tagged and grouped, and what data to look at (e.g. bugs instead of regressions)

- Additionally, I consulted with my wife, who has a master's degree in statistics from Penn State University for what sort of statistical methodology would be justified for this very limited data set, while still giving as much information as possible.

- I know the website looks like we stereotypically consider vibe-coded websites to look, but I actually explicitly asked for that. The original HTML design looked like a website from 1995, and I just prefer how this looks. It's pretty!

jchw

9 hours ago

I really struggle to believe you wrote text like:

> A simple distributional analysis of every rsync release with bug data. No model. No assumptions. Just placement.

logicprog

9 hours ago

No, I didn't write the text itself. I'm typically significantly more verbose and elliptical, and more than that, the numbers and methodology changed often enough over the course of the last couple days I was working on this because I was trying to get it to be as accurate and fair as possible that trying to keep the whole thing up to date manually would have been problematic.

jchw

8 hours ago

Sorry to say but I'm absolutely certain I would've preferred to read your worst attempt at a write-up over the grating utter shite LLMs output. It's not even a question, this is unreadable.

logicprog

8 hours ago

That's interesting; IME, most people get equally angry and are as likely to disengage with a superior tone over my autism-infodump verbose essay prose as with LLM output.

ok_dad

3 hours ago

At least when I write an autistic info dump people know I wrote it. Why give your voice over to a corpo slop factory?

Heck, I use LLM assistance for coding and I’ve even coded up whole features with the clankers, but giving it the right to speak for me is too much.

I should also add that I read and understand every line of clanker output that I publish for others, so I’m not a vibe coder either, just adhd.

skeledrew

8 hours ago

I read it perfectly fine. I see content, not style.

grey-area

3 hours ago

Style is also part of the content. Word choice, grammar, register, and tone all affect meaning and communication of that meaning. The medium is part of the message.

So your statement betrays a significant misunderstanding - there is no neat clean divide between style and content.

Also, LLMs often generate text that is plausible, but wrong, in ways big and small.

skeledrew

3 hours ago

Well, I got the meaning in the article fine, and have no complaints.

> Also, LLMs often generate text that is plausible, but wrong, in ways big and small.

So do humans. Always have, always will.

grey-area

2 hours ago

Humans acting with intention do it a lot less. The difference is that LLMs don’t act with intention.

skeledrew

2 hours ago

No, the difference is in the education/experience of any given human, which is mostly gated by age. Like you'd generally expect someone young to make a lot of mistakes, and as time went on they'd learn and make fewer. Pretty much the same with LLMs, which have been around for... a bit over 5 years now? What would you expect of a 5 year old acting with intention? Or 10? Or even a 15 year old?

jchw

7 hours ago

When you say, "I see content, not style," you are separating what is being said from how it is being said. While it is great that you can extract the core message, you are missing a fundamental truth about writing: style and content are rarely completely separate. Writing involves both.

Poor prose does not just make writing ugly — it creates friction, obscures nuance, and introduces ambiguity.

You can eat a gourmet meal out of a dirty paper bowl. You still get the calories, but the delivery mechanism definitely impacts the experience and the perceived value of the food. Same food, different response.

See? I can write slop too, I don't even need to burn down a forest to do it. If you are OK with every fucking thing being written exactly like this, good for you. I am not.

skeledrew

3 hours ago

The internet is going to really suck for you if you keep that attitude, because LLM use will only increase. Though also maybe not too much as the LLM-isms will likely be fine-tuned out of them to the point that the only way you'll be sure something is done with one is if the author left a note saying such. But maybe that'll make it suck even more as then you'd be without a definite target most of the time, always wondering how much of the thing you're reading is by human and how much by LLM...

jchw

2 hours ago

Uh... huh.

I waited a minute to make sure you weren't going to delete this post because frankly, if I had written it, I would have. Guess not, so... Here goes.

No. It is not the fault of my "attitude" that the Internet is going to suck. That is a complete reversal of the reality. The fact that even people without bad intent are already spreading slop everywhere should be enough evidence to essentially prove that there was never any hope. If this is what good actors are doing, what exactly do you expect from bad actors?

Also, to stress it yet again, I don't care if people use LLMs in general. I'll even say that I don't particularly care very much if people use them without disclosing it in most cases. If you're using it like a normal tool and not merely just dumping the output verbatim there is not any particular need to disclose it any more than you'd disclose other tools, though I think people would prefer if you did just for transparency.

My chief complaint is just how bad LLM slop writing is. It simply is not good at all. It would literally be much better for the Internet if they weren't so turboshit at writing. There is almost no writing style I don't prefer over garbage LLM writing. I'm dead serious. Early LLMs were worse at almost everything else, but they were a lot better at writing for sure. Something went wrong somewhere.

But I do also believe that it is inherently bad to dump prose as-if you are communicating as a human, but said prose isn't actually written by a human. If someone shows me a cool drawing that they made, that means that they sat there and went through the process of sketching, possibly multiple drafts, inking, coloring/shading/painting/etc. to create an expression. This involves many human skills that take years to hone, and every detail carries someone's explicit intention. I think that this is cool, and shows a great degree of skill and effort.

When you, of course, generate some crap from an image generator, it may very well look similar. It may emulate some actual defects that make it look like someone really drew it. But someone didn't. A model went directly from a text prompt and dumped out pixels on screen. No sketching. No layers. No thought processes about how to frame things or what details to include. That doesn't mean zero effort went in: I'm sure in many cases someone sat around and fudged with LoRas and inpainting for a couple hours and pulled the slot machine lever to get good seeds and etc. That doesn't mean that an AI model does not have some model for how to structure an appealing image: it does, that's obviously why the results can look decent to begin with. But when you dump out an image from an image generator and you wink wink nudge nudge present it as your own and people evaluate it as if you drew it, this is basically fraud. Everyone looking at it who doesn't know it is AI generated actually believes you went through the normal effort of drawing that image and all of the years of practicing skills and acquiring knowledge that takes. That's bullshit, and it takes away from the actual accomplishments of people who put in the work like cheating in sports does.

Like yeah, a lot of people are cheating at chess, by passing off engine play as their own, but does that really make it okay? When the entire point is using your brain and not just the raw outputs themselves, doesn't that hit you as a problem?

For generative AI, I personally draw this line at what I feel are expressions of creativity. If you use AI for drawing references, whatever. If you use AI to generate globs of repetitive code, whatever. Code can be creative but I do not view it as an expression of creativity and almost any tool is fair game. If you are using ML models for motion capture or some other data processing thing where humans had to do repetitive work before, whatever. Maybe these tools sometimes do devalue the work, but the LLMs are not doing the interesting part here, they're doing the boring part. (This is, in part, an admission that actually writing code is often pretty boring in and of itself, something that I realize programmers have been inconsistent with in an attempt to justify their value. But, I still believe it to be true.)

So okay fine. People are reluctant to disclose that they used AI to generate text because they fear the backlash that it will get them. This is understandable. What upsets me about this is that well-meaning people are apparently falling back to the idea that because LLM backlash is strong, what would be better than either trying to just simply write your own damn posts or be honest about your usage of LLMs... Is to just try to wink wink nudge nudge pass off more or less verbatim LLM writing as if it's a post that you wrote.

I am not ruining the Internet. There is literally nothing I or any group of angry mobs could do that would even remotely slow down the decay of the Internet even if we desperately wanted to.

So in fact, I'm not even trying to not ruin the Internet. I don't particularly care if my attitude is not helping or hurting. I'm not having an attitude as part of some grand strategy to save or destroy the internet. I'm having an attitude, because I am pissed off.

And I am pissed off because I am tired of reading posts the author probably only skimmed themselves.

aozgaa

8 hours ago

In general, it seems HN does not like to read llm-generated articles. I ran into this myself when using an llm to edit some stuff I wrote.

At the time, I found this a bit irritating, but with a few weeks time I see the merit. The informational content tends to fall into “derivative” territory when LLM’s write stuff. And people are here for novelty and some socialization.

Also LLM prose seems optimized for engagement rather than concise communication. Takes longer to sift through linguistic boilerplate to get to the point. (The quoted bit being a case in point)

fireflash38

3 hours ago

Why would anyone spend time reading something that someone couldn't even spend the time to write themselves?

jchw

7 hours ago

I just find it to be utter dreck. It has one of the most agitating prose styles I've ever seen. I would legitimately rather read actual broken English than the cliché polished turds Claude pops out. I am not an LLM hater, I think these tools are pretty impressive and often even useful, but even if I didn't care about the fact that I want to read communication from humans and not robots (and I do care about that, FWIW) I just find the current LLMs are horrid at writing.

And while the comments are always flooded with people like me, the upvotes seem to tell a different story; clearly LLM writing really does appeal to some people. Or idk, maybe a lot of people who vote on stories and don't comment don't actually read them. Hard to say for sure.

grey-area

3 hours ago

I think it’s just people don’t read before voting, they upvote on the headline and then come to discuss it here.

otabdeveloper4

3 hours ago

I don't even know what "just placement" is.

(I need a better model to translate from llmese.)

grey-area

3 hours ago

Sometimes the things word generators say just don’t make sense.

CuriouslyC

9 hours ago

I'd suggest writing the lead-in yourself and boxing AI prose separately from your prose in the analysis for future articles. You can give the humanized summary/eli5/key points, then have "details according to AI" boxes that go into nitty-gritty. People seem to dislike AI ghostwriting, but most of these people still use AI, so perhaps keeping authorship clear and separate will avoid some of the flak.

logicprog

8 hours ago

This seems fair. Of course, now that I've posted this here once, I doubt it'll get constructive engagement again, but I can at least improve this for the future

bri3k

8 hours ago

Even if everything in the article is true you should not use AI to write this. A analogy would be tobacco company report on how smoking isn’t so bad for you.

ex-aws-dude

3 hours ago

So the original unfounded claim has 400+ comments because its perfect HN ragebait

The author provides evidence to the contrary and the HNers won't even engage with it instead just talking about the writing of the article in classic HN bikeshedding fashion.

How about after that we talk about the formatting of the website and the colors?

This site is really going down hill

Where is the accountability for your own opinions?

Are you guys only upvoting things that confirm your existing gripes?

dang

3 hours ago

Comments like this do more of what they complain about, only with an extra layer of judgment.

It would be preferable if someone would seed a better discussion by engaging with the article's claims/observations.

ex-aws-dude

2 hours ago

Why did you the admin allow such ragebait to stay on the front page then?

Is that the kind of low effort posts we want around here? Just a link to a github comment of a screenshot?

You're complicit here in fueling the harassment of an open source project

dang

2 hours ago

I don't have enough background info to understand what you're referring to here.

Even if you're right, though, you shouldn't be posting comments that break the site guidelines.

dang

4 hours ago

This submission was heavily flagged, presumably because the article sounded like genai. But the article now says the following:

> After posting this on Hacker News and recieving almost no substantive input, discussion, or response on the actual content of the article, I decided to rewrite all of the prose in my own voice.

I've therefore turned off the flags and hopefully people can actually now discuss the claims/findings being reported.

hypfer

3 hours ago

> I decided to rewrite all of the prose in my own voice.

Soo... it didn't just sound like genai but was genai?

___

Huh. From the article:

> If anyone complains about my verbosity or sentence structure — as they usually do, which is the reason I originally let the AI write the prose, among other reasons obsoleted by templating — they can go fuck themselves.

This is kinda sad, honestly. But also should show the author that doing what people try to bully you into doing will not stop them from bullying you.

Just stick with your unique voice man. If people don't want to read that that's fine. They do not have to. You're fine

.. what are those em-dashes doing there though?

ellyagg

3 hours ago

Right so it’s gonna be a litmus test for knowledge workers going forward if they can separate style over substance. Genai tells are style. You have to be able to evaluate the ideas.

dang

3 hours ago

I doubt that you can separate style from substance in that way, because you can't separate writing from thinking.

I agree that it will be interesting to see how this develops going forward. One can imagine wildly varying scenarios.

hypfer

3 hours ago

Hm. Nah. Why?

Why should I care? If it's a good thought, chances are it appears without slop around it. If it doesn't re-appear, life will still go on regardless.

No need to shift through noise just to avoid FOMO.

logicprog

3 hours ago

> .. what are those em-dashes doing there though?

You're literally doing exactly the bullying I was trying to avoid, even while denouncing it. I like em-dashes. I have AuDHD, and they help me represent how I think.

hypfer

3 hours ago

> You're literally doing exactly the bullying I was trying to avoid

Uhm, no. Really just no. And, frankly, I find it shameful that you'd throw such an accusation at me.

But I guess we can stop here.

Idk man. The internet can be a bit too much sometimes. I truly get that, but this was too much from your side.

Wish you all the best.

skeledrew

2 hours ago

Why did you point at the em-dashes? It looks very much as though you're accusing the author of an update that was also generated (possible but they seem sincere enough about wanting honest feedback on the content, and making changes for that). Or you're saying the author - and maybe everyone in general? - should no longer use em-dashes because they're a LLM smell. Yeah I'd feel offended too. It's a real pity I can't find em-dashes on my keyboard, or I'd stick them in this comment.

ajkjk

3 hours ago

The em dashes are fine.

If someone gives them shit about their writing, that's on the critic for being shitty. If they use AI to write, that's on them for being fake. But, to write online at all requires being ready to have people be shitty to you and ideally not reacting in a way that makes the situation worse. Sounds like they need work on that part.

Anyway it is basically always possible for someone to find something legitimately bad about anything a person does. The question is, how much of an issue is that? Not much actually. So you have flaws. Fine, just be flawed. It had no affect on your life beyond your reaction to the attack. And putting aside that reaction is a prerequisite for learning anything useful (or discerning that there is nothing to learn) from the experience.

Good people will trust good intentions through the flaws, while shitty people will write off your work and your intentions because of the flaws (and try to make sure you feel bad about it in the process). But it's always they're too weak to express disagreement maturely, or sometimes because they're bitter and threatened by your good intentions directly. Either way, it's their flaw, not yours.

hypfer

3 hours ago

I don't think that you can successfully dismiss an obvious AI writing marker with

"No these are fine, now look over there!! <lotsoftext>"

Pay no attention to the man behind the curtain?

logicprog

3 hours ago

Great, so I rewrite everything in my own prose, and now it's still "obvious AI writing," just because I'm literate.

ajkjk

an hour ago

What? You are confused--human beings write em dashes also. Also you're being a dick to the OP, grow up.

otabdeveloper4

3 hours ago

> I decided to rewrite all of the prose in my own voice

"Claude, rewrite all of the prose in my own voice."

The funny part is that it probably works.

roywiggins

9 hours ago

> A simple distributional analysis of every rsync release with bug data. No model. No assumptions. Just placement.

If you want me to read your analysis, you are going to have to make it not read like Claude wrote it. What does "placement" even mean here?

rroblak

9 hours ago

Yeah, made me chuckle that an LLM— probably Claude— was used to write this.

The use of "regime shift" is what gave it away for me. I've never seen a human write that, but Claude does from time to time.

At least they removed occurrences of "load-bearing".

roywiggins

8 hours ago

"quietly" seems to be the new one recently

genxy

4 hours ago

Ohhh, quietly load-bearing is the real just. No noise. Pure fact. Delivered robustly.

gamegod

9 hours ago

It's the ultimate product for marketers. It inserts itself as an advertisement into every conversation now and defends itself against criticism. Just crazy. There's no hope for the rest of us.

logicprog

9 hours ago

It's not defending itself here, both because I used GLM 5.1, not Claude, and because I was the one who decided to do this analysis, iterated through six or seven different methodologies to try to find the one that was most honest with the data that I had (all of the methodologies showed directionally and often in magnitude the exact same thing, but I wanted to do something that fit the purpose, in consultation with my wife, who, as I've mentioned elsewhere, has a master's degree in statistics), and, of course, I specifically chose all of the metrics and sources for the data.

If you don't want to read the LLM prose, you can just go to the GitHub of my project, grab the scripts, and run the full pipeline. It will gather the data, build the database, and run the analysis from scratch for you, and you can look at the numbers directly. It's all repeatable.

logicprog

9 hours ago

"Placement" as in where the Claude-driven releases exist within the existing distribution of bugs per 100 commits. If they're not OOD, then nothing is unusual.

Also, it wasn't written by Claude FWIW, GLM 5.1.

jrflowers

2 hours ago

Tl;dr:

Yes, it did. Here is some math showing that you shouldn’t care about that.

the_real_cher

9 hours ago

Is there a non vibe coded fork of rsync?

MYEUHD

an hour ago

There is openrsync, which the OpenBSD re-implementation of rsync

It's not a fork, but it's 8 years old, and is already shipped by default in OpenBSD and macOS.

logicprog

an hour ago

To quote Tridge:

> As to all the people saying “I’m going to package openrsync for platform XXX and we’ll use that!”. I find that rather amusing. If you do decide to go down that path I’d suggest you try the new rsync test suite on openrsync if you can stomach something that an AI has helped write. I tried it today and openrsync currently fails 85 of 98 tests, so I’m sure it won’t take you long to get it up to speed. You run it like this “./runtests.py — rsync-bin=../openrsync/openrsync — use-tcp”. Admittedly a lot of the failures are just features openrsync doesn’t have, but still, it’s not a great result.

SoftTalker

an hour ago

It's also not feature-equivalent to rsync. But if it meets your needs, it's an option.

MagicMoonlight

3 hours ago

Typical AI slop post. It’s pretty hilarious that he just added spaces before the emdashes and claims it’s human written.

If I’m hiring and I see this kind of slop, I ain’t hiring you.

logicprog

3 hours ago

I rewrote all the AI prose several hours ago with purely my own. I like em-dashes, and specifically use them with spaces as a habit. I don't know what to tell you.

Etheryte

3 hours ago

Emdashes don't really tell you much anything these days tbh. Many languages use them regularly and those people often bring the habit with them when they write in English — me included. Plus I would imagine every major model has tuned them way down at this point due to the backlash.

wookmaster

9 hours ago

Claude is just a tool ? The developers who merged that code and didn't properly test increased the bugs.

everdrive

9 hours ago

"Did cars increase traveling deaths?"

"Cars are just a tool. The drivers who piloted the vehicles and weren't careful enough [are responsible for the deaths.]"

roywiggins

9 hours ago

If something's a bad tool that misleads people into doing bad work, it would be good to know that.

Angostura

9 hours ago

This tool is claimed to be able to find and fix bugs.

runarberg

2 hours ago

Feels like something a bad (and potentially dangerous) tool would say.

ebiederm

8 hours ago

Please read the article.

The unsolicited security reports are the issue.

mwkaufma

2 hours ago

Smokescreen of highly-contingent analysis and appeals to authority over a premotivated-conclusion.

logicprog

2 hours ago

- Appeals to... what authority, exactly? My fucking wife? That's me a) being really proud I married such a baddie and b) explaining the process and where the ideas came from. Which people seemed to want. Damned if you do, damned if you don't, I guess.

- All analysis is contingent.

- How do you know the conclusion was premotivated, and does it matter if the analysis, which is attempting to be as objective and extremely reproducible as possible, holds up?

- The whole point is that there's no actual evidence for what you are claiming, so why does it being highly-contingent cause a problem for me, when that just further shows there's no evidence for what the anti-AI crowd is saying?

- Why do the anti-AI crowd get to state wide, absolute, objective claims with cherry-picked anecdotes as their only evidence, but the pro-AI crowd is not allowed to respond the same way, and when we then go out of our way to respond in a far more thorough, rigorous, and objective way than you ever did, that's just more evidence for our guilt? It's a Kafka trap. You can't win.

vintagedave

an hour ago

> My fucking wife? That's me a) being really proud I married such a baddie

Good for you. I really mean that. I think people are winding you up in this thread, but keep your cool, and I admire publicly crediting and being proud of your wife. That’s a healthy relationship. Good for you.

bakugo

an hour ago

Your analysis was so thorough, rigorous, and objective, that you couldn't be bothered to write it yourself.

Do you genuinely believe an article written by AI defending itself is going to convince anyone who wasn't already on your side? All you're doing is giving more fuel to the "anti-AI crowd" you hate so much.

logicprog

an hour ago

Okay, so you didn't respond to any of my rebuttals — like the double standard between anti-AI and pro-AI claims, one of which gets to make claims based on cherry-picked anecdotes, and the other which must produce rigorous studies — you're just going to insult me/my work. Cool.

> Your analysis was so thorough, rigorous, and objective, that you couldn't be bothered to write it yourself. Do you genuinely believe an article written by AI defending itself is going to convince anyone who wasn't already on your side?

Except that I did. I spend days comparing and manually deciding on metrics and methodology – I did not use the AI to decide what I would do or how I would do it, so it is not "the AI defending itself" — then refining things, adding more angles to analyze, and, as I literally say in the opening section, I rewrote all the prose in the entire document just to satisfy critics like you. That sounds like "could be bothered" to me. But people like you will never be satisfied.

Also, even if I hadn't done all that work, that wouldn't make it not rigorous (it clearly is) or objective (it is as objective as it can be with so little data). You're bikeshedding to avoid the point.

bakugo

32 minutes ago

> like the double standard between anti-AI and pro-AI claims, one of which gets to make claims based on cherry-picked anecdotes, and the other which must produce rigorous studies

This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecdotes with zero empirical evidence to back it up. It's way, way too late to claim hypocrisy here. As I stated under the original submission about this topic, irrational anti-AI behavior is usually just an equal and opposite reaction to irrational pro-AI behavior.

> I rewrote all the prose in the entire document just to satisfy critics like you.

And that doesn't help. If anything, editing the AI output to make it read less like blatant slop just comes off as deceptive, like you're trying to hide the fact that the analysis was AI generated. Looking at the commits, you were adding more AI generated text less than 2 hours ago[0] before quickly editing out one of the most blatantly sloppy sentences I've ever read[1].

Regardless, the final contents of the article are not the main issue. Even if we ignore the bias clearly on display there, the premise alone is enough to dismiss the entire thing as heavily biased and chasing a pre-determined conclusion - of course someone who is so dependent and trustful of AI that they decide such an analysis on the bugginess of AI code should itself be written by AI is going to steer the conclusion towards "actually AI code is good and you luddites are overreacting". The entire concept is so tone-deaf that failing to notice it or predict the criticism before publishing is enough to prove the bias.

[0] https://github.com/alexispurslane/rsync-analysis/commit/e029...

[1] https://github.com/alexispurslane/rsync-analysis/commit/740b...

logicprog

a minute ago

> This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecdotes with zero empirical evidence to back it up. It's way, way too late to claim hypocrisy here. As I stated under the original submission about this topic, irrational anti-AI behavior is usually just an equal and opposite reaction to irrational pro-AI behavior.

I'm talking about the double standard on the anti-AI side. I'm aware LinkedIn Lunatics and Steve Yegge are also being crazy. And it seems to me that even your response here is engaging in a bit of a double standard, or something akin to it, in that you think the irrational anti-AI behavior should be given a pass — and the conclusions perhaps even taken seriously — just because pro-AI people did it too.

> And that doesn't help. If anything, editing the AI output to make it read less like blatant slop just comes off as deceptive, like you're trying to hide the fact that the analysis was AI generated.

Okay, so, if I don't spend the time to write everything myself, that's bad because it's AI slop. If I do rewrite everything myself, then it's evidence of deceptiveness... despite being asked by multiple people to do that, and being extremely explicit about my methods and process and the commit history being (as you've shown), very public.

> Looking at the commits, you were adding more AI generated text less than 2 hours ago[0] before quickly editing out one of the most blatantly sloppy sentences I've ever read[1].

A small amount of the text being AI written, where it is in and around UI elements, doesn't seem like the knockdown you think it is; especially since the conclusions come directly from the data (again, templated in, so hallucinations won't happen) and the framing and much of the phrasing is explicitly prompted by me, because it's easier to let it relay my words when they're intertwined with a bunch of new added code. And if going through and reading what it writes well enough to make editorial decisions and eliminate slop isn't spending effort on making the thing good, why doesn't it count?

> Even if we ignore the bias clearly on display there, the premise alone is enough to dismiss the entire thing as heavily biased and chasing a pre-determined conclusion - of course someone who is so dependent and trustful of AI that they decide such an analysis on the bugginess of AI code should itself be written by AI is going to steer the conclusion towards "actually AI code is good and you luddites are overreacting".

That's not ignoring the bias, that's literally restating that you think the bias is there. But if you really think that my bias meaningfully "steered the results," then show me how that happened. Tell me how you would've proven the Claude releases were meaningfully worse, or unusual, at all, or how the methods I chose biased the data against that result, or literally anything except shifting the goalposts and using accusations of "bias" as a get-out-of-jail-free-card.

> The entire concept is so tone-deaf that failing to notice it or predict the criticism before publishing is enough to prove the bias.

And you're so committed to your preconceived notions that anything made with AI must be bad, wrong, or not worth your time, that you'll spend your entire time begging the question ("it's made with AI, therefore it's wrong") and shifting the goalposts instead of engaging meaningfully.

Also, I certainly predicted the criticism (in general, anyway, to the fact that it was made with AI; not the prose being AI) but I made it this way anyway, because if someone is so AI-blinded that they can't read and evaluate the actual metrics, methodology, and provide meaningful criticism to it, and instead can only see that it was made with AI, and they're so it doesn't matter.

Nothing you have said makes the analysis wrong. At this point, you're essentially just resorting to ad home