We didn't collectively decided, we've got this forced down our throats to apply a novel tool to any imaginable situation because the execs got antsy about being left behind.

A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.

That's it.

The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".

I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.

But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...

Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.

Agents are providing to employees the long overdue benefits limited liability companies long enjoyed: Gambling with upside for themselves and other peoples downsides.

I’ve never had respect for the industry as a whole, only individuals within. There has a been a serious lack of rigor and professionalism in software engineering for as long as I’ve been a part of it

I think it might be because we (or at least I) used to associate insecure actions with people, not computers. Computers should know better, right? Recently, I spotted that Opus 4.6 found config files for one of its tools and gave itself access to my whole filesystem. Similarly, Gemini CLI will rewrite itself if you let it.

It’s a nightmare… the problem is it’s far too easy for people to set these agents up - without understanding the security implications.

We’ve covered so many issues already on our blog (grith.ai)

The number of wasted hours spent talking about code quality and patterns has to be astronomical.

> cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter

I saw the sea change in 2008 when quality process got replaced with velocity and testing tasks. I've watched everything from Experian and health record data leaks to Windows 11 since that change. Software quality hasn't mattered for a long time.

The media isn’t helping. This wasn’t a “rogue AI”. It was a system that was given permission by a human operator.

We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.

The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.

People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.

My thinking is, this will increase the demand for backup and other resilience solutions.

> Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter

Is this new to people? I figured this out when I first entered the industry. The messages have never been particularly subtle.

How can you respect an industry that doesn't respect itself?

Turns out all of the frenzy of the ZIRP era is piddling compared to what happens when ZIRP is taken away.

I think it's batshit crazy. That's why I wrote yoloAI, so I could sandbox it up properly and control EXACTLY what comes out of that sandbox, diff style.

https://github.com/kstenerud/yoloai

I can't go back anymore. Going back to a non-sandboxed Claude feels like going back to a non-adblocked browser.

The whole agent ecosystem is a ridiculous shitshow. All of this because you need to ASAP find something believable to sell your overinflated, bullshit machine to the masses. Otherwise the bubble will burst.

Yeah, a nothingburger for clicks.

From Schlock Mercenary: "Oh, I love aiming. It's my very favorite thing to do before firing."

AI use without checking its output (at least at the moment) is firing without aiming. Sure, you can fire really fast. But who cares if you don't hit what you need to? The point wasn't to just shoot bullets, the point was to hit your target!

I mean, you might make a case that enough of them hit the target that shooting fast is a net win, and accept the occasional friendly fire incident. That might possibly be true. Or it might not. I'm not sure that everyone trying to run fast has really done the calculation, though.

But funny enough the person who was responsible for setting up the bot will likely face no repercussions. In fact they will probably be rewarded for transitioning their team's workflows to AI.

I mean, only if it leads to embarrassment right off the bat.

If there is a year or two between writing your security fuck up and it being discovered the likelihood of repercussions drops significantly.

A machine doesn’t need food, leisure time, or vacations. It doesn’t care.

It also doesn’t care.

If you have to do all that, then what's the point of the AI? I'm joking, but I'm afraid many others say the same thing 100% seriously

A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available.

If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?

> The fix for this common pattern is to reason about LLM outputs before making use of them.

That is politics. Not engineering.

Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.

If you have to check the AI output every single time, the AI is pointless. You can just check immediately.

However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.

If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.

It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer

When organizational incentives penalize NOT using AI and firing the bottom x% regularly then are you really surprised LLM outputs aren't being scrutinized?

It's already there for a dollar to the right data broker. Could probably pull your doctor visit info from last week (example).

Very soon, and at this point I'm not sure even that would cure the delusions of the few who practically worship LLMs

Why do we keep calling these things "agents" then? Or using the term "agentic"?

> AI is a tool with no agency

Claw AIs absolutely do have agency in the sense of being able to independently perform actions on their own, based on their "understanding" of a goal given by a "principal". I can't think of a better word than "agent" for that.

Then the human should write the code.

I hit back, clicked the link again, and it let me through

https://archive.is/A2hmz

Read TFA. It's not "Someone vibe coded too hard and leaked data"