I wonder how robust these linters are. E.g. In C you might find some library function that allocates and abuse it to get dynamic memory.

At least in the USA, blame (or at least responsibility for damages) will often fall on the customer. We’ve allowed Caveat Emptor to become this awful fundamental basis of the economy. We just cross our fingers and hope that The Invisible Hand of the market stops companies from putting out defective products.

Depends. Back in the day, I worked on development tools for a commericial avionics flight management computer. The headcount for embedded hardware/software verification was greater than for development.

That was never true for software development in other domains I worked in i.e. Fortune 500 enterprise tools, compiler/design tools, networking, and semiconductor.

Isn't that why there's such investment into code verification agents in the cicd pipeline?

If the future is agentic, human review will be core bottleneck that needs to be streamlined.

And everything to a beat by Daft Punk.

This is all good stuff to verify of course, but why would you not continually add the useful agent-created tests and verifications into a deterministic suite to ensure consistency of coverage?

For instance:

>Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).

Ignores the places where it's always been important enough to have automated checks on many things.

> Why these are not becoming standard or even required practices is beyond me.

It sounds like you haven't actually tried to do what your proposing then.

> tries to use the software using desktop control

This is not a solved problem yet. I've tried a number of solutions and none of them were actually good enough for what you're proposing. Don't get me wrong, I'm not saying its an unsolvable problem. Only that it isn't sufficiently solved yet.

I agree. IN addition I think many businesses/industries are still not aware of the full capability of AI yet, that is why they still do things the old way. I have several friends who work as QA. When they describe the things they do daily, I tell myself quietly that AI agents can do all of them now.

I just don't understand the premise. If we're fully committed to agents, what does it mean to talk about a test being hard to maintain or requiring hardcore plumbing? "Hey Claude, can you please mock out Component X and write a test suite capturing behaviors A, B, C of component Y. Thanks!" When in the process of typing this would I go "ughh", and why would I prefer to receive a SCENARIOS.md as output rather than a test suite?

A lot of the risk is insider risk. A hacked credential, an annoyed engineer going to a competitor, etc.

Think embezzlement. Think someone with a political agenda who changes the homepage on the way out the door.

Having a single point of control is asking for trouble.

There's a reason verification and code review shouldn't be done by the person who wrote the code, and those reasons don't magically go away because a code generation tool did it instead.

Of course, since AI will be doing the writing and maintaining, there's really no reason for that any more. Let the AI do it all and wait for management to realize they're paying senior salaries to people that are little more than entering prompts, a job that really only requires someone to be able to read and write English.

This is mathematically challenging as your system gets more complicated.

If you give up on trying to reduce the number of permutations you need to test, you're gonna also (even if not intentionally) give up on testing them.

We knew this, of course. But everything old is new again.

what are the guardrails?

if there are many of them (e.g. unit tests), who generates those guardrails and who verifies their correctness?

if they are a few, then how can they cover vast product capability surface?

What "guardrails" specifically ensure that? I assume you have a more interesting idea than simply suggesting more tests.

“Make no mistake.”

[dead]

This LLM as compiler analogy comes up here and there and it’s not hard to poke a hole in it.

If I generate a piece of code with LLM, and verify it to be correct; can you trust the LLM output, with the same prompt, in a new session based on my verification of my code, with the same certainty that you’d trust LLVM output for the same source code in a similar environment, ran independently? If no, why?

That would be compiler writers, which seems unlikely.

kek