hackernews client

Code mode yields a 99.2% cost reduction in our systems

60 pointsposted 6 hours ago

72 Comments

shireboy

4 hours ago

A maxim I’ve arrived at working w llm every day is “put deterministic things in code, non deterministic in llm”. I do wish the harnesses would be more helpful in this regard. For example I’ve seen tons of scheduled jobs that people wrote in Claude/copilot/etc that could easily have been scripts. They aren’t scripts because the author doesn’t know how to script and they stop at the point the llm does what they want. It isn’t hard to then tell the llm “make a script to do all of the tool calls this prompt needs and then pass the result back to the llm for this non-deterministic part”.

agentdev001

4 hours ago

The way I try to illustrate this to my peers, in the context of automating with llms, is to "do as much of the deterministic work as possible before and after involving an agent". Tbf this is largely a restatement of your comment;

Ie instead of

"hey agent, use the api to get X, then do unpredictable mutation Y to it, and email it to me",

it should be;

"Scheduled task runs code to get X then place it in workspace. Task then starts agent in workspace and is prompted to do mutation Y to it. Post-flight code then sends final product."

The difference here being that the pre/post flight stuff is called deterministically, rather than called by an llm.

firasd

4 hours ago

An absurd example is how harnesses like Codex, Claude code have no 'cut/paste this range of chars' affordance so if you want to move a block of code the LLM galaxy brain rewrites it in token space in the new file.

It ~can~ be done using standard bash utils but there just isn't a kind of standard pattern for 'move this text block via CLI tools' so the LLM and human both overlook it

agentdev001

4 hours ago

Anecdotally, when I see coding agents preform this action- I see them using bash. IMO less tools is better, if the agent has a shell- so not having a dedicated cut/paste tool is good.

jwally

4 hours ago

Preach!

Dedicated Infrastructure > automation > ad hoc scripts > non deterministic llms from a dependability perspective alone. Using an LLM to do something that prettier could have done is analogous to using a 3d printer to nail up a fence.

j45

4 hours ago

I’d expect the llm to write the script and run it.

The tooling can remain generic and token spending focused.

somnium_sn

4 hours ago

I am still a bit surprised that this is not more widely used. Normal inference + tool calls is effectively composition via serialized natural language, whereas Code Mode / Programmatic Tool Calling is using purpose built languages (aka programming languages) for composition and concurrency. It somewhat feels obvious to me that the latter is way more token and latency efficient.

asdfsa32

4 hours ago

In the world of AI hype, everything is to be learned again and we are going to see many many breakthroughs like this. You can see this shift already in the tone of conversation in HN for example.

The basics of computer engineering is now novel approaches. We are discovering why programming languages exist, starting with a worst version of Cobal running on a very expensive and probabilistic VM.

We will shoehorn every square, rectangular, and all forms of peg through the AI hole.

Diogenesian

4 hours ago

"You know, as impressive as these systems are, they sure are bafflingly stupid. I am wondering if perhaps there is more to human intelligence than technical mastery of human language."

Drew McDermott, 1976, "Artificial Intelligence Meets Natural Stupidity": https://dl.acm.org/doi/10.1145/1045339.1045340

  I wish to rail against a pervasive sloppiness in our thinking, the tendency to see in natural language a natural source of problems and solutions. Many researchers tend to talk as if an internal knowledge representation ought to be closely related to the "corresponding" English sentences; and that operations on the structure should resemble human conversation or "word problems". Because the fault here is a disregard for logic, it will be hard for my criticism to be logical and clear.

Note that 1976 had some cool AI, including theorem-provers and story-tellers (Knuth wrote a story-telling program, I think it's an exercise in TAoCP). Yet the underlying theory of intelligence was "intelligence is logical processing of symbolic information"; this is obviously inadequate, but the ideology of analytic philosophy is a helluva drug for a scientist. Likewise with the 21st century "intelligence is about correctly predicting things based on past information,"; this is obviously inadequate, but the ideology of big data technocracy is a helluva drug for a scientist.

cyanydeez

4 hours ago

to be fair, people are also discovering why vaccines, democracy and human decency are desirable traits; so it's not just programming in the age of LLMs.

There's clearly a paradox that happens when people have to face things they treat as essentially magic because the magic has wiped away the pain and suffering that borne them. So too will be another age when people have to relearn everything the LLMs do for us.