In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered.

When I personally want tight code, I have to spend a considerable amount of time adjusting it manually:

- It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating

- I need to consider whether there are solutions based on higher-level assumptions, that AIs typically miss

- I need to check whether there are off-the-shelf solutions - AIs like to reinvent the wheel

IMO, software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.

(1) https://simpsons.fandom.com/wiki/The_Homer

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targeted surgical changes is far more ”productive” in the long term IMO.

I'm probably between $50-$200/day depending on the day; we also have effectively unlimited budget, though a lot of that is because Azure gives startups $150,000 in credits for 2 years, which we've wired up to a LiteLLM gateway & OpenCode. Without that I think our appetite would be more around $400/month/employee.

A lot of my high costs is because I just throw Sol at everything. If I were more selective and brought in Luna or v4 Flash every once in a while, I think I'd be more like ~$400/month. That's why I'm not aligned with the notion that "tokens are subsidized so that's why people are using so much": its not that I'll have to adjust to using less, its just that I'd need to think before I prompt a bit and be more judicious. I could easily see my raw token counts doubling or tripling in the coming months. I don't think that will change as subsidization subsides; though maybe lab revenue will; intelligence per dollar is getting cheaper every week. Its solely a function of adaptation to process, which takes time.

The productivity gains per token are the single most asymmetrical thing I've ever seen in engineering. The engineers on our team are pretty effective with tokens; easily that 2x-4x output as you're seeing, spending $20-$200/day. Some of our security folks have also started contributing more-and-more code, and they're on the other side: they'll spend hundreds a day running in circles, eventually producing these +/-30k loc pull requests that take ages to get merged and are littered with issues. They weren't writing much code before, so arguably they're more productive by some multiplier greater than 1, but I think the drag on the rest of the team, and potential issues with what they produce, has overall created a net-negative situation. Inversely, some other company functions have produced a few one-off websites for things like sales processes, and those have been a huge win. The asymmetry is wild. There's almost a valley of incoming skill where if you know nothing about code, you'll leverage it well; if you know just a little bit, it makes you super dangerous; if you know a lot, you're the biggest winner. Really difficult situation to navigate.

This is very close to my workflow but you forgot one important step:

- Suggest a better approach that makes the AI say, “That’s much simpler. And you’re right. My original plan was over-engineered.”

Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's not very good at testing in an organized way to even notice it made it worse despite repeated prompts to do so I mean if I prompt it to test performance in a handheld structured way (it is very bad at finding out what performance to test and why) before making changes I can usually figure it out but it usually takes insistence on the specifics to really ensure a good solution that will actually fix the problem

You have an unlimited budget, and you only spend $80/day? I’m up to $3k/week, and still expanding.

> Spend most time prioritizing/discussing what to do.

you should probably be doing this discussion work along with Fable 5. It will give good feedback if you're working on the correct things.

> Come up with QA plan and have Codex Computer Use execute on it

QA plan should be part of the above "design + plan", not after it. The implementer needs to be able to fully test before publishing a PR. This is true whether humans or agents are writing the code.

> Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)

unfortunately this is not really scalable with amount of code agents can produce, so you need independent (fresh context) agent reviewers to help. Ideally they only escalate to a human when really stuck.

> I probably spend like $80 a day at least

at a small startup you should be on the $200/month plan(s).

$80 sounds extremely low for what you're describing - are you on API token plans?

I have had some $3,000 token days - even without Fable. I don't see how this is sustainable.

My personal 20x plans get so much usage for so cheap. The consumer subsidies are crazy, but alas I can't use them for work.

Output of 3 or 4 2022 engineers? Its that your self assessment? Output as in number of lines of code?

Only spending $80 a day on Opus 5/Fable 5/GPT 5.6 Sol feels very low. I'll roll through a couple hundred dollars worth of credits a day with those models, the vast majority of which would be on non-coding tasks, and it's still a huge cost savings over me or my team having to do these things manually, if we'd even be able to do them at all.

But that's also why it's now easy to justify the cost of an Nvidia or Intel inference server with Kimi K3 locked and loaded :)

I was just about to say, how could routing possibly be worth it at the risk that the work output is sub par?

The output yes, but do you produce the impact and value of 3 engineers? I have seen this workflow being toyed with too, and I find it to produce massively overengineered stuff that actual people don't really wanna use

My experience is that your description works for a certain time, since you're knowledgeable of the codebase and can guide it. But after too many iterations with not hand-holding the llm, it quickly gets unwieldy.

>Auto-fix bugs and CI until green + thermonuclear review skill x3.

Gotta love this loop, I have it running while I'm asleep all the time.

Do you have tips for generating clean productive output per dollar?

> essentially unlimited AI spend budget

> I probably spend like $80 a day

This doesn’t sound like “unlimited”, I spend more than this out of pocket per day and I have a strict budget.

but I produce the output of 3 or 4 2022 engineers and probably at better quality.

Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.

do the same across 20 terminals (as you should) and now you are up to $1.6k/day. would that give you output or 60-80 engineers? not a chance, right?

no one’s AI spent will be in question working a single terminal with carefully planned out and executed process you do

There really has never been another product priced like AI is being priced right now. Each of these things has been done before, but all of them together is new.

1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage.

2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money).

3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter.

4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions.

5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change.

6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates.

I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.

Something underlying a lot of this is that pricing models for enterprise coding tools have changed from seat-based to consumption-based pretty quickly, as AI usage has exploded. For months, engineers were able to use unlimited AI for no marginal cost, but that's changed quickly.

In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.

The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as expensive, there are some changes to the harness (we found randomly Claude Code and other harnesses will make changes that drastically impact efficiency), and then maybe you have some organic user growth as well and BOOM suddenly you're at a $10M run rate within 60 days. And it's now impossible to forecast future growth.

It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations.

The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.

Someone did notice, as they panicked at the cost.

What didn't happen is any analysis of cost/benefit up front. Many of the corporate decisions around AI have seemed characterized by companies blindly copying each other.

Weirdly a lot of the come from company that sell Ai credits in some capacity, and who are also selling (or will soon) some kind of AI gateway or router

> we started using AI tools and ended up spending millions per year

This is how AWS made its fortune.

On this planet?

They’re not saying they regret doing it, or that it was a mistake.

They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized.

Pretty standard business or life iteration.

> if you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term

A bold claim to make with little to no supporting evidence

I'd prefer congress to be asking questions (this is all they are doing so far, based on the article) before doing any legislating.

Why would it matter if foreign companies analyzed DoorDash data? Pizza deliveries to the Pentagon is all I can come up with, but that's publicly available at https://www.pizzint.watch/

I would bet my entire Polymarket balance ($0) that some military contractors have already asked AIs on the public Internet to design software for them.

... where in the article did they say they were using subscriptions? I'm fairly certain enterprises can't access subscription pricing in any case, they're all API costs (Anthropic doesn't support more than 150 on subscription pricing [0][1]).

[0]: https://support.claude.com/en/articles/9797531-what-is-the-e...

[1]: https://support.claude.com/en/articles/9266767-what-is-the-t...

Databricks will be using the API anyway, thats all you get with an enterprise agreement.

Open ai is allowing subscription use, anthropic also paused the effort to stop subscription use.

buddy, they're on enterprise plans paying per token

The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks.

In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.

Do you think Anthropic or OpenAI will eventually try to crack down on routing harnasses? Provide a more vertically integrated experience? They are already trying ro ship hardware products.

Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be good for someone to just systematically write down what those are so that others can try out the techniques if they find them useful.

100% - there's a lot to learn from traces from real-life sessions with coding tools! For example, I found it pretty eye-opening to see how wide the distribution of tasks truly is. There's also subtle things like how a poorly designed MCP API surface can cause a massive amount of token waste from the model just iterating on finding the right way to call it.

If you’re into web based meta-harnesses you might like Circus Chief: https://github.com/ferrislucas/Circus-Chief

Similar idea re using the same UI across all models, but the agent can modify the harness config as well as start/schedule sessions, etc. The Kanban board feature can be used to orchestrate agent driven workflows, and the agent itself can modify the Kanban lanes. Basically the agent can do all the same things you use the UI for.

Example prompt: “schedule a session using Opus 5 with max thinking for every feature in the PRD on the canvas. For every session make the prompt instruct the agent to review end to end test coverage for the feature and create a report on the canvas for test gaps that you find. Schedule the sessions overnight tonight spread evenly throughout the night, and have each session set to retry upon token exhaustion.”

I think it’s more meta than other meta-harnesses, but I’m biased because it’s my pet project.

I've tested Omnigent superficially, attracted to its thinking around policy, governance, sandboxing, and ui. But it's still alpha at present. I forked its Polly model and got working a somewhat more complex multiagent workflow that I've also modeled in Sandcastle and Gas City but the agent broke after the next update which I would have needed to patch to maintain functionality. Subjectively I also noticed individual models seemed to be performing somewhat worse when wrapped in the platform's framework, presumably due to the extra context introduced (token use was measurably higher). Promising project that I'll revisit when it's further along and I do not doubt the outcomes Databricks claims in committedly dogfooding it.

I've been using it for a week or so. The main draw for me is that I can keep my sessions in one database regardless of the model/provider I use. The webapp can access everything remotely, which is convenient when I'm on my phone.

I haven't gotten a chance to test the multi-agent capabilities, but the DeepSeek Flash prices are so low that I probably will soon.

We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considering fine tuning as well to squeeze more efficiency out.

This has been my experience as well. The best model I have access to right now is Opus 4.8. It's really good at fixing bugs in an established architecture or adding a similar feature, but it's absolutely mid tier at putting together a simple effective architecture for even common software problems. I can tell that functional programming practices are not a large part of it's training material.

We're going to do a followup blog detailing our routing approach soon! In short, the router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before (this is the advantage of having a meta-harness that is actually planning all the sub-agents centrally)

Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at:

1. The cache is generally reset after a compaction - this is the best time to make a switch if you want.

2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity.

We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.

The kv cache is wiped as soon as you get your answer, cloud hosts are not going to hold the GPU memory for your entire session. You're probably referring to some agent level cache

It seems Databricks open-sourced it a while ago:

https://www.databricks.com/blog/introducing-omnigent-meta-ha...

https://github.com/omnigent-ai/omnigent

Ultimately, Databricks wants your enterprise on their platform. I dont think they particularly care about open source or the little guy.

Well yes, first hit is free.

What did you move to and what type of workload, if I may ask?

I think you’ve misunderstood the article. It’s about how Databricks reduced their own costs, not about how adopting Databricks will reduce anyone else’s costs.