Heavily biased a kube admin but make kube clusters is easy now - its literally one command (aws eks create-cluster/gcp ... something something haven't done it in awhile after doing it multiple times a day in a past life) - I think its more familiarity with the tools that is the challenge - creating a vm is also easy if you know how, brew/apt/yum install is easy if you know how, setup.exe is easy etc)

If you are approaching anything looking like an enterprise environment, you are likely to have Kubernetes nearby. ko is a weird requirement. ax requires agent substrate, so it makes sense that it's needed. Agent Substrate also has experimental support in kagent, so it makes sense that it is growing.

I don't really like the oversubscription of agent pods though, as you can no longer trust the k8s pod identity as being from a singular workload. Haven't seen a solution to this for ax yet and it is a barrier to adoption for us.

Easy as in "Google cloud console interface" easy

Yes perhaps easy refers to when actually using AX as a platform for deploying and managing agents, not actually installing and managing AX itself. I think it's awesome they targeted kubernetes but it's not easy from a human perspective. However with documentation and an AI assistant working with Kubernetes is so nice. I have worked with all the big clouds and Kubernetes a lot. But with AI it's so much faster and nicer to work with Kubernetes. It's so easy to debug and find issues in Kubernetes since the models know's Kubernetes so well. It quickly find issues and just fixes them. With the cloud deployment and changes are usually pretty slow and also it's sometimes tricky to debug if ex enough info isn't exposed. With Kubernetes every little atom is available for inspection by the AI.

As with everything in software engineering, any average dev can fire up a few containers with Docker, but running anything in a real environment takes a whole team of people who actually understand the platforms involved in order to deliver a fully functional service, hopefully via properly designed code (TF) for ease of reusability. Most devs simply don't care to think about that part, and then throw it over the fence for "someone else" to deal with. Just as long as they can say "DONE!" (I created a thing!), that's all that matters.

It is easier in that if you have infrastructure already, it's trivial to add this on top. The primitives look also very familiar.

Classic AI solution, you do the hardwork so that you can chat to an agent to do the easy part.

I agree, also the problem is the substrate is still in beta. So this is a beta on another beta: it seems not fully mature.

I'd prefer a K8s Operator or plugin (like Istio) to get all the pack.

Obligatory https://youtu.be/3t6L-FlfeaI

Google operates at such a scale with a wide surface area of serious production considerations that even “ergonomic” solutions internally feel extremely heavyweight externally.

Source: I’m an Xoogler

Easier from enterprise perspective.

I don't think there is really any convergence going on. The agentic ecosystem is continuing to multiply on a daily basis and everyone and their grandma has written a new agent framework--people are stepping over each other to get these new projects out the door.

That said, I think Google's ADK ecosystem and this new AX platform is promising--I would expect Google to maintain this and other tooling around this for years to come.

To the Googlers out there: is Google using this at any capacity for internal projects?

I'm working on something in the "cloud VMs for agents" space[1], so I have some battle scars and opinions!

IMO, you want the flexibility to create either: (a) permanent devbox VMs, and (b) per-task VMs

Agent sandbox platforms tend to be tuned for the latter, which sometimes involves VMM hackery for fast boot, snapshotting VM filesystem and RAM, etc.

Some workflows are a lot simpler if the multiple agents share a VM. These are workflows where agents must share state. A simple one we have: making related changes in our public OSS repo and our private repo, and then testing the change.

And other times you want to split up the tasks onto isolated VMs so they don't interfere with each other (ie run two dev servers without database or port collisions).

I tweeted a bit about this (https://x.com/dbmikus/status/2099264325231771878) and had a little debate with folks about ephemeral vs persistent VMs for agents

[1]: https://github.com/gofixpoint/amika

Basically everyone has a sandbox of some sort to run agents inside. Everyone has a registry of some sort for tools. Everyone has a way of running agents inside a sandbox and giving it some tools.

Now the stuff people are coming up with is: how do you do authorization in this model? do you need a full sandbox all the time or can it be a workflow? how do you specify an agent is it a prompt or does it have some kind of control flow structure? How do you coordinate among many running agents?

I would say thats where we are now is there’s loads of people all solving the same problems a bit like when CoreOS, Kube etc. were all competing.

From what I've seen, the vast majority of agent sandboxes with funding aren't for developers to use when coding, but for production applications that want to have LLMs do work. It's just a different model - APIs are better than a great terminal experience for a coding harness.

I've been working on https://lullabot.github.io/sandbar/latest/ which works with Proxmox for VMs (and lima for locals or regular linux hosts over ssh). There's a diagram in https://lullabot.github.io/sandbar/latest/why/#recommended-w... with what we're currently recommending. Though, after some feedback, I'm in the process of integrating a colleague's web-based review tool as it turns out many preferred fully reviewing locally instead of using draft PRs.

It's got some opinions in terms of default tools for our team and industry so it may not fit yours. Forgive some of the AI-isms in the docs, I want to get the UX and feature set to a solid place before doing a full review.

"Is giving the agent a temporary scratchbox really that valuable?"

Yes, but, wrong layer here. Giving the agent a computer use (a la bash) is what folks are after. A temporary sandbox with lots of control knobs and security bits is how you do that in (as you noted) an enterprise.

This sort of works.

The problem is that you'll end up wanting to run 2 or 3 (or 20, 100, 10,000) agents at once and that gets very hard with a single VM.

There's also an argument that you should be using a separate sandbox for each code operation a LLM performs (or at least each set of related operations). That's even harder to do with conventional VMs.

I have no idea, I just live in my terminal at this point, on linux. Don't use Visual Studio or Jetbrains products anymore at all. I use visual studio code to view the occasional diff and run sql queries. Other than that, zero desktop apps, just terminal & cli tooling. Its great! I have about 20 terminals open at any given time though. I have no idea how some of my colleagues stay productive as they are messing around with all these workflow tools, desktop apps etc.

For my workflow I need both, a permanent-ish VM and emphemeral sandboxes. Both have their place and pros and cons.

VMs are better for personal assistant work, GUI clicktesting, investigating bugs in your personal dogfooding dev instance and anything you haven‘t yet made repeatable and fast to set up.

Sandboxes are better when you need resource isolation or security and have a graph of tasks to work through. My agents often starve each other on one VM, so if they don‘t need any of the above it‘s just easier to isolate them.

Everyone is working in this area, including me [0], but either option really isn‘t that convenient to use yet. It‘s a bit of a „isn‘t Dropbox just FTP on a VM“ moment right now.

[0] https://github.com/madeinorbit/podium

> I've been still just like, making VM's with proxmox, then putting my agent in the machine and letting it run free

I outgrew this when I wanted to bring different sets of skills and templates to different machines, wanted to be able to share a small number of credentials, different agents in different machines, different egress rules etc. I wrote https://github.com/pjlsergeant/byre which gives you a TUI and some machinery for doing this easily on top of Docker or Podman.

My advice: stop your fomo. Just get useful work done.

The goal is "ticket in JIRA -> solution in production" without human intervention. Right now agentic frameworks are multiplying to bring us closer to that solution like JavaScript frameworks did ~10y ago. We still haven't uncovered the "React" of this space yet, the one that business decides is good enough to standardize on.

> What's wrong with that?

Nothing at all. You'll know when you've outgrown it.

> what the general workflow is now that people are converging to?

Graph-based workflows where agents pick up work as it becomes available, structured output, while you manage the work queue and outcomes. Maybe? IDK really, it's all moving quite fast.

I wrote a little program to spin up isolated rootless workspaces: https://github.com/jgillich/tpd

Works pretty well for me but I haven't put any effort into promoting it

An even simpler "sandbox" is to create a separate user and home dir for your LLM.

The general workflow I have seen for non-technical people building software is something like:

- Do multiple tasks in the same context window / session, conflate different changes into the same prompt

- Repo mixed with old markdown files from previous tasks, excel and word docs and 300 playwright screenshots

- 5 tools all calling each other, test and deployment scripts are all markdown skills

Personally I prefer a ticketing system and isolated work trees

Personally I think microVM are the future but nothing wrong with a solid proxmox setup.

Probably we'll converge on a virtualised IO / Storage layer running microVMs beneath for isolation and security. Keep the network and storage layer separate for compatibility running a variety of stuff and a second security boundary.

Compared to enterprise yours is missing egress control and secrets management, if you make the isolation watertight you cripple the agent's performance, and then the careful game of whack-a-mole begins when you stand up local package mirrors, authentication brokers etc. etc.

I think you might be confusing "what people should be doing" from "what some people think they can make money from". imho the whole "agent sandboxing" thing is vastly premature and un-thought-out.

From their own example "Setting up a Python 3 environment" is best it can do.

There's no convergence, but there sure as hell is a lot of pseudo-scientific voodoo and overengineering going on.

[dead]

[dead]

I'm stuck on Windows, so oh-my-pi has been really nice. The others I've tried such as kilo do alright but tool calling can mess up a bit.

Only complaint is that connecting the agent harness to my local model took more work getting configured right than I'd like, but that's been true of most harnesses I've tried as well. Most assume you're using a cloud model and local model configuration is a bit of an afterthought.

What are you using Jules for? I want to like it but it fails too often. If I could use Gemini 3.8 I would be happy but 3.6 rarely succeeds.

Since local models are largest constrained by context window you want to have a tiny system prompt. I know Hax: https://github.com/OleksandrChekhovskyi/hax was designed with local models in mind, but I haven't used it.

Give https://juggler.studio a shot if you want a nice desktop UX with all the extensibility of things like Pi

Deepseek Harness is great, at least with deepseek.

Does agy has auto mode?

goose has now native support for local models

pi.

Always Pi.

It is on Google's github https://github.com/google/ax and the title comes from there.

It's got a formal announcement on the blog. 4 months ago.

https://cloud.google.com/blog/products/ai-machine-learning/a...

I have no insider knowledge but https://x.com/rakyll is working on it and she is tweeting about it and I got the impression there is a quite a team behind it. It looks like an effort in GCP.

This /looks/ at least more official. Most unofficial Google projects have a disclaimer in the repo.

>The reality with releases like this is that I'm 90% sure most Google bigwigs have never heard of it

Google has around 200,000 employees. They probably haven't heard of most things Google releases.

the repo description https://github.com/google/ax is "Google's open agentic orchestrator"

Yeah especially with their tendency to depreciate stuff they’ve grown tired of

Yeah, anyone should think of this as an experimental side project that may be forgotten in 15min and make careful decisions about using them in production environments.

We have used a few of Google's (smaller) open source projects, and in the last 2-3 years most of them are getting fewer updates if any updates at all. Some became very bad tech debt and we had to spend a lot of time migrating them.

Of course, that is the nature of open source projects (written in the license terms), and there is nothing to complain. But it's important to point out these days Google's open source project are not any more trustworthy than a one man's project in terms of support and maintainability. Personally I would stay away from them as far as possible. Especially if you look at what happened to Android, Gemini CLI etc.

(To be honest, even if it were officially supported by Google, that barely means anything. https://killedbygoogle.com/)

https://github.com/google/ax/blob/main/docs/concepts.md

>A Model is not a model. It is a named model configuration: ...

Remember kids, a model is not a model.

> Disclaimer: I'm one of the co-creators of this project and working on Kubernetes so my opinions may be biased.

We met quite a large number of customers in the last few months, and several teams inside Google, who are asking for a stack that runs on any cluster.

There are several reasons for a layer like this.

(a) Today, it's extremely hard to deliver large agentic applications to someone else's compute, and it kills adoption for products that need to run closer on customer data plane, e.g. large scale code scanning product.

(b) It's tedious to build systems that set up the environment, deal with networking policies, provide integrations for session storage, memory curation, skills retrieval and mounting, etc while all you want to do is to focus on building applications.

(c) People want a truly transparent stack for compliance/auditing, and all other non functional capabilities most people don't usually think about until they have to.

If there's a criticism here, it's that they don't mention k8s right up front. It's completely opaque what this tool is. "Orchestration" can mean literally anything.

The web page presents ax as a typical developer tool, but it's actually not for developers.

it feels like the kind of interface a devops engineer who hates AI would design

Fully agreed. Google just can't stop losing.

I have 6 and ended up needing to use my gaming PC for a build server.

I think you could get by with 1 computer, but it’ll have to have a pretty decent machine.

Between agents running tests, CI, docker image builds, an average $400 mini PC won’t cut it.

Don’t forget also many older mini PCs don’t support KVM. Some newer ones don’t support AVX/ mongodb.

It’s not so easy to buy any old hardware sadly.

I think it will be, but I don't think you need a standalone machine! If you run things inside a VM, you can get safety and control over access and networks

A standalone machine is nice if you need more compute resources or if you want an always-on machine you can connect to from your laptop, phone, etc.

It doesn't look like Google's AX is quite the plug-and-play fit for running agents on a computer you own, since it requires setting up a K8S cluster, etc.

I think what's needed is something like a zero-setup combo of Tailscale and Firecracker

I'm trying to work towards that with my startup (https://github.com/gofixpoint/amika) but the bring-your-own-computer part doesn't work quite yet.

[flagged]

> you are opting in to a pretty huge stack that is just Agent Substrate, that is their runners, their harness, their substrate

The website makes me think the contrary: It is described as “low opinion” and explicitly mentions that the running tasks don’t even have to be AI agents. Can you explain in what ways you’re more locked in than the website suggests?

Scion at the same time talks much more about concrete agents, giving me the opposite initial impression.

thanks for the kind comments (lead Scion architect here), Scion is still very much meant for teams of people to interact with teams of agents. it is not just about scheduling agents on infrastructure. it’s been getting some new UX improvements: https://youtu.be/1WYsoSXt9QQ

I assume they expect agents will be writing most of those

It depends on where they decide to go with it, I guess. Kubernetes, Go, Tensorflow, Chromium, gRPC are some examples that obviously went incredibly well.

Well, they're not above forking their own project to patch security holes and never upstreaming the fixes.

https://grapheneos.social/@GrapheneOS/117282080803799576

> Google should not be gatekeeping security patches to the standard Android platform code from Android OEMs but that's what they've started doing.

Maintain it briefly then slowly let it die.

Everything is always better with more YAML, are you perhaps new to this industry?

Next we also need an instruction style guide and CoC. It's important to treat your agents with respect. I almost forgot, the YAML template meta-language to YAML the YAML. Then we will need a foundation employing 12 FTEs to maintain it all and of course to run the certification process. You are certified, right? Statistics show a 10x increased chance of an agent going rogue and hacking competitors if it has been mistreated or been run in an unvalidated sandbox. It goes without saying the sandbox certification process is separate and must be repeated yearly by a trusted third party auditing company.

The best part of all of this is the most if not all people have no idea what the hell is going on when every day a new paradigm/tooling/harness emerges. It's hard to keep up. On the plus side, it's a great equalizer.

Offload the work of an LLM agent to a box in the cloud, so it doesn’t run on your laptop. This has security benefits (no access to your laptop’s files) and you can scale it up (run a lot of agents at the same time).

Then put a “sandbox” around these agents, that word has many meanings. In this case they fence the network traffic, so likely some kind of allowlist for network requests so that the agent doesn’t exfil crap to random websites. They also limit the resource limits of the sandbox, so that is beneficial to the cost of running these agents.

k8s, but for agents

must look cool for people who want to solve every problem with k8s

it starts with interesting misnomers like "Task" which is not a work item but a sandbox.

"billions of tasks" is a "solution" to problem nobody has (maybe some RL labs? but they solve it other way and with orders of magnitude better optimizations).

freezes design too early – unless they'll actually focus on developing it and make tons of breaking changes it looks shit.

shared state in the same workspace, identity, authority, etc – stuff like that needs to be solved

Same. I don't get it.

Not sure if this is connected, but I've found Gemini pretty hopeless within its own notebooks. I've been doing some job applications, and added cv and docs into a notebook. I then create a new chat to say "here is a job description, help me write a cover letter" or some such.

After about 3 messages in any given chat, a follow-up to "rewrite that with a more friendly tone" will result in a letter for a completely different job from another chat within the notebook.

On your Sheets + Gemini integration point, I've genuinely tried to give the Gemini integration into Google Docs & Google Sheets a chance. It is so incompetent that it is fully useless to me. I have not gotten a single correct solution each time I tried to use it, even something I consider table stakes. I often write my work reports in Vim in Markdown format, but they need to go to the corporate Google space. No matter how hard I tried, no matter how many prompts I have, it was completely unable to manage the command to "convert the Markdown format markers into native Google Docs markers". And I want to note, this was 2 pages of extremely simple Markdown with no "advanced" patterns, like tables or quotes, I think all I used was heading-marks, bolding, italicizing, and code blocks. This is something I would expect even GPT 3.5 to succeed in, and even more so Luna, but somehow it destroyed the formatting throughout half the document. This leads me to believe that they apply the absolute cheapest model they have there, or they have the model a harness which can barely be considered working. I found it absurd when I found out that they suddenly made this Gemini integration an additional paid plan recently, there's absolutely no way I can consider that in good faith.

This is an Apache 2.0 open source project

Making Kubernetes a centre of everything. This one, they won’t sunset, because it helps selling GCP services.

[dead]

I wonder if Google stole this code too like they did with minitap

you can run it yourself, it's open source, you can use any harness (req. custom image), you can use any token vendor (config)

> Is the throughput of these objects too high for etcd?

Yes (perhaps not etcd, but the combination of kube-apiserver and etcd)

Substrate does not use CRDs for anything on the hot path of actor scheduling or resumption.

IIUC agent-sandbox and agent-substrate (the one ax builds upon) are similar. Agent-sandbox is more k8s-native, while agent-substrate is less so.

Personally I think that this kind of workload is better off not being tied too much into kubernetes. I've worked with crossplane and other controller who put a lot of load on the k8s-apiserver and etcd and can easily slow the whole machinery down / grind them to a halt.

btw, agent-substrate is in the process of being moved to CNCF: https://github.com/cncf/sandbox/issues/523

Not sure about billions, but companies doing evals or RL or training will create really big bursty agent workloads. I think they are the best fit for AX, as opposed to individual dev teams building software, etc.

billions? who is running BILLIONS of agents?

tens, hundreds, maybe a couple thousand at a time? absolutely.

Google already has gVisor running in Kubernetes as a product (GKE Sandbox), which provides the security guarantees necessary for secure sandboxes (regular k8s isn't great in this respect). They also have pod snapshots running at scale (which run on gVisor), so you can spin up process(es) and snapshot the memory and fs of a pod at a point in time, ship it to a blob in GCS, and then rehydrate those snapshots very quickly (or fork into new instances), which allows for the fast/cheap startup and suspend times and the instant scaling they advertise here. One of these snapshots can be created in one cluster and spun up in another.

Not sure if this is an extension of tech they already have had in their systems, but I've experimenting with it to build my own orchestrator and it's been a pretty neat set of tools and abstractions so far.

Future of platforms is operators in k8s to abstract the developer need to the underlying systems. On local it maps to kvm, on gke it maps to their stuff, on AWS to RDS. It's "interfaces" on a platform level so devs can just ask for a thing.

Overall I agree though, this is a bit of an abuse of that concept.

EDIT: I'm sure op is familiar with this workflow but I'm being overly verbose to clarify what I think they mean and my thoughts.

I can launch Astra to deploy changes to my homelab via creating Forgejo issues from my phone. That kind of system is pretty hard to set up without kubernetes.

Even if you confine yourself to a dev workstation, having 5 agents concurrently building testing deploying code makes your computer loud and/or hot.

Agree. Probably an unpopular opinion, but I strongly dislike YAML.

EDIT: if I HAD to use YAML, I'd prefer KYAML: https://dev.to/mechcloud_academy/goodbye-yaml-hell-meet-kyam...

The OP is not really a workflow manager, it’s a workspace manager that facilitates creating controlled environments where your skills can run. Everything you said is compatible with (and complementary of) the OP project.

With that said, I’ll somewhat disagree with you. I’ve been down the path you’re talking about and while it is incredibly flexible and powerful, it became too difficult to maintain, and too inconsistent between workflow runs, and a pretty hefty waste of tokens to use AI on things that could instead be handled by deterministic scripts. I ended up creating an orchestrator for myself that uses skills as the primary way to tell agents how to execute a step in a workflow, but also directly orchestrates running scripts and managing state in a deterministic way rather than leaving it all up to agents.

How does your criticism relate to the specifics of what OP posted? https://github.com/google/ax/blob/main/docs/concepts.md#work... This says it has skill registries.

You really, really need different skills depending on the model.

If you are using Qwen 27B you need very prescriptive skills.

If you are using Astra you usually want very minimal skills (because it will follow them but be unnecessarily constrained in some contexts)

If you are using Fable then it depends - it will take the skills as general guidelines but ignore them a lot more than Astra does. Sometimes this is good, sometimes not at all.

Dropbox comment

Didn't you see the example on the website? It can set-up a Python 3 environment. Duh!

Restoring memory is useful if, when you resume an agent VM, you want the apps to be in the exact state when they were suspended.

But for most things, I find resuming with memory is more trouble than it's worth. If you always resume from memory, you lose the ability to control the state of a VM. It's much easier to define which services should run than to define which active RAM state should be purged

Similar to why "did you try turning it on and off again?" is good for system reliability.

> I don't get why they to restore the RAM of the agent env. Maybe to fully optimize resumption. Idk, I don't have that much RAM in my homelab, my agents use a ton, testing stuff in Chromium making screenshots for me. I can't keep RAM for 100 workspaces from the past 24 hours in RAM.

This is going to be decomposed. I believe the plan is to offer resumption with disk state only as well (and the RAM snapshots will need to be discarded from time to time, if you update the underlying code of the agent, or switch CPU types).

By the way, the RAM snapshots are not kept in RAM, they are serialized to disk, or uploaded to object storage.

For microVM, smolvm is quite impressive.

For further isolation, I like to use nono inside a smolvm instance.

For microvms, take a look at https://smolmachines.com

what about msb? https://github.com/superradcompany/microsandbox

[dead]

what are your points of comparison ?

[dead]

I believe it's a stylized lo-fi rendition of an axolotl.

It's most likely an axolotl.

Axolotl

K8s jobs don’t run in a secure runtime. K8s jobs don’t give you dynamic zero trust permissions scopes. Restoring a harness in 500ms is really fast, much faster than naively creating a new job downloading session and ‘codex resume’ etc.

https://developers.googleblog.com/why-go-is-an-ideal-languag...

Ah yes, k8s8n.

No to confuse with Ax, the DSPy inspired agent framework

https://axllm.dev/