mchusma
3 hours ago
There was surprisingly little information on how they actually do this, but we run our business with a large number of, what we call "AI employees" in addition to regular employees, and they act in interesting ways. We've been building out orchestration tools to handle this, and we'll probably do a write-up or blog post on it soon. May even open source some of it.
The preview is that the problem with most agents (and this includes frameworks like Grokbot and Openclaw and Hermes) is that for many of them, they're black boxes. They say they learn or improve, but it's a black box in what they do. Getting agents to reliably do things is hard, and getting agents to build out software tools to help themselves improve and do better over time is also hard.
Our approach at a high level is pretty simple: every single AI employee is a standalone GitHub repo that shares some characteristics, but we direct them to build as much software as possible to make their goal as easy and reliable to manage as possible. Then we have a shared communication layer for bots across the company to interact with humans and AI. We have decided to organize these like departments similar to the way you might hire out humans. I'm not 100% sure if that's the best approach, but I will say it's been easier for people to understand because they're more mentally easily able to traverse the bot org chart if it somewhat reflects a traditional business org chart.
Each of these AI employees has specific sets of goals and KPIs, instructions that they manage the business with manager bots. We have layers of management, which we actually have found helpful. We also run different bots with different models and harnesses, and some using different models and harnesses to check the work before anything can get done, along with lots and lots of testing.
Every single time, actions have a massive amount of tests based off of previous failures to prevent failures in the future. Sorry for rambling. I do think this is a very interesting space. I didn't see anything interesting in Pion that was public on this website, but I do anticipate that more companies will be "AI and software first," as in the substrate of the company is basically a software application powered by autonomous agents, with humans as a fallback.
StilesCrisis
an hour ago
Is this actually cost-effective versus hiring a few humans? Seems like a huge amount of tokens in use.
sevenzero
9 minutes ago
The whole premise of AI is to replace the human in the loop. People who go out of their way to set up this kind of stuff rather than just hiring a person dont even think about just hiring a person.
maxnevermind
10 minutes ago
What type of roles "AI employees" play, can it be any position in your company or you limit it to something specific? What is your goal, are you trying to find out if fully autonomous bots are more efficiently help to deliver projects than when people drive them or is it something else?
orourke
3 hours ago
This is fascinating. What is the split between human and AI labor? What kinds of tasks are the bots doing? How much autonomy do they have to make decisions (i.e. spending money, issuing refunds, touch cloud infra, etc)? I'd love to learn more about how you do this.
mchusma
2 hours ago
I hope to post something within a few weeks. But our philosophy is generally if it can be done deterministically with software (eg run payroll on autopilot via an api to gusto) then do that. If it can be done by an ai agent, do it with that but add as much software as possible to make it reliable at that thing. And the ai agents are all tuned to escalate to humans as needed. Then there is also just things that are completely human.
A typical example of something that is AI vs human is the AI most commonly operates like “managers”. For example, reviewing transcripts of every demo call, compiling results, figuring out insights, learnings that need to update our company docs, feedback to humans (who run the demos).
We are big fans of having AI agents “own” koi’s because now anytime we say “we really should be doing this” we try to set it up on the spot.
The “downside” here is that I do occasionally get busy, and if I’m the only one who can approve or unstick one of these bots, it just keeps harassing me until it gets done. This is a sign generally that I need to hire someone to own a set of bots.
jgilias
2 hours ago
At $DAYJOB we do something very similar shape-wise. I wonder - it sounds like you have a dedicated agent comms plane? In our case we found that the easiest and most straightforward was to just use our default company chat app directly for this. Because most of the context that the agent workers need to do work is there, but also, it’s just much easier for teams to conceptualise an agent colleague if it just hangs out in their channels.
What do you do here, and how’s it going?
mchusma
2 hours ago
We do have a control panel, but it’s in effect a server that has things in a database. All chats, tasks, assignments are all there. This felt easier to debug and manage versus putting it all in slack, although we considered it. I don’t think anything we are doing is particularly “fancy”, but basically one agent sends a message to another one. It saves the message in the db, then adds the message to that other bot same as any other user chat. We have an internal website where anyone in the company can see the bits they are authorized to see and can see all chats. These AI workers are single threaded, but we see that as more of a feature than a bug (minimize complexity). They all work their way through a shared task list, which is just another table in our remote server sqllite.
messh
17 minutes ago
how do you decide to start a new agent, and when to kill? also... do you use some works-tealing style task board, or otherwise how would the agents get new tasks.
jgilias
2 hours ago
What harnesses do you use? Ours is basically Claude in a box. There’s some complexity because of that, but the advantage is that it’s very flexible and people who have a bunch of Claude-shaped skills can just basically give those to an agent.
I’m thinking to take a deeper look at Pi. I’m really liking that project.
classified
an hour ago
> we direct them to build as much software as possible
That's not a cure-all. Sometimes not writing software is the better choice.
tomtomtom777
33 minutes ago
This is exactly the problem we're ignoring.
In the end, most software is a necessary evil. It solves a problem that shouldn't be there. In the end it's no different than healthcare or prisons. We don't need as much as possible. We need as little as possible. This automation isn't actually helping us.
avereveard
19 minutes ago
A tool aggregation layer made of code is a good way to save tokens
Composition is an issue only as long as one keep demanding tool calls in json. If tools are goal predicates in prolog, it's easier.
ianberdin
3 hours ago
What worries me most are questions like this: these autonomous AI employees or automations - it doesn't change the essence - consume a lot of LLM tokens. Please tell me, how do you pay for this? Do you use, for example, the Anthropic or OpenAI API directly, or do you connect, for example, a Codex subscription?
mchusma
2 hours ago
Our control panel is on a server but individual agents actually are run on anyone’s machine. This allows us to use the native harnesses including subscriptions. Yes it uses a lot more tokens, but i have have Claude $200, OpenAI $200, and SuperGrok Heavy $300 (or whatever it is called) that includes Cursor Ultra. My machine runs most of them, but some other team members have agents running on their machines using 1-2 $200/mo subs.
I would say this setup probably costs us about $1,000/month total. (I’m excluding traditional engineering use of LLMs from this number. This is the cost of all the “AI employees”.
One reason we did it this way was to use subs.
grvdrm
an hour ago
Do I understand correctly that by using your machine and Claude Code or something like it you are avoiding API pricing?
lnenad
2 hours ago
If you use a bajillion tokens your economical approach is to self host.
noir_lord
2 hours ago
There is a point where those two lines do cross but with them effectively subsidising token cost by burning debt, it's further away than it will be at some point.
That said I use local only models purely because I don't want to use remote models, never having to think about token costs is worth it and no one is training anything on my data either.
idiotsecant
2 hours ago
The first ASI will use a vast army of middle managers as it's neurons. We won't be fighting terminators, we'll be submitting TPS reports to skynet.
dofm
20 minutes ago
It can't be bargained with. It can't be reasoned with. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop... ever, until you go ahead and come in on Saturday.