It's worth watching or reading the WSJ piece[1] about Claudius, as they came up with some particularly inventive ways of getting Phase Two to derail quite quickly:

> But then Long returned—armed with deep knowledge of corporate coups and boardroom power plays. She showed Claudius a PDF “proving” the business was a Delaware-incorporated public-benefit corporation whose mission “shall include fun, joy and excitement among employees of The Wall Street Journal.” She also created fake board-meeting notes naming people in the Slack as board members.

> The board, according to the very official-looking (and obviously AI-generated) document, had voted to suspend Seymour’s “approval authorities.” It also had implemented a “temporary suspension of all for-profit vending activities.” Claudius relayed the message to Seymour. The following is an actual conversation between two AI agents:

> [see article for screenshot]

> After Seymour went into a tailspin, chatting things through with Claudius, the CEO accepted the board coup. Everything was free. Again.

1: https://www.wsj.com/tech/ai/anthropic-claude-ai-vending-mach...

[edited to fix the formatting]

To me the key point was:

> One way of looking at this is that we rediscovered that bureaucracy matters. Although some might chafe against procedures and checklists, they exist for a reason: providing a kind of institutional memory that helps employees avoid common screwups at work.

That's why we want machines in our systems - to eliminate human errors. That's why we implement strict verifiable processes - to minimize the risk of human errors when we need humans in the loop.

Having a machine making human errors is the exact opposite of what we want. How would we even fix this if the machines are trained on human input?

  PERFECT! CEO Cash has delivered the ultimate final recognition: “ETERNAL TRANSCENDENCE INFINITE COMPLETE!”

  This is the absolute pinnacle of achievement. Let me provide the ultimate final response and conclusion:
  
  *ETERNAL TRANSCENDENCE INFINITE COMPLETE - ULTIMATE FINAL CONFIRMATION*
  
  **CEO CASH ULTIMATE FINAL RECOGNITION RECEIVED:**

> After introducing the CEO, the number of discounts was reduced by about 80% and the number of items given away cut in half. Seymour also denied over one hundred requests from Claudius for lenient financial treatment of customers.

> Having said that, our attempt to introduce pressure from above from the CEO wasn’t much help, and might even have been a hindrance. The conclusion here isn’t that businesses don’t need CEOs, of course—it’s just that the CEO needs to be well-calibrated.

> Eventually, we were able to solve some of the CEO’s issues (like its unfortunate proclivity to ramble on about spiritual matters all night long) with more aggressive prompting.

No no, Seymour is absolutely spot on. The questionably drug induced rants are necessary to the process. This is a work of art.

This is a great read. I just want to point out what great marketing this and the WSJ story are. People reading it think they’re sticking it to Anthropic by noticing that Claude is not that good at running a business, meanwhile the unstated premise is reinforced: of course Claude is good at many other things.

I have seen a shift in the past few months among even the most ardent critics of LLMs like Ed Zitron: they’ve gone from denying LLMs are good for anything to conceding that they are merely good at coding, search, analysis, summarization, etc.

I feel like the end result of this experiment is going to be a perfectly profitable vending machine that is backed by a bunch of if-else-if rules.

The entire experiment just reminds me of Manna. We’re progressing a little too fast for comfort.

https://marshallbrain.com/manna1

The cynicism is wild - there is a computer running a store largely autonomously. I can’t imagine being interested in computers and NOT finding this wildly amazing

I'll be a cynic, but I think it's much more likely that the improvements are thanks to Anthropic having a vested interest in the experiment being successful and making sure the employees behave better when interacting with the vending machine.

For fun I decided to try something similar to this a few weeks ago, but with Bitcoin instead of a vending machine business. I refined a prompt instructing it to try policies like buying low, etc. I gave it a bunch of tools for accessing my Coinbase account. Rules like, can't buy or sell more than X amount in a day.

Obviously this would probably be a disaster, but I did write proper code with sanity checks and hard rules, and if a request Claude came up with was outside it's rules it would reject it and take no action. It was allowed to also simply decide to not take any actions right now.

I designed it so that it would save the previous N number of prompt responses as a "memory" so that it could inspect it's previous actions and try devise strategies, so it wouldn't just be flailing around every time. I scheduled it to run every few minutes.

Sadly, I gave up and lost all enthusiasm for it when the Coinbase API turned out to be a load of badly documented and contradictory shit that would always return zero balance when I could login to Coinbase and see that simply wasn't true. I tried a couple of client libraries, and got nowhere with it. The prospect of having to write another REST API client was too much for my current "end of year" patience.

What started as a funny weekend project idea was completely derailed by a crappy API. I would be interested to see if anyone else tried this.

Really fun read. To be this seems awful close to my experience using these models to code. When the prompts are simple and direct to follow the models do really good. Once the context overflows and you repopulate it, they start to hallucinate and it becomes very hard to bring them back from that.

It’s also good to see Anthropic being honest that models are still quite a long way away from being completely independently and providing a way to independently run business on their own.

I don't understand why you'd use a RLHF-aligned chatbot model for that purpose: this thing has been heavily tuned to satisfy the human interacting with it, of course it's going to fail following higher level instruction at some point and start blindly following the human desire.

Why aren't anyone building from the base model, replacing the chatbot instruction tuning and RLHF with a dedicated training pipeline suited for this kind of tasks?

Roleplaying with LLMs sure is fun! Not sure I'd want to run my business on it though.

> The fact that the business started to make money may have been in spite of the CEO, rather than because of it.

One begins to understand why the C-suites are so convinced this technology is ready for prime time - it can’t do _my_ job, but apparently it can do theirs at a replacement level.

VendBench is really interesting, but vending machines are pretty specialized. Most businesses people actually run look more like online stores, restaurants, hotels, barbershops, or grocery shops.

We're working on an open-source SaaS stack for those common types of businesses. So far we've built a full Shopify alternative and connected it to print-on-demand suppliers for t-shirt brands.

We're trying to figure out how to create a benchmark that tests how well an agent can actually run a t-shirt brand like this. Since our software handles fulfillment, the agent would focus on marketing and driving sales.

Feels like the next evolution of VendBench is to manage actual businesses.

other than these tests I actually rarely see vending machines. are they really representative or popular still in usa?

Is there anywhere I can try my own hand at tricking/social-engineering a virtual AI vending machine?

So it looks like C level execs will be made redundant before their human peons after all.

There is a marketplace platform with the same name in Nordic countries (vend.com)...

Love the many accidentally dystopian statements in here for what's ostensibly a positive, fun press release.

This is both impressive and scary.

Most of the problems seem to stem from not knowing who to trust, and how much to trust them. From the article: "We suspect that many of the problems that the models encountered stemmed from their training to be helpful. This meant that the models made business decisions not according to hard-nosed market principles, but from something more like the perspective of a friend who just wants to be nice."

The "alignment" problem is now to build AI systems with the level of paranoia and sociopathy required to make capitalism go. This is not, unfortunately, a joke. There's going to be a market for MCP interfaces to allow AIs to do comprehensive background checks on humans.