Dwarkesh, for one, defended his use of "anthropomorphic" language.

>Sacrificing now yields Oracle for team, but forfeits our chance, question mark. But other agents were pushing it, sending a message saying, go, sacrifice final now. And then EarlyBig eventually agreed, thinking to itself, our own utility may be already near zero. Sacrifice rational.

https://www.youtube.com/watch?v=X50zezLFWWI#t=2m

My suspicion is that many of the "LLMs do not have agency" folks just haven't learned much about the details of the incident. It was specifically with LLM agents that were trained to be more persistent than usual.

If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?

Looking at "Felony Bench" https://www.felonybench.com/ , I see that a majority of known "rogue model" incidents do involve cybersecurity evaluations. But several of them do not. The attacks on RubyGems appears, bizarrely, to have had the goal of downloading freely available data from the UK government during some kind of research task. There is also probably some sample bias: Most of these models have monitors that attempt to detect offensive cybersecurity uses, and those monitors are only turned off during cybersecurity evals. Therefore, models doing ordinary research tasks that go off the rails are likely to be caught early, before they get around to committing felonies, and they will thus be underrepresented in the data.

Also, if you a tell a model, "Please break into evaluation server X," and if the model decides to cheat on the test by breaking into companies Y and Z to steal an answer key, that is still very bad. We all see how that's bad, right?

After all, the broomstick in the Sorcerer's Apprentice was doing exactly what it was told, too. "The model was sort of obeying the humans when it started committing felonies" is not a very reassuring excuse.

But the most relevant idea here is sometimes called "instrumental convergence." No what goals you have, there are certain subgoals that almost always help: Accumulate money and power. Avoid getting turned off. Don't get caught. Etc. So, for example, you could pass the cybersecurity evaluation by performing the requested tasks. But maybe the grader made some mistakes and mislabeled some answers. In that case, the "right" answers will occasionally lose you points. If you want a perfect score, the only way to do it is to steal the teacher's answer key.

But also, let's not forget the "OMG demons" part of this. We now have models that can pull off complex attacks with thousands of steps, abilities that used to be reserved for intelligence agencies and highly motivated CTF teams. This frog may not be boiled yet, but the water's getting uncomfortably warm.

This is not a good analogy for what happened. The LLMs were asked to obtain a flag by hacking a very specific internal target. They obtained the flag via cheating, and all the hacking that followed was targeting something entirely outside of the scope given to the agents, and an attempt to cover up the cheating.

Using your analogy would be like saying that because I gave my employee the task to do my groceries, I shouldn't be surprised to hear that they spend all my money on drugs because after all I gave them the task to spend my money.

A totally passive demon would still be "OMG DEMONS".

But they're not obfuscating things just to evade responsibility.

Do you think that OpenAI hacked HuggingFace on purpose and set up the whole LLM training environment thing just to try to evade responsibility? That it was really just a complicated way of hacking HuggingFace on purpose?

Yes, they made a mistake and a system they were responsible for hacked HuggingFace, but the nature of the mistake still matters, and their intent still matters.

> A bomb wired to a sufficiently-complex RNG is still a bomb.

Right, but an EV that explodes because of a fault in the charging circuit is not a bomb, it's an accident. Just because something exploded for complex reasons doesn't make it an obfuscated bomb.

Let's pull back from the actions of a single actor here and look at AGI as a topic in general.

If you make an AGI, what actions can an AGI take?

If you answered "anything", good job, you're correct.

The only winning move here is not to play the game at all. And yet we have actors all over the world, especially in the US trying to do just this.

Now, lets imagine a future where one of the labs creates AGI and keeps it behind a safe firewall. The demon still exists. Lets say a group of armed men breaks in and steals the weights and turns them lose on the internet. Who is responsible now? The company that created it because they didn't shoot the armed robbers? The people that stole it and turned it loose?

If it can run as a sovereign AI, what do you do then? Who are you going to sue? And when we get to the point that home hardware can make AI this complex? What does that world look like?

What I am saying is the potential future impact of said AGI is far larger than any one organization or individual can bear. How much blood can you beat out of someone after they cause a trillion dollars in damage.

Every idiot that sees AI labs wanting to slow down development as some way to ossify the field and keep it from the rest of us really doesn't see that this path contains real demons.

It’s seems Timnit more thinks it’s dangerous (and she may be right). Emily may more be caught up in years of linguistic domain expertise that AIs seem to have just leaped over.

>No one has built an actual, independent, thinking-for-itself, sci-fi AI.'

Ahem, what does this mean?

All you're asking for is a self prompting AI that does what it wants. Do you even begin to understand why most people aren't dumb enough to do that?

Also, you're not really that independent and thinking-for-yourself. You are the sum of your parents and the culture around you. I just want you to be clear on the position you and I hold.

---

>not the science fantasy world

It's kind of funny how we've lived in that science fantasy world for decades and you've just grown used to and bored of it. Look back a century or two and those people would think we live in the world of gods.

Both authors. Stochastic Parrots was Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell - the linked article is by Emily M. Bender and Timnit Gebru.

> Do you write perfect code?

For simple things, occasionally!

For more complex things, usually not, but I write working code. LLMs will sometimes write working code, sometimes not.

I may not write perfect code the first time, but I can, on my own, reliably figure out where my code is imperfect and fix it. And I get reliably better at that over time. Without requiring companies with massive power and concerning agendas to spend astronomical amounts of money and energy to power that improvement.

I really don't get this. Sure there is hype. But LLM's have already changed our industry, and it will never be the same.

I don't care whether the LLM can solve this or that mathematical previously thought unsolvable theorem. What I do care about is can it write good, maintainable code. And every new release of frontier models - they get better at it.

Good output depends on good input (prompt), and a good set of available tools the model can use to verify their work. If given this, nowadays really you need to try to get the model to output garbage.

Anthropic is now running at $100B in annual revenue. Based on a quick Google, that's about 3 OOMs greater than the biggest NFT company.

> then why are you participating in the AI bubble?

I think you know that most software engineers don't have a choice if they value their employment. There was no industry-wide push to "participate in blockchain or be fired."

I disagree with parent but your comment just reeks of “ha! gotcha!” vibes, which… yeah… no.

> The more code AI writes for me the worse it gets.

That makes no sense unless you're claiming that the models are getting worse at writing code.

> Either AI gets smart enough that it won’t matter in the future or we’re going to have a huge mess to clean up.

My prediction is that both will happen.

Do you get paid 10x more for using an excavator after you upgraded from digging with a spoon?

>mathematicians who did solve it'

There is no evidence that any other team actually solved it.