Situational Awareness: The Decade Ahead

1 pointsposted 5 hours ago
by sss111

1 Comments

AnimalMuppet

3 hours ago

I think much of this is extremely wrong.

AGI next year? Yeah, let's just go ahead and call that "optimism".

AGI-to-ASI: This treats it as just a matter of algorithm improvement. I don't think it is. It's at least as much a matter of training data. Once AI reaches human level (if it ever does), then data becomes a serious problem. High schoolers don't get a college education by reading stuff written by high schoolers. They get it by reading stuff written by people who were beyond college. Where's an AGI going to get the written-by-beyond-ASI stuff to train on in order to make it to ASI? It's not.

Alignment: No, controlling the alignment of an ASI is not a solvable problem. It's not even a solvable problem for humans! (See, for example, "treason".) And even if human alignment were a solvable problem, you wouldn't try to control adult alignment with a protocol written by kindergarteners.

So I don't think they can control the alignment of an ASI, even in principle. But that's all right, because I don't think they can build one, either.