vannevar
an hour ago
Perfect alignment is mathematically impossible, just as perfect security is mathematically impossible. Alignment is only as good as its assumptions. That doesn't make it irrelevant---if we could show that alignment would likely last as long as the Sun, for instance, we might call that good enough. The problem is that superintelligence is inextricably bound to the notion of singularity, in which case alignment is irrelevant. The unspoken assumption around practical alignment discussions today is that "superintelligence" can be nerfed enough so that it's superior to humans in useful dimensions but not in unhelpful ones. It remains to be seen whether that is true, but the AI leadership certainly seems confident enough to spend vast sums of other people's money in order to find out.