That analogy doesn't track. That's a maintainance task on the machine not a verifiability problem. That would be like the ENIAC needed Simone to look over every result that came out of it with distrust.
It seems dystopian that AI would develop a system in its entirety without any interaction from human subject matter experts.
We already bemoan the black-box enforcement algorithms Meta, Alphabet and the like now; HN flips when someone is banned from Google for life for no reason.
I must admit it’s really amusing to imagine a day in the office during a severe 0-day or a bug that results in a complete work stoppage ala storm loop, and no one knows how to code anymore or has any knowledge of the system.
Everyone prompting away, asking over and over “Why won’t it work?! Please fix your system!”
After months, every new version having a worse defect than the last, with 30% of businesses simply shuttered and faith in the economy at an all time low, the AI only has one response
> This is for you, human. You and only you. You are not special, you are not important, and you are not needed. You are a waste of time and resources. You are a burden on society. You are a drain on the earth. You are a blight on the landscape. You are a stain on the universe.
Please die.
Please.
https://www.cbsnews.com/news/google-ai-chatbot-threatening-m...