I tackle these ideas/questions head-on using Claude to help book-shape the result.
https://spine.zice.app/jeremy/the-smaller-infinity
In my view, the accumulating density of useful language/token patterns is the real story, not who/what is processing them. There is no "why" other than the universe's physics permit such accumulation, but that doesn't make the "how" any less interesting.
You can click the header and copy the whole thing to clipboard, ask your favored agent for summary/conversation.
> but Western culture has lost the vocabulary to express their true sentiment. They are grasping for words but can't reach for "soul" and other religious terms because that would sound ridiculous in a secular modern world
Ironic to implicitly accuse Stross of this; as I recall, he's one of the voices describing The Singularity as Christian Eschatology/Book of Revelation/the rapture for atheist nerds.
> Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not.
One of his own paragraphs explains his non-financial perspective and reason:
I do not want or need a large language model to write my fiction for me. I write fiction compulsively—before I was published I wrote for many years as a hobbyist—so why on earth would I pay someone else to take my fun away?
I may not have quite the level of obsessive desire to write as he has, but I have some, and it is different from the desire to simply consume someone (or some*thing*) else's work.
Without criticism, he is also clearly upset about the economic impact this will have on him (he says specifically he is "one of the parties to the settlement in the class action lawsuit against Anthropic AI for pirating ebooks to train their LLMs" and "train an LLM that is intended to compete for revenue with the authors of the works they stole"). Writing is not a high-earning job (with the exception of a few superstars who may as well be lottery winners), and he has written in support of very long copyright durations that can be inherited.
> People will rather talk about fake water consumption problems than touch the topic of the human soul or spirit.
For a living, he writes about Cthulhu and their ilk doing mind control; I want to say I don't think this would put him off, but if it did he'd hardly be the only professional author who can write a fictional story about something without being able to apply such ideas to rhetoric. (He'll hate being compared to Rowling here, but consider how Rowling hates trans people while her fiction has a variety of shape-shifting spells).
> Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not.
I sure would. If you want a religious tone to it… It would be a signal suggesting that the human spirit(tm) had finally transcended.
If humans can someday build a machine that can truly carry a (trans)human mind, and let some future souls live unconstrained by all the quirks and limitations of our preexisting biology, exercising more control of their own lives - that’d be wonderful.
I obviously have no idea wherever language models can lead to such dream, of would be a dead end.
I think attitudes like this are traitorous to the human species. You’re not talking about a transcendence, or at least, not one that I recognise. Our biology is an innate part of us - to improve, to fix, sure, but not to cast off. Who is directing this “transcendence”, and who benefits? I absolutely do not want a religious tone to any of this. Awful vibes.
I find this "traitorous to the human species" interesting. You could not form these thoughts without an accumulated knowledge stored/accessed outside human biology. No other species forms such explicit thoughts because they lack access to that accumulated pattern of useful sequences. We've already "transcended" our biology in that sense. Who is directing... that becomes interesting in new ways once outside-a-body sequences start to accumulate. My guess is that it's that accumulation that sets the trajectory, just as DNA's "accumulation" of useful sequences produced forms fit for the local environment(s).
Sure, we're cyborgs, but still very much of the flesh, and I think we have to draw a line somewhere.
> Our biology is an innate part of us - to improve, to fix, sure, but not to cast off.
Right now, necessarily so.
The more mysteries that science solves, the more we unweave the rainbow*, the worse the tension between rationality and our sense of what we are. Am I the atoms, or the pattern betwixt? Am I the present moment alone, my memories mere shadows of past patterns and my current form doomed to be a shadow to a future pattern of these self-same atoms?
Sometimes even the question is unbearable.
* https://en.wikipedia.org/wiki/Unweaving_the_Rainbow
What are you talking about… You can use the word soul without believing in a religious concept of soul. It’s done all the time in secular societies. It’s even a very common term used to describe the problem with LLM generated content, that it lacks soul or human spirit, it lacks the human meaning, intent
It's rare. It's mostly, "but it will never be able to do X" (when it can already do X), or it uses too much water (while having no idea or reference point to what amount is much), or other climate things, or non-progressive bias, or copyright (when training is fair use) etc.
Just be upfront. 1) It threatens your livelihood that people can get the thing done faster and cheaper than what they used to pay you, 2) our relationship to an AI shaped entity is spiritually risky and a esthetically off putting, and we have to talk about golems and the tower of babel and stuff. If you think I'm being incoherent, you're in for a ride. Society will have to come to terms with major major issues, not just job replacement and bad slop art.
I think it is true that people who obviously dislike AI for ideological reasons often lead with ecological concerns or doubt about the capabilities, leading to an unproductive discussion.
Have you considered engaging on those ecological concerns? There is quite a lot to dig into and can be pretty interesting. For example, try to project how much electricity will be needed to power all the datacenters in construction, then compare that to the existing grid and infra, then figure out the cooling capacity. If done well you will see that we have a pretty massive resource problem here
I had a look now, and apparently we are projected to increase by 2.3x datacenter power consumption until 2030 from 415 TWh to 945 TWh.
That is going from 1.5% of global electricity use to 3%.
It is less than I expected.
Maybe if AI does work out better than expected for knowledge work or becomes applicable to robotics and fields like construction, we are going to see a larger buildout.
> That is going from 1.5% of global electricity use to 3%.
> It is less than I expected.
I suspect this is because of the word "global". Are not most of these currently planned specifically for the USA? Sure, the USA has a disproportionate use of electricity, but IIRC it's like 16% (1/6th) of global?
3% of global electricity is roughly the same as 18% of US electricity; a bit less because not all the new DCs will be in the US, but that's something that I consider with regards to the economics and why I'm bearish on this topic, as I think it will be a case of Dutch disease in the USA: https://en.wikipedia.org/wiki/Dutch_disease
Okay nice, after looking at the macroview look at the immediate impact into local communities and how much data these companies refuse to, rightfully I might add, share with the public.
Weird how all these concerns can simply go away if they just simply shared the information the public wants (water usage, sound generation, power usage) rather than refuse to do so.