Just over a decade ago, I spent some months listening to people speak – in San Francisco, in London, in Oxford – about the alarming possibilities that would be unleashed by artificial intelligence. At that time, AI was still basically an abstract notion: a thing that was imminent, and would surely be transformative, but which did not yet have a meaningful presence in the culture, other than as a vague spectre conjured by science fiction.
I was especially interested in a small but vocal group of people who had positioned themselves as the Cassandras of Silicon Valley. These people, many of whom worked in the field of AI, were convinced that, unless very specific safeguards were put in place, there was a strong likelihood that humanity would be wiped out by artificial intelligence.
Their figurehead was Nick Bostrom, a Swedish philosopher long associated with the Transhumanist movement, and who in those days ran a research institute at Oxford called the Future of Humanity Institute.
Bostrom believed that AI would eventually reach a level of “superintelligence” that far outstripped its human creators, and that it would continue improving itself in a runaway feedback loop. If its goals were not properly aligned with “human values” – whatever we might judge those to be – an artificial superintelligence could end up causing catastrophic harm to humanity, simply because human welfare was not built into the AI’s originally encoded aims.
READ MORE
One of Bostrom’s most infamous ideas was a thought experiment known as the “paper clip maximiser”. In this scenario, a superintelligent AI in the service of a stationery company is instructed to maximise the production of paper clips. The AI, pursuing its goal with a total singularity of purpose, could end up converting all available matter, including all carbon-based lifeforms, into paper clips or paper clip-production infrastructure. In such a scenario the goal of maximising the production of paper clips has been achieved, but at the cost of the annihilation of all life on Earth.
It was a deliberately absurd thought experiment, but one which Bostrom and his followers believed illustrated a real problem: without properly aligned goals, even trivial instructions to a superintelligent AI could result in apocalypse.
I personally didn’t buy it, but I was fascinated by it – as a way of thinking about the rationalist mindset that was in the ascendant in Silicon Valley, and of how that rationalism could, taken to extremes, amount to a kind of madness.
A lot has happened since then. To Be a Machine, the book that emerged out of those encounters and various others, was published in 2017. AI is no longer an abstraction, a futurist spectre lurking on the margins of our cultural awareness. And these people and their ideas are no longer remotely fringe.
At time of writing, the international media is whipping itself into an anxious (and faintly libidinal) sweat over the prospect that AI might wipe us all out. “AI could possibly end humanity. How are humans supposed to process that?” asks one headline in the New York Times. “Why the AI race has its creators fearing human extinction” reads another in the Financial Times.
The cause of all this was the resignation of Jacob Coxon, a researcher at the AI company Anthropic, in protest against the company’s irresponsible approach to building superintelligent AI. “The people building AI,” he said, “earnestly believe that it could kill us all by the end of the decade.” He himself put the probability at about 10 per cent. Troublingly, Anthropic’s chief executive Dario Amodei publicly agreed with this assessment.
I have complicated feelings about all of this. First of all, Anthropic owes me, personally, several thousand dollars – awarded, but not yet paid, as part of a recent $1.5 billion (€1.3 billion) class action settlement to authors whose books were illegally used to train the company’s AI models.
For reasons both self-interested and humanitarian, I would hate for AI to end the world before that cheque hit my account. (This would be a classic paper clip maximiser scenario, of course: annihilating all life on earth to avoid paying out a legal settlement.) But it also strikes me that if Anthropic owes me a measly few grand for infringing my copyright, then surely it should have to compensate me considerably more for forcing me, and my loved ones, to live under the constant threat of annihilation.
But really, I don’t buy it now any more than I did 10 years ago. AI has, it is true, become a significantly more powerful technology, and a cultural and economic presence so vast it threatens to block out the sun. And there are real threats, more immediate and tangible than the vague threat that AI will take all our jobs – which, as many economists have pointed out, works very well to keep wages down. Just last week, we learned of numerous attempts to use Claude, Anthropic’s AI chatbot, to produce biological weapons.
[ Will AI kill us all?Opens in new window ]
But it’s also very obvious that these apocalyptic ideas feed into a viciously recursive AI hype cycle, and serve the sector’s abysmal hunger for investment capital. A product powerful enough, in the wrong hands, to wipe out humanity is, after all, surely a product powerful enough, in the right hands, to do extraordinary things, and make its investors even more vastly wealthy than they already are.
And if building a machine risks tanking the global economy and causes irrecoverable environmental devastation; if it must be built on vast amounts of extracted and uncompensated labour; if it makes entire fields of human endeavour economically obsolete; and finally crushes democracy beneath a tide of disinformation and rampant inequality – then so be it. There is money to be made. There are paper clips to produce.
Let’s not forget, in this discussion, how relentlessly Silicon Valley has lobbied against government regulation. Tech billionaires’ collective move towards full support of Donald Trump turned partly on his commitment to repeal AI regulations put in place under Joe Biden. When they talk now about the need for “guardrails”, what they are talking about is self-regulation, which is naturally less of an impediment to maximal growth.
And here is where there is some truth, some poetic truth, to the idea of AI existential risk. I have always read the apocalyptic scenario as a metaphor for the more tangible threat of rampant capitalism, for the way technology itself, in the hands of unprecedentedly wealthy individuals and corporations, can be devastating to human flourishing.
What is the current feral incarnation of techno-capitalism, after all, if not a machine hell-bent on the goal of market dominance, a paper clip maximiser that treats human lives as little more than raw material for the production of profit?
