England, 1009. King Æthelred the Unready faced an existential nightmare in the form of Thorkell the Tall and his great Viking army. The names were a bit of a spoiler. The King had ordered the construction of a vast fleet, an early Royal Navy. The project went so well that part of it went rogue and ended up raiding the south coast itself, while the rest was destroyed trying to stop it.
Thorkell’s army landed unopposed in Kent, where local nobles immediately paid off the marauding Vikings with silver and pointed them toward London. Lesser kings might have panicked or resorted to rash schemes, but Æthelred had a brilliant plan. He would lead a three-day national effort to pray the Vikings away. People would fast, and walk barefoot to church, where they would hold masses. Special ‘lamb of god’ pennies were minted. This would make the Vikings think twice, which they did before rampaging up the Thames to Oxford and burning it.
By 1014, Æthelred had fled to Normandy. Wulfstan, the Archbishop of York, delivered his infamous Sermon of the Wolf: “Beloved ones, know what is true: this world is in haste, and it is nearing the end.”
Spoiler alert: it wasn’t. Sure, life at the turn of the millennium was bloody and brutal, with peasants at the whim of roaming warlords; but grandiose visions of an apocalypse driven by divine retribution were largely an elite phenomenon. Ordinary people were busy dealing with whichever in a long-line of noble bastards had turned up in their village this week.
After all, Thorkell the Tall wasn’t the only wealthy despot to set fires in Oxford - seven years earlier Æthelred had ordered the killing of Danes living in England, convinced they would assassinate him. In Oxford they ran into St. Fridewide’s church for protection, barricading themselves in until Æthelred’s supporters set fire to it and burned them all alive.
Recently, an assortment of Lords, MPs and the Bishop of Hereford addressed their own apocalyptic sermon to the Prime Minister. Under the banner of ‘ControlAI’ they warned Burnham that AI could kill us all in a decade and urged him to enact a bill that would ban the creation of Artificial Superintelligence (ASI): a technology that doesn’t exist, isn’t being developed in Britain, and is based on a word that has no operable meaning. It followed the resignation of Mrinank Sharma, a former researcher at Anthropic, who declared that: “The world is in peril. And not just from AI, or bioweapons, but from a whole series of interconnected crises unfolding in this very moment.”
The bill they propose would regulate not just ASI but ‘precursor’ technologies or “any resource … that may reasonably be expected to be used in, or to materially assist" the development of ASI.” It is drafted so broadly that arguably even the chips in your phone or the graphics card in your games console could be regulated like nuclear material. A shipping container full of rare books could conceivably be in scope if it were destined to be used for training. Punishments would range up to life in prison.
The language comes via the Swedish philosopher Nick Bostrom, whose Future of Humanity Institute was headquartered just a few hundred yards from where the people of Oxford burned his fellow Scandinavians a millennium earlier. In the 90s he set out a definition of superintelligence as “an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills." Bostrom’s work added an academic veneer to the ideas of Eliezer Yudkowsky, a self-taught philosopher and fellow graduate of the late-90s transhumanist movement, known for his nuclear-grade pessimism. It was Yudkowsky who proposed, in a 2023 article in Time magazine, that we should:
“Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. […] No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold…”
This is where the policy put forward by ControlAI was cribbed from, but strangely they missed the best part:
“If intelligence says that a country outside the agreement is building a GPU cluster… be willing to destroy a rogue datacenter by airstrike. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.”
Which is odd, because according to the nobles’ letter, “superintelligent AI poses risks to international security and an extinction threat to humanity.” If they genuinely believe this and they’re not just cosplaying, then surely Yudkowsky’s logic - the logic of Æthelred’s massacre of the Danes - holds?
Of course military action would need public support, and what’s striking is how divorced all this is from the British peasantry. I started this post because I’d seen a growing number of friends succumb to a form of AI mania. Not because they’d decided Claude was their girlfriend; rather they genuinely seemed to believe an apocalypse was coming. Language like, “AI is a threat to the lives of everyone you hold dear,” was cropping up in my feeds. So I set out to see if this was a growing sentiment in the wider public. But it doesn’t seem to be, at least yet.
That’s not to say Brits aren’t depressed about the world. Of course we are, look at it. In the sunny words of YouGov last year: “The public think the UK is in a bad state, is worse than it was ten years ago, and will get worse still over the next year.” That year happened, and the result was: “Three quarters of Britons say Britain is getting worse as a place to live.”
That said, we are relatively optimistic about our own lives and communities. The ONS reported this year that “64.4% of adults reported feeling hopeful about their future.” Whatever concerns we have aren’t apocalyptic in nature. Surveys that prime people on the ‘dangers of AI’ tend to provoke a response, but in unprompted surveys like the Ipsos Issues Index, the topic simply hasn’t featured until now.
So if the apocalyptic vibe isn’t coming from their constituents, who or what is driving these politicians? Inevitably, it’s rich American AI lobbyists. ControlAI is funded in the US by the billionaire Jaan Tallin, who made his fortune from Skype before sitting on the boards of DeepMind and later Anthropic, and believes “we need to realise that this is a suicide race. This is not a race with winners.”
ControlAI is a vehicle for the American AI ‘Doomer’ movement, a subset of the overlapping AI and rationalist communities. Like Wulfstan they believe the apocalypse is nigh, and the most extreme predictions come from the top: leading rationalists, trans-humanists and the upper echelons of companies like Anthropic or OpenAI.
You could argue this lends credence to the idea. After all, these are the experts, the people closest to the technology - they should know. Yet the same people would argue that experts in other fields are terrible at judging the potential impact of AI; that the very nature of their experience and in-group heuristics makes them unable to adapt their views based on outside evidence. Why wouldn’t this apply to the AI community?
Philip Tetlock, the father of superforecasting, classed many experts as ‘hedgehogs’: people obsessed with ‘one big theory’ who apply it to a one-dimensional view of the world, making bold predictions that wildly overstate change. The bolder the claim, the easier it is to attract media coverage. Careers, funding and reputations depend on buying into these claims. Being clever doesn’t make you immune from these effects - if anything it allows you to rationalise your beliefs more effectively. It would be surprising if AI research were the only field to be exempt from these effects.
Which brings me to how utterly weird this particular expert community has become. Bostrom’s output in recent years has degenerated into sophomoric word salad about ‘cosmic hosts’. In a 2024 essay he argued that humanity is encompassed within an all-powerful construct that sets the moral norms of the universe. “The cosmic host might conceivably consist of galactic-scale civilizations, simulators, superintelligences, and/or a divine being or beings.”
In Bostrom’s mind, the only sensible thing for us amoeba-like humans to do is show our respect and loyalty to these hypothetical powers. To metaphorically take off our shoes and walk to church. In most walks of life this sort of statement would have you gently led out of the auditorium; in AI it’s standard rhetoric. Groups of employees at OpenAI are said to have openly talked about the impending rapture: “Literally the rapture.” Freddie deBoer observed “the armies of rabid AI fans online who sit around on Reddit muttering darkly about the imminent AI rupture that, they believe, will devastate the people they don’t like and enrich themselves.”
Peter Thiel delivered rambling private lectures this year, talking about the antichrist and coming armageddon. Guillaume Verdon and the ‘effective accelerationists’ who riddled top AI companies in recent years believe that it is the duty of humanity to give as much money as possible to tech companies because something something thermodynamics something something promote the spreading of new forms of consciousness throughout the universe. Former Uber and Google engineer Anthony Levandowski set up a church dedicated to AI worship. Multiple cult-like groups like the Zizians have sprung up from the rationalist community, sucking in young tech workers.
Set against this bubbling mania, it seems almost quaint and normal when Sam Altman speaks of going “from solving high-energy physics one year to beginning space colonization the next year,” or rationalists bet tens of thousands of dollars on AI causing US GDP to grow 15% in a single year. Of course gambling has always been a hobby for bored rich people, and therein lies a clue. I don’t think these people are making predictions based on rational thought. They seem more like the aristocrats who gambled away their fortunes at the tables of John Aspinall. Anything to escape the inanity of their cosseted lives.
The worldview of AI elites is riddled with anxiety and paranoia, people who believe they are “manning the barricades of civilisation,” to quote Hofstadter. This culture has serious consequences for the rank-and-file of the AI industrial-philosophical complex, and the glimpses we see into the health of that group are quite shocking.
Ihor Kendiukhov wrote this month about the need for ‘psychological support for AI safety researchers.’ He describes a community gripped by stress and defeatism, where anxious, burnt-out people sit prompting Claude for twelve hours a day, unable to make life plans because they believe the apocalypse Singularity could happen at any moment, terrified by the “high probability of dying soon.” Incredibly Kendiukhov believes some aren’t anxious enough:
“I think there is still some kind of taboo on acting with heroic sacrifice or encouraging others to act so. While I agree that heroic sacrifice, extreme application of effort, and working at or over the limit of your capacity are not for everyone and shouldn’t be a requirement, I think people should be allowed to do so, and the Overton window should be pushed towards an explicit realisation that we are in an extraordinary situation which requires extraordinary effort. It should be talked about more. We should pretend less that we are doing a normal job, for we are not.”
It’s hard not to see this as collective narcissism. Once you’ve decided that you’re part of the few hundred people who can prevent human extinction, any evidence against that thesis becomes a challenge to your own self-importance, to your identity. Disagreement becomes personal insult, and grandiose stakes justify grandiose means.
Or just tedious everyday evil. Sonia Joseph’s account of her time in the San Francisco AI scene describes a culture rife with heavy drug use, in which work, life and sex are blurred, young female engineers are encouraged “to sexualise themselves for access into the heart of the industry,” and where those who raise concerns are dismissed - or worse. “I had three OpenAl and Anthropic researchers debate whether I was mentally ill on a Google document.”
This all leads to deeply dysfunctional, cult-like behaviour as Ozy Brennan pointed out in their essay, Why Are There So Many Rationalist Cults? Brennan describes a community of would-be-martyrs, enmeshed in a deep paranoid fantasy that they alone can stave off the end of the world. They live together, work together, sleep together, take LSD together and seemingly talk about nothing else: “All the minor details of their lives felt like they had the fate of humanity or all sentient life as the stakes.”
The ur-text for this scene is a series of blog posts by Eliezer ‘we might have to bomb data centres’ Yudkowsky called ‘The Sequences’. As Brennan put it, “[they] make certain implicit promises. There is an art of thinking better, and we’ve figured it out. If you learn it, you can solve all your problems, become brilliant and hardworking and successful and happy, and be one of the small elite shaping not only society but the entire future of humanity.” You can see why this would appeal to people with high IQs and low satisfaction, adrift in San Francisco, searching for meaning.
These are textbook conditions for exploitation and groupthink: smart people in thrall to larger-than-life leaders, isolated from outside context or opinions, engaged in endless collective rationalisation, and driven by a sense of moral mission. The promises were hollow of course. The reality, as Brennan found in interview after interview, was isolation, hostility to ‘unenlightened’ outsiders, and grandiose self-delusion: “As long as you’re sitting in a room taking LSD with the blackout curtains over the windows, you can be Superman.”
Still, at least they’re not minting symbolic tokens with animals on them.
Perhaps these people are being driven slowly mad because they’ve peered into the abyss, they have seen the future better than any of us, and this hysteria is actually proportionate response. After all, the Vikings were real! But if AI does represent a threat to our way of life, this still is still not rational or constructive behaviour. I don’t believe that people talking to Claude all day and rambling to their friends about the rapture all night are going to help us any more than Wulfstan’s apocalyptic sermons helped the English in 1014.
Let me put my cards on the table though: I don’t really believe in superintelligence.
I think it’s a meaningless term, rooted in a rationalist culture that prizes individual brainpower above all else; assumes that mastery and creativity in all fields is within the entitlement that brainpower confers; and fantasises that with more IQ points they could basically be omnipotent. Ideally without leaving the safety of their bedroom.
This nerd’s fallacy just isn’t how progress works in any field of human endeavour. Ideas are cheap and thinking is disposable. The true work of innovation is active, social, incremental, defined by real-world limitations and constrained by the speed of feedback. You can design a million new drugs tomorrow, you still have to test them, fund and execute trials, learn from the successes and failures, and roll them out. Things have to be built, in the world, from physical materials, and tested by people. Knowledge is accumulated and built on over time.
To the extent that you can assign a value or an upper limit to ‘human intelligence’, that number is derived not from our individual IQs but the information bandwidth of our society. It is limited by the speed with which we can collectively explore the problem spaces our species encounters; problems that almost always require us to make, build, test, find or explore something in the real world before we can make the next painstaking increment of progress. It is limited even further by the speed with which we can implement something useful based on that discovery, and spread that change across the world.
Civilisation is the gearbox for the engines of our minds; human intelligence is its attribute. The scale of that intelligence is not governed by the person or entity with the highest IQ. You can’t make a car that goes at 3000mph by just putting a rocket in it. I’m not sure it’s possible to be superintelligent. To me it’s like saying you’re going to make the ocean superwet by pouring more water into it.
Rationalists desperately wish this wasn’t the case, because it means the promise of the AI priesthood, the promise of a world where being smart and thinking better magically solves all your problems - could lead you to the gates of heaven itself! - is a lie: that the world isn’t changed by thinkers, but doers. That’s why every contrived thought experiment that involves an AI triggering the end of humanity involves, ultimately, human beings doing something incredible stupid with real stuff, like shipping a super-virus to an angry teenager or firing a nuclear arsenal. The boring risks of human agency that we already face today.
I could be wrong; but if I am then one of the deeper reasons for my heretical complacency is that I think destroying the world is a fundamentally stupid act. The obvious problem with Bostrom’s paperclip maximiser thought experiment is it requires a sort of Schrödinger’s intelligence - an AI so smart and capable that it can escape all attempts to control it, yet so dumb and tragic that it can’t transcend its slavish devotion to a single, human-imposed rule.
Of course paperclip maximisers already exist. They are all around us in bureaucracies and institutions. They’re so common that they prompted Goodhart's Law: “Every measure which becomes a target becomes a bad measure.” They are not the product of intelligent thought, but of dumb algorithms applied at scale - something that’s been a feature of our world for centuries now. If we do go extinct in the near future, I doubt it will happen because of a surplus of intelligence.
In 1524, astrologers in London became obsessed with the idea that an upcoming planetary conjunction would cause a biblical flood. The recent arrival of the printing press allowed the learned elites to put their predictions into print for the first time, distributing pamphlets to the masses, and this new media allowed hysteria to spread. Twenty thousand people fled London for Highgate, Hampstead and Blackheath. The year turned out to be drier than normal, but the astrologers simply revised their predictions.
It would be nice if this latest hysteria were neatly contained within a few square miles of San Franciscan real estate, or the walls of an Anglo-Saxon church for that matter. But in recent weeks it seems to have escaped containment, driven by the new media of our age, and now it occupies the minds of politicians and an increasing proportion of the public.
What a lovely distraction! In Britain we’re approaching the start of a third decade of stagnation, watching out children grow up poorer than their parents. The global trade we depend on is increasingly disrupted. The eastern borders of Europe are under constant threat. Large numbers of young people are trapped in a state of arrested development, unable to buy property or find love or engage in meaningful work.
Behind every martyr is a narcissist, and I wonder to what extent these smart people are drawn to AI precisely because of its grandiosity; because fixing the boring problems of society like healthcare or inequality is beneath people as smart as them. As Brennan put it: “As soon as you have to accomplish specific goals in the real world, the mundanity of everyday life comes flooding in, with its endless slog of tasks variously boring, frustrating, and annoying.” The appeal of this mindset to British politicians is obvious: if the world is ending, nobody needs to explain why Britain can't build a railway, house its young or raise wages; nobody needs to bother with the dull thankless drudgery of fixing it.
Will the AI apocalypse come for us all? I can’t tell you that it won’t, in the same way I can’t say an asteroid won’t collide with us five years from now. What I can say is that none of the people claiming so inspire my confidence, none of them seem to think particularly deeply about the world and how it works, or offer plausible accounts of how this could happen. GDP is not rising by double digits, employment remains high, my Claude is as stubbornly low-witted as ever. Every prediction made by this cohort in my lifetime has been wrong by orders of magnitude. I see little reason to take them seriously.
More than that, I find these people and their worldview profoundly boring. The AI apocalypse is the sort of mono-maniacal junk-intellectualism people resort to when they live a dull uneventful life devoid of stimulating interests and engagements. It is one-dimensional thought, like that guy you know on Facebook who responds to every thread with dreary rants about ULEZ. None of it leads to anything useful or interesting. I’m reminded of the famous 1948 essay by C.S. Lewis, On Living in an Atomic Age:
“In one way we think a great deal too much of the atomic bomb. “How are we to live in an atomic age?” I am tempted to reply: “Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.”
England became Britain became an empire which rose and fell and then we got WiFi. Æthelred’s lamb pennies, intended to ward off the apocalypse, live in Scandinavia now. The Vikings made jewellery out of them and took them home as souvenirs. Life goes on, or maybe not. Either way, let’s get on with it.
“If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs.”
Atomic weapons and energy aren’t a “that all worked out all right so don’t worry about it” sort of thing. A lot of people had to, and still have to, think about, talk about and work on atomic safety to allow us to live rather than die in the atomic age. Thinking and talking about and working on AI safety is the same.
I think this piece is often mistaken insofar as it engages in object-level discussion of AI capabilities and threat.
You say that an AI could not destroy the world because "destroying the world is a fundamentally stupid act". This is an obviously terrible argument. Destroying the world would be a stupid act for humans because it is contrary to our goals. If it helped you to achieve your goals, destroying the world would be the intelligent thing to do. We can easily imagine an intelligent alien species destroying the world for such a reason. There's no reason to believe that the "smart" goals are just whatever humans happen to want.
You say that AI could not exceed human intelligence because the upper limit on collective intelligence comes "not from our individual IQs but the information bandwidth of our society" and because the information bandwidth of society is capped by the physical world. But it's just a fact that human information throughput is very often bottlenecked by our own cognitive capacity first. Nothing physical prevented us from discovering, for example, calculus, Newton's laws, the theory of computation, germ theory, and so on, sooner than we did.
The claim that "Every prediction made by this cohort in my lifetime has been wrong by orders of magnitude" is false. Eliezer Yudkowsky's predictions about how and why AI agents would become agentic and misaligned have been completely vindicated recently by the Hugging Face incident and other similar incidents. Since you mention superforecasting, you should appreciate how accurate the forecasts of the AI 2027 authors have been with respect to the current state of the art in 2026. Some people may have made exaggerated claims about X% GDP growth by Y year but these types of claim really aren't central to the AI risk case.
The claim "my Claude is as stubbornly low-witted as ever" doesn't survive contact with the state-of-the-art in mathematics and coding. The rate of progress has been extraordinary. Three years ago AI couldn't reliably add or subtract and now AI agents are solving major open problems in mathematics. A very large fraction of coding tasks have been automated by AI agents.
It would be a mistake to dismiss AI risk without appreciating these points.