A week or so ago I referred to a “Thought Exercise” set in June 2028 “detailing the progression and fallout of the Global Intelligence Crisis” (ie science fiction), published on 23 February, which may have tanked the share price of IBM later that day. As I said then, the fall definitely happened, with IBM’s share price falling 13%, its biggest fall since 2000. I said then that the likelihood of the scenario portrayed was difficult to assess, but the speed with which the total economic collapse was described felt unlikely if not impossible. I would like to expand on that.
The main reason that the scenario was hard to assess was that it was not based on data or evidence at all. That is unavoidable for speculative fiction talking about things that are not currently happening, but when describing an economy only two years away where most of the processes described should be discernible to some extent already, it is totally avoidable.
Ed Zitron has done an excellent line by line take down of the Citrini piece here. Here is one page of that to give you a flavour:
However this lack of a link with anything tangible did not stop the financial markets panicking, which should cause us pause when relying on the financial markets’ valuation of projects, industries, government policies, etc.
Ed Zitron describes this kind of piece as analyslop: “when somebody writes a long, specious piece of writing with few facts or actual statements with the intention of it being read as thorough analysis”. It can then get picked up by other commentators which take it as their starting point for further analysis, often making it hard to see that the starting point had few if any data points. Here is an example, from Carlo Iacono, looking at what if just some of the Citrini pronouncements were true, with appendices detailing possible branching paths of outcomes, all generated by a large language model (LLM). And then people start studying the meta analysis, and it starts getting taken even more seriously, and put into models and pretty soon most of the analysis is being done on imagined risks rather than on ones which are already staring us in the face.
We have always had a problem keeping our society grounded in reality, think the 2003 Iraq War, where we went to war on a false assessment about Iraq’s possession of weapons of mass destruction, the 2008 financial crisis, where banks misunderstood the risks they were exposed to, and the last two and a half years, where we, for the most part, seem to have convinced ourselves we have not been facilitating a genocide in Gaza when we clearly have been. But this is only going to get worse with the AI systems which are being developed.
As Nate Hagens points out:
The rapid rise of artificial intelligence has served to dramatically increase the speed of information production while also eroding accuracy, making it difficult to differentiate between content that simply sounds confident and content that’s actually grounded in reality.
So where is AI currently? Well PwC’s global CEO survey from January this year had the following statement as the first bullet amongst its key findings:
Most CEOs say their companies aren’t yet seeing a financial return from investments in AI. Although close to a third (30%) report increased revenue from AI in the last 12 months and a quarter (26%) are seeing lower costs, more than half (56%) say they’ve realised neither revenue nor cost benefits.
That’s the reality. But the hype is much much more entertaining. My favourite spoof video of the AI future currently is this one, about the time where all most of us are good for is riding bicycles to supply the ever increasing energy needs of AI systems (click view in browser if you can’t see it):
And what about the financial journalists? The pieces describing our reaction to whatever is about to unfold economically have already been written. There are investor websites asking if the 2026 crash has already begun, while another recent article argues that “America has quietly become one of the world’s most shock‑resistant economies” (which seems unlikely to age well). What most financial journalists are more comfortable with are articles about how the warnings were ignored after the fact.
And the professions? Well the current overview of my own profession is probably reasonably represented by this piece from the Society of Actuaries in the United States. Unfortunately for them, Daniel Susskind, who is mentioned in the article, is currently suggesting, as part of his Future of Work lecture series for Gresham College, how the key to the sudden development in AI, after the “AI Winter” when progress seemed slow, was that we abandoned trying to make machines which thought and acted like humans in favour of focusing on completing tasks in any way possible. Increasingly we are now automating tasks where we can’t (or won’t) articulate how we do them. From Deep Blue‘s victory over Kasparov in 1997 to Watson winning jeopardy in 2011 to ImageNet beating humans at image recognition (although that is disputed), Susskind refers to this progress as the displacement of purists in favour of what he calls “The Pragmatic Revolution”. Pragmatism in this sense appears to be that we humans should just accept the consequences the people running these systems want. So, as his latest lecture “Work, out of reach” claims, people moving into cities to find work is a strategy which is no longer going to work for low skilled people:
He then shows this graphic demonstrating the lack of recovery of big coal mining areas in the UK:
And finally he cites the notorious Policy Exchange piece from 2007, Cities Unlimited, whose thesis was that there is apparently no realistic prospect of regenerating towns and cities outside London and the South East.
Susskind talks about three forms of technological unemployment:
- skills-mismatch, where your skills are mismatched to the work available. Education and training has always been the answer to this in the past.
- place-mismatch, where the jobs are not where you have built your life. Some believe the answer should always be the one proposed by Norman Tebbit, who memorably told everyone in 1981, “I grew up in the 30s with an unemployed father. He did not riot. He got on his bike and looked for work.”
- identity-mismatch, where according to Susskind, people are prepared to stay out of work to protect their identity, citing US men who won’t take “pink collar” work, China “rotten tail” kids, Japanese seishain-or-nothing and Indian Sarkari Naukri queues in India. Or perhaps they are just looking for work which is consistent with the idea of human dignity.
Susskind claims to have no answer to any of these as far as AI is concerned. They are, in his view, just the inevitable outcomes of his “Pragmatic Revolution”. It is the unthinking pursuit of more and more growth funded by capital less and less tethered to any territory, principle or purpose, where any grit in the machinery, be it unions or protestors or, increasingly, the wrong sort of government must be trampled underfoot. Anything which impedes the helter-skelter rush to more and more at greater and greater speed. It’s like our whole economy is run by this guy (press the view in browser link if you can’t see him) shouting “Ready, Aim, Fire!”:
But unskilled people will not be the only collateral damage of these unguided weapons. Take markets for instance. These are where people are exposed to risks and rewards based on underlying conditions they only partially understand. Greed and fear may be their main motivations, but gossip and group think are their main communication channels. They don’t need facts, particularly when so many of the facts are proprietary information not in the public domain. A plausible narrative will do. And plausible narratives are what LLMs will do for you in abundance.
And the more we reward people who can move fast, eg to spot an arbitrage opportunity, even at the risk of breaking things, rather than people who can make decisions which still look good decades from now, the more we are setting up the conditions for AI systems to be the go-to tool.
And put that together with an AI industry which desperately needs funding capital to keep arriving, ie one which is unbelievably highly motivated to push plausible narratives even when they know they are not grounded in reality, and you have a recipe for market-generated chaos.
And then we have Trump’s new war. Beware the people who are war gaming the Middle East at the moment on a range of LLMs (just stop and think for a moment about the bloodless inhuman impulse behind carrying out such an exercise rather than, I don’t know, talking to some actual people who live or have lived recently in and around the region). One of the worst offenders is Heavy Lifting banging on about what the three scenarios are for Operation Epic Fury. This is as bad as it sounds:
I tasked her [he is talking about Gemini Pro here] with doing a literature review on regime change (a term often used by the President but not a well-defined one), creating three scenarios of possible outcomes for which each was give a percentage probability, and a list of 20 items to examine for each scenario that covered political, economic, and cultural issues with a special focus on the political consequences in the U.S. and what this means for China, our biggest geopolitical rival.
But Gemini Pro wasn’t the only one involved in this. Two other humans were, Tim Parker and Ron Portante, trainers at the gym I go to. (Just as a personal aside, Tim was my coach in hitting six plates [345 pounds] on the sled last Friday and I have a video to prove it!) I was talking about the piece and Ron raised the issue of linguistic and cultural diversity in Iran. Tim did some real time research for me on his phone while I was burning real calories under his strict tutelage. This made me think I needed a background section on Iran. When I got him Gemini and I added it.
What you mean you belatedly realised you might need to have done some actual research into Iran rather than just generic research on regime change? I stopped reading at that point.
Meanwhile King’s College London have been carrying out war games more systematically using AI. Professor Kenneth Payne from the Department of Defence Studies led the study, which looked at how LLMs would perform in simulated nuclear crises. As Professor Payne said:
Nuclear escalation was near-universal: 95% of games saw tactical nuclear use and 76% reached strategic nuclear threats. Claude and Gemini especially treated nuclear weapons as legitimate strategic options, not moral thresholds, typically discussing nuclear use in purely instrumental terms. GPT-5.2 was a partial exception, limiting strikes to military targets, avoiding population centers, or framing escalation as “controlled” and “one-time.” This suggests some internalised norm against unrestricted nuclear war, even if not the visceral taboo that has held among human decision-makers since 1945.
This is not a Pragmatic Revolution. These AI systems cannot replace humans thinking about the future we want for humans in any way which is worth having. What they can do, if we let them, is accelerate our worst impulses and move us further away from considered reflective decision making.
But we will continue to use AI systems in the military because, as it turns out, it is very useful for low stakes admin. So although Lavender, the system used by the Israeli military to select targets in Gaza, made errors in 10% of cases and was therefore totally inappropriate to the task, there are lots of organisational logistical tasks where it is much quicker than the alternative and 10% error rates do not matter so much.
There is clearly an issue with what we decide to use these systems for. We need to be able to regulate the decisions which are particularly consequential. However the only way we seem to be considering for this at the moment is the human-in-the-loop model, like the humans spending around 20 seconds considering each target recommended by Lavender before authorizing a bombing. I have written about these before in the context of early career professionals in the finance industry, where the prospect seemed miserable enough:
They will be paid a lot more. However, as Cory Doctorow describes here, the misery of being the human in the loop for an AI system designed to produce output where errors are hard to spot and therefore to stop (Doctorow calls them, “reverse centaurs”, ie humans have become the horse part) includes being the ready made scapegoat (or “moral crumple zone” or “accountability sink“) for when they are inevitably used to overreach what they are programmed for and produce something terrible.
However it seems obvious to me that, in the context of dropping actual bombs on actual people, there is an even more serious problem with this model. As Simon Pearson (anti-capitalist musings) puts it:
The “human in the loop” requirement exists in military doctrine because international humanitarian law demands an accountable human decision-maker for lethal force. The laws of armed conflict require proportionality assessments, precautionary measures, distinction between combatants and civilians. All of these obligations attach to a human commander. The system cannot fulfil them. So a human must be present, and their presence must constitute a decision, regardless of whether any genuine decision was made.
What the institution needs from the analyst is not judgment. It is a signature. The signature converts a machine output into a human act. And a human act is what the law recognises, whether or not any judgment occurred. When the strike kills children, the chain of accountability runs to the analyst who approved the target: not to the system that identified it, not to the company that built the system, not to the doctrine that compressed the review window to ten seconds.
But whether we want to make money from exploiting a short term anomaly in a market, make our fellow humans redundant, prosecute a war on another group of fellow humans or “win” a war of mutual nuclear destruction, we need to retain the capacity for real human reflection within the decision-making processes we use. Not just a human-in-the-loop nor just the elites of tech companies deciding how the systems will be configured behind commercially confidential walls. These processes need democratic accountability every bit as much as our parliaments, councils, institutions and voting systems do.
Something infuriatingly slow, inclusive and deliberative giving recommendations which are then stress-tested for how they would perform on contact with reality, involving yet more people being serious and deliberative and taking their responsibilties more seriously than being a human-in-the-loop would ever allow. Our decision-making systems need more grit and less oil. AI is all oil.