Seven years ago I wrote about Catch 22 and actuarial practice, concluding, rather piously:
If we want far fewer actuaries to be employed in not growing alfalfa in the future and far more working on making the finance structures of our economy work better, whether to support a Green New Deal or more generally, we first need to embrace the idea that our current economic priorities are indeed insane.
So imagine my excitement at finding Catch 22 grabbed out of the pages of fiction and informing US foreign policy. Not convinced? Compare two passages. The first, from Catch 22, in 1961:
This time Milo had gone too far. Bombing his own men and planes was more than even the most phlegmatic observer could stomach, and it looked like the end for him. High-ranking government officials poured in to investigate. Newspapers inveighed against Milo with glaring headlines, and Congressmen denounced the atrocity in stentorian wrath and clamored for punishment. Mothers with children in the service organized into militant groups and demanded revenge. Not one voice was raised in his defense. Decent people everywhere were affronted, and Milo was all washed up until he opened his books to the public and disclosed the tremendous profit he had made. He could reimburse the government for all the people and property he had destroyed and still have enough money left over to continue buying Egyptian cotton. Everybody, of course, owned a share. And the sweetest part of the whole deal was that there really was no need to reimburse the government at all.
This week, the US Treasury lifted all oil sanctions on Iran. For 30 days. 140 million barrels of Iranian crude, sitting on ships at sea, may now be sold freely on the global market. Including to the United States itself.
In yuan.
The United States is purchasing, with Chinese currency, oil from the country it is currently bombing?! The same oil that funds the missiles that just shot down an F-35 for the first time. The same missiles that are redecorating allied oil infrastructure.
Treasury Secretary Bessent called this “narrowly tailored”. Narrow like in white, and tailored as in card, apparently.
In the same OFAC filing, Russian oil sanctions were lifted as well. And Belarus potash too, because apparently the universe was running low on irony and needed to top up.
The logic, insofar as there is any, goes like this: the war has crashed the global oil market so hard that the administration needs the enemy’s oil to keep gasoline prices from eating the midterms. They are unsanctioning the people they’re bombing because the bombing is working too well at the thing they didn’t want it to do. The sanctions were necessary to stop Iran funding the war, but the war made the sanctions too effective, so the sanctions had to be lifted to fund the war effort against the country that no longer needs sanctions because the oil revenues that sanctions were preventing are now required to prevent the economic damage caused by preventing those revenues, which is itself a consequence of the military campaign designed to make the sanctions unnecessary by making Iran the kind of country that doesn’t need sanctioning, which it would be, if the sanctions hadn’t been lifted to pay for making it that.
There have been many names thrown at Trump since he arrived in US politics. My personal favourite is probably the Tangerine Tyrant. Many people are currently relying on TACO (Trump Always Chickens Out) to resolve the Middle East crisis he has instigated. However, until now, I had not heard of anyone likening him to Milo Minderbender. But once you see it, it is difficult to un-see it.
Trump likes to give himself and everyone else nicknames. From the very stable genius of his first term, to more recently Honest Don and the Tariff King, whereas Milo, as M&M Enterprises (the company he started as the mess officer) expands, becomes the Mayor of Palermo, Assistant Governor-General of Malta, Vice-Shah of Oran, Caliph of Baghdad, Mayor of Cairo, and the god of corn, rain, and rice.
Trump likes to use his presidency to enrich himself, from his Trump coin to the Amazon documentary about his wife to his Board of Peace to all of his merchandise. Milo’s catchphrase is “what is good for M&M is good for the country”.
Trump doesn’t appear to believe in safety nets for ordinary people. Meanwhile Milo secretly replaces the CO2 cartridges in emergency life vests and the morphine in first aid kits with printed notes to the effect that what is good for M&M is good for the country.
Milo Minderbender is a war profiteer trying to convince himself that he is a free market fundamentalist. So what does that make Trump? Well hold that thought, because today’s Guardian has provided a partial answer I think, with a history of military targeting.
This introduces the concept of the kill chain, ie the process between detecting something and destroying it. Trying to shortcut the kill chain has been a perennial preoccupation of militaries through the ages. In the Vietnam War, Operation Igloo White dropped 20,000 acoustic and seismic sensors along the Ho Chi Minh trail, which transmitted data to relay aircraft, which then fed the signals to the IBM 360 computers at Nakhon Phanom airbase in Thailand. These analysed the data, predicted where the convoys would be and strikes were directed to those locations. The Viet Cong realised quickly that this system could not detect the difference between military vehicles and ox carts and therefore:
They played recordings of truck engines, herded animals near the sensors to trigger vibration detection, and hung buckets of urine in trees to set off the chemical detectors.
There was no way to independently check what they were destroying. The air force claimed 46,000 trucks were destroyed or damaged, which the CIA calculated exceeded the total number of trucks believed to exist in all of North Vietnam.
…air force personnel invented a creature to explain the absence. They called it the “great Laotian truck eater”.
Last time I talked about military targeting, I focused on the human in the loop, but let’s instead focus on the actual destruction going on for a moment, shall we? Trump’s assault on Iran hit 6,000 targets in two weeks. The kill chain had, apparently, been compressed so much that it allowed 1,000 decisions an hour. The school he hit, killing between 175 and 180 people, most of them girls between the ages of seven and 12, had changed its use to a school since at least 2016 and was visible on Google Maps. Old target lists had been reached for and noone had had the time or the inclination to check them before bombing them.
This is what you can expect from a Milo Minderbender presidency. It has been obvious, since at least the 1960s, that the US system requires enormous strength of purpose from its executive to hold its industrial-military complex in check. That is why so many of them have been so keen to install a Trump.
It feels as if, far from embracing the idea that our current economic priorities are indeed insane, as I fervently hoped seven years ago, we are instead doubling down on the insanity.
A week or so ago I referred to a “Thought Exercise” set in June 2028 “detailing the progression and fallout of the Global Intelligence Crisis” (ie science fiction), published on 23 February, which may have tanked the share price of IBM later that day. As I said then, the fall definitely happened, with IBM’s share price falling 13%, its biggest fall since 2000. I said then that the likelihood of the scenario portrayed was difficult to assess, but the speed with which the total economic collapse was described felt unlikely if not impossible. I would like to expand on that.
The main reason that the scenario was hard to assess was that it was not based on data or evidence at all. That is unavoidable for speculative fiction talking about things that are not currently happening, but when describing an economy only two years away where most of the processes described should be discernible to some extent already, it is totally avoidable.
Ed Zitron has done an excellent line by line take down of the Citrini piece here. Here is one page of that to give you a flavour:
However this lack of a link with anything tangible did not stop the financial markets panicking, which should cause us pause when relying on the financial markets’ valuation of projects, industries, government policies, etc.
Ed Zitron describes this kind of piece as analyslop: “when somebody writes a long, specious piece of writing with few facts or actual statements with the intention of it being read as thorough analysis”. It can then get picked up by other commentators which take it as their starting point for further analysis, often making it hard to see that the starting point had few if any data points. Here is an example, from Carlo Iacono, looking at what if just some of the Citrini pronouncements were true, with appendices detailing possible branching paths of outcomes, all generated by a large language model (LLM). And then people start studying the meta analysis, and it starts getting taken even more seriously, and put into models and pretty soon most of the analysis is being done on imagined risks rather than on ones which are already staring us in the face.
We have always had a problem keeping our society grounded in reality, think the 2003 Iraq War, where we went to war on a false assessment about Iraq’s possession of weapons of mass destruction, the 2008 financial crisis, where banks misunderstood the risks they were exposed to, and the last two and a half years, where we, for the most part, seem to have convinced ourselves we have not been facilitating a genocide in Gaza when we clearly have been. But this is only going to get worse with the AI systems which are being developed.
The rapid rise of artificial intelligence has served to dramatically increase the speed of information production while also eroding accuracy, making it difficult to differentiate between content that simply sounds confident and content that’s actually grounded in reality.
So where is AI currently? Well PwC’s global CEO survey from January this year had the following statement as the first bullet amongst its key findings:
Most CEOs say their companies aren’t yet seeing a financial return from investments in AI. Although close to a third (30%) report increased revenue from AI in the last 12 months and a quarter (26%) are seeing lower costs, more than half (56%) say they’ve realised neither revenue nor cost benefits.
That’s the reality. But the hype is much much more entertaining. My favourite spoof video of the AI future currently is this one, about the time where all most of us are good for is riding bicycles to supply the ever increasing energy needs of AI systems (click view in browser if you can’t see it):
And what about the financial journalists? The pieces describing our reaction to whatever is about to unfold economically have already been written. There are investor websites asking if the 2026 crash has already begun, while another recent article argues that “America has quietly become one of the world’s most shock‑resistant economies” (which seems unlikely to age well). What most financial journalists are more comfortable with are articles about how the warnings were ignored after the fact.
And the professions? Well the current overview of my own profession is probably reasonably represented by this piece from the Society of Actuaries in the United States. Unfortunately for them, Daniel Susskind, who is mentioned in the article, is currently suggesting, as part of his Future of Work lecture series for Gresham College, how the key to the sudden development in AI, after the “AI Winter” when progress seemed slow, was that we abandoned trying to make machines which thought and acted like humans in favour of focusing on completing tasks in any way possible. Increasingly we are now automating tasks where we can’t (or won’t) articulate how we do them. From Deep Blue‘s victory over Kasparov in 1997 to Watson winning jeopardy in 2011 to ImageNet beating humans at image recognition (although that is disputed), Susskind refers to this progress as the displacement of purists in favour of what he calls “The Pragmatic Revolution”. Pragmatism in this sense appears to be that we humans should just accept the consequences the people running these systems want. So, as his latest lecture “Work, out of reach” claims, people moving into cities to find work is a strategy which is no longer going to work for low skilled people:
He then shows this graphic demonstrating the lack of recovery of big coal mining areas in the UK:
Source: Left – Sheffield Hallam University map of coal mining areas; Right – % employment from Overman and Xu (2022)
And finally he cites the notorious Policy Exchange piece from 2007, Cities Unlimited, whose thesis was that there is apparently no realistic prospect of regenerating towns and cities outside London and the South East.
Susskind talks about three forms of technological unemployment:
skills-mismatch, where your skills are mismatched to the work available. Education and training has always been the answer to this in the past.
place-mismatch, where the jobs are not where you have built your life. Some believe the answer should always be the one proposed by Norman Tebbit, who memorably told everyone in 1981, “I grew up in the 30s with an unemployed father. He did not riot. He got on his bike and looked for work.”
identity-mismatch, where according to Susskind, people are prepared to stay out of work to protect their identity, citing US men who won’t take “pink collar” work, China “rotten tail” kids, Japanese seishain-or-nothing and Indian Sarkari Naukri queues in India. Or perhaps they are just looking for work which is consistent with the idea of human dignity.
Susskind claims to have no answer to any of these as far as AI is concerned. They are, in his view, just the inevitable outcomes of his “Pragmatic Revolution”. It is the unthinking pursuit of more and more growth funded by capital less and less tethered to any territory, principle or purpose, where any grit in the machinery, be it unions or protestors or, increasingly, the wrong sort of government must be trampled underfoot. Anything which impedes the helter-skelter rush to more and more at greater and greater speed. It’s like our whole economy is run by this guy (press the view in browser link if you can’t see him) shouting “Ready, Aim, Fire!”:
But unskilled people will not be the only collateral damage of these unguided weapons. Take markets for instance. These are where people are exposed to risks and rewards based on underlying conditions they only partially understand. Greed and fear may be their main motivations, but gossip and group think are their main communication channels. They don’t need facts, particularly when so many of the facts are proprietary information not in the public domain. A plausible narrative will do. And plausible narratives are what LLMs will do for you in abundance.
And the more we reward people who can move fast, eg to spot an arbitrage opportunity, even at the risk of breaking things, rather than people who can make decisions which still look good decades from now, the more we are setting up the conditions for AI systems to be the go-to tool.
And put that together with an AI industry which desperately needs funding capital to keep arriving, ie one which is unbelievably highly motivated to push plausible narratives even when they know they are not grounded in reality, and you have a recipe for market-generated chaos.
And then we have Trump’s new war. Beware the people who are war gaming the Middle East at the moment on a range of LLMs (just stop and think for a moment about the bloodless inhuman impulse behind carrying out such an exercise rather than, I don’t know, talking to some actual people who live or have lived recently in and around the region). One of the worst offenders is Heavy Lifting banging on about what the three scenarios are for Operation Epic Fury. This is as bad as it sounds:
I tasked her [he is talking about Gemini Pro here] with doing a literature review on regime change (a term often used by the President but not a well-defined one), creating three scenarios of possible outcomes for which each was give a percentage probability, and a list of 20 items to examine for each scenario that covered political, economic, and cultural issues with a special focus on the political consequences in the U.S. and what this means for China, our biggest geopolitical rival.
But Gemini Pro wasn’t the only one involved in this. Two other humans were, Tim Parker and Ron Portante, trainers at the gym I go to. (Just as a personal aside, Tim was my coach in hitting six plates [345 pounds] on the sled last Friday and I have a video to prove it!) I was talking about the piece and Ron raised the issue of linguistic and cultural diversity in Iran. Tim did some real time research for me on his phone while I was burning real calories under his strict tutelage. This made me think I needed a background section on Iran. When I got him Gemini and I added it.
What you mean you belatedly realised you might need to have done some actual research into Iran rather than just generic research on regime change? I stopped reading at that point.
Meanwhile King’s College London have been carrying out war games more systematically using AI. Professor Kenneth Payne from the Department of Defence Studies led the study, which looked at how LLMs would perform in simulated nuclear crises. As Professor Payne said:
Nuclear escalation was near-universal: 95% of games saw tactical nuclear use and 76% reached strategic nuclear threats. Claude and Gemini especially treated nuclear weapons as legitimate strategic options, not moral thresholds, typically discussing nuclear use in purely instrumental terms. GPT-5.2 was a partial exception, limiting strikes to military targets, avoiding population centers, or framing escalation as “controlled” and “one-time.” This suggests some internalised norm against unrestricted nuclear war, even if not the visceral taboo that has held among human decision-makers since 1945.
This is not a Pragmatic Revolution. These AI systems cannot replace humans thinking about the future we want for humans in any way which is worth having. What they can do, if we let them, is accelerate our worst impulses and move us further away from considered reflective decision making.
But we will continue to use AI systems in the military because, as it turns out, it is very useful for low stakes admin. So although Lavender, the system used by the Israeli military to select targets in Gaza, made errors in 10% of cases and was therefore totally inappropriate to the task, there are lots of organisational logistical tasks where it is much quicker than the alternative and 10% error rates do not matter so much.
There is clearly an issue with what we decide to use these systems for. We need to be able to regulate the decisions which are particularly consequential. However the only way we seem to be considering for this at the moment is the human-in-the-loop model, like the humans spending around 20 seconds considering each target recommended by Lavender before authorizing a bombing. I have written about these before in the context of early career professionals in the finance industry, where the prospect seemed miserable enough:
They will be paid a lot more. However, as Cory Doctorow describes here, the misery of being the human in the loop for an AI system designed to produce output where errors are hard to spot and therefore to stop (Doctorow calls them, “reverse centaurs”, ie humans have become the horse part) includes being the ready made scapegoat (or “moral crumple zone” or “accountability sink“) for when they are inevitably used to overreach what they are programmed for and produce something terrible.
However it seems obvious to me that, in the context of dropping actual bombs on actual people, there is an even more serious problem with this model. As Simon Pearson (anti-capitalist musings) puts it:
The “human in the loop” requirement exists in military doctrine because international humanitarian law demands an accountable human decision-maker for lethal force. The laws of armed conflict require proportionality assessments, precautionary measures, distinction between combatants and civilians. All of these obligations attach to a human commander. The system cannot fulfil them. So a human must be present, and their presence must constitute a decision, regardless of whether any genuine decision was made.
What the institution needs from the analyst is not judgment. It is a signature. The signature converts a machine output into a human act. And a human act is what the law recognises, whether or not any judgment occurred. When the strike kills children, the chain of accountability runs to the analyst who approved the target: not to the system that identified it, not to the company that built the system, not to the doctrine that compressed the review window to ten seconds.
But whether we want to make money from exploiting a short term anomaly in a market, make our fellow humans redundant, prosecute a war on another group of fellow humans or “win” a war of mutual nuclear destruction, we need to retain the capacity for real human reflection within the decision-making processes we use. Not just a human-in-the-loop nor just the elites of tech companies deciding how the systems will be configured behind commercially confidential walls. These processes need democratic accountability every bit as much as our parliaments, councils, institutions and voting systems do.
Something infuriatingly slow, inclusive and deliberative giving recommendations which are then stress-tested for how they would perform on contact with reality, involving yet more people being serious and deliberative and taking their responsibilties more seriously than being a human-in-the-loop would ever allow. Our decision-making systems need more grit and less oil. AI is all oil.
The Actuary magazine recently had a debate about whether the underlying data or the story you wove around it was more important. I’m not sure there is always a clear distinction between the two, as Dan Davies rather neatly illustrates here, but my view is that, if a binary choice has to be made, it is always going to be the story. And there was a great example of this which popped up recently in the FT.
The FT article was ‘Is university still worth it?’ is the wrong question, by John Burn-Murdoch, with great graphs as usual by John. However, as is sometimes the case, I feel that a very different and more convincing story could be wrapped around the same datasets he is showing us.
The article’s thesis is as follows:
The graduate earnings premium, ie how much more on average graduates earn than non-graduates, has only fallen in the UK as the proportion going to university has risen. It has risen in other countries:
In the UK, we have had much weaker productivity growth than the other comparator countries, and also “the steady ramping up of the minimum wage has squeezed the earnings premium from the lower end too”:
We have also had a much smaller increase in the percentage of managerial and professional jobs than a different group of comparator countries (they haven’t mentioned Germany before), meaning graduates are forced to take lower salaried jobs elsewhere:
So the answer according to the FT? We should focus on economic growth rather than “tweaking” higher education intake and funding. Then graduate earnings would be higher, student loans could be more generous(!) and students would have more chance of getting a good job.
Well perhaps. But here’s a different framing of the same data that I find more persuasive.
Let’s start by addressing that point about the minimum wage. According to the House of Commons Library report on this, the UK’s minimum wage is broadly comparable to that of France and the Netherlands, although higher than Canada’s and much higher than that of the United States. The employers who are the FT’s constituency would obviously like us lower down this particular chart:
The main economic framing here is the progress myth of the UK’s business community: economic growth. All problems can be solved if we can just get more economic growth. Apparently we need more inequality in pay between graduates and non-graduates which we can get by generating more economic growth. This is honest of them at least, although I don’t see much evidence that the economic growth they crave will go into skilled job creation rather than stock buy backs (according to Motley Fool, “Companies spent $249 billion on stock buybacks in Q3 2025, and $777 billion over the first three quarters of 2025.”).
There are a lot of problems with framing every economic question with respect to economic growth, memorably illustrated by Zack Polanski of the Green Party in this less than 3 minute video recently (I strongly recommend you watch it before you read on – click on the read in browser link if you can’t see it):
Economic growth is increasingly without purpose, wasteful of energy and poorly distributed. It is chasing outputs, literally any outputs, whatever the cost to the environment, our health system, our education system, our social support systems and our communities. Looking at the framing above, you can see that economic growth as currently pursued will always see anything which stops the concentration of wealth amongst the already wealthy, like a higher national minimum wage or a totally made-up concept like a lower graduate earnings premium (which in itself is a framing trying to make reducing inequality seem undesirable) as a problem. Lack of productivity growth, itself a proxy for this kind of economic growth (because if you ask why we need more productivity the answer is always to get more economic growth), is usually directed as a criticism at “lazy” UK workers, rather than under-investing and over-extracting UK business owners.
But what if, instead of economic growth, your progress myth was reducing inequality? Or growing equality within the economy?
Source: World Inequality Database wid.world
If you focused on inequality rather than economic growth, then you would find it correlates with everything we say we don’t want. Unlike economic growth, having equality as an aim actually has the advantage of having an evidence base for the claim that it improves society:
If you focused on inequality, then you would be pleased that we have had an increase in our minimum wage. You would think that the same FT article’s admission that UK graduates’ skills levels are higher than those in the United States was more important than something called a graduate earnings premium.
Burn-Murdoch is right to say asking whether university is worth it is the wrong question.
However economic growth is the wrong answer.
And I thought I would probably be stopping there for this week. But then something odd happened. A “Thought Exercise” set in June 2028 “detailing the progression and fallout of the Global Intelligence Crisis” (ie science fiction), published on 23 February, may have tanked the share price of IBM later that day. The fall definitely happened, with IBM’s share price falling 13%, its biggest fall since 2000, alongside smaller falls in other tech stocks.
Investors have recently seized on social media rumours and incremental developments by small AI companies to justify further selling, with a widely circulated blog post by Citrini Research over the weekend describing how AI could hypothetically push the US unemployment rate above 10 per cent by 2028, proving the latest catalyst.
The likelihood of the scenario portrayed is difficult to assess, but the speed with which the total economic collapse happens subsequently as described feels unlikely if not impossible. However the fact that the markets are this jittery tells us something I think. As Carlo Iacono puts it:
We are living through a period in which the gap between “plausible narrative” and “tradeable signal” has collapsed to nearly nothing. When a scenario feels real enough to model, and the underlying anxiety is already there waiting to be organised, fiction and forecast become functionally indistinguishable.
The data underlying the markets hasn’t changed, but the story has. I rest my case.
The book I finished this weekend had a passage about a first experiment with a new substance which could shield against gravity. Mr Cavor, the rather unworldly scientist, is explaining to Mr Bedford, a man with no obvious talents other than to look for a quick buck where he can find one, what would have happened if his substance, Cavorite, had not got dislodged fairly quickly from where they had positioned it:
“You perceive,” he said, “it formed a sort of atmospheric fountain, a kind of chimney in the atmosphere. And if the Cavorite itself hadn’t been loose and so got sucked up the chimney, does it occur to you what would have happened?”
I thought. “I suppose,” I said, “the air would be rushing up and up over that infernal piece of stuff now.”
“Precisely,” he said. “A huge fountain—”
“Spouting into space! Good heavens! Why, it would have squirted all the atmosphere of the earth away! It would have robbed the world of air! It would have been the death of all mankind! That little lump of stuff!”
“Not exactly into space,” said Cavor, “but as bad—practically. It would have whipped the air off the world as one peels a banana, and flung it thousands of miles. It would have dropped back again, of course—but on an asphyxiated world! From our point of view very little better than if it never came back!”
I stared. As yet I was too amazed to realise how all my expectations had been upset. “What do you mean to do now?” I asked.
“In the first place if I may borrow a garden trowel I will remove some of this earth with which I am encased, and then if I may avail myself of your domestic conveniences I will have a bath. This done, we will converse more at leisure. It will be wise, I think”—he laid a muddy hand on my arm—“if nothing were said of this affair beyond ourselves. I know I have caused great damage—probably even dwelling-houses may be ruined here and there upon the country-side. But on the other hand, I cannot possibly pay for the damage I have done, and if the real cause of this is published, it will lead only to heartburning and the obstruction of my work. One cannot foresee everything, you know, and I cannot consent for one moment to add the burden of practical considerations to my theorising…”
In case you are in any doubt, Dario Amodei is our Mr Cavor here. I can just imagine his response to the first disaster attributed to AI research being prefaced by “one cannot foresee everything, you know…”. And there are too many Mr Bedfords out there to shake a stick at, trying to sell you anything they can possibly attribute to AI just to keep the whole thing rolling along.
I am with the fog people. The FT seem to be too, with this pair of diagrams attached to this article.
First the US, where there are tentative signs of something they can possibly use as a proxy for productivity growth as a result of using AI:
And so it was this foggy sensibility about AI which I took with me to the Bletchley Park Museum last weekend, site of the AI Safety Summit in November 2023 which drew in the US Vice President, Kamala Harris, European Commission President Ursula von der Leyen, Elon Musk, then UK Prime Minister Rishi Sunak, Open AI’s Sam Altman, Meta’s Nick Clegg and Prof Yann LeCun, Meta’s chief AI scientist, amongst around 100 guests invited to suck their teeth about AI.
The thing that particularly struck me at Bletchley Park is that it demystified the emergence of the computer for me. The forerunner, which was the mechanisation using punch cards of the process of sorting the massive amounts of data the centre was receiving in war time, smacks of a group of people who had just run out of wall to spread their webs of cards and strings across. It was a crime investigation which had got out of hand.
A highlight for me was Alan Turing’s very prescient little note about AI, written in 1940 but anticipating the arguments which would be raging by 2026 (and how poignant that the man who probably did more than anyone to transform what we are able to do by punching a keyboard was chained to one that could only press hunks of metal against a strip of carbon onto a piece of paper):
There is also a hilarious secrecy pledge from the ancestors of the safety summit people, telling you all the ways in which you just need to shut up:
“There is an English proverb none the worse for being seven centuries old:” it thunders.
Wicked tongue breaketh bone,
Though the tongue itself hath none.
Words to live by, I’m sure we’d all agree.
What Bletchley Park was less good at was explaining how the Enigma code was cracked, despite an excellent collection of the hardware involved. For that, I recommend Simon Singh’s The Code Book.
Here was the world’s first “intelligence factory”, scaling up intelligence gathering and analysis as never before and by so doing also changing the way governments would interact with their populations, with just as many implications for our current times as the development of AI. This cluster of huts around a country house rebranded as GCHQ and moved to Cheltenham a few years after World War 2.
Path dependence is a term which describes a situation where past events or decisions constrain later events or decisions. Bletchley Park feels like the Museum of Path Dependence to me.
And the legacy of the safety summit? Well my “hot take” would be: when you are a little lost in the fog, it is generally advisable to slow down a bit and take steps to reduce your risk of breaking things. I wonder if I can get that on a bumper sticker.
In ordering #5, self-driving cars will happily drive you around, but if you tell them to drive to a car dealership, they just lock the doors and politely ask how long humans take to starve to death.Source: https://m.xkcd.com/1613/
My attention was drawn this week to an article by Dario Amodei, co-founder of Anthropic (a spin off from OpenAI, which was co-founded by Elon Musk and heavily invested in by Microsoft so very much part of the Magnificent 7 architecture), the creator of the large language model Claude, called The Adolescence of Technology. It is hard to overemphasise how much I disagree with everything Dario has written here, but also useful in that it is a long article, which covers a lot of ground, and allows me to define my views in opposition to it.
The irritations start pretty much straight away. So Dario quotes from a science fiction classic (Carl Sagan’s First Contact), but then follows this up under the heading of “Avoid doomerism” with this:
…but it’s my impression that during the peak of worries about AI risk in 2023–2024, some of the least sensible voices rose to the top, often through sensationalistic social media accounts. These voices used off-putting language reminiscent of religion or science fiction, and called for extreme actions without having the evidence that would justify them.
Notice the word “sensible” doing the heavy lifting there. Only science fiction endorsed by Dario will be considered. Dario wants us to consider the risks of AI in “a careful and well-considered manner”, which sounds reasonable, but then his 3rd and final bullet under this (after “avoid doomerism” and “acknowledge uncertainty”) goes as follows:
Intervene as surgically as possible. Addressing the risks of AI will require a mix of voluntary actions taken by companies (and private third-party actors) and actions taken by governments that bind everyone. The voluntary actions—both taking them and encouraging other companies to follow suit—are a no-brainer for me. I firmly believe that government actions will also be required to some extent, but these interventions are different in character because they can potentially destroy economic value or coerce unwilling actors who are skeptical of these risks (and there is some chance they are right!).
So reflexively anti regulation of his own industry, of course. And voluntary actions by corporations, an approach to solving problems which has been demonstrated not to work repeatedly, is apparently “a no-brainer”. Also it is automatically assumed that government actions will destroy value. Only market solutions will be endorsed by Dario, pretty much until they have messed up so badly you are forced to bring governments in:
To be clear, I think there’s a decent chance we eventually reach a point where much more significant action is warranted, but that will depend on stronger evidence of imminent, concrete danger than we have today, as well as enough specificity about the danger to formulate rules that have a chance of addressing it. The most constructive thing we can do today is advocate for limited rules while we learn whether or not there is evidence to support stronger ones.
There is then the expected sales pitch about what he has seen within Anthropic about the relentless “increase in AI’s cognitive capabilities”. And then the man who warned about sensationalist science fiction is off:
I think the best way to get a handle on the risks of AI is to ask the following question: suppose a literal “country of geniuses” were to materialize somewhere in the world in ~2027. Imagine, say, 50 million people, all of whom are much more capable than any Nobel Prize winner, statesman, or technologist.
And the rest of the article is then off solving this imaginary problem in all its facets, rather than the wealth and power concentration problem that we actually have. The only legislation he seems to be in favour of seems to be something called “transparency legislation”, legislation which of course Anthropic would help to write.
However, after suggesting everything from isolating China and using “AI to empower democracies to resist autocracies” to private philanthropy as the solutions to his imagined problems, Dario finally and reluctantly concludes government intervention might after all be necessary as follows:
…ultimately a macroeconomic problem this large will require government intervention. The natural policy response to an enormous economic pie coupled with high inequality (due to a lack of jobs, or poorly paid jobs, for many) is progressive taxation. The tax could be general or could be targeted against AI companies in particular. Obviously tax design is complicated, and there are many ways for it to go wrong. I don’t support poorly designed tax policies. I think the extreme levels of inequality predicted in this essay justify a more robust tax policy on basic moral grounds, but I can also make a pragmatic argument to the world’s billionaires that it’s in their interest to support a good version of it: if they don’t support a good version, they’ll inevitably get a bad version designed by a mob.
That, by the way, is what Dario thinks of democracy: “a bad version designed by a mob” rather than the “good version” that he and his fellow billionaires could come up with in their own self interest. The mask has really slipped by this point. And the following section, on “Economic concentration of power”, just demonstrates that he has no effective answers at all that he deems acceptable on this. It’s just an inevitability for him.
This is what Luke Kemp’s excellent Goliath’s Curse refers to as a “Silicon Goliath”. Goliaths are dominance hierarchies which spread by dominating the areas around them. They need three conditions (which Luke calls “Goliath fuel”): lootable resources (ie resources which can be easily stolen off someone else), caged land (ie land difficult to escape from) and monopolizable weapons (ie ones which require processes which can be developed to give one society an edge over another). We are all Goliath-dwellers in “The West” now, looting resources from other countries in unequal exchanges which impoverish the Global South, with weapons (eg nuclear weapons) available only to the elite few countries and operating within the cages of heavily-policed national boundaries. The Silicon Goliath which is developing will have data as its lootable resource, mass surveillance systems providing its cages and monopolizable weapons such as killer drones. The resultant killbot hellscapes which people like Dario Amodei laughably imagine they have defences against through things like their Claude’s Constitution are almost pitiful in their inadequacy.
Nate Hagens takes Dario’s claims for AI’s cognitive capabilities much more seriously than me, and then considers the risks in a less adolescent way here. As he says:
And here’s what his essay has almost nothing about. Energy, water, materials, or ecological limits.
And also nowhere does Dario talk about the 99% of people who are just spectators in his world, other than to describe them as “the mob”. This is quite a blind spot, as Luke Kemp points out in his exhaustive study of the collapses of “Goliaths” over the last 5,000 years. “The extreme levels of inequality” predicted by Amodei in his essay are not just things we have to put up with, but the reasons the world he predicts is likely to be hugely unstable. Not created by AI, but accelerated by it. Kemp describes it as “diminishing returns on extraction”:
We see a pattern re-emerging across case studies. Societies grow more fragile over time and more prone to collapse. Threats that they had always faced such as invaders, disease and drought seem to take a heavier toll.
As societies grew bigger:
They still faced the underlying (and ongoing) problem of rising inequality creating societies where and institutions more extractive power was more concentrated.
And eventually:
The result is more extractive institutions creating growing instability, internal conflict, a drain of resources away from government, state capture by private elites, and worse decision-making. Society – especially the state – becomes more fragile. Private elites tend to take a larger share of extractive benefits. The state, and many of the power structures it helps prop up, then usually falls apart once a shock hits: for Rome it was climate change, disease, and rebelling Germanic mercenaries; for China it was often floods, droughts, disease and horseback raiders; for the west African kingdoms it was invaders and a loss of trade; for the Maya it was drought and a loss of trade; and for the Bronze Age it was drought, a disruption of trade and an earthquake storm.
The only real answer to combatting existential risks in the hands of adolescents like the Tech Bros is more democracy: over control of decision-making, over control of resources, over control of the threat of violence and over control of information. We are a long way from achieving these within our own particular Goliath at the moment, and indeed there is no sign at all that our elites are interested in achieving them. The Magnificent 7 are propping up the US stock exchange. The promise of perpetual economic growth is the progress myth of our time and leaders who do not provide it will lose the “Mandate of Heaven” in just the same way as Chinese rulers did when they were unable to prevent floods and droughts. Adam Tooze sees the signs of the inner demons of our elites starting to detach them from reality in the latest disclosures from the Epstein files:
Are we, like [Larry] Summers, fantasizing about stabilizing our desires and needs in an inherently dangerous and uncertain world? Are we kidding ourselves?
But, without those controls in place, we would need a lot more than Dario’s Anthropic playing nicely to allow this particular adolescent to grow up. And this is where I am forced to take Nate Hagens’ assessment more seriously. Because if our rulers’ Mandates of Heaven are dependent on eternal economic growth on their watch and they, rightly, think that this is not possible in our current non-AI-enhanced world but, wrongly, think it is possible in a future AI-enhanced world, then that is the way they are going to demand we go. And, if the Larry Summers fantasists really are kidding themselves, it may be very hard to talk them out of it.
Happy new year all! New year, new banner, courtesy of my brilliant daughter who presented me with a plausible 3-D model of my very primitive cartoon of a reverse-centaur over Christmas. And I thought I would kick off with a relatively uncontentious subject: examinations!
“Back to normal!” That was the cry throughout education when the pandemic had finally ended enough for us to start cramming students into rooms again. The universities had all leveraged themselves to the maximum, and perhaps beyond, to add to the built estate, so as to entice students in both the overseas and the uncapped domestic market to their campuses, and one by-product of this was they had plenty of potential examination halls. So let’s get away from all of that electronic remote nonsense and get everyone in a room together where you can keep an eye on them and stop them cheating. This united the purists who yearned for the days of 10% of the cohort turning up for elite education via chalk and talk rather than the 50% we have today, senior management needing to justify the size of the built estate and politicians who kept referring to traditional exams in an exam hall as the “gold standard”.
So, in a time when students have access to information, tools, how to videos of everything imaginable, the entire output of the greatest minds of thousands of years of human history, as well as many of the less than great minds, in short anything which has ever caught anyone’s attention and been committed to some form of media: in this of all times, we want to sort the students into categories for the existing job market based on how they answer academic questions about what they can remember unaided about the content of their lecture courses and reading lists with a biro on a pad of paper perched precariously on a tiny wooden table surrounded by hundreds of other similar scribblers, for a set period of time as minders wander the floors like Victorian factory owners.
And for institutions that thought the technology we fast-tracked for education delivery and assessment in the pandemic would surely be part of education’s future? Or perhaps they just can’t afford to borrow half a billion or have the the land available to construct more cathedrals of glass and brick to house more examination halls? Simple! We just create the conditions for that gold standard examination right there in the student’s own bedroom or the company they work for!
There are 54 pages to the Institute and Faculty of Actuaries’ (IFoA’s) guidance for remotely invigilated candidates. It covers everything from the minimum specification of equipment you need, including the video camera to watch your every movement and the microphone to pick up every sound you make, to the proprietary spying software (called “Guardian Browser”) you will need to download onto your own computer, how to prove who you are to the system, what you are allowed to have in your bedroom with you and even how you need to sit for the duration of the exam (with a maximum of two 5 minute breaks) to ensure the system has sufficient visibility of you at all times:
These closed book remote arrangements replaced the previous open book online exams which most institutions operated during the pandemic. The reason given was that the exam results shot up so much that widespread cheating was suspected and the integrity of the qualifications was at risk. The IFoA’s latest assessment regulations can be found here.
The belief in examinations is very widespread. A couple of months ago I was discussing the teacher assessments which replaced them briefly during the pandemic with a secondary business studies teacher. He took great pride in the fact that he based his assessments solely on mock results, ie an assessment carried out before all of the syllabus had been covered and when students were unaware it would be the final assessment. But still in his mind more “objective” than any opinion he might have of his own students.
If a large language model can perform enormously better in an examination than your students can without it, what it actually demonstrates is that the traditional examination is woefully unprepared for the future. As Carlo Iacono puts it:
The machines learned from us.
They learned what we actually valued and it turned out to be different from what we said we valued.
We said we valued originality. We rewarded conformity to genre. We said we valued depth. We measured surface features. We said we valued critical thinking. We gave higher marks to confident assertion than to honest uncertainty.
So now the machines produce what the world trained them to produce: fluent, confident, passable output that fits the shapes we reward.
And we’re horrified. Not because they stole something from us. Because they showed us what the systems were selecting for all along.
The scandal isn’t that a model can imitate student writing. The scandal is that we built an educational and professional culture where imitation passes as competence, and then acted shocked when a machine learned to imitate faster.
We trained the incentives. We trained the rubrics. We trained the career ladders.
The pattern recognition which gets you through most formal examinations is just too cheap and easy to automate now. It is no longer a useful skill, even by proxy. It might as well be Hogwarts’ sorting hat for all the use it is in a post scarcity education world. If the machines have worked out how to unlock the elaborate captcha system we have placed around our gold standard assessments, an arms race of security measures protecting a range of tests which look increasingly narrow compared to the capabilities which matter does not seem like the way to go.
What instead we are doing is identifying which students are prepared to put themselves through literally anything to get the qualification. Companies like students like that. They will make ideal reverse-centaurs. The description of life as a reverse-centaur even sounds like the experience of a proctored exam:
Like an Amazon delivery driver, who sits in a cabin surrounded by AI cameras, that monitor the driver’s eyes and take points off if the driver looks in a proscribed direction, and monitors the driver’s mouth because singing isn’t allowed on the job, and rats the driver out to the boss if they don’t make quota.
The driver is in that van because the van can’t drive itself and can’t get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance. But the driver is human, so the van doesn’t just use the driver. The van uses the driver up.
And, even if you are OK with all of that, all of these privacy intrusions don’t even work to prevent cheating! The ACCA, the world’s largest accounting professional body, has just announced it is stopping all remote exams after giving up the arms race against the cheats, facilitated in some cases seemingly by their Big Four employers lying about what had gone on.
Actuarial exams started in 1850, only 2 years after the Institute of Actuaries was established (Dermot Grenham wrote about them recently here). This pre-dated the establishment of the first examination boards by a few years (1856 Society of Arts, the Society for the encouragement of Arts, Manufactures and Commerce, later the Royal Society for the encouragement of Arts, Manufactures and Commerce (Royal Society of Arts); 1857: University of Oxford Delegacy of Local Examinations (founded by the University of Oxford); and 1858: University of Cambridge Local Examinations Syndicate (UCLES, founded by the University of Cambridge)), so keen were actuaries to institute examinations. However it was the massive expansion of the middle classes as the Industrial Revolution disrupted society in so many ways that led to the need for a new sorting hat beyond the capacity of the oral examinations that had previously been the norm.
Now people seem to be lining up to drag everyone back into the examination hall. Any suggestion of a retreat from traditional exams is met by howls of outrage from people like Sam Leith at The Spectator about lack of “rigour”. However, in my view, they are wrong.
Yes of course you can isolate students from every intellectual aid they would normally use, as a centaur, to augment their performance, limit the sources they can access, force them to rely on their own memories entirely, and put them under significant time pressure. You will definitely reduce marks by doing that. So that has made it harder and therefore more rigorous and more objective, right?
Well according to the Merriam-Webster dictionary, rigorous is a synonym of rigid, strict or stringent. However, while all these words mean extremely severe or stern, rigorous implies the imposition of hardship and difficulty. So promoting exams above all as an exercise in rigour reveals their true nature as a kind of punishment beating in written form, for which the prize for undergoing it is whatever it qualifies you for. Suddenly the sorting hat looks relatively less arbitrary.
The problems of traditional exams are well known, but the most important ones in my view are that they measure a limited range of abilities and therefore are unlikely to show what students can really achieve. Harder does not mean more objective. It is like deciding who can act by throwing students out, one at a time, in front of a baying mob of, let’s say for argument, readers of The Spectator. Sure, some of the students might be able to calm the crowd, some may even be able to redirect their anger towards a different target. But are the people who can play Mark Antony for real necessarily the best all-round actors? And has someone who can only stand frozen on the spot under those circumstances really proved that they could never act well?
It also means that education ends a month or more before the exams, to allow the appropriate cramming, followed by engaging all of the teaching staff in the extended exercise of marking, checking and moderating what has been written in answer to academic questions about what the students can remember unaided about the content of their lecture courses and reading lists with a biro on a pad of paper perched precariously on a tiny wooden table surrounded by hundreds of other similar scribblers, for a set period of time as minders wander the floors like Victorian factory owners. But what if instead the assessment was part of the teaching process? What if students felt that their assessment had been a meaningful part of their educational experience? What if, instead of arguing the toss over whether they scored 68% or 70% on an assessment, students could see for themselves whether they had demonstrated mastery of their subject.
One model of assessment which is getting a lot of attention at the moment, one I am a big fan of having used it at the University of Leicester on some modules, is something called interactive oral assessment, where students meet with a lecturer or tutor, individually or in a small group, and answer questions about work they have already submitted. It is a highly demanding form of assessment, for both the students and the assessors, but it means the final assessment is done with the student present and, with careful probing from the assessors, who will obviously need to have done a close reading of the project work beforehand, you can be highly confident of the degree to which the student understands the work they have submitted. It also allows the student to submit a piece of work of more complexity and ambition than can be accommodated by a traditional exam. And it needn’t take any more time if the interviews are carried out online when set against the exam marking time of the traditional exam. Something which all the technology we developed through the pandemic allows us to do, without the need for spyware.
There are other models which also assess the technological centaurs we wish our students to become rather than the reverse-centaurs we are currently dooming too many to become. It is looking like it may be time to start telling students to stop writing and to put down their pens on the traditional exam. And perhaps the actuarial profession, who led us into the era of professional written examinations so enthusiastically 175 years ago, might now want to take the lead in navigating our way out of them?
The rear view mirror isn’t going to help us any more Source: Wikimedia Commons: Shattered right-hand side mirror on a 5-series BMW in Durham, North Carolina by Ildar Sagdejev
I would like to start this week’s post with a quote from Carlo Iacono, from a Substack piece he did a couple of weeks ago called The Questions Nobody Is Funding:
What is a human being for? What do we owe the future? What remains worth the difficulty of learning?
These are not questions you will find in the OECD’s AI Literacy Framework. They are not addressed in the World Economic Forum’s Education 4.0 agenda. They do not appear in the competency matrices cascading through national education systems. Instead, we get learning objectives and assessment criteria. Employability outcomes and digital capabilities. The language of preparation, as if the future were already decided and our job were simply to ready people for it.
I think this articulates well the central challenge of AI for education. Whether you think this is the beginning of a future where augmented humans move into a different type of existence to any we have known before; or you believe very little will be left behind in the rubble from the inevitable burst of the AI bubble when it comes and will be, at least temporarily, forgotten in the most devastating stock market crash and depression for a century; or you hold both these beliefs at the same time; or you are somewhere in between, it is difficult to see how the orderly world of competency matrices, learning objectives, assessment criteria, employability outcomes and digital capabilities can easily survive the period of technological, cultural, economic and political disruption which we appear to have entered. Looking in the rear view mirror and trying to extrapolate what you see into the future is not going to work for us any more.
Whether you think, like Cory Doctorow, in his recent speech at the University of Washington called The Reverse Centaur’s Guide to Criticizing AI, that:
AI is the asbestos in the walls of our technological society, stuffed there with wild abandon by a finance sector and tech monopolists run amok. We will be excavating it for a generation or more.
Technologies such as LLMs are neither going to transcend humanity as the holdouts on one side still hope, nor disappear, as other holdouts might like. We’re going to have to figure out ways to talk about them better and more clearly.
We are certainly going to have to figure out ways to talk about LLMs and other forms of AI more clearly, so that the decisions we need to make about how to accommodate them into society can be made with the maximum level of participation and consensus. And this seems to be the key for me with respect to education too. We do need people graduating from our education system understanding clearly what LLMs can and cannot do, which is a tricky path to navigate at the moment as a lot of money is being concentrated on persuading you that it can do pretty much anything. One example here has created a writers’ room of four LLMs where they are asked to critique each other by pushing the output from one into the prompts for the others, reminiscent of The Human Centipede. Which immediatel reminded me of this take from later in that Cory Doctorow speech:
And I’ll never forget when one writer turned to me and said, “You know, you prompt an LLM exactly the same way an exec gives shitty notes to a writers’ room. You know: ‘Make me ET, except it’s about a dog, and put a love interest in there, and a car chase in the second act.’ The difference is, you say that to a writers’ room and they all make fun of you and call you a fucking idiot suit. But you say it to an LLM and it will cheerfully shit out a terrible script that conforms exactly to that spec (you know, Air Bud).”
So, back to Carlo’s little questions:
What is a human being for?
A lofty question certainly, and not one I am going to tackle in a blog post. But perhaps I can say a bit about what a human being is not for. This is the key to Henry Farrell’s piece which is his take on the humanist critique of AI. We are presumably primarily designing the future for humans. All humans. Not just Tech Bros. And the design needs to bear that in mind. For example, a human being is not, in my opinion, for this (from the Cory Doctorow link):
Like an Amazon delivery driver, who sits in a cabin surrounded by AI cameras, that monitor the driver’s eyes and take points off if the driver looks in a proscribed direction, and monitors the driver’s mouth because singing isn’t allowed on the job, and rats the driver out to the boss if they don’t make quota.
The driver is in that van because the van can’t drive itself and can’t get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance. But the driver is human, so the van doesn’t just use the driver. The van uses the driver up.
The first task of the education establishment, I think, is to attempt to protect the graduate from becoming the reverse-centaur described above, whether a deliver driver, a coder (where additionally the human-in-the-loop becomes the accountability sink for everything the AI gets wrong) or a radiologist. This will often be resisted by the employers you are currently very sensitive to the needs of as educators (many of who are senior enough to get to use the new technologies as a centaur rather than be used by them as a reverse-centaur, tend to struggle to put themselves in anyone else’s shoes and, frankly, can’t see what all the fuss is about) but, remember, the cosy world of employability outcomes is over. The employers are not sticking to the implicit agreement to employ your graduates if you delivered the outcomes and therefore neither should you. Your responsibility in education is to the students, not their potential future employers, now their interests no longer appear to be aligned.
What do we owe the future?
This depends on what you mean by “the future” of course. If it is some technological dystopia of diminished opportunities for most (even for making friends as seemingly envisioned by some of the top Tech Bros), then nothing at all. But if it is the future which is going to support your children and their children, you obviously owe it a lot. But what do you owe it? What is owed is often converted into money by the political right, and used to justify not running up public debt in the present so as not to “impoverish” future generations. What that approach generally achieves is to impoverish both the current and future generations.
But if you think of owing resources, institutions and infrastructure to the next generation, then that is a responsibility that we should take seriously. And part of that is to produce an educated generation with tools, systems, institutions and infrastructure. The education institutions must take steps to make sure they survive in a relevant way, embedded in systems which support individuals and proselytising the value of education for all. They must ensure that their graduates understand and have facility with the essential tools they will need, and have developed the ability to learn new skills as they need them, and realise when that is. This is about developing individuals who leave no longer dependent on the institutions, able to work things out for themselves rather than requiring never-ending education inside an institution.
What remains worth the difficulty of learning?
The skills already mentioned will be the core ones for everyone, and these will need to be hammered out in terms everyone can understand. But in the world of post scarcity education, which is here but which we have not yet fully embraced, the rest will be up to us. A large part of the education of the future will need to be about equipping us all to understand what we now have access to and when and how to access it. We will all have different things we are interested in, or end up involved with and needing to be educated about. It will be up to each of us to decide which things are worth the difficulty of learning, but to make those decisions we will need education that can support the development of judgement.
For education institutions, the question will be what is not worth the difficulty of learning? Credentialising based on now relatively meaningless assessment methods will not cut it. This is where the confrontation with employers and politicians is likely to come. Essential skills and their related knowledge will be better developed and assessed via more open-ended project work and online assessment of it to check understanding. These will need to become the norm, with written examinations becoming less and less prevalent. Not because of fear of cheating and plagiarism, but because an outcome which can be replicated that easily by AI is not worth assessing in the first place.
So this is my 42nd blog post of the year and the 8th where I have referenced Cory Doctorow. Thought it was more to be honest, so influential has he been on my thought, particularly as I have delved deeper into what, how and why the AI Rush is proceeding and what it means for the people exiting universities over the next few years.
Yesterday Cory published a reminder of his book reviews this year. He is an amazing book reviewer. There are 24 on the list this year, and I want to read every one of them on the strength of his reviews alone.
I would like to repay the compliment by reviewing his latest book: Enshittification (the other publication this year – Picks and Shovels – is also well worth your time by the way). Can’t believe this wasn’t the word of the year rather than rage bait, as it explains considerably more about the times we are living in.
I have been a fan of Doctorow for a couple of years now. I had had Walkaway sat on my shelves for a few years before I read it and was immediately enthralled by his tale of a post scarcity future which had still somehow descended into an inter-generational power struggle hellscape. I moved on to the Little Brother books, now being reenacted by Trump with his ICE force in one major US city after another. Followed those up with The Lost Cause, where the teenagers try desperately to bridge the gap across the generations with MAGA people, with tragic results along the way but a grim determination at the end “the surest way to lose is to stop running”. From there I migrated to the Marty Hench thrillers, his non-fiction The Internet Con (which details the argument for interoperability, ie the ability of any platform to interact with another) and his short fiction (I loved Radicalised, not just for the grimly prophetic Radicalised novella in the collection, but also the gleeful insanity of Unauthorised Bread). I highly recommend them all.
I came to Enshittification after reading his Pluralistic blog most days for the last year and a half, so was initially disappointed to find very little new as I started working my way through it. However what the first two parts – The Natural History and The Pathology – are is a patient explanation of the concept of enshittification and how it operates assuming no previous engagement with the term, all in one place.
Enshittifcation, as defined by Cory Doctorow, proceeds as follows:
First, platforms are good to their users.
Then they abuse their users to make things better for their business customers.
Next, they abuse those business customers to claw back all the value for themselves.
Finally, they have become a giant pile of shit.
So far, so familiar. But then I got to Part Three, explaining The Epidemiology of enshittification, and the book took off for me. The erosion of antitrust (what we would call competition) law since Carter. “Antitrust’s Vietnam” (how Robert Bork described the 12 years IBM fought and outspent the US Department of Justice year after year defending their monopolisation case) until Reagan became President. How this led to an opening to develop the operating system for IBM when it entered the personal computer market. How this led to Microsoft, etc. Then how the death of competition also killed Big Tech regulation ( regulating a competitive market which acts against collusion is much easier than regulating one with a small number of big players which absolutely will collude with each other).
And then we get to my favourite chapter of the book “Reverse-Centaurs and Chickenisation”. Any regular reader of this blog will already be familiar with what a reverse centaur is, although Cory has developed a snappy definition in the process of writing this book:
A reverse-centaur is a machine that uses a human to accomplish more than the machine could manage on its own.
And if that isn’t chilling enough for you, the description of the practices of poultry packers and how they control the lives of the nominally self-employed chicken farmers of the US, and how these have now been exported to companies like Amazon and Arise and Uber, should certainly be. The prankster who collected up the bottled piss of the Amazon drivers who weren’t allowed a loo break and resold it on Amazon‘s own platform as “a bitter lemon drink” called Release Energy, which Amazon then recategorised as a beverage without asking for any documentation to prove it was fit to drink and then, when it was so successful it topped their sales chart, rang the prankster up to discuss using Amazon for shipping and fulfillment – this was a rare moment of hilarity in a generally sordid tale of utter exploitation. My favourite bit is when he gets on to the production of his own digital rights management (DRM) free audio versions of his own books.
The central point of the DRM issue is, as Cory puts it, “how perverse DMCA 1201 is”:
If I, as the author, narrator, and investor in an audiobook, allow Amazon to sell you that book and later want to provide you with a tool so you can take your book to a rival platform, I will be committing a felony punishable by a five-year prison sentence and a $500,000 fine.
To put this in perspective: If you were to simply locate this book on a pirate torrent site and download it without paying for it, your penalty under copyright law is substantially less punitive than the penalty I would face for helping you remove the audiobook I made from Amazon’s walled garden. What’s more, if you were to visit a truck stop and shoplift my audiobook on CD from a spinner rack, you would face a significantly lighter penalty for stealing a physical item than I would for providing you with the means to take a copyrighted work that I created and financed out of the Amazon ecosystem. Finally, if you were to hijack the truck that delivers that CD to the truck stop and steal an entire fifty-three-foot trailer full of audiobooks, you would likely face a shorter prison sentence than I would for helping you break the DRM on a title I own.
DMCA1201 is the big break on interoperability. It is the reason, if you have a HP printer, you have to pay $10,000 a gallon for ink or risk committing a criminal offence by “circumventing an access control” (which is the software HP have installed on their printers to stop you using anyone else’s printer cartridges). And the reason for the increasing insistence on computer chips in everything from toasters (see “Unauthorised Bread” for where this could lead) to wheelchairs – so that using them in ways the manufacturer and its shareholders disapprove of becomes illegal.
The one last bastion against enshittification by Big Tech was the tech workers themselves. Then the US tech sector laid off 260,000 workers in 2023 and a further 100,000 in the first half of 2024.
In case you are feeling a little depressed (and hopefully very angry too) at this stage, Part 4 is called The Cure. This details the four forces that can discipline Big Tech and how they can all be revived, namely:
Competition
Regulation
Interoperability
Tech worker power
As Cory concludes the book:
Martin Luther King Jr once said, “It may be true that the law cannot make a man love me, but it can stop him lynching me, and I think that’s pretty important, also.”
And it may be true that the law can’t force corporate sociopaths to conceive of you as a human being entitled to dignity and fair treatment, and not just an ambulatory wallet, a supply of gut bacteria for the immortal colony organism that is a limited liability corporation.
But it can make that exec fear you enough to treat you fairly and afford you dignity, even if he doesn’t think you deserve it.
And I think that’s pretty important.
I was reading Enshittification on the train journey back from Hereford after visiting the Hay Winter Weekend, where I had listened to, amongst others, the oh-I’m-totally-not-working-for-Meta-any-more-but-somehow-haven’t-got-a-single-critical-word-to-say-about-them former Deputy Prime Minister Nick Clegg. While I was on the train, a man across the aisle had taken the decision to conduct a conversation with first Google and then Apple on speaker phone. A particular highlight was him just shouting “no, no, no!” at Google‘s bot trying to give him options. He had already been to the Vodaphone shop that morning and was on his way to an appointment which he couldn’t get at the Apple Store on New Street in Birmingham. He spotted the title of my book and, when I told him what enshittification meant, and how it might make some sense out of the predicament he found himself in, took a photo of the cover.
My feeling is that enshittification goes beyond Big Tech. It is the defining industrial battle of our times. We shouldn’t primarily worry about whether it is coming from the private or the public sector, as enshittification can happen in both places: from hollowing out justice to “paying more for medicines… at the exact moment we can’t afford to pay enough doctors to prescribe them” in the public sector, where we already reside within the Government’s walled garden, to all of the outrages mentioned above and more in the private sector.
The PFI local health hubs set out in last week’s budget take us back to perhaps the ultimate enshittificatory contracts the Government ever entered into, certainly before the pandemic. The Government got locked into 40 year contracts, took all the risk, and all the profit was privatised. The turbo-charging of the original PFI came out of the Blair-Brown government’s mania for keeping capital spending off the balance sheet in defence of Gordon Brown’s “Golden Rule” which has now been replaced by Rachel Reeves’ equally enshittifying fiscal rules. All the profits (or, increasingly, rents, as Doctorow discusses in the chapter on Varoufakis’ concept of Technofeudalism) from turning the offer to shit always seem to end up in the private sector. The battle is against enshittification from both private and, by proxy, via public monopolies.
Enshittification is, ultimately, a positive and empowering book which I strongly recommend you buy, avoiding Amazon if you can. We can have a better internet than this. We can strike a better deal with Big Tech over how we run our lives. But the surest way to lose is to stop running.
And next time a dead-eyed Amazon driver turns up at your door, be nice, they are probably having a worse day than you are.
A couple of weeks ago I wanted to find an article I had written about heat pumps to check something. So I Googled weknow0 and heat pump. This did give me the article, from December 2022, I was after, but also an “AI overview” that I hadn’t requested. The above is what it told me.
Now this is inaccurate on a number of counts. Firstly, I have published 226 articles over the more than 12 years I have been writing on weknow0.co.uk and I have only mentioned heat pumps in two of these. These articles did focus on the points mentioned in 3 of the 4 bullet points above and in one of them I also set out how the market at the time (December 2022) was stacked against anyone acquiring a heat pump, a state of affairs which has thankfully improved considerably since. However to claim that my blog “provides a consumer-focused perspective in the practicalities and challenges of domestic heat pump adoption in the UK” is clearly hilarious.
In fact anyone seeing that would assume I talked about little other than heat pumps, so I decided to do a search on something else that I talk about infrequently and see what I got (I searched “weknow0 science fiction”):
This seems a considerably better summary of the recent activity on the blog, which is also unrecognisable as the blog summarised in response to the previous search.
Right at the end, it suggests a reason for the title of the blog which isn’t an unreasonable guess from a regular reader. But guess it still is, and it does not appear to have processed the significant number of blog posts with variants of we know zero in the title to fine tune its take.
So someone using the AI overview as a research tool would get a completely different view of what the blog was about depending upon which other word they used alongside weknow0. Perhaps that doesn’t matter too much to anyone other than me in this case, but it is part of a broader issue. It is not summarising the website it is suggesting it is summarising.
Of course many of you will now be shouting at me that I need to give the system more focused prompts. There is now a whole area of expertise, lectured in and written about at considerable length, called “prompt engineering”. There are senior professionals who have rarely given their juniors the time of day for years, giving the tersest responses to their completely reasonable queries about the barely intelligible instructions they have given for a piece of work, suddenly prepared to spend hours and hours on prompt engineering so that the Metal Mickey in their phone or laptop can give them responses closer to what they were actually looking for.
At this point, perhaps we should perhaps hear from Sundar Pichai, the Google CEO:
As part of Faisal Islam’s slightly gushing interview with Pichai, we learn that the AI overview on Google is “prone to errors” and needs to be used alongside such things as Google search. “Use them for what they are good at but don’t blindly trust them” he says of his tools which he admits to currently investing $90 billion a year in. This is of course a problem, as one of the reasons people are reluctantly resorting to the AI overview is because the basic Google search has become so enshittified.
And that kind of echoes what Cory Doctorow has said about Google. Google need to maintain a narrative about growth. You will have picked this up if you watched the Pichai interview above, from the breathless stuff about “one of the most powerful men in the world” “perhaps being one of the easier things for AI to replicate one day” to:
You don’t want to constrain an economy based on energy. That will have consequences.
To the even more breathless stuff about us being 5 years from quantum computing being where generative AI is now.
The reason for all the growth talk, according to Doctorow, is that Google needs to be growing for it to be able to maintain a price earnings ratio of 20 to 1, rather than the more typical 4 to 1 of a mature business. So it’s all about the share price. As Doctorow says:
Which is why Google is so desperately sweaty to maintain the narrative about its growth. That’s a difficult narrative to maintain, though. Google has 90% Search market-share, and nothing short of raising a billion humans to maturity and training them to be Google users (AKA “Google Classroom”) will produce any growth in its Search market-share. Google is so desperate to juice its search revenue that it actually made search worse on purpose so that you would have to run multiple searches (and see multiple rounds of ads) before you got the information you were seeking.
Investors have metabolized the story that AI will be a gigantic growth area, and so all the tech giants are in a battle to prove to investors that they will dominate AI as they dominated their own niches. You aren’t the target for AI, investors are: if they can be convinced that Google’s 90% Search market share will soon be joined by a 90% AI market share, they will continue to treat this decidedly tired and run-down company like a prize racehorse at the starting-gate.
This is why you are so often tricked into using AI, by accidentally grazing a part of your screen with a fingertip, summoning up a pestersome chatbot that requires six taps and ten seconds to banish: companies like Google have made their product teams’ bonuses contingent on getting normies to “use” AI and “use” is defined as “interact with AI for at least ten seconds.” Goodhart’s Law (“any metric becomes a target”) has turned every product you use into a trap for the unwary.
So here we are. AI isn’t meant for most of you, its results are “prone to errors” and need to be used alongside other corroborating material or “human validation”. It needs you to take a course in prompt engineering even if you never did the same to manage any of your human staff. It is primarily designed to persuade investors to keep the share price up to the levels the Board of Alphabet Inc have become accustomed to.
New (left) and old (right) Naiku shrines during the 60th sengu at Ise Jingu, 1973, via Bock 1974
In his excellent new book, Breakneck, Dan Wang tells the story of the high-speed rail links which started to be constructed in 2008 between San Francisco and Los Angeles and between Beijing and Shanghai respectively. Both routes would be around 800 miles long when finished. The Beijing-Shanghai line opened in 2011 at a cost of $36 billion. To date, California has built only a small stretch of their line, as yet nowhere near either Los Angeles or San Francisco, and the latest estimate of the completed bill is $128 billion. Wang uses this, amongst other examples to draw a distinction between the engineering state of China “building big at breakneck speed” and the lawyerly society of the United States “blocking everything it can, good and bad”.
Europe doesn’t get much of a mention, other than to be described as a “mausoleum”, which sounds rather JD Vance and there is quite a lot about this book that I disagree with strongly, which I will return to. However there is also much to agree with in this book, and none more so than when Wang talks about process knowledge.
Wang tells another story, of Ise Jingu in Japan. Every 20 years exact copies of Naiku, Geku, and 14 other shrines here are built on vacant adjacent sites, after which the old shrines are demolished. Altogether 65 buildings, bridges, fences, and other structures are rebuilt this way. They were first built in 690. In 2033, they will be rebuilt for the 63rd time. The structures are built each time with the original 7th century techniques which involve no nails, just dowels and wood joints. Staff have a 200 year tree planting plan to ensure enough cypress trees are planted to make the surrounding forest self-sufficient. The 20 year intervals between rebuilding are the length of the generations, the older passing on the techniques to the younger.
This, rather like the oral tradition of folk stories and songs, which were passed on by each generation as contemporary narratives until they were all written down and fixed in time so that they quickly appeared old-fashioned thereafter, is an extreme example of process knowledge. What is being preserved is not the Trigger’s Broom of temples at Ise Jingu, but the practical knowledge of how to rebuild them as they were originally built.
Process knowledge is the know-how of your experienced workforce that cannot easily be written down. It can develop where such a workforce work closely with researchers and engineers to create feedback loops which can also accelerate innovation. Wang contrasts Shenzhen in China where such a community exists, with Silicon Valley where it doesn’t, forcing the United States to have such technological wonders as the iPhone manufactured in China.
China has recognised the supreme importance of process knowledge as compared to the American concern with intellectual property (IP). IP can of course be bought and sold as a commodity and owned as capital, whereas process knowledge tends to rest within a skilled workforce.
This may then be the path to resilience for the skilled workers of the future in the face of the AI-ification of their professions. Companies are being sold AI systems for many things at the moment, some of which will clearly not work with few enough errors, or without so much “human validation” (a lovely phrase a good friend of mine actively involved in integrating AI systems into his manufacturing processes used recently) that they are not deemed practical. For early career workers entering these fields the demonstration of appropriate process knowledge, or the ability to develop it very quickly, may be the key to surviving the AI roller coaster they face over the next few years. Actionable skills and knowledge which allow them to manage such systems rather than being managed by them. To be a centaur rather than a reverse-centaur.
Not only will such skills make you less likely to lose your job to an AI system, they will also increase your value on the employment market: the harder these skills and knowledge are to acquire, the more valuable they are likely to be. But whereas in the past, in a more static market, merely passing your exams and learning coding might have been enough for an actuarial student for instance, the dynamic situation which sees everything that can be written down disappearing into prompts in some AI system will make such roles unprotected.
Instead it will be the knowledge about how people are likely to respond to what you say in a meeting or write in an email or report, and the skill to strategise around those things, knowing what to do when the rules run out, when situations are genuinely novel, ie putting yourself in someone else’s shoes and being prepared to make judgements. It will be the knowledge about what matters in a body of data, putting the pieces together in meaningful ways, and the skills to make that obvious to your audience. It will be the knowledge about what makes everyone in your team tick and the skills to use that knowledge to motivate them to do their best work. It will ultimately be about maintaining independent thought: the knowledge of why you are where you are and the skill to recognise what you can do for the people around you.
These have not always been seen as entry level skills and knowledge for graduates, but they are increasingly going to need to be as the requirement grows to plug you in further up an organisation if at all as that organisation pursues its diamond strategy or something similar. And alongside all this you will need a continuing professional self-development programme on steroids going on to fully understand the systems you are working with as quickly as possible and then understand them all over again when they get updated, demanding evidence and transparency and maintaining appropriate uncertainty when certainty would be more comfortable for the people around you, so that you can manage these systems into the areas where they can actually add value and out of the areas where they can cause devastation. It will be more challenging than transmitting the knowledge to build a temple out of hay and wood 20 years into the future, and will be continuous. Think of it as the Trigger’s Broom Process of Career Management if you like.
These will be essential roles for our economic future: to save these organisations from both themselves and their very expensive systems. It will be both enthralling and rewarding for those up to the challenge.
