I didn’t understand statistics until I started taking actuarial exams that required me to master particular statistical techniques, decide which ones were appropriate to the problem I was looking at, apply them accordingly and be able to interpret what the results did and didn’t tell me. An A at GCE O level in maths, and A at GCE A level in both maths and further maths and a maths degree from Oxford did not give me those abilities.
I didn’t understand economics until I started putting together modules which could be taught on both BSc and MSc courses and then teaching them. My economics module on the way to qualifying as an actuary, based very heavily on Economics by Begg, Fischer and Dornbusch, did not give me that understanding.
There is a pattern here: the most impactful experiences we have are frequently at a bit of a distance from the credentials we present to the world. Your education is not a paragraph on your CV, it is your lived experience, sometimes assisted by, sometimes actively hindered by and often pursued completely independently of the educational institutions you have had a relationship with during your life.
Daniel Susskind has marched into this often fraught relationship between credentials and actual actionable skills, knowledge and experience in the last lecture of his Gresham College series on The Future of Work, called Education – And Its Limits. His analysis is a very clear expression of the problems that will be created if AI systems prove to be half as capable and long-lasting as people from OpenAI and Anthropic are telling us they will be.
Susskind argues that trying to future-proof students through education was a hopeless task and that working on the assumption of unresolvable uncertainty was the better way forward. He suggests a “no regret” strategy for education would focus on the basics, which he describes as literacy, numeracy and critical thinking, and the critical use of AI. Others in the audience suggested some other “basics”: communication skills for instance.
The other strand of Susskind’s basics was critical use of AI. And his challenge to the audience was whether we can teach AI without losing the basics in the process.
After a bit about the need to make continuing education in later life more accessible, he looked at his list from a previous lecture (which I briefly touched on here) of problems for a post AI future:
Distribution (replacing wages);
Contribution (how do you “pull your weight”);
Power (domination by Big Tech on economics, politics, liberty, social justice and democracy); and
Meaning (fulfilment in life).
And this is why I continue to watch Susskind’s output, because he is the unusual combination of an extremely orthodox economist (see his book Growth: A Reckoning for proof of this) and someone who has been wrestling with the challenges of more capable systems to our way of doing things for over 10 years. What Susskind gives you is a peek at how the actual economists advising our governments would deal with things if OpenAI and Anthropic are proved right. It is as if you had someone who both thought, as Ben Bernanke, Chair of the Federal Reserve, did in October 2007, that “the banking system is healthy” and also that the banking collapse was going to happen anyway.
Because what he reveals is that, if Anthropic are right, orthodox economists have really got no policy prescriptions worthy of the name.
On 1. Distribution: Susskind acknowledges that, if the labour market could not redistribute wealth effectively any more, then a bigger state would be needed to do so (but he was at pains to emphasise that this would not be the 20th century central planning type of state).
On 2. Contribution: Susskind thinks that perhaps we should allow people to make non-economic contributions! As if all of the activity in society which economists routinely ignore really wasn’t already happening!
On 3. Power: Susskind says the political power of Big Tech with regard to liberty, social justice and democracy is a problem. We have anti-trust legislation that can deal with Big Tech’s economic power, but not its political power. I don’t know what kind of political power he thinks Big Tech would have without the economic power that has been granted it by a steady erosion of that anti-trust legislation over recent years.
On 4. Meaning: he has little to say other than something about us currently having policies for work but not for leisure.
And in all of this, there is the implicit underlying assumption of a stable future environment for all of this tech to operate within.
For me it brought to mind something a friend of mine who was in a tent with Cory Doctorow at the How The Light Gets In festival at Hay-on-Wye last week. “He has a theory of change” he said.
He really does, including about how to break the economic power of the Big Tech companies. You all need to read Enshittification for the full account, but my review of the book here is a sneaky peek.
Susskind really really does not have a theory of change. Which tells me that the economics profession does not have one either.
However I continue to watch him as I find he goads me into thinking what some better answers might be to the questions he asks. And perhaps also a better question than whether we can teach AI without losing the basics in the process. I think a better question would be what do we need to learn when the future is uncertain.
First of all, let’s remind ourselves of the problem. If noone cares how your advice was constructed, but your client can get advice that ticks the compliance box more cheaply and quickly from an AI system, while the experienced professional still has some role in managing the process, it may increasingly be a struggle to justify the cost of the junior colleague. So the future education system is going to need to help that future junior colleague demonstrate their value in ways they haven’t historically needed to. AI has not brought new problems, it has accelerated existing ones. And the gap between actual actionable skills, knowledge and experience and the credentials which are supposed to represent them is currently the key one as far as that future junior colleague is concerned.
The temptation is to rush into syllabus changes towards what currently looks like the cutting edge activity. I agree with Susskind here that this would be a mistake. He cites the example of Michael Gove, amongst many other education ministers at the time, mandating the teaching of coding in 2014. Now we find that the new AI systems (despite the problems highlighted by Hannah Fry, Kyle Kingsbury and others I talked about here) are most suited to writing code and Anthropicclaim that Claude is now writing 80% of its own code. Programmers are saying that, on the famous XKCD cartoon above, they are now living on the theory curve.
But both higher education institutions and the professions who still want to be in the game of developing the next generation of professionals can do a lot more both to reduce the gap between credentials and actual actionable skills, knowledge and experience and to make it clearer to employers that they have done so. That will be the subject of my next post.
Source: Nick Foster – December 2013 – originally drawn to deride George Osborne’s austerity as Pugwash Economics, repurposed now as I am worried about the actuarial profession pulling up the ladder on the next generation
The “black box” was a constant refrain when I was working as an actuarial consultant. It was where the results from a process were being accepted without any understanding of how they were arrived at. Something we felt that any self-respecting actuarial consultant should challenge in their own work and everybody else’s.
However when you came to actually present analysis or arguments to a client, you expected a certain amount of that expertise to be taken as read, to effectively be inside a black box as far as the client was concerned. They couldn’t be expected to understand all of the aspects of what you were talking about, otherwise they wouldn’t need you. Good practice was always to put them in a position where they could understand and make decisions about the key aspects of your advice without needing to engage with the other parts. As the expert, you decided what was in the black box.
Now the black box is back with a vengeance for all the professionals who have relied upon them in their working lives. As Dan Davies puts it:
The same black-box property which stops you from being second guessed or overruled means that nobody is interested in your explanations for your decisions; it is definitional of being a black box that you are going to be judged by results.
And, if you are in the business of advising in the teeth of uncertainty, as actuaries are, then this is likely to be a real problem. If noone cares how your advice was constructed, but they can get advice that ticks the compliance box your client has to complete more cheaply and quickly than you can, the more automated black box is going to win the business. The experienced professional still has a role in managing this process, verifying the results coming out of the black box and determining what can still be kept out of the black box, but he may be increasingly struggling to justify the cost of his junior colleague.
Well things don’t look so bad in the UK right now according to the Office for National Statistics (ONS), reverting to close to the average after a post pandemic surge in the finance and insurance sector:
However, if we look at the United States, which tends to show us where the UK finance sector is going, it looks far more ominous:
Yesterday Sky News ran a story about Standard Chartered‘s CEO who, in his desperation not to describe over 7,500 job losses as cost cutting, said this:
It’s not cost-cutting. It’s replacing in some cases lower-value human capital with the financial capital and the investment capital we’re putting in.
We may need to sit with that statement for a little while.
Daniel Susskind talks about this risk in his latest lecture entitled A World Without Work: in summary, to paraphrase only slightly, sure relatively junior white collar roles may already be particularly hard hit by AI, but he is optimistic because of the impact on GDP and we cannot pause because of China. He then goes on to talk about the four problems he sees for a post AI future:
Distribution (replacing wages);
Contribution (how do you “pull your weight”);
Power (domination by Big Tech on economics, politics, liberty, social justice and democracy); and
Meaning (fulfilment in life).
Susskind has gone from thinking that the fear that AI is coming for your jobs was overblown and that it was just task encroachment that we faced, to now thinking that it may encroach on all the tasks in most fields. Jevons Paradox (that technological innovation that increases the efficiency of a resource’s use leads to a rise in consumption of that resource) is no comfort if that new demand is robot-met.
Carlo Iacono suggests that the move of junior roles to AI may be subtle to begin with:
The weakness among young workers may appear as fewer people entering employment from outside the workforce. Firms may not fire large numbers of juniors; they may simply hire fewer of them.
That matters. The labour market can look healthy while the entry path narrows. Senior workers stay employed. Output rises. Productivity improves. There is no dramatic wave of redundancies.
Yet the first rung is being taken out.
It may also be masked by the fact that there remains a shortage for actuaries beyond the entry roles. There is almost a hint of desperation to approaches like this looking for introductions from a retired actuary like myself:
(followed by a list of clients he is working for)
Source: recruitment consultant who will remain anonymous. I am assuming “candies” are candidates
And, even if you think the risk of the AI Bubble bursting soon, taking down the global stock markets underpinned by the Magnificent 7, is exaggerated, you do need to be suspicious about the current abilities of AI to replace junior staff. My experience with another, somewhat earlier, actuarial technology, the pensions valuation engine, would suggest that the outputs need to be analysed very carefully before sharing with a client: it often had dependencies between what should have been independent variables hidden in the programming, or vagaries in the setup which left out non-standard benefit rules for your particular scheme, for instance. Or the student who had set it up initially (a complicated process usually) might have made a mistake or you might have not communicated with them very well to start with. Or a hundred other things.
For whatever reason, there was often still a lot to do after the valuation engine had produced some output.
Can this sort of thing happen with agentic AI? Well think about that student programming the valuation engine, but on steroids. Its patchy capabilities combined with its basic psychopathy leads to, as Hannah Fry entertainingly demonstrates here, some serious problems arising with the agent’s relentless to and fro with the large language models it depends upon, asking them what it should do next. As Hannah says:
I built an AI agent. She opened a shop selling novelty mugs, emailed a journalist without being asked, and then leaked our passwords to a total stranger.
As Kyle Kingsbury wrote about having an AI agent as a colleague in a programming team:
Imagine a co-worker who generated reams of code with security hazards, forcing you to review every line with a fine-toothed comb. One who enthusiastically agreed with your suggestions, then did the exact opposite. A colleague who sabotaged your work, deleted your home directory, and then issued a detailed, polite apology for it. One who promised over and over again that they had delivered key objectives when they had, in fact, done nothing useful. An intern who cheerfully agreed to run the tests before committing, then kept committing failing garbage anyway. A senior engineer who quietly deleted the test suite, then happily reported that all tests passed.
You would fire these people, right?
Yet despite all this, the money continues to pour in to the construction of AI infrastructure. There are already websites up and running for all of the parts of tasks AI cannot encroach upon.
The bottom rung of the actuarial ladder is clearly in danger. This is a particular problem for the actuarial profession, which has traditionally relied on longer periods of work-based training for its future qualified actuaries than many other professions. Training to become an actuary takes a long time. Median time to fellowship is still around six years, with some taking up to ten or giving up. The exams are hard to pass. There have been attempts by the profession to tackle some of these disincentives: the Chartered Actuary designation to make a destination of the generalist qualification before the specialisation of the fellowship, championed on this blog and launched in the teeth of opposition by some fellows, being one example.
It has led to a culture within actuarial firms around managing the extended time in training, with rituals around study leave and results days. One of the fears expressed in opposition to the introduction of the Chartered Actuary designation was that, if this could be achieved almost entirely within formal education at universities, the value of working alongside experienced actuaries would be lost.
It has led to a culture within the profession itself of managing large parts of its education system in house. Half of its revenue and around 30% of its expenditure are on “pre-qualification learning and development”. Sometimes it looks more like an education business with a professional side hustle.
But then the new AI toys have come along, and it turns out that many of those experienced actuaries may be less keen on graduates coming in and needing supervision from them after all. Many of them may rather spend hours on AI prompts than on developing another human being.
I fear that, increasingly, companies are not going to accommodate actuarial students in their work plans without significant persuasion. And, if the number of students studying while in work falls, the profession itself is going to struggle to finance its own bespoke education system at an acceptable cost to its members.
It will be hard for the profession to challenge this too: it is going to be good for many of those already established in their roles as the market for more experienced actuaries, when the market has no interest in developing the actuaries of the future, becomes increasingly competitive.
If the actuarial profession does accept the challenge of protecting the pipeline of future experienced actuaries it will need to review its entire education syllabus through this lens. It will also need to engage with other partners involved in what is in effect a problem of capital formation and collective action: government incentives may be needed to encourage firms to continue to train early career professionals and discourage free-riding. There may be no way back for the student with no actuarial qualifications learning on the job. The universities may be needed to plug people in at a different career point, which will require them to innovate themselves even further into the professional training role than ever before. As Carlo Iacono points out:
educational institutions may be pushed to simulate more of the apprenticeship environment. That does not mean adding a thin “AI literacy” module. It means creating settings where students practise judgement under uncertainty, in realistic workflows, with feedback that is close enough to hurt and useful enough to teach.
It will not be at all easy. But the alternative is a future without opportunity for those who do not already have it and an ageing profession withering on the vine it refused to nurture.
The Wetherspoons pub The Mary Shelley in Bournemouth
A few months ago I decided to read Mary Shelley’s Frankenstein for the first time. I also watched Guillermo del Toro’s Frankenstein, on a big screen, despite, according to The New Yorker, it having been “Netflixed down to size”.
Shelley’s book is largely monologues of interior thoughts of Frankenstein and his creation, with wildly careering emotions and death, death, death everywhere – perhaps unsurprising from an author whose mother died 10 days after giving birth to her, who lost a child and whose half sister died by suicide while she was working on Frankenstein, with much more tragedy to follow after its publication. There is a word Mary Shelley uses more than I have read in any other book: variants of sympathy/sympathise/sympathies turn up 32 times. Because of course one of the many things the book is all about is mutual incomprehension of the creator and the created.
Last week I was in Bournemouth as a last minute substitute for Lanzarote, something I may come back to at a later date, and I stumbled across the churchyard of St Peter’s Church in which Mary Shelley was buried, along with the cremated heart of her husband Percy Shelley, at the age of 53. There is also a pub in Bournemouth named after her (above) but whose sign depicts the monster from her most famous piece of writing.
As we enter another time of mutual incomprehension of the creator and the created, I have been reading the surprisingly-difficult-to-access paper by Kyle Kingsbury (the systems engineer, not the MMA guy) called The Future of Everything is Lies, I Guess. I will put a link to an X account which shared it here, as going to the aphyr.com site to read it seems to generate this message:
Once you can read it though, it starts to sketch out a likeness of our current monster and chip away a little at the human side of the mutual incomprehension. I am talking, of course, about what people are currently calling “AI”, which Kingsbury defines as:
…a family of sophisticated Machine Learning (ML) technologies capable of recognizing, transforming, and generating large vectors of tokens: strings of text, images, audio, video, etc. A model is a giant pile of linear algebra which acts on these vectors. Large Language Models, or LLMs, operate on natural language: they work by predicting statistically likely completions of an input string, much like a phone auto-complete. Other models are devoted to processing audio, video, or still images, or link multiple kinds of models together.
The article sets out how this is a technology where nobody really understands why it has been successful or how to make it better, which falls into strange loops or attractors, has odd gaps in its capabilities and is highly sensitive to slight changes in its formatting. It is a technology which is simultaneously highly capable and an idiot. And Kingsbury worries that our culture is not ready for such a technology. As he says:
As LLMs etc are deployed in new situations, and at new scale, there will be all kinds of changes in work, politics, art, sex, communication and economics. Some of these effects will be good. Many will be bad. In general, ML promises to be profoundly weird.
Buckle up.
He continues:
Most people seem concerned with conscious, motivated threats: AIs could realize they are better off without people and kill us. I am concerned that ML systems could ruin our lives without realizing anything at all.
There follow extensive examples of the problems the various ML applications are already starting to cause and some speculation about where things may be going in various areas of our lives before we get to the chapter on work. And the subject of hiring “AI employees”. This is probably my favourite bit:
Imagine a co-worker who generated reams of code with security hazards, forcing you to review every line with a fine-toothed comb. One who enthusiastically agreed with your suggestions, then did the exact opposite. A colleague who sabotaged your work, deleted your home directory, and then issued a detailed, polite apology for it. One who promised over and over again that they had delivered key objectives when they had, in fact, done nothing useful. An intern who cheerfully agreed to run the tests before committing, then kept committing failing garbage anyway. A senior engineer who quietly deleted the test suite, then happily reported that all tests passed.
You would fire these people, right?
Kingsbury sees the two extremes of the possible range of outcomes as:
ML systems continue to hallucinate, cannot be made reliable, and ultimately fail to deliver on the promise of transformative, broadly-useful “intelligence”. Or they work, but people get fed up and declare “AI Bad”…a lot of ML people lose their jobs, defaults cascade through the financial system, but the labor market eventually adapts and we muddle through. ML turns out to be a normal technology.
In the other extreme, OpenAI delivers on Sam Altman’s 2025 claims of PhD-level intelligence, and the companies writing all their code with Claude achieve phenomenal success with a fraction of the software engineers. ML massively amplified the capabilities of doctors, musicians , civil engineers, fashion designers, managers, accountants, etc, who briefly enjoy nice paychecks before discovering that demand for their service is not as elastic as once thought, especially once their clients lose their jobs or turn to ML to cut costs. Knowledge workers are laid off en masse and MBAs start taking jobs at McDonalds or driving for Lyft, at least until Waymo puts an end to human drivers. This is inconvenient for everyone: the MBAs, the people who used to work at McDonalds and are now competing with MBAs, and of course bankers, who were rather counting on the MBAs to keep paying their mortgages. The drop in consumer spending cascades through industries. A lot of people lose their savings, or even their homes. Hopefully the trades squeak through. Maybe the Jevons paradox kicks in eventually and we find new occupations.
In the following chapter Kingsbury speculates on what some of those new occupations might be:
Incanters. People who can prompt LLMs into getting what is wanted.
Process Engineers. People who help catch LLM errors. They build quality control processes – training people, identifying where more intense review is needed, assessing the cost-benefit trade offs of automating tasks, etc
Statistical Engineers. People who try and measure, model and control variability in ML systems.
Model Trainers. This will become increasingly difficult as the amount of false content or “slop” increases across the internet.
Meat Shields. People who are accountable for the errors of the ML systems they supervise.
Haruspices. People responsible for going through the model inputs, outputs and internal states of a ML system which has done something terrible to try and give a plausible reason for its behaviour.
But ultimately Kingsbury concludes that we should just stop using these systems. To return to the original analogy, the monster cannot be understood. There is often nothing actually there to understand. And it is certainly not in the business of understanding you. Although it may be very very good at convincing you otherwise.
On balance I think my view is currently at the muddle-through-with-ML-as-a-normal-technology end, which still looks likely to cause a disruption considerably bigger than 2008. My main reason is the already collapsing trust in many of the Big Tech companies. Trust which is going to be required even if their technology really can do some of this stuff. It is the scenario where we all get fed up and declare “AI Bad”. Like when we read about the people running Meta showing nowhere near the social responsibility commensurate with their current level of market power.
Or when, as last week, we have days and days of breathless commentary about Anthropic’s Mythos and Project Glasswing, and how its immense capabilities caused the company not to release it, sparking a meeting of central bankers to discuss the threat such technologies posed to financial systems. Only to finally read an account of attempts to verify any of what Anthropic have been saying. It is quite a technical piece, which I by no means understand all of, but the final paragraph is fairly arresting:
The most important thing in the Mythos release is not the model. It is the precedent. Anthropic has established, without discussion and without pushback, that a private company can unilaterally classify a capability as too dangerous for the public, grant selective access to the largest incumbents in the affected industry, and construct a parallel disclosure regime outside any democratic accountability structure. That precedent is exclusivity for abuse. It will be used by companies with worse judgment than Anthropic and narrower definitions of “partner” than the Glasswing consortium. The time to object to the shape of this thing is while it is still being built, not after it has removed all transparency and accountability.
How might Claude or ChatGPT respond to being designated “AI Bad”? Well Mary Shelley’s monster put it this way:
Once I falsely hoped to meet with beings who, pardoning my outward form, would love me for the excellent qualities which I was capable of unfolding. I was nourished with high thoughts of honour and devotion. But now crime has degraded me beneath the meanest animal. No guilt, no mischief, no malignity, no misery, can be found comparable to mine. When I run over the frightful catalogue of my sins, I cannot believe that I am the same creature whose thoughts were once filled with sublime and transcendent visions of the beauty and the majesty of goodness. But it is even so; the fallen angel becomes a malignant devil. Yet even that enemy of God and man had friends and associates in his desolation; I am alone.
Seven years ago I wrote about Catch 22 and actuarial practice, concluding, rather piously:
If we want far fewer actuaries to be employed in not growing alfalfa in the future and far more working on making the finance structures of our economy work better, whether to support a Green New Deal or more generally, we first need to embrace the idea that our current economic priorities are indeed insane.
So imagine my excitement at finding Catch 22 grabbed out of the pages of fiction and informing US foreign policy. Not convinced? Compare two passages. The first, from Catch 22, in 1961:
This time Milo had gone too far. Bombing his own men and planes was more than even the most phlegmatic observer could stomach, and it looked like the end for him. High-ranking government officials poured in to investigate. Newspapers inveighed against Milo with glaring headlines, and Congressmen denounced the atrocity in stentorian wrath and clamored for punishment. Mothers with children in the service organized into militant groups and demanded revenge. Not one voice was raised in his defense. Decent people everywhere were affronted, and Milo was all washed up until he opened his books to the public and disclosed the tremendous profit he had made. He could reimburse the government for all the people and property he had destroyed and still have enough money left over to continue buying Egyptian cotton. Everybody, of course, owned a share. And the sweetest part of the whole deal was that there really was no need to reimburse the government at all.
This week, the US Treasury lifted all oil sanctions on Iran. For 30 days. 140 million barrels of Iranian crude, sitting on ships at sea, may now be sold freely on the global market. Including to the United States itself.
In yuan.
The United States is purchasing, with Chinese currency, oil from the country it is currently bombing?! The same oil that funds the missiles that just shot down an F-35 for the first time. The same missiles that are redecorating allied oil infrastructure.
Treasury Secretary Bessent called this “narrowly tailored”. Narrow like in white, and tailored as in card, apparently.
In the same OFAC filing, Russian oil sanctions were lifted as well. And Belarus potash too, because apparently the universe was running low on irony and needed to top up.
The logic, insofar as there is any, goes like this: the war has crashed the global oil market so hard that the administration needs the enemy’s oil to keep gasoline prices from eating the midterms. They are unsanctioning the people they’re bombing because the bombing is working too well at the thing they didn’t want it to do. The sanctions were necessary to stop Iran funding the war, but the war made the sanctions too effective, so the sanctions had to be lifted to fund the war effort against the country that no longer needs sanctions because the oil revenues that sanctions were preventing are now required to prevent the economic damage caused by preventing those revenues, which is itself a consequence of the military campaign designed to make the sanctions unnecessary by making Iran the kind of country that doesn’t need sanctioning, which it would be, if the sanctions hadn’t been lifted to pay for making it that.
There have been many names thrown at Trump since he arrived in US politics. My personal favourite is probably the Tangerine Tyrant. Many people are currently relying on TACO (Trump Always Chickens Out) to resolve the Middle East crisis he has instigated. However, until now, I had not heard of anyone likening him to Milo Minderbender. But once you see it, it is difficult to un-see it.
Trump likes to give himself and everyone else nicknames. From the very stable genius of his first term, to more recently Honest Don and the Tariff King, whereas Milo, as M&M Enterprises (the company he started as the mess officer) expands, becomes the Mayor of Palermo, Assistant Governor-General of Malta, Vice-Shah of Oran, Caliph of Baghdad, Mayor of Cairo, and the god of corn, rain, and rice.
Trump likes to use his presidency to enrich himself, from his Trump coin to the Amazon documentary about his wife to his Board of Peace to all of his merchandise. Milo’s catchphrase is “what is good for M&M is good for the country”.
Trump doesn’t appear to believe in safety nets for ordinary people. Meanwhile Milo secretly replaces the CO2 cartridges in emergency life vests and the morphine in first aid kits with printed notes to the effect that what is good for M&M is good for the country.
Milo Minderbender is a war profiteer trying to convince himself that he is a free market fundamentalist. So what does that make Trump? Well hold that thought, because today’s Guardian has provided a partial answer I think, with a history of military targeting.
This introduces the concept of the kill chain, ie the process between detecting something and destroying it. Trying to shortcut the kill chain has been a perennial preoccupation of militaries through the ages. In the Vietnam War, Operation Igloo White dropped 20,000 acoustic and seismic sensors along the Ho Chi Minh trail, which transmitted data to relay aircraft, which then fed the signals to the IBM 360 computers at Nakhon Phanom airbase in Thailand. These analysed the data, predicted where the convoys would be and strikes were directed to those locations. The Viet Cong realised quickly that this system could not detect the difference between military vehicles and ox carts and therefore:
They played recordings of truck engines, herded animals near the sensors to trigger vibration detection, and hung buckets of urine in trees to set off the chemical detectors.
There was no way to independently check what they were destroying. The air force claimed 46,000 trucks were destroyed or damaged, which the CIA calculated exceeded the total number of trucks believed to exist in all of North Vietnam.
…air force personnel invented a creature to explain the absence. They called it the “great Laotian truck eater”.
Last time I talked about military targeting, I focused on the human in the loop, but let’s instead focus on the actual destruction going on for a moment, shall we? Trump’s assault on Iran hit 6,000 targets in two weeks. The kill chain had, apparently, been compressed so much that it allowed 1,000 decisions an hour. The school he hit, killing between 175 and 180 people, most of them girls between the ages of seven and 12, had changed its use to a school since at least 2016 and was visible on Google Maps. Old target lists had been reached for and noone had had the time or the inclination to check them before bombing them.
This is what you can expect from a Milo Minderbender presidency. It has been obvious, since at least the 1960s, that the US system requires enormous strength of purpose from its executive to hold its industrial-military complex in check. That is why so many of them have been so keen to install a Trump.
It feels as if, far from embracing the idea that our current economic priorities are indeed insane, as I fervently hoped seven years ago, we are instead doubling down on the insanity.
A week or so ago I referred to a “Thought Exercise” set in June 2028 “detailing the progression and fallout of the Global Intelligence Crisis” (ie science fiction), published on 23 February, which may have tanked the share price of IBM later that day. As I said then, the fall definitely happened, with IBM’s share price falling 13%, its biggest fall since 2000. I said then that the likelihood of the scenario portrayed was difficult to assess, but the speed with which the total economic collapse was described felt unlikely if not impossible. I would like to expand on that.
The main reason that the scenario was hard to assess was that it was not based on data or evidence at all. That is unavoidable for speculative fiction talking about things that are not currently happening, but when describing an economy only two years away where most of the processes described should be discernible to some extent already, it is totally avoidable.
Ed Zitron has done an excellent line by line take down of the Citrini piece here. Here is one page of that to give you a flavour:
However this lack of a link with anything tangible did not stop the financial markets panicking, which should cause us pause when relying on the financial markets’ valuation of projects, industries, government policies, etc.
Ed Zitron describes this kind of piece as analyslop: “when somebody writes a long, specious piece of writing with few facts or actual statements with the intention of it being read as thorough analysis”. It can then get picked up by other commentators which take it as their starting point for further analysis, often making it hard to see that the starting point had few if any data points. Here is an example, from Carlo Iacono, looking at what if just some of the Citrini pronouncements were true, with appendices detailing possible branching paths of outcomes, all generated by a large language model (LLM). And then people start studying the meta analysis, and it starts getting taken even more seriously, and put into models and pretty soon most of the analysis is being done on imagined risks rather than on ones which are already staring us in the face.
We have always had a problem keeping our society grounded in reality, think the 2003 Iraq War, where we went to war on a false assessment about Iraq’s possession of weapons of mass destruction, the 2008 financial crisis, where banks misunderstood the risks they were exposed to, and the last two and a half years, where we, for the most part, seem to have convinced ourselves we have not been facilitating a genocide in Gaza when we clearly have been. But this is only going to get worse with the AI systems which are being developed.
The rapid rise of artificial intelligence has served to dramatically increase the speed of information production while also eroding accuracy, making it difficult to differentiate between content that simply sounds confident and content that’s actually grounded in reality.
So where is AI currently? Well PwC’s global CEO survey from January this year had the following statement as the first bullet amongst its key findings:
Most CEOs say their companies aren’t yet seeing a financial return from investments in AI. Although close to a third (30%) report increased revenue from AI in the last 12 months and a quarter (26%) are seeing lower costs, more than half (56%) say they’ve realised neither revenue nor cost benefits.
That’s the reality. But the hype is much much more entertaining. My favourite spoof video of the AI future currently is this one, about the time where all most of us are good for is riding bicycles to supply the ever increasing energy needs of AI systems (click view in browser if you can’t see it):
And what about the financial journalists? The pieces describing our reaction to whatever is about to unfold economically have already been written. There are investor websites asking if the 2026 crash has already begun, while another recent article argues that “America has quietly become one of the world’s most shock‑resistant economies” (which seems unlikely to age well). What most financial journalists are more comfortable with are articles about how the warnings were ignored after the fact.
And the professions? Well the current overview of my own profession is probably reasonably represented by this piece from the Society of Actuaries in the United States. Unfortunately for them, Daniel Susskind, who is mentioned in the article, is currently suggesting, as part of his Future of Work lecture series for Gresham College, how the key to the sudden development in AI, after the “AI Winter” when progress seemed slow, was that we abandoned trying to make machines which thought and acted like humans in favour of focusing on completing tasks in any way possible. Increasingly we are now automating tasks where we can’t (or won’t) articulate how we do them. From Deep Blue‘s victory over Kasparov in 1997 to Watson winning jeopardy in 2011 to ImageNet beating humans at image recognition (although that is disputed), Susskind refers to this progress as the displacement of purists in favour of what he calls “The Pragmatic Revolution”. Pragmatism in this sense appears to be that we humans should just accept the consequences the people running these systems want. So, as his latest lecture “Work, out of reach” claims, people moving into cities to find work is a strategy which is no longer going to work for low skilled people:
He then shows this graphic demonstrating the lack of recovery of big coal mining areas in the UK:
Source: Left – Sheffield Hallam University map of coal mining areas; Right – % employment from Overman and Xu (2022)
And finally he cites the notorious Policy Exchange piece from 2007, Cities Unlimited, whose thesis was that there is apparently no realistic prospect of regenerating towns and cities outside London and the South East.
Susskind talks about three forms of technological unemployment:
skills-mismatch, where your skills are mismatched to the work available. Education and training has always been the answer to this in the past.
place-mismatch, where the jobs are not where you have built your life. Some believe the answer should always be the one proposed by Norman Tebbit, who memorably told everyone in 1981, “I grew up in the 30s with an unemployed father. He did not riot. He got on his bike and looked for work.”
identity-mismatch, where according to Susskind, people are prepared to stay out of work to protect their identity, citing US men who won’t take “pink collar” work, China “rotten tail” kids, Japanese seishain-or-nothing and Indian Sarkari Naukri queues in India. Or perhaps they are just looking for work which is consistent with the idea of human dignity.
Susskind claims to have no answer to any of these as far as AI is concerned. They are, in his view, just the inevitable outcomes of his “Pragmatic Revolution”. It is the unthinking pursuit of more and more growth funded by capital less and less tethered to any territory, principle or purpose, where any grit in the machinery, be it unions or protestors or, increasingly, the wrong sort of government must be trampled underfoot. Anything which impedes the helter-skelter rush to more and more at greater and greater speed. It’s like our whole economy is run by this guy (press the view in browser link if you can’t see him) shouting “Ready, Aim, Fire!”:
But unskilled people will not be the only collateral damage of these unguided weapons. Take markets for instance. These are where people are exposed to risks and rewards based on underlying conditions they only partially understand. Greed and fear may be their main motivations, but gossip and group think are their main communication channels. They don’t need facts, particularly when so many of the facts are proprietary information not in the public domain. A plausible narrative will do. And plausible narratives are what LLMs will do for you in abundance.
And the more we reward people who can move fast, eg to spot an arbitrage opportunity, even at the risk of breaking things, rather than people who can make decisions which still look good decades from now, the more we are setting up the conditions for AI systems to be the go-to tool.
And put that together with an AI industry which desperately needs funding capital to keep arriving, ie one which is unbelievably highly motivated to push plausible narratives even when they know they are not grounded in reality, and you have a recipe for market-generated chaos.
And then we have Trump’s new war. Beware the people who are war gaming the Middle East at the moment on a range of LLMs (just stop and think for a moment about the bloodless inhuman impulse behind carrying out such an exercise rather than, I don’t know, talking to some actual people who live or have lived recently in and around the region). One of the worst offenders is Heavy Lifting banging on about what the three scenarios are for Operation Epic Fury. This is as bad as it sounds:
I tasked her [he is talking about Gemini Pro here] with doing a literature review on regime change (a term often used by the President but not a well-defined one), creating three scenarios of possible outcomes for which each was give a percentage probability, and a list of 20 items to examine for each scenario that covered political, economic, and cultural issues with a special focus on the political consequences in the U.S. and what this means for China, our biggest geopolitical rival.
But Gemini Pro wasn’t the only one involved in this. Two other humans were, Tim Parker and Ron Portante, trainers at the gym I go to. (Just as a personal aside, Tim was my coach in hitting six plates [345 pounds] on the sled last Friday and I have a video to prove it!) I was talking about the piece and Ron raised the issue of linguistic and cultural diversity in Iran. Tim did some real time research for me on his phone while I was burning real calories under his strict tutelage. This made me think I needed a background section on Iran. When I got him Gemini and I added it.
What you mean you belatedly realised you might need to have done some actual research into Iran rather than just generic research on regime change? I stopped reading at that point.
Meanwhile King’s College London have been carrying out war games more systematically using AI. Professor Kenneth Payne from the Department of Defence Studies led the study, which looked at how LLMs would perform in simulated nuclear crises. As Professor Payne said:
Nuclear escalation was near-universal: 95% of games saw tactical nuclear use and 76% reached strategic nuclear threats. Claude and Gemini especially treated nuclear weapons as legitimate strategic options, not moral thresholds, typically discussing nuclear use in purely instrumental terms. GPT-5.2 was a partial exception, limiting strikes to military targets, avoiding population centers, or framing escalation as “controlled” and “one-time.” This suggests some internalised norm against unrestricted nuclear war, even if not the visceral taboo that has held among human decision-makers since 1945.
This is not a Pragmatic Revolution. These AI systems cannot replace humans thinking about the future we want for humans in any way which is worth having. What they can do, if we let them, is accelerate our worst impulses and move us further away from considered reflective decision making.
But we will continue to use AI systems in the military because, as it turns out, it is very useful for low stakes admin. So although Lavender, the system used by the Israeli military to select targets in Gaza, made errors in 10% of cases and was therefore totally inappropriate to the task, there are lots of organisational logistical tasks where it is much quicker than the alternative and 10% error rates do not matter so much.
There is clearly an issue with what we decide to use these systems for. We need to be able to regulate the decisions which are particularly consequential. However the only way we seem to be considering for this at the moment is the human-in-the-loop model, like the humans spending around 20 seconds considering each target recommended by Lavender before authorizing a bombing. I have written about these before in the context of early career professionals in the finance industry, where the prospect seemed miserable enough:
They will be paid a lot more. However, as Cory Doctorow describes here, the misery of being the human in the loop for an AI system designed to produce output where errors are hard to spot and therefore to stop (Doctorow calls them, “reverse centaurs”, ie humans have become the horse part) includes being the ready made scapegoat (or “moral crumple zone” or “accountability sink“) for when they are inevitably used to overreach what they are programmed for and produce something terrible.
However it seems obvious to me that, in the context of dropping actual bombs on actual people, there is an even more serious problem with this model. As Simon Pearson (anti-capitalist musings) puts it:
The “human in the loop” requirement exists in military doctrine because international humanitarian law demands an accountable human decision-maker for lethal force. The laws of armed conflict require proportionality assessments, precautionary measures, distinction between combatants and civilians. All of these obligations attach to a human commander. The system cannot fulfil them. So a human must be present, and their presence must constitute a decision, regardless of whether any genuine decision was made.
What the institution needs from the analyst is not judgment. It is a signature. The signature converts a machine output into a human act. And a human act is what the law recognises, whether or not any judgment occurred. When the strike kills children, the chain of accountability runs to the analyst who approved the target: not to the system that identified it, not to the company that built the system, not to the doctrine that compressed the review window to ten seconds.
But whether we want to make money from exploiting a short term anomaly in a market, make our fellow humans redundant, prosecute a war on another group of fellow humans or “win” a war of mutual nuclear destruction, we need to retain the capacity for real human reflection within the decision-making processes we use. Not just a human-in-the-loop nor just the elites of tech companies deciding how the systems will be configured behind commercially confidential walls. These processes need democratic accountability every bit as much as our parliaments, councils, institutions and voting systems do.
Something infuriatingly slow, inclusive and deliberative giving recommendations which are then stress-tested for how they would perform on contact with reality, involving yet more people being serious and deliberative and taking their responsibilties more seriously than being a human-in-the-loop would ever allow. Our decision-making systems need more grit and less oil. AI is all oil.
The Actuary magazine recently had a debate about whether the underlying data or the story you wove around it was more important. I’m not sure there is always a clear distinction between the two, as Dan Davies rather neatly illustrates here, but my view is that, if a binary choice has to be made, it is always going to be the story. And there was a great example of this which popped up recently in the FT.
The FT article was ‘Is university still worth it?’ is the wrong question, by John Burn-Murdoch, with great graphs as usual by John. However, as is sometimes the case, I feel that a very different and more convincing story could be wrapped around the same datasets he is showing us.
The article’s thesis is as follows:
The graduate earnings premium, ie how much more on average graduates earn than non-graduates, has only fallen in the UK as the proportion going to university has risen. It has risen in other countries:
In the UK, we have had much weaker productivity growth than the other comparator countries, and also “the steady ramping up of the minimum wage has squeezed the earnings premium from the lower end too”:
We have also had a much smaller increase in the percentage of managerial and professional jobs than a different group of comparator countries (they haven’t mentioned Germany before), meaning graduates are forced to take lower salaried jobs elsewhere:
So the answer according to the FT? We should focus on economic growth rather than “tweaking” higher education intake and funding. Then graduate earnings would be higher, student loans could be more generous(!) and students would have more chance of getting a good job.
Well perhaps. But here’s a different framing of the same data that I find more persuasive.
Let’s start by addressing that point about the minimum wage. According to the House of Commons Library report on this, the UK’s minimum wage is broadly comparable to that of France and the Netherlands, although higher than Canada’s and much higher than that of the United States. The employers who are the FT’s constituency would obviously like us lower down this particular chart:
The main economic framing here is the progress myth of the UK’s business community: economic growth. All problems can be solved if we can just get more economic growth. Apparently we need more inequality in pay between graduates and non-graduates which we can get by generating more economic growth. This is honest of them at least, although I don’t see much evidence that the economic growth they crave will go into skilled job creation rather than stock buy backs (according to Motley Fool, “Companies spent $249 billion on stock buybacks in Q3 2025, and $777 billion over the first three quarters of 2025.”).
There are a lot of problems with framing every economic question with respect to economic growth, memorably illustrated by Zack Polanski of the Green Party in this less than 3 minute video recently (I strongly recommend you watch it before you read on – click on the read in browser link if you can’t see it):
Economic growth is increasingly without purpose, wasteful of energy and poorly distributed. It is chasing outputs, literally any outputs, whatever the cost to the environment, our health system, our education system, our social support systems and our communities. Looking at the framing above, you can see that economic growth as currently pursued will always see anything which stops the concentration of wealth amongst the already wealthy, like a higher national minimum wage or a totally made-up concept like a lower graduate earnings premium (which in itself is a framing trying to make reducing inequality seem undesirable) as a problem. Lack of productivity growth, itself a proxy for this kind of economic growth (because if you ask why we need more productivity the answer is always to get more economic growth), is usually directed as a criticism at “lazy” UK workers, rather than under-investing and over-extracting UK business owners.
But what if, instead of economic growth, your progress myth was reducing inequality? Or growing equality within the economy?
Source: World Inequality Database wid.world
If you focused on inequality rather than economic growth, then you would find it correlates with everything we say we don’t want. Unlike economic growth, having equality as an aim actually has the advantage of having an evidence base for the claim that it improves society:
If you focused on inequality, then you would be pleased that we have had an increase in our minimum wage. You would think that the same FT article’s admission that UK graduates’ skills levels are higher than those in the United States was more important than something called a graduate earnings premium.
Burn-Murdoch is right to say asking whether university is worth it is the wrong question.
However economic growth is the wrong answer.
And I thought I would probably be stopping there for this week. But then something odd happened. A “Thought Exercise” set in June 2028 “detailing the progression and fallout of the Global Intelligence Crisis” (ie science fiction), published on 23 February, may have tanked the share price of IBM later that day. The fall definitely happened, with IBM’s share price falling 13%, its biggest fall since 2000, alongside smaller falls in other tech stocks.
Investors have recently seized on social media rumours and incremental developments by small AI companies to justify further selling, with a widely circulated blog post by Citrini Research over the weekend describing how AI could hypothetically push the US unemployment rate above 10 per cent by 2028, proving the latest catalyst.
The likelihood of the scenario portrayed is difficult to assess, but the speed with which the total economic collapse happens subsequently as described feels unlikely if not impossible. However the fact that the markets are this jittery tells us something I think. As Carlo Iacono puts it:
We are living through a period in which the gap between “plausible narrative” and “tradeable signal” has collapsed to nearly nothing. When a scenario feels real enough to model, and the underlying anxiety is already there waiting to be organised, fiction and forecast become functionally indistinguishable.
The data underlying the markets hasn’t changed, but the story has. I rest my case.
The book I finished this weekend had a passage about a first experiment with a new substance which could shield against gravity. Mr Cavor, the rather unworldly scientist, is explaining to Mr Bedford, a man with no obvious talents other than to look for a quick buck where he can find one, what would have happened if his substance, Cavorite, had not got dislodged fairly quickly from where they had positioned it:
“You perceive,” he said, “it formed a sort of atmospheric fountain, a kind of chimney in the atmosphere. And if the Cavorite itself hadn’t been loose and so got sucked up the chimney, does it occur to you what would have happened?”
I thought. “I suppose,” I said, “the air would be rushing up and up over that infernal piece of stuff now.”
“Precisely,” he said. “A huge fountain—”
“Spouting into space! Good heavens! Why, it would have squirted all the atmosphere of the earth away! It would have robbed the world of air! It would have been the death of all mankind! That little lump of stuff!”
“Not exactly into space,” said Cavor, “but as bad—practically. It would have whipped the air off the world as one peels a banana, and flung it thousands of miles. It would have dropped back again, of course—but on an asphyxiated world! From our point of view very little better than if it never came back!”
I stared. As yet I was too amazed to realise how all my expectations had been upset. “What do you mean to do now?” I asked.
“In the first place if I may borrow a garden trowel I will remove some of this earth with which I am encased, and then if I may avail myself of your domestic conveniences I will have a bath. This done, we will converse more at leisure. It will be wise, I think”—he laid a muddy hand on my arm—“if nothing were said of this affair beyond ourselves. I know I have caused great damage—probably even dwelling-houses may be ruined here and there upon the country-side. But on the other hand, I cannot possibly pay for the damage I have done, and if the real cause of this is published, it will lead only to heartburning and the obstruction of my work. One cannot foresee everything, you know, and I cannot consent for one moment to add the burden of practical considerations to my theorising…”
In case you are in any doubt, Dario Amodei is our Mr Cavor here. I can just imagine his response to the first disaster attributed to AI research being prefaced by “one cannot foresee everything, you know…”. And there are too many Mr Bedfords out there to shake a stick at, trying to sell you anything they can possibly attribute to AI just to keep the whole thing rolling along.
I am with the fog people. The FT seem to be too, with this pair of diagrams attached to this article.
First the US, where there are tentative signs of something they can possibly use as a proxy for productivity growth as a result of using AI:
And so it was this foggy sensibility about AI which I took with me to the Bletchley Park Museum last weekend, site of the AI Safety Summit in November 2023 which drew in the US Vice President, Kamala Harris, European Commission President Ursula von der Leyen, Elon Musk, then UK Prime Minister Rishi Sunak, Open AI’s Sam Altman, Meta’s Nick Clegg and Prof Yann LeCun, Meta’s chief AI scientist, amongst around 100 guests invited to suck their teeth about AI.
The thing that particularly struck me at Bletchley Park is that it demystified the emergence of the computer for me. The forerunner, which was the mechanisation using punch cards of the process of sorting the massive amounts of data the centre was receiving in war time, smacks of a group of people who had just run out of wall to spread their webs of cards and strings across. It was a crime investigation which had got out of hand.
A highlight for me was Alan Turing’s very prescient little note about AI, written in 1940 but anticipating the arguments which would be raging by 2026 (and how poignant that the man who probably did more than anyone to transform what we are able to do by punching a keyboard was chained to one that could only press hunks of metal against a strip of carbon onto a piece of paper):
There is also a hilarious secrecy pledge from the ancestors of the safety summit people, telling you all the ways in which you just need to shut up:
“There is an English proverb none the worse for being seven centuries old:” it thunders.
Wicked tongue breaketh bone,
Though the tongue itself hath none.
Words to live by, I’m sure we’d all agree.
What Bletchley Park was less good at was explaining how the Enigma code was cracked, despite an excellent collection of the hardware involved. For that, I recommend Simon Singh’s The Code Book.
Here was the world’s first “intelligence factory”, scaling up intelligence gathering and analysis as never before and by so doing also changing the way governments would interact with their populations, with just as many implications for our current times as the development of AI. This cluster of huts around a country house rebranded as GCHQ and moved to Cheltenham a few years after World War 2.
Path dependence is a term which describes a situation where past events or decisions constrain later events or decisions. Bletchley Park feels like the Museum of Path Dependence to me.
And the legacy of the safety summit? Well my “hot take” would be: when you are a little lost in the fog, it is generally advisable to slow down a bit and take steps to reduce your risk of breaking things. I wonder if I can get that on a bumper sticker.
In ordering #5, self-driving cars will happily drive you around, but if you tell them to drive to a car dealership, they just lock the doors and politely ask how long humans take to starve to death.Source: https://m.xkcd.com/1613/
My attention was drawn this week to an article by Dario Amodei, co-founder of Anthropic (a spin off from OpenAI, which was co-founded by Elon Musk and heavily invested in by Microsoft so very much part of the Magnificent 7 architecture), the creator of the large language model Claude, called The Adolescence of Technology. It is hard to overemphasise how much I disagree with everything Dario has written here, but also useful in that it is a long article, which covers a lot of ground, and allows me to define my views in opposition to it.
The irritations start pretty much straight away. So Dario quotes from a science fiction classic (Carl Sagan’s First Contact), but then follows this up under the heading of “Avoid doomerism” with this:
…but it’s my impression that during the peak of worries about AI risk in 2023–2024, some of the least sensible voices rose to the top, often through sensationalistic social media accounts. These voices used off-putting language reminiscent of religion or science fiction, and called for extreme actions without having the evidence that would justify them.
Notice the word “sensible” doing the heavy lifting there. Only science fiction endorsed by Dario will be considered. Dario wants us to consider the risks of AI in “a careful and well-considered manner”, which sounds reasonable, but then his 3rd and final bullet under this (after “avoid doomerism” and “acknowledge uncertainty”) goes as follows:
Intervene as surgically as possible. Addressing the risks of AI will require a mix of voluntary actions taken by companies (and private third-party actors) and actions taken by governments that bind everyone. The voluntary actions—both taking them and encouraging other companies to follow suit—are a no-brainer for me. I firmly believe that government actions will also be required to some extent, but these interventions are different in character because they can potentially destroy economic value or coerce unwilling actors who are skeptical of these risks (and there is some chance they are right!).
So reflexively anti regulation of his own industry, of course. And voluntary actions by corporations, an approach to solving problems which has been demonstrated not to work repeatedly, is apparently “a no-brainer”. Also it is automatically assumed that government actions will destroy value. Only market solutions will be endorsed by Dario, pretty much until they have messed up so badly you are forced to bring governments in:
To be clear, I think there’s a decent chance we eventually reach a point where much more significant action is warranted, but that will depend on stronger evidence of imminent, concrete danger than we have today, as well as enough specificity about the danger to formulate rules that have a chance of addressing it. The most constructive thing we can do today is advocate for limited rules while we learn whether or not there is evidence to support stronger ones.
There is then the expected sales pitch about what he has seen within Anthropic about the relentless “increase in AI’s cognitive capabilities”. And then the man who warned about sensationalist science fiction is off:
I think the best way to get a handle on the risks of AI is to ask the following question: suppose a literal “country of geniuses” were to materialize somewhere in the world in ~2027. Imagine, say, 50 million people, all of whom are much more capable than any Nobel Prize winner, statesman, or technologist.
And the rest of the article is then off solving this imaginary problem in all its facets, rather than the wealth and power concentration problem that we actually have. The only legislation he seems to be in favour of seems to be something called “transparency legislation”, legislation which of course Anthropic would help to write.
However, after suggesting everything from isolating China and using “AI to empower democracies to resist autocracies” to private philanthropy as the solutions to his imagined problems, Dario finally and reluctantly concludes government intervention might after all be necessary as follows:
…ultimately a macroeconomic problem this large will require government intervention. The natural policy response to an enormous economic pie coupled with high inequality (due to a lack of jobs, or poorly paid jobs, for many) is progressive taxation. The tax could be general or could be targeted against AI companies in particular. Obviously tax design is complicated, and there are many ways for it to go wrong. I don’t support poorly designed tax policies. I think the extreme levels of inequality predicted in this essay justify a more robust tax policy on basic moral grounds, but I can also make a pragmatic argument to the world’s billionaires that it’s in their interest to support a good version of it: if they don’t support a good version, they’ll inevitably get a bad version designed by a mob.
That, by the way, is what Dario thinks of democracy: “a bad version designed by a mob” rather than the “good version” that he and his fellow billionaires could come up with in their own self interest. The mask has really slipped by this point. And the following section, on “Economic concentration of power”, just demonstrates that he has no effective answers at all that he deems acceptable on this. It’s just an inevitability for him.
This is what Luke Kemp’s excellent Goliath’s Curse refers to as a “Silicon Goliath”. Goliaths are dominance hierarchies which spread by dominating the areas around them. They need three conditions (which Luke calls “Goliath fuel”): lootable resources (ie resources which can be easily stolen off someone else), caged land (ie land difficult to escape from) and monopolizable weapons (ie ones which require processes which can be developed to give one society an edge over another). We are all Goliath-dwellers in “The West” now, looting resources from other countries in unequal exchanges which impoverish the Global South, with weapons (eg nuclear weapons) available only to the elite few countries and operating within the cages of heavily-policed national boundaries. The Silicon Goliath which is developing will have data as its lootable resource, mass surveillance systems providing its cages and monopolizable weapons such as killer drones. The resultant killbot hellscapes which people like Dario Amodei laughably imagine they have defences against through things like their Claude’s Constitution are almost pitiful in their inadequacy.
Nate Hagens takes Dario’s claims for AI’s cognitive capabilities much more seriously than me, and then considers the risks in a less adolescent way here. As he says:
And here’s what his essay has almost nothing about. Energy, water, materials, or ecological limits.
And also nowhere does Dario talk about the 99% of people who are just spectators in his world, other than to describe them as “the mob”. This is quite a blind spot, as Luke Kemp points out in his exhaustive study of the collapses of “Goliaths” over the last 5,000 years. “The extreme levels of inequality” predicted by Amodei in his essay are not just things we have to put up with, but the reasons the world he predicts is likely to be hugely unstable. Not created by AI, but accelerated by it. Kemp describes it as “diminishing returns on extraction”:
We see a pattern re-emerging across case studies. Societies grow more fragile over time and more prone to collapse. Threats that they had always faced such as invaders, disease and drought seem to take a heavier toll.
As societies grew bigger:
They still faced the underlying (and ongoing) problem of rising inequality creating societies where and institutions more extractive power was more concentrated.
And eventually:
The result is more extractive institutions creating growing instability, internal conflict, a drain of resources away from government, state capture by private elites, and worse decision-making. Society – especially the state – becomes more fragile. Private elites tend to take a larger share of extractive benefits. The state, and many of the power structures it helps prop up, then usually falls apart once a shock hits: for Rome it was climate change, disease, and rebelling Germanic mercenaries; for China it was often floods, droughts, disease and horseback raiders; for the west African kingdoms it was invaders and a loss of trade; for the Maya it was drought and a loss of trade; and for the Bronze Age it was drought, a disruption of trade and an earthquake storm.
The only real answer to combatting existential risks in the hands of adolescents like the Tech Bros is more democracy: over control of decision-making, over control of resources, over control of the threat of violence and over control of information. We are a long way from achieving these within our own particular Goliath at the moment, and indeed there is no sign at all that our elites are interested in achieving them. The Magnificent 7 are propping up the US stock exchange. The promise of perpetual economic growth is the progress myth of our time and leaders who do not provide it will lose the “Mandate of Heaven” in just the same way as Chinese rulers did when they were unable to prevent floods and droughts. Adam Tooze sees the signs of the inner demons of our elites starting to detach them from reality in the latest disclosures from the Epstein files:
Are we, like [Larry] Summers, fantasizing about stabilizing our desires and needs in an inherently dangerous and uncertain world? Are we kidding ourselves?
But, without those controls in place, we would need a lot more than Dario’s Anthropic playing nicely to allow this particular adolescent to grow up. And this is where I am forced to take Nate Hagens’ assessment more seriously. Because if our rulers’ Mandates of Heaven are dependent on eternal economic growth on their watch and they, rightly, think that this is not possible in our current non-AI-enhanced world but, wrongly, think it is possible in a future AI-enhanced world, then that is the way they are going to demand we go. And, if the Larry Summers fantasists really are kidding themselves, it may be very hard to talk them out of it.
Happy new year all! New year, new banner, courtesy of my brilliant daughter who presented me with a plausible 3-D model of my very primitive cartoon of a reverse-centaur over Christmas. And I thought I would kick off with a relatively uncontentious subject: examinations!
“Back to normal!” That was the cry throughout education when the pandemic had finally ended enough for us to start cramming students into rooms again. The universities had all leveraged themselves to the maximum, and perhaps beyond, to add to the built estate, so as to entice students in both the overseas and the uncapped domestic market to their campuses, and one by-product of this was they had plenty of potential examination halls. So let’s get away from all of that electronic remote nonsense and get everyone in a room together where you can keep an eye on them and stop them cheating. This united the purists who yearned for the days of 10% of the cohort turning up for elite education via chalk and talk rather than the 50% we have today, senior management needing to justify the size of the built estate and politicians who kept referring to traditional exams in an exam hall as the “gold standard”.
So, in a time when students have access to information, tools, how to videos of everything imaginable, the entire output of the greatest minds of thousands of years of human history, as well as many of the less than great minds, in short anything which has ever caught anyone’s attention and been committed to some form of media: in this of all times, we want to sort the students into categories for the existing job market based on how they answer academic questions about what they can remember unaided about the content of their lecture courses and reading lists with a biro on a pad of paper perched precariously on a tiny wooden table surrounded by hundreds of other similar scribblers, for a set period of time as minders wander the floors like Victorian factory owners.
And for institutions that thought the technology we fast-tracked for education delivery and assessment in the pandemic would surely be part of education’s future? Or perhaps they just can’t afford to borrow half a billion or have the the land available to construct more cathedrals of glass and brick to house more examination halls? Simple! We just create the conditions for that gold standard examination right there in the student’s own bedroom or the company they work for!
There are 54 pages to the Institute and Faculty of Actuaries’ (IFoA’s) guidance for remotely invigilated candidates. It covers everything from the minimum specification of equipment you need, including the video camera to watch your every movement and the microphone to pick up every sound you make, to the proprietary spying software (called “Guardian Browser”) you will need to download onto your own computer, how to prove who you are to the system, what you are allowed to have in your bedroom with you and even how you need to sit for the duration of the exam (with a maximum of two 5 minute breaks) to ensure the system has sufficient visibility of you at all times:
These closed book remote arrangements replaced the previous open book online exams which most institutions operated during the pandemic. The reason given was that the exam results shot up so much that widespread cheating was suspected and the integrity of the qualifications was at risk. The IFoA’s latest assessment regulations can be found here.
The belief in examinations is very widespread. A couple of months ago I was discussing the teacher assessments which replaced them briefly during the pandemic with a secondary business studies teacher. He took great pride in the fact that he based his assessments solely on mock results, ie an assessment carried out before all of the syllabus had been covered and when students were unaware it would be the final assessment. But still in his mind more “objective” than any opinion he might have of his own students.
If a large language model can perform enormously better in an examination than your students can without it, what it actually demonstrates is that the traditional examination is woefully unprepared for the future. As Carlo Iacono puts it:
The machines learned from us.
They learned what we actually valued and it turned out to be different from what we said we valued.
We said we valued originality. We rewarded conformity to genre. We said we valued depth. We measured surface features. We said we valued critical thinking. We gave higher marks to confident assertion than to honest uncertainty.
So now the machines produce what the world trained them to produce: fluent, confident, passable output that fits the shapes we reward.
And we’re horrified. Not because they stole something from us. Because they showed us what the systems were selecting for all along.
The scandal isn’t that a model can imitate student writing. The scandal is that we built an educational and professional culture where imitation passes as competence, and then acted shocked when a machine learned to imitate faster.
We trained the incentives. We trained the rubrics. We trained the career ladders.
The pattern recognition which gets you through most formal examinations is just too cheap and easy to automate now. It is no longer a useful skill, even by proxy. It might as well be Hogwarts’ sorting hat for all the use it is in a post scarcity education world. If the machines have worked out how to unlock the elaborate captcha system we have placed around our gold standard assessments, an arms race of security measures protecting a range of tests which look increasingly narrow compared to the capabilities which matter does not seem like the way to go.
What instead we are doing is identifying which students are prepared to put themselves through literally anything to get the qualification. Companies like students like that. They will make ideal reverse-centaurs. The description of life as a reverse-centaur even sounds like the experience of a proctored exam:
Like an Amazon delivery driver, who sits in a cabin surrounded by AI cameras, that monitor the driver’s eyes and take points off if the driver looks in a proscribed direction, and monitors the driver’s mouth because singing isn’t allowed on the job, and rats the driver out to the boss if they don’t make quota.
The driver is in that van because the van can’t drive itself and can’t get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance. But the driver is human, so the van doesn’t just use the driver. The van uses the driver up.
And, even if you are OK with all of that, all of these privacy intrusions don’t even work to prevent cheating! The ACCA, the world’s largest accounting professional body, has just announced it is stopping all remote exams after giving up the arms race against the cheats, facilitated in some cases seemingly by their Big Four employers lying about what had gone on.
Actuarial exams started in 1850, only 2 years after the Institute of Actuaries was established (Dermot Grenham wrote about them recently here). This pre-dated the establishment of the first examination boards by a few years (1856 Society of Arts, the Society for the encouragement of Arts, Manufactures and Commerce, later the Royal Society for the encouragement of Arts, Manufactures and Commerce (Royal Society of Arts); 1857: University of Oxford Delegacy of Local Examinations (founded by the University of Oxford); and 1858: University of Cambridge Local Examinations Syndicate (UCLES, founded by the University of Cambridge)), so keen were actuaries to institute examinations. However it was the massive expansion of the middle classes as the Industrial Revolution disrupted society in so many ways that led to the need for a new sorting hat beyond the capacity of the oral examinations that had previously been the norm.
Now people seem to be lining up to drag everyone back into the examination hall. Any suggestion of a retreat from traditional exams is met by howls of outrage from people like Sam Leith at The Spectator about lack of “rigour”. However, in my view, they are wrong.
Yes of course you can isolate students from every intellectual aid they would normally use, as a centaur, to augment their performance, limit the sources they can access, force them to rely on their own memories entirely, and put them under significant time pressure. You will definitely reduce marks by doing that. So that has made it harder and therefore more rigorous and more objective, right?
Well according to the Merriam-Webster dictionary, rigorous is a synonym of rigid, strict or stringent. However, while all these words mean extremely severe or stern, rigorous implies the imposition of hardship and difficulty. So promoting exams above all as an exercise in rigour reveals their true nature as a kind of punishment beating in written form, for which the prize for undergoing it is whatever it qualifies you for. Suddenly the sorting hat looks relatively less arbitrary.
The problems of traditional exams are well known, but the most important ones in my view are that they measure a limited range of abilities and therefore are unlikely to show what students can really achieve. Harder does not mean more objective. It is like deciding who can act by throwing students out, one at a time, in front of a baying mob of, let’s say for argument, readers of The Spectator. Sure, some of the students might be able to calm the crowd, some may even be able to redirect their anger towards a different target. But are the people who can play Mark Antony for real necessarily the best all-round actors? And has someone who can only stand frozen on the spot under those circumstances really proved that they could never act well?
It also means that education ends a month or more before the exams, to allow the appropriate cramming, followed by engaging all of the teaching staff in the extended exercise of marking, checking and moderating what has been written in answer to academic questions about what the students can remember unaided about the content of their lecture courses and reading lists with a biro on a pad of paper perched precariously on a tiny wooden table surrounded by hundreds of other similar scribblers, for a set period of time as minders wander the floors like Victorian factory owners. But what if instead the assessment was part of the teaching process? What if students felt that their assessment had been a meaningful part of their educational experience? What if, instead of arguing the toss over whether they scored 68% or 70% on an assessment, students could see for themselves whether they had demonstrated mastery of their subject.
One model of assessment which is getting a lot of attention at the moment, one I am a big fan of having used it at the University of Leicester on some modules, is something called interactive oral assessment, where students meet with a lecturer or tutor, individually or in a small group, and answer questions about work they have already submitted. It is a highly demanding form of assessment, for both the students and the assessors, but it means the final assessment is done with the student present and, with careful probing from the assessors, who will obviously need to have done a close reading of the project work beforehand, you can be highly confident of the degree to which the student understands the work they have submitted. It also allows the student to submit a piece of work of more complexity and ambition than can be accommodated by a traditional exam. And it needn’t take any more time if the interviews are carried out online when set against the exam marking time of the traditional exam. Something which all the technology we developed through the pandemic allows us to do, without the need for spyware.
There are other models which also assess the technological centaurs we wish our students to become rather than the reverse-centaurs we are currently dooming too many to become. It is looking like it may be time to start telling students to stop writing and to put down their pens on the traditional exam. And perhaps the actuarial profession, who led us into the era of professional written examinations so enthusiastically 175 years ago, might now want to take the lead in navigating our way out of them?
The rear view mirror isn’t going to help us any more Source: Wikimedia Commons: Shattered right-hand side mirror on a 5-series BMW in Durham, North Carolina by Ildar Sagdejev
I would like to start this week’s post with a quote from Carlo Iacono, from a Substack piece he did a couple of weeks ago called The Questions Nobody Is Funding:
What is a human being for? What do we owe the future? What remains worth the difficulty of learning?
These are not questions you will find in the OECD’s AI Literacy Framework. They are not addressed in the World Economic Forum’s Education 4.0 agenda. They do not appear in the competency matrices cascading through national education systems. Instead, we get learning objectives and assessment criteria. Employability outcomes and digital capabilities. The language of preparation, as if the future were already decided and our job were simply to ready people for it.
I think this articulates well the central challenge of AI for education. Whether you think this is the beginning of a future where augmented humans move into a different type of existence to any we have known before; or you believe very little will be left behind in the rubble from the inevitable burst of the AI bubble when it comes and will be, at least temporarily, forgotten in the most devastating stock market crash and depression for a century; or you hold both these beliefs at the same time; or you are somewhere in between, it is difficult to see how the orderly world of competency matrices, learning objectives, assessment criteria, employability outcomes and digital capabilities can easily survive the period of technological, cultural, economic and political disruption which we appear to have entered. Looking in the rear view mirror and trying to extrapolate what you see into the future is not going to work for us any more.
Whether you think, like Cory Doctorow, in his recent speech at the University of Washington called The Reverse Centaur’s Guide to Criticizing AI, that:
AI is the asbestos in the walls of our technological society, stuffed there with wild abandon by a finance sector and tech monopolists run amok. We will be excavating it for a generation or more.
Technologies such as LLMs are neither going to transcend humanity as the holdouts on one side still hope, nor disappear, as other holdouts might like. We’re going to have to figure out ways to talk about them better and more clearly.
We are certainly going to have to figure out ways to talk about LLMs and other forms of AI more clearly, so that the decisions we need to make about how to accommodate them into society can be made with the maximum level of participation and consensus. And this seems to be the key for me with respect to education too. We do need people graduating from our education system understanding clearly what LLMs can and cannot do, which is a tricky path to navigate at the moment as a lot of money is being concentrated on persuading you that it can do pretty much anything. One example here has created a writers’ room of four LLMs where they are asked to critique each other by pushing the output from one into the prompts for the others, reminiscent of The Human Centipede. Which immediatel reminded me of this take from later in that Cory Doctorow speech:
And I’ll never forget when one writer turned to me and said, “You know, you prompt an LLM exactly the same way an exec gives shitty notes to a writers’ room. You know: ‘Make me ET, except it’s about a dog, and put a love interest in there, and a car chase in the second act.’ The difference is, you say that to a writers’ room and they all make fun of you and call you a fucking idiot suit. But you say it to an LLM and it will cheerfully shit out a terrible script that conforms exactly to that spec (you know, Air Bud).”
So, back to Carlo’s little questions:
What is a human being for?
A lofty question certainly, and not one I am going to tackle in a blog post. But perhaps I can say a bit about what a human being is not for. This is the key to Henry Farrell’s piece which is his take on the humanist critique of AI. We are presumably primarily designing the future for humans. All humans. Not just Tech Bros. And the design needs to bear that in mind. For example, a human being is not, in my opinion, for this (from the Cory Doctorow link):
Like an Amazon delivery driver, who sits in a cabin surrounded by AI cameras, that monitor the driver’s eyes and take points off if the driver looks in a proscribed direction, and monitors the driver’s mouth because singing isn’t allowed on the job, and rats the driver out to the boss if they don’t make quota.
The driver is in that van because the van can’t drive itself and can’t get a parcel from the curb to your porch. The driver is a peripheral for a van, and the van drives the driver, at superhuman speed, demanding superhuman endurance. But the driver is human, so the van doesn’t just use the driver. The van uses the driver up.
The first task of the education establishment, I think, is to attempt to protect the graduate from becoming the reverse-centaur described above, whether a deliver driver, a coder (where additionally the human-in-the-loop becomes the accountability sink for everything the AI gets wrong) or a radiologist. This will often be resisted by the employers you are currently very sensitive to the needs of as educators (many of who are senior enough to get to use the new technologies as a centaur rather than be used by them as a reverse-centaur, tend to struggle to put themselves in anyone else’s shoes and, frankly, can’t see what all the fuss is about) but, remember, the cosy world of employability outcomes is over. The employers are not sticking to the implicit agreement to employ your graduates if you delivered the outcomes and therefore neither should you. Your responsibility in education is to the students, not their potential future employers, now their interests no longer appear to be aligned.
What do we owe the future?
This depends on what you mean by “the future” of course. If it is some technological dystopia of diminished opportunities for most (even for making friends as seemingly envisioned by some of the top Tech Bros), then nothing at all. But if it is the future which is going to support your children and their children, you obviously owe it a lot. But what do you owe it? What is owed is often converted into money by the political right, and used to justify not running up public debt in the present so as not to “impoverish” future generations. What that approach generally achieves is to impoverish both the current and future generations.
But if you think of owing resources, institutions and infrastructure to the next generation, then that is a responsibility that we should take seriously. And part of that is to produce an educated generation with tools, systems, institutions and infrastructure. The education institutions must take steps to make sure they survive in a relevant way, embedded in systems which support individuals and proselytising the value of education for all. They must ensure that their graduates understand and have facility with the essential tools they will need, and have developed the ability to learn new skills as they need them, and realise when that is. This is about developing individuals who leave no longer dependent on the institutions, able to work things out for themselves rather than requiring never-ending education inside an institution.
What remains worth the difficulty of learning?
The skills already mentioned will be the core ones for everyone, and these will need to be hammered out in terms everyone can understand. But in the world of post scarcity education, which is here but which we have not yet fully embraced, the rest will be up to us. A large part of the education of the future will need to be about equipping us all to understand what we now have access to and when and how to access it. We will all have different things we are interested in, or end up involved with and needing to be educated about. It will be up to each of us to decide which things are worth the difficulty of learning, but to make those decisions we will need education that can support the development of judgement.
For education institutions, the question will be what is not worth the difficulty of learning? Credentialising based on now relatively meaningless assessment methods will not cut it. This is where the confrontation with employers and politicians is likely to come. Essential skills and their related knowledge will be better developed and assessed via more open-ended project work and online assessment of it to check understanding. These will need to become the norm, with written examinations becoming less and less prevalent. Not because of fear of cheating and plagiarism, but because an outcome which can be replicated that easily by AI is not worth assessing in the first place.
