I developed a business continuity plan for the Birmingham office I worked in once. It involved a mainly paper exercise, establishing where the various links required were, having a gawp at the spare office space available to us somewhere near Leicester if the shit went down, but also preparing a bag containing torches and a copy of our plan. I think we might even have sprung for a hi-viz jacket.

Now apparently we are all suddenly expected to do something similar on an individual basis. In the face of risks unspecified, we have been told that 3 in fact really IS The Magic Number. Food and water for three days and nights, by which time presumably it will be safe to venture outside (!). Fresh from the success of See It – Say It – Sorted we now have three phrases depending upon whether we are running into the house (Go In – Stay In – Tune In) or out of the house (Get Out – Stay Out – Ring 999). We are all encouraged to fill out a Household Emergency Plan. This includes space to fill in the number of your GP and insurance company details, like you might do in the aftermath of an unsatisfactory holiday. Also it is suggested you note down the FM frequency of radio stations you would turn to for national or local news, like when you need to know whether schools are closed. All couched in familiar terms so as, presumably, not to frighten the horses.

By failing to prepare you are preparing to fail, as Benjamin Franklin almost certainly did not say.

Well, neigh.

It all starts to go tits up when you look at your local area’s resilience forum. Here’s mine:

They have a range of advice across these risks:

I went straight to the Power failure one, as there seemed to be more attention given to this risk than any other in the emergency supplies section of my National Emergency Plan:
● Torch (battery/wind up) and spare batteries – torches are safer then candles
● FM radio (battery/wind up) to get updates during a power cut
● Portable power bank to charge phone and medical devices (if compatible)

One of the 5 headline pieces of advice under Power failure is, hilariously, “Consider an analogue landline phone that will work in a power outage”.

No it won’t. Apparently they have not been keeping up to date with the activities of openreach, whose website cheerfully lays out the sell by date for this particular piece of advice:

It is not the only problem with the information provided by West Midlands Resilience Forum. Their list of organisations to turn to in a crisis includes integrated care boards which exist in name only in the current chaos at the top of the NHS and the wrong name for the region’s main airport, amongst the issues I could immediately spot:

So it is tempting to perhaps suggest that they update their own national emergency plan before demanding them from us. Particularly as Wes Streeting’s contributions appear to have pushed up sales of tinned goods at Iceland by 40% over the weekend, thereby making gaps on the shelves more likely for everyone else.

Remember Raymond Briggs’ parody of the government’s Protect and Survive pamphlet from the mid 70s – supposedly allowing us to individually prepare for nuclear war – When The Wind Blows? It feels like we are right back there:

And yet. Carol Cadwalladr has a different take:

A confession: I’ve been a covert prepper since a conversation with an Irish open-source researcher, Gavin Sheridan, two years ago. It was at the time Russian spy ships that were circling Britain and Ireland and the anchor of a shadow fleet tanker had “accidentally” severed a cable between Finland and Estonia.

What’s the worst thing that could happen if they did succeed in cutting the cables between Britain and America, I asked Gavin. He thought for a moment “Well, I guess the first thing would happen would be the total collapse of the global economy,” he said cheerfully. I told him I was thinking of buying a wind-up radio. I’d contemplated the idea of not knowing what was happening but at the same time I didn’t want to become one of those people US cable networks make documentaries about.

“I’ve got two!” Gavin said. “One for me and one for my parents.” When I saw him this summer he told me that he’d now got a Starlink antenna stashed away in his cupboard. Another of my friends, a world-renowned European tech expert, told me a year and a half ago that she keeps cash in the house and the tank of her car full. I would never have admitted publicly that I followed suit with the cash or that after European intelligence chiefs warned of an imminent attack by Russia this September, I got some water containers in too. It would have been much too embarrassing. And yet here we are with an extensive list the UK government has just put out – blankets, torches, three days’ supply of food, a power bank – it makes my own efforts look frankly pathetic.

There are risks out there. And most other European countries gave their citizens similar advice several years ago.

Then there is Cory Doctorow on the risk of the United States deciding to “brick” essential goods and services because we have done something Trump doesn’t like:

Now that Trump has made it clear that the US no longer has allies or trading partners, only rivals and adversaries, everyone in the world is trying to figure out whether they can trust American tech infrastructure with their governments, businesses and personal data.

The answer is a resounding “no”. Just look at the international criminal court, which ditched Microsoft Office for a European alternative after Trump sanctioned its officials for issuing an arrest warrant for the génocidaire Benjamin Netanyahu. Directly after Trump denounced the court, its justices lost access to all their Microsoft accounts – emails, documents, calendar, address book. The court was, in effect, “bricked”. Microsoft denies this, but between the justices of the international criminal court and the US tech monopolist, I know who I trust.

Trump is capable of weaponising the US’s tech companies, and there’s no telling where it will end. Remember that when Russian looters stole millions of dollars’ worth of John Deere tractors and spirited them away to Chechnya, the company was able to push a kill signal to the tractors that rendered them inoperable.

Until we repeal the anti-circumvention law, we can’t reverse-engineer the US’s cloud software, whether it’s a database, a word processor or a tractor, in order to swap out proprietary, American code for robust, open, auditable alternatives that will safeguard our digital sovereignty. The same goes for any technology tethered to servers operated by any government that might have interests adverse to ours – say, the solar inverters and batteries we buy from China.

And we also know from Paul Behrens’ contribution to the National Emergency Briefing that the UK is “woefully unprepared” for food system failure.

So some level of individual preparation, while certainly nowhere near sufficient, may perhaps be necessary. And here the West Midlands Resilience Forum are quite right that all of us who are able to need to “reduce the need for support from others and enable you to support the vulnerable members of your street and community”.

Now where did I put that torch?

Source: Universal Pictures Publicity, Fair use, https://en.wikipedia.org/w/index.php?curid=71354716

10 years ago I was working my way through Cathy O’Neill’s excellent Weapons of Math Destruction, the first really loud warning about the inappropiate application of algorithms, and blogging about it. In the book she set out the case against devolving important decisions to mathematical models without adequate feedback loops. Yes, 10 years ago!

I concluded that post by paraphrasing Robert Oppenheimer’s words about the Manhattan Project:

If mathematical models are to be the dominant regulatory tool of a financial world, and of the consultancies and financial firms competing in that world, then the time will come when mankind will curse the names of the highly paid professionals who followed inappropriate rules rather than exercising their own expert judgement when it mattered.

I think that is still a work in progress. Although at least they have made a film about the Manhattan Project in the meantime. And you could say the United States certainly appear to be embracing the analogy:

Meanwhile Cathy O’Neill has done a lot more than just wring her hands about these risks. She has set up her own consultancy (O’Neil Risk Consulting & Algorithmic Auditing or ORCAA) to audit your algorithms for you. As she says:

Whether it’s a hiring algorithm, healthcare AI, predictive scoring system, or generative AI platform, we are here to think about how it could fail, for whom, and what you can do to monitor and mitigate these risks. We are actively developing frameworks and governance approaches to help companies and organizations use algorithms and AI safely – confirming these technologies perform as intended and operate within sensible guardrails. We help our clients realize the transformative benefits of AI, while avoiding discrimination, bias, and other problems.

Their offering includes:

  • Algorithmic audit, including generative AI, automated decision systems, predictive models, and facial recognition;
  • Quantitative testing for regulatory compliance;
  • AI Governance and Risk Management Consultation;
  • What they call “Cockpit design” – “To make an effective cockpit, you must understand what can go wrong in flight, and include dials/gauges that monitor for those risks. We help build cockpits for your AI systems”; and
  • Education and Training.

ORCAA have been going for 7 years now, with a range of high profile clients, including AirBnB, Siemens and the City of Amsterdam.

This seems to me a good example of expertise being used in the right way.

I think I may have been naive.

A couple of weeks ago I took the IFoA’s AI Manifesto to task for wanting to be on the Red Team rather than the Blue Team, ie the one supposed to be stopping as many AI errors as possible. As I said then:

The Red Team breaks into systems by exploiting a single vulnerability. That is comparatively easy. The Blue Team needs to reassure its clients that it has removed ALL vulnerabilities.

But what if you are not working to protect people from vulnerabilities? What if you are working to protect and accumulate capital instead?

Today I was alerted to the latest article from the IFoA GenAI Working Party, Agent Skills in the Enterprise: Making Actuarial Process Knowledge Reusable. It explains “how an agent skill package can connect reusable instructions with scripts, controlled references, templates, examples and tests”. This will allow organisations to “preserve not only completed models, reports and workpapers, but also the process checks, review decisions and handling of exceptions that produced them using the “agent skill” framework”. As the report puts it:

The problem is not simply that organisations need better prompts. They are preserving the outputs of professional work while leaving much of the procedure that produced them implicit. A completed report records what was concluded. It rarely records every data check, exception, escalation point and review decision needed to reach the conclusion reliably next time.

I can immediately see the motivation. I have been that person trying to replicate a previous valuation result produced by a former colleague I can’t contact, with incomplete data and a-no-longer-supported valuation system which keeps going on the blink. But it was the mention of process knowledge which caught my attention.

I wrote about process knowledge and its importance around a year ago, citing the example of the shrines at Ise Jingu in Japan (which I had learned about from Dan Wang’s Breakneck). This is a now over 1,300 year project to preserve the traditional techniques of building these shrines which involve no nails, just dowels and wood joints. Skills are passed from one generation to the next, there is a tree planting programme which has survived a dozen or more historical eras, pandemics, volcanic eruptions, earthquakes, invasions and oh so many wars over that period. As well as two nuclear bombs.

The important thing is not the ability to build a shrine over 1,300 years later exactly as it was built then of course. It is a proof of concept. It has preserved intricate difficult skills over more than a millenium. And it has involved people, lots and lots of people.

By contrast, the IFoA project has a different, rather shorter term, goal, which is laid out as follows:

This is the first of five articles. The second examines how completed work can become reusable packages. The third introduces SkillOps as the discipline for governing agent skill packages in actuarial work. The fourth considers the consequences for consulting when expertise previously carried through human capital is codified in reusable agent skill packages and becomes intellectual property. The fifth places skills within the wider challenge of continual learning.

It is the fourth article which really bothers me. Once not just the details of a project, but all of the other parts which do not normally get passed on (ie the process knowledge) are captured in this system, it becomes intellectual property, which Cory Doctorow defines as:

any rule, law or policy that allows a company to exert control over its critics, competitors or customers

Up until now, process knowledge has not been given much attention, mainly because it can’t be turned into intellectual property. Process knowledge has always been what remains in the heads of the skilled technicians employed by businesses after everything not nailed down which can be taken ownership of by those businesses has been. Process knowledge has always been the nailed down part. It is the part that has allowed the last vestiges of worker power within most corporations to survive.

Now a group of actuaries think they have come up with a plan to monetize that for the firm as well. To pull up those last remaining nails of worker power.

Think how the IFoA GenAI Working Party would have approached the Ise Jingu shrines.

First you outsource the tree planting programme to a specialist forestry manager. Minimum tender of course: need to get maximum value for money here. Don’t worry about having control over the delivery of timber of just the right type in just the right condition for a unique project. Just leave that to the specialists working on a budget.

Then you sit with the old guys taking you through the various processes involved. Might need some video capture for some of it. Make sure everything they say is captured and then cross-referenced for easier access subsequently. Don’t take too long over it as time is money. Don’t actually feel what it’s like to work with the tools, just note down everything about tool selection. Don’t actually apply the required pressure in the right direction with respect to the grain of the wood, its dryness, the precise cut from the trees selected. Just make sure the video is high enough quality.

I would be surprised if this project survived 20 years. In fact, in view of the fantasy economics of the AI industry, I would be surprised if it lasted even five years if the AI Agents involved in this project relied to any degree on the survival of any of the Magnificent 7.

And this is just to build a shrine out of wood.

These two projects are doing two different things. One is trying to preserve a human activity over a non-trivial period of time. The other is attempting to monetize it so that it can be included in the bottom line for the next quarter. And perhaps to float the idea that your business might be able to dispense with those pesky new employees acquiring process knowledge.

Which brings me to another alarm call. This time from an anonymous group of senior professionals in the UK and European insurance and reinsurance industries called Sounding the Alarm and published in English, French and German. It is not long, so I urge you to read it, but I want to focus on this passage about how the insurance industry is responding to the climate crisis:

Our industry’s strategy is to manage renewals in order to limit our exposure as the physical risk manifests so that premiums are sufficient to cover the rising claims load. This is not a malicious act but a simple response to the commercial realities we are faced with.

We will do this in three ways which many clients are unaware of and governments across Europe are failing to internalise the consequences of:

Firstly, we will reprice our cover. We will simply make policies much more expensive to reflect this elevated physical risk and uncertainty. This will have wider consequences for economic inequality, making it harder for less affluent households to access insurance. It will also increase costs for critical industries such as farming or shipping , creating a significant social and financial burden for government s and, in turn, its taxpayers.

Secondly, we will limit coverage to exclude elevated risks such as wildfires as well as other less predictable risks . The nature of insurance being re – negotiated on a 12 – month basis allows us to do this without friction. Ensuring customers really understand that this is happening is critical.

Thirdly, if there is no way to bring premiums and claims into balance or introduce risk reduction measures we will – for certain geographies and sectors – simply withdraw coverage . This will particularly affect coastal, mountain and flood-prone communities.

This approach is our way of managing our own financial exposure and maintaining profitability at a time of increasing physical risk and increasing uncertainty. Doing so is critical to ensure we can continue to pay claims, therefore fulfilling both our fiduciary duty and our role in society.

This is the flipside of the capitalisation of process knowledge of course. If it can no longer be monetized then ditch it, all the while spouting about “our role in society”. The people on the receiving end will not care whether it is a malicious act or not, it will feel like one.

Both of these stories tell us which side many actuaries are on, and it is the side of capital rather than the side of the people from who it has been extracted. It is not malicious, and it is how many actuaries understand fiduciary duty. To the contract rather than to public interest.

I look back on my own actuarial career. Some time as a scheme actuary where I really felt I was safeguarding the pensions of ordinary people, some time protecting the wealth of already fairly wealthy people from the tax authorities, some time in insurance feeling really quite remote from customers of any kind. And then 10 years helping students into actuarial work. Have I really just been a tool of capital this whole time? Is this why a IFoA sustainability WhatsApp group, which is all about connecting sustainability thought leaders together and raising awareness of the impact of the climate emergency across the finance sector (where I first saw the Sounding the Alarm memo), is so small?

Dan Davies, highlighting Robin Wriggleworth’s new book, A Fabulous Debt: The Epic Story of How Bonds Built the Modern World, wonders whether life insurance actuaries and general insurance actuaries (referred to as casualty actuaries here) have a similar relationship as bond traders and equity traders. It’s a good read. There is even a reference to Craig Turnbull’s magisterial A History of British Actuarial Thought! And this is the conclusion:

Which leads me to the paradox of the title. Reading the histories of actuaries, there seems to be quite a similar needle between the life and non-life insurers; one group place their trust in the higher mathematics, the other deals with heuristics, rules of thumb and vibes. It all comes to a head when you have to draw up the accounts, of course – the big intellectual battle seems to have been the casualty actuaries’ practice of setting reserves depending on the sum of each individual policy, which is really contrary to the spirit of treating risks as draws from a single distribution.

So it makes me think that there is some potential general principle here. Fields of human endeavour where management makes most use of complicated mathematics are likely to be less intrinsically complex and difficult. We use quantitative techniques only when we are content with regarding something as quantifiable.

This feels like a possible explanation. If your acknowledged expertise is in clever mathematics, you are likely to favour techniques which make more things quantifiable and therefore manipulable with clever mathematics. And the more actuaries flood into new fields, the further the quantifying will go.

Capitalism has always been in the business of quantifying things, allocating them to someone (often using considerable violence to do so) and then selling them as property. But late-stage capitalism has a now almost cartoonish obsession with quantifying even things that cannot be easily or successfully quantified, extracting ownership of the thing at the point of quantifying it and then selling it on.

When all the process knowledge has been turned into intellectual property, proprietary knowledge behind paywalls that hide the fact that it doesn’t really work any more, businesses will start to realise that they still have important gaps that are now nearly impossible to fill. There may be a Sounding the Alarm memo about it from anonymous insiders.

But it will be too late by then.

We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.

T.S. Eliot – Four Quartets

I feel like I am having a bit of a T.S. Eliot week. I have been writing about the OBR since the beginning of this blog in 2013, with their repeated failures as a forecaster forming the banner for the blog for many years.

What has brought me back to the OBR, as if for the first time, is recalling during the recent Funding the Future conference the words of the very good and wise friend of mine who was there with me. I had invited him to talk to one of my undergraduate groups at the University of Leicester during a professional skills session a few years ago. We were looking at past disciplinary cases of the Institute and Faculty of Actuaries and discussing what the actuaries involved could have done better.

In one of the case studies we considered, an actuary had produced a quarterly reserving report for a company, estimating the reserves on a ‘high’ basis of £74.5 million, a ‘low’ basis of £36.9 million and a ‘best’ basis of £52.1 million. The company then produced financial statements for the year, approved by the board of which he was a part, containing a reserve figure of £15.5 million. My friend told the students that the way to challenge in a situation like this is not to focus on the numbers, but instead see that each different number reflects a different story being told. So you need to look at what the story would need to be for the £15.5 million to be an appropriate figure and take issue with the things that don’t make sense within that story.

It was great advice then and still is, and it is definitely the way to look at the OBR I think. We find a story unsatisfactory if the characters are shallow, do not behave in a credible way, and the plot oversimplifies a much richer situation (something which every story must do to some extent to be manageable to the reader) in a deeply unsatisfying way.

Several other commentators have talked about the inability of the OBR to forecast anything (including me) – this is the equivalent of insufficiently fleshed out characters behaving in unbelievable ways. Richard Murphy, for instance, pointed out how ridiculous the proposed character plotline for inflation was at the last budget:

Whereas in fact of course the downward trajectory predicted at the last budget didn’t look remotely likely even then. The Consumer Prices Index (CPI) rose by 3.1% in the 12 months to August 2026, up from 2.9% the previous month.

However it is Dan Davies’ recent couple of Substack posts (one and two) which get to the heart of the problem in my view. Dan Davies’ contention is that it is the plot oversimplification which is the main problem here. As he says:

The system we have set up only seems to be able to communicate two states though – the signal is either “COMPLIANT – WAR CHEST” or “NONCOMPLIANT – BLACK HOLE”. It would be a significant improvement even for the OBR to add some measure of urgency or non-urgency to its press release – a sentence like “this surplus is well within the forecast error, however, so it should not be used to justify a change in policy”.

But, in his view, it is even worse than that:

In general, most things which a government spends money on will tend to have effects on the economy which go well beyond the five year forecast. The OBR uses an information set which contains only a small and quite arbitrary subset of government decisions – it is, literally, not capable of representing the system it is meant to control at an acceptable level of accuracy.

The justification for ignoring the future consequences of current spending seems to be no more sophisticated than what we pejoratively call “Treasury Brain” – the belief that a) spending departments always claim that their budget should be seen as an investment, b) that they systematically overestimate the long term benefits of current costs and c) that the best solution to a) and b) is to completely ignore any future benefits from investment or future costs caused by noninvestment. It’s just not good enough.

It is this dog’s breakfast that apparently tells us whether we are on track to meet the plotline the government has set for itself. Its three fiscal rules:

Rule 1. The current budget should be on course to be in balance or surplus by 2029/30 (‘stability rule’);

Rule 2. Net financial debt should fall as a share of the economy in 2029/30 (‘investment rule’); and

Rule 3. Some types of welfare spending must remain below a pre-specified level (the ‘welfare cap’).

And if that gruesome threesome doesn’t set your heart racing about the country’s future story, you are not alone. Because the people banging on about fiscal space will always push for governments to spend less, regardless of the circumstances. For instance, at the height of the pandemic in December 2020, Moody’s were saying this (with excess deaths already estimated at over 30,000 in the UK by Public Health England at that point):

However, compared to the government’s March budget (that was quickly overtaken by events), there are some initial signs that fiscal policy outside of investment is likely to be less expansive than previously announced. What remains unclear is whether this ambition will be able to withstand the political pressures that seem to be inevitable given the government’s previous commitments. Even before the Spending Review, longer-term spending commitments for health, education, and defence had already been announced. Together, these three areas account for around 60% of total expenditure.

One of the themes from the Funding the Future conference was, to quote from Educating Rita:

“There must be better songs to sing than this”.

I agree. One possibility is proposed by the New Economics Foundation, who suggest categorising policies by their level of multiplier effect (ie how much economic activity is increased by an increase in spending) and replacing fiscal rules with a “fiscal referee” who can make a more holistic judgement addressing many of the omissions Dan Davies complains about. This would then lead to a discussion rather than the compliant/non-compliant cliff edge we currently have, and allow the government to make its case. And it would not automatically incentivise cuts like the current system does.

Other possibilities exist. As Dan Davies describes it:

any accounting system is a mental prison of some kind, and one which needs to be periodically escaped from

In my view, a jailbreak is long overdue.

A no entry sign over an image of a reverse-centaur where the robot is in control

Last month the Institute and Faculty of Actuaries (IFoA) launched what it calls its “AI Manifesto“. I thought I would take a look.

The Manifesto is admirably short and to the point. Its pitch can be summarised by this final line of its Introduction and context:

For the benefit of society, we encourage those responsible for developing and deploying AI systems to ensure that there is an actuary in the loop.

Think of it as a massive “Gissa job” plea to those companies currently investing in AI systems. But what sort of jobs are we pleading for?

The idea is that actuaries have been working with models which use machine learning for many years, that we have statistical expertise, are comfortable working with data, used to stress-testing systems and also to looking at the risk appetite of a business as a whole.

I think the first hint that there might be a problem here is when the Manifesto moves onto ethics and the public interest. As it says:

Actuaries help teams balance commercial aims with societal expectations, offering constructive solutions rather than imposing limits.

It is difficult to square this with Anthropic‘s recent IPO filing, where they warn its technology may pose “existential risks to humanity”. I am pretty confident that Society expects not to be put at existential risk. So if Anthropic thinks its technology is that dangerous, there is no balance to be found between commercial aims and societal expectations here. Societal expectations are paramount. And limits will need to be imposed rather than “constructive solutions”.

But, in my view, the main problem here is what “real world” the IFoA feel they are serving. This comes through in the role for “professionalism” they envisage:

Projects with sound validation, honest documentation and clear accountability clear governance more quickly, avoid late-stage rework, and fail in testing rather than in production. Professionalism is what lets an organisation say “yes” to AI with confidence – and protects it from the adverse outcomes that would otherwise force it to say “no”.

This, remember, is in an environment where the latest large language models like Anthropic’s Claude Opus 5.5 or OpenAI’s GPT-6 Astra, are proprietary models where validation, documentation, accountability and governance are all just what they let you see. A framework which allows companies to say yes with misplaced confidence is unlikely to be serving the public interest. And translating a capacity to apply a regulatory model to an insurance company where you have all the data and can make informed judgements about the things that you don’t know (Actuaries are used to working with large, imperfect, transactional datasets subject to audit and regulatory scrutiny. We clarify what data is needed, what quality level is acceptable, and how data should be checked, reconciled and monitored over time) is emphatically not the same as working with a model where you don’t know precisely how it works, cannot accurately predict the errors it will make and do not have enough time to look in all of the places you need to.

Take bias for instance. The Manifesto says:

We examine how a model behaves for different groups and sub-groups, test whether training data is representative of the people the system will affect, and help manage the biases we find responsibly through fairness metrics for predictive models, and output audits and red-teaming for generative systems.

Nita Farahany in the law and policy course documented in her Thinking Freely Substack demonstrates the limits of that bias management in practice, and this is Cory Doctorow on red-teaming (from Red Team Blues):

That’s the problem with blue teaming it—you need to be perfect, while—The red team only has to find a single error

Yes, that’s right, the IFoA haven’t even understood that they are on the Blue Team when it comes to AI errors, ie the one supposed to be stopping as many of them as possible. The Red Team breaks into systems by exploiting a single vulnerability. That is comparatively easy. The Blue Team needs to reassure its clients that it has removed ALL vulnerabilities.

That will be the role of any “actuary in the loop”. And the chances are it won’t be possible. The three examples given by the IFoA are as follows:

A generative AI customer assistant
An insurer wants an LLM-based assistant to answer customer queries. An actuary designs the evaluation
framework: a curated test set of realistic queries, explicit tolerances for error, escalation rules for high-stakes topics, and live monitoring of answer quality. The project clears internal governance at the first attempt because approvers can see exactly what “good enough” means and how it is evidenced.

A machine-learning decision model
A lender deploys an ML model to score applications. An actuary tests behaviour across customer sub-groups,
identifies a data artefact that disadvantages one segment, and works with the data science team to correct it. The documentation of that judgement becomes the centrepiece of a constructive conversation with the regulator.

AI risk across an organisation
A Chief Risk Officer faces AI adoption in a dozen business units at once. An actuary helps articulate an enterprise AI risk appetite, maps where the organisation depends on the same foundation models, and builds board reporting that distinguishes experimentation from deployment. The organisation accelerates adoption in low-risk areas precisely because it now knows where the high-risk areas are.

None of these examples look unreasonable in isolation, but, to justify the eye-watering expense of this technology, it will need to increase speed and capacity significantly. So, rather than a single generative AI customer assistant, imagine a client relationship management role where a junior actuary or senior student manages 10 times the current expected number of clients at the same time, because the generative AI customer assistant makes that possible?

Will the skills of an actuary versed in the intricacies of Solvency 2 or pensions regulation be so revered by your client that you would be given more than enough time to rubber stamp a system serving 30 or 40 clients? How long do you think you’d need to check the output on all of them properly? I think it unlikely you will be given that long.

This is the risk of becoming a reverse centaur, where you work for the assistant rather than the other way around. And my view is that the actuarial education system is positively encouraging the development of reverse centaurs amongst our current students.

It gets worse. The early evidence suggests that working with AI systems encourages rubber stamping. In one study of AI-generated audits with an auditor in the loop, 150 of the 160 reports were submitted without any changes made by the human. The performance of humans in the loop may also be adversely affected by working with AI systems, leading to “agency decay”:

Agency decay is the gradual erosion of a person’s ability and willingness to observe carefully, think independently, choose deliberately, and act responsibly. It does not arise because AI is inherently harmful. It arises when convenience becomes the default setting for cognition. A tool that first helps us think can begin to think around us, then for us, then without us noticing what has been weakened.

There are lessons to be learned from other professions who have been working for much longer with expert systems. The aviation industry for example. As Craig Bright, Co-Chief Operating Officer at Barclays has posted:

So the test shouldn’t be whether a human appears somewhere on the process map.

It’s whether the system remains understandable, controllable and recoverable when the agent is wrong, uncertain or unavailable.

Safety doesn’t come from leaving a human in the loop.

It comes from designing a system the human can still control.

The IFoA’s position appears contradictory. On the one hand, they clearly don’t believe that AI is as powerful or as dangerous as Anthropic are claiming in order to push up the IPO share price, or I assume they would not be pushing so hard to get an actuary on every team “developing and deploying AI systems”. I think they are right not to believe it, for a number of reasons, but Naomi Alderman’s AI predictions: seven ways you can tell useful thinking from sci-fi fantasies gives probably the most entertaining ones.

On the other hand, if this is just another technology for which the price is going to need to adjust to reality at some point, we need to be very careful as a profession not to be part of the mob urging their clients to adopt it at a faster rate than they would otherwise. Particularly if it is damaging our own professionals in the process.

AI in some form will survive the inevitable crash, but the shape and scope of the wreckage that will be left behind is currently hard to predict. Let’s be careful what we wish for.