10 years ago I was working my way through Cathy O’Neill’s excellent Weapons of Math Destruction, the first really loud warning about the inappropiate application of algorithms, and blogging about it. In the book she set out the case against devolving important decisions to mathematical models without adequate feedback loops. Yes, 10 years ago!
I concluded that post by paraphrasing Robert Oppenheimer’s words about the Manhattan Project:
If mathematical models are to be the dominant regulatory tool of a financial world, and of the consultancies and financial firms competing in that world, then the time will come when mankind will curse the names of the highly paid professionals who followed inappropriate rules rather than exercising their own expert judgement when it mattered.
I think that is still a work in progress. Although at least they have made a film about the Manhattan Project in the meantime. And you could say the United States certainly appear to be embracing the analogy:
Meanwhile Cathy O’Neill has done a lot more than just wring her hands about these risks. She has set up her own consultancy (O’Neil Risk Consulting & Algorithmic Auditing or ORCAA) to audit your algorithms for you. As she says:
Whether it’s a hiring algorithm, healthcare AI, predictive scoring system, or generative AI platform, we are here to think about how it could fail, for whom, and what you can do to monitor and mitigate these risks. We are actively developing frameworks and governance approaches to help companies and organizations use algorithms and AI safely – confirming these technologies perform as intended and operate within sensible guardrails. We help our clients realize the transformative benefits of AI, while avoiding discrimination, bias, and other problems.
Their offering includes:
- Algorithmic audit, including generative AI, automated decision systems, predictive models, and facial recognition;
- Quantitative testing for regulatory compliance;
- AI Governance and Risk Management Consultation;
- What they call “Cockpit design” – “To make an effective cockpit, you must understand what can go wrong in flight, and include dials/gauges that monitor for those risks. We help build cockpits for your AI systems”; and
- Education and Training.
ORCAA have been going for 7 years now, with a range of high profile clients, including AirBnB, Siemens and the City of Amsterdam.
This seems to me a good example of expertise being used in the right way.