By Rahim Hirji
Every board has someone who says it: we keep a human in the loop. It is the most reassuring sentence in corporate governance, and under European law, on its own, it protects nobody.
Why is the real risk of AI not automation, but accountability? Rahim Hirji, author of SuperSkills: The Seven Human Skills for the Age of AI explains. The line every board uses, “we keep a human in the loop,” protects nobody on its own under European law. The danger is a decision passing through a model and a chain of people until no one can honestly say they made it. Drawing on a landmark ruling and seven years of research across 200 organisations, this piece shows how accountability disappears, and the one test that restores it.
Frances Walter was an eighty-five-year-old in Wisconsin, with a shattered left shoulder and an allergy to most pain medication. An algorithm called nH Predict, matching her against a database of six million patients, estimated she would be ready to leave her nursing home in 16.6 days. A reviewer entered that estimate in her file. Two days before payment stopped, a medical director cited it again in finding she no longer met her insurer’s coverage rules, and payment stopped on the seventeenth day. Notes from that week record her pain at the top of the scale and that she could not dress herself unaided. A judge later called the denial, at best, speculative.
There was a human in that loop. There were two. The case is American, and we know the detail only because it went to court. The arrangement is not American at all.
No shoulder heals to a tenth of a day. But precision reads as authority, and everyone in the chain that followed treated the estimate as though somebody had thought hard about it.
So who actually made that decision?
The tool’s makers say it does not make coverage determinations and serves only as a guide. UnitedHealth, which later bought them, has been blunter: medical necessity determinations are made by physicians, not by AI. Take that at face value, because it is probably true. The algorithm did not decide, it produced an estimate. The reviewer did not decide, she recorded it. The medical director applied criteria to an assessment already made for him, and signed. There is no villain in that sequence, and no decision-maker in it either.
Boards are worried about the wrong thing. AI launders accountability. Responsibility goes in at one end, plausibility comes out at the other, and the trail between them is gone.
Why is automation the wrong thing to fear?
Automation is a decoy because it is visible. It is budgeted, it is scheduled, and it happens to jobs, which guarantees somebody fights about it. Accountability gaps have none of those properties. Nobody proposes one, nobody funds one, and nobody argues about one, because there is nobody to argue with. A workflow is amended, a step is added, and the organisation chart still shows the same names in the same boxes.
How does responsibility actually disappear?
It thins at each handoff. The analyst accepts the model’s output, and not out of laziness: a colleague’s poor reasoning can be argued with, whereas a model’s output arrives finished, and disagreeing with it feels less like judgement than accusation. The manager approves the analyst, the director approves the manager, and the board notes the outcome. Everyone acted reasonably, the aggregate is unreasonable, and you cannot fix that by hiring better people. There are no better people available. The audit trail, meanwhile, shows four approvals and no objections, which is exactly what it was designed to show.
Then the incentives arrive. STAT News reported that the tool’s owner set case managers a target of keeping patients’ stays within 3 per cent of the algorithm’s projection, later tightening that to 1 per cent. The reviewer’s job remained, formally, to exercise independent judgement, while her performance was measured by how closely she matched the machine. Nobody had to be instructed to defer. Deference was simply the cheaper of the two available behaviours, and it was recorded in the file as agreement.
Doesn’t AI governance already cover this?
Serious organisations do have the machinery: ethics boards, model risk committees, human-in-the-loop policies, and people sincere about all of it. UnitedHealth has an internal AI review board. In March, a US federal magistrate had to order the company to disclose who sat on it, and to name the medical directors who denied coverage to three hundred members of a proposed class. Its objection is the most instructive document in the file. Identifying everyone involved in issuing the denial notices, it argued, would impose a disproportionate burden, because that would encompass “thousands of current and former employees.” Thousands of them. Not one of them decided anything.
That case is unresolved and no court has found the company liable. Most AI governance frameworks, in my view, are built to make failure survivable for the people who approved them rather than less likely. Across seven years of research in more than 200 organisations, I have asked leaders to walk me through a decision their AI touched, and I have never once been given a name without a pause coming first. The pause is the finding.
What does accountability actually require?
A person. There are three places to put one in relation to a machine’s decision, and the middle position is the one everybody names.
Humans at the start. Before a model is pointed at a problem, one named person defines what a good outcome looks like, what the tool may conclude, and what would cause them to override it. That is where the decision is made. Everything after it is administration, however senior the person performing it.
Humans in the loop. This is the phrase boards reach for and the weakest of the three, and European regulators dealt with it years ago. Guidance endorsed by the European Data Protection Board holds that a controller cannot escape Article 22 of the GDPR by manufacturing human involvement, and that oversight must come from someone with “the authority and competence to change the decision”. In SCHUFA, a German woman was refused a loan on the strength of an automated credit score. The Court of Justice held that producing the score can itself amount to the automated decision, where the lender draws strongly on it. That closes the other exit. A person placed inside the process who has no power to change the outcome is a witness to it.
Humans at the end. This is the position most organisations think they already hold, and it is not a signature. It is an owner: somebody who can explain the outcome without reference to the tool, and who had the standing to refuse it. If you want to automate, you need this position and the first one. The loop between them will not save you.
The same word runs through all of it. The EU AI Act requires deployers of high-risk systems to give oversight to people with the competence, training and authority to act on it. The data protection test for meaningful human involvement turns on authority. Under the Senior Managers and Certification Regime, British financial services has run the principle as law for a decade: every senior manager holds a Statement of Responsibilities naming what is theirs, and Conduct Rule 3 requires that any delegation of it goes to an appropriate person whom you then oversee. Authority is the word organisations skip, and a model is not an appropriate person. Neither is a signatory who cannot say no.
Four things, then, before a model touches a consequential decision:
- One named individual owns it, in writing. Not the model, not the function, not the committee.
- That person can explain it without reference to the tool. If the only justification is that the system said so, there is none.
- That person has the authority to overrule it, and the time, and is not marked down for using either.
- If no name can be attached, the decision is not ready to be automated.
For directors this is not housekeeping. In the UK, section 174 of the Companies Act requires reasonable care, skill and diligence, personally, and that duty does not transfer to a vendor when the reasoning moves into software.
None of this is free. Naming an owner slows decisions down, and it will cause capable people to refuse to sign things they ought to have signed. Some refusals will be wrong, they will cost real money, and nobody has been able to tell me how much. Anyone selling named accountability as a free good is not being straight with you.
Somewhere in your organisation is a decision from this quarter that nobody can honestly claim. It was probably fine, and most of them are. But an organisation that cannot name who made a decision has not automated that decision. It has abandoned it.
Rahim Hirji
