In 1985 and 1986 I was a senior software engineer in the education division of FACOM, now Fujitsu. While I was there, I was trained as one of the first formally certified “knowledge engineers” in Australia. I always hated the term. The language was Lisp, and the systems we built were what were then called expert systems.
One of the experts I worked with was a senior statistician. He told me something in passing that I have never forgotten: “The problem with a computer screen is that you can’t scribble a note in the margin.”
I thought of this last week.
The algorithm nobody can overrule
On 26 October the Commonwealth Ombudsman, Iain Anderson, released his report on the algorithm that decides how much home support an older Australian receives.[1] Since 1 November last year, an assessor sits with the person, fills in a questionnaire, and the algorithm turns the answers into a level of funding. More than a million people receive services through that system.[1]
The assessor who sat in the kitchen cannot change the answer. The report puts it plainly: “Under the law, these automated decisions cannot be overridden and changed, even if they are clinically inappropriate.”[2] It also says there is no public information that “meaningfully explains how the algorithm converts inputs into outcomes. Without this, the IAT is a technological black box.”[2]
The ombudsman found the algorithm was put in place “without any meaningful risk assessment, evaluation or analysis, despite the obvious risks that would arise with deterministic automated decision-making”.[1] Testing was done on an original version built by a consultancy; the version in use was then modified by the department, with no external validation.[1] The department does not accept that it failed to test or validate the algorithm.[1] Appeals rose more than five-fold.[1]
Of his ten recommendations, six were accepted. The first two, to change the law so a human can override the algorithm, were only “noted”.[1] Instead there is an “escalation pathway” for exceptional cases, which the ombudsman is concerned will not be adequate.[2]
This is not an AI story
It would be easy to file this under artificial intelligence. It isn’t. As far as I can tell, this is a deterministic, rule-based system, the old-fashioned kind. It is much closer to what I was building in Lisp forty years ago than to anything that writes essays.
That makes it more worrying, not less, because we have known for decades how systems like this behave.
We built ours by talking to experts, over and over, to capture their rules. Most were of the form “if this, then that”, with a twist: rules were ranked by importance, and whether one rule applied could depend on a sequence of others. Every new example we worked through exposed a case the rules didn’t handle. The experts would add a clarification, which usually meant more rules, which made the network more complex.
At some point, both the experts and the knowledge engineers lost the ability to follow the logic path through the system. Improving it became a loop: try a rule, see if it works, check for regressions, assess the result, try again. We stopped when all the worked examples gave satisfactory answers.
Notice what that means. The system was right for the cases we had thought of. Nobody could promise anything about the cases we hadn’t.
The margin is the safety net
That was the statistician’s point. A person working with whatever tools they need, even a pencil and paper, can handle the edge case the procedure manual never considered. They can scribble in the margin: this doesn’t fit, here’s why, here’s what I did instead. A computer system can’t.
There are always unforeseen situations and edge cases. With older people and their health, there are more of them than anywhere else. So any system that makes decisions about people, whether about funding, health, aged care or employment, must allow a human to review the decision and override it. Not as a favour. As part of the design.
A few weeks ago I wrote about an AI model that submitted a false tip to a police website about an unsolved murder.[3] It went nowhere, because the police process required a person to review every tip before it went anywhere near an investigation. The human check was the safeguard, and it worked.
The aged care system was built the other way round. The human is in the room, has seen the person, and has been told by law that their judgement doesn’t count.
And earlier this week I wrote about how hard it is to argue with a number once a computer has produced it.[4] Here, the law itself takes the computer’s side.
The fix doesn’t need new technology. It needs a margin: a way for the assessor to write “this is wrong, and here is why”, and have it count.
Anne Connolly, “Aged care funding algorithm neither fair nor consistent, ombudsman finds”, ABC News, 26 October 2026. https://www.abc.net.au/news/2026-10-26/ombudsman-finds-automation-in-aged-care-lacks-safeguards/107308730
Guardian Australia, 26 October 2026, quoting the Commonwealth Ombudsman’s report and an interview with Iain Anderson. https://www.theguardian.com/australia-news/2026/oct/26/damning-investigation-labor-algorithm-tool-aged-care-assessment
“When an AI Agent Invents a Witness”, 11 October 2026. https://blog.karabatsos.com/p/when-an-ai-agent-invents-a-witness
“Would You Mind Leaving the Room?”, 26 October 2026. https://blog.karabatsos.com/p/would-you-mind-leaving-the-room