Ceremonial Compliance
First defined in Outpaced by AI by Waydell D. Carvalho.
- Committees meet, audits run, and reports are filed, but the system never stops.
- Each oversight body reaches its own conclusion and closes while the system keeps running.
- "We followed every procedure" is offered as proof the outcome was sound.
Outcome: A complete paper trail of oversight sits on top of a system nothing was able to halt.
A governance process that runs on schedule, produces records, and closes without escalation describes itself. It does not, by that fact alone, describe the system it is supposed to govern. Ceremonial Compliance is the belief that because the oversight exists and operates, it has to be constraining what the system does.
The Dutch childcare-benefits scandal is the case. From around 2013 the Dutch Tax and Customs Administration used an algorithmic risk model to decide which childcare-benefit claims to flag, and Amnesty International later characterized it as a self-learning system that treated a foreign nationality as a reason for suspicion. Tens of thousands of families were branded fraudsters and ordered to repay years of benefits at once. The system ran for seven years inside a country with functioning courts, a functioning press, and a functioning parliament.
Every one of those institutions operated. Legal review ran. Data-protection oversight ran. Parliament scrutinised the programme and the press reported on it. And the system kept running through all of it, because each channel reached its own conclusion and closed, and none of them had a mechanism that actually stopped the machine between conclusions. It ended by bringing down the government, not by being switched off.
That is the trap. Oversight that produces records is easy to mistake for oversight that produces constraint. The records are real. The meetings happened. But a meeting that cannot halt the system, an audit that cannot change it, and a report the system outlives are governance in form only. The activity becomes its own evidence, and the question of effect never gets asked.
AI raises the stakes, because the records are even easier to generate and even easier to trust. Model cards, review boards, audit logs, and sign-off checklists all accumulate. If none of them can stop a deployment that is causing harm, they are ceremony. The test of governance is not whether it ran. It is whether, on the day the system needed to be stopped, anything in the structure could stop it.