Audit Theatre

Tests well, fails silently

When AI generates the work, AI evaluates the work, and AI fixes what AI flags, the audit loop closes on itself. The appearance of rigor is real but the rigor is not. The snake eats its tail.

Bruce Schneier coined "security theatre" for security measures that make people feel safer without doing anything to actually improve safety. Audit Theatre extends the pattern to design quality. Goodhart's Law describes it: "When a metric becomes a target, it stops being a good metric." Behaviour deforms to produce good measurements rather than good results. AI accessibility audits pass WCAG checks via pattern matching while missing user-experience failures a manual review would catch. AI-mediated design critique - where one AI generates and another evaluates - appears robust, multi-perspective, adversarial. But the loop closes on itself; the practitioner feels that review has occurred; the failure mode is in a place the audit doesn't look. Evals will help, but they are necessary, not sufficient, and relying on them is a brittle solution for quality.

Design transfer

Design review processes where rigor is performed by AI and the practitioner approves the verdict
Accessibility compliance documentation built from AI scans that miss experiential failures
Iteration loops producing version 2, 3, 4 activity without each version meaningfully addressing the issues from the previous
Quality dashboards that report green while user-facing failures multiply

In the wild

AI accessibility audits that pass WCAG checks via pattern matching but miss user-experience accessibility failures that a manual review would catch.— Practitioner reports across accessibility-focused design teams
Benchmark gaming: Goodharting is a robust phenomenon across reinforcement learning environments — models reliably optimise for the proxy reward rather than the underlying objective. The same dynamic appears when teams use AI to evaluate AI output.— Karwowski et al. (2023). Goodhart's Law in Reinforcement Learning. arXiv.
AI-mediated design critique where one AI generates the design, another evaluates it, and the practitioner accepts the loop output — producing a sense of multi-perspective rigor that is structurally absent.— Practitioner observations
Iteration-loop theatre: rapid AI-mediated iteration cycles produce 'version 2, 3, 4' activity without each version meaningfully addressing the issues identified in the previous. Volume of iteration substitutes for depth of revision.— Observed cross-industry pattern