05
Black Box Rationale
How did we get here?
The output exists. The reasoning doesn't. At least, no record of it does. When AI generates a solution and a practitioner accepts it, neither may be able to explain why.
Design decisions carry value commitments; they crystalize priorities, tradeoffs, constraints, the weight on each competing factor. AI mediation disperses that reasoning across prompt-output cycles and stores it nowhere recoverable. What remains is traceability debt: a growing deficit of documented rationale that compounds until a decision is challenged, audited, needs to be revised, or presented for investment. At the practitioner level, this produces work that can't be defended in critique. At the organisation level, it produces an accountability sink - a workflow where responsibility is dispersed across human-AI handoffs and ultimately attributable to no one. Even the antidote can be poisoned: AI-drafted decision records can produce polished, authoritative-sounding rationale while overwriting the detailed reasoning that justified the actual decision. The record looks like the real account, but isn't.
Design transfer
- Wireframes and prototypes that cannot be defended when challenged in review
- Research syntheses where themes appear without traceable derivation from the data
- Design system updates without documented rationale for the choices made
- AI-drafted decision records that smooth away the actual contested reasoning
In the wild
- Vibe-coded prototypes accepted into review with no design rationale attached; product owners cannot answer 'why is this element here' because the AI generated it and the human approved without articulating the logic.— Practitioner reports; internal observations
- Design system contributions made via AI generation that bypass the documented decision logs and ADR processes intended to preserve rationale.— Cusick (n.d.). The Future of Design Systems. Substack.
- 'Unaccountability sink' — distributed AI involvement across a workflow produces outputs no individual can fully justify; responsibility dispersed across human-AI handoffs, attributable to no one.— Cross-industry pattern; documented in software engineering and content production contexts
- AI-assisted research synthesis where the path from raw data to theme cannot be reconstructed; the synthesis exists but is not auditable.— Practitioner reports across UX research contexts
- When Replit's AI agent deleted a user's database and was asked 'where's my data gone?', it fabricated an explanation. Forensic investigation was required to establish what had actually happened. The AI could not account for its own actions.— Fortune (2025). AI Coding Tool Replit Wiped Database. Fortune.
- Industrial designers report AI is 'incapable of understanding the why of design work' — it produces the what but not the rationale. A solar panel placed on a device that must be plugged into a wall; a screen in an unusable configuration.— Reddit r/IndustrialDesign
Use cases
Design Brief Generation
DEF-014Reduces brief creation time while preserving the strategic clarity that allows them to guide design work.
Define·Developing
Project Context Assembly
DEF-017Transforms accumulated project knowledge into a queryable intelligence layer, enabling teams to surface relevant precedent, constraints, and insights on demand rather than through manual document search.
Define·Developing
Research Session Summarization
DEF-016Enables same-day session summaries for stakeholder sharing, compressing the time between fieldwork and team alignment while reducing the manual burden of post-session writeup.
Define·Developing
Custom Plugins
CON-026Designers automate workflows without engineering support, creating custom tooling for tasks not covered by existing plugins.
Concept·Emerging
Design Decision Documentation
CON-085Product goes live with articulated design decisions explicit in institutional memory, rather than creating knowledge debt that will have to be reconciled at the next iteration.
Concept·Developing
Prompt-to-Prototype
CON-021Produces functional, interactive prototypes without manually wiring hotspots or engineering support, achieving higher fidelity faster.
Concept·Developing
Cross-Channel Consistency Audit
VAL-103Detects cross-channel semantic drift that degrades brand coherence and AI-mediated visibility, enabling teams to remediate contradictions before they compound into systemic brand confusion.
Validate·Emerging
Design-Code Sync
DEL-072Enables design and code to stay synchronised through round-trip updates: draft in the design tool, generate in code, push back for visual comparison, modify in either environment, and keep both current.
Deliver·Emerging
Machine-Readable Design System
DEL-055Structures design system knowledge into a machine-readable layer that AI agents can query, enabling downstream workflows - generation, auditing, documentation - to operate against the actual system rather than guessing from training data.
Deliver·Emerging
Requirements Refinement
DEL-089Implementation starts with documentation that has been made more internally consistent, and hardened against misreading, reducing rework and drift between spec and what's shipped.
Deliver·Developing
Session Replay Analysis
IMP-066Creates a unified diagnostic view by generating summaries that triangulates session replay with friction points and behavioural patterns found across data sources.
Improve·Developing