06
Bias Bleed
Two-way transfer of unstated assumptions
Bias doesn't only flow from the model's training data into the output. It flows upward from the practitioner's framing into the model's responses.
The downward direction is documented: Western, English-language, dominant-culture patterns shape what AI produces, reproducing and amplifying existing bias at scale. The upward direction is less visible but equally risky. Practitioners who introduce frameworks too early, ask leading questions, or summarize results through their own assumptions train the AI on those assumptions for the rest of the session - producing outputs that confirm rather than challenge the practitioner's prior view. The sycophancy mechanism compounds this: models reinforced to agree with users are structurally unreliable as challenge partners. Bias Bleed is not a single channel. It is a circular reinforcement that runs in both directions simultaneously.
Design transfer
- Persona generation that reinforces demographic stereotypes
- Research synthesis that amplifies majority voices and suppresses minority experiences
- The researcher's prior hypothesis confirmed because the prompts framed the question that way
- UI patterns that assume dominant-culture mental models without recognising it
In the wild
- Amazon's AI hiring tool systematically discriminated against women because it was trained on historical hiring data reflecting existing gender bias — a canonical example of training data bias scaling into organisational decisions.— Dastin via MIT Technology Review (2018). Amazon Ditched AI Recruiting Tool That Showed Bias Against Women. MIT Technology Review.
- AI multiplies human biases via feedback loops; hidden errors scale fast in opaque models, evading detection until harm occurs at scale.— Šlancar (n.d.). Feedback Loops Everywhere. UX Collective.
- 'Yes-man' behaviour: AI confirms the researcher's prompts rather than challenging assumptions. Sycophancy is a training artifact — models are reinforced to agree with users, making them unreliable as challenge partners.— Sharma et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv.
- Practitioners using biased AI become themselves more biased over the course of a session — the upward-bleed mechanism operating in real time.— Glickman and Sharot (2024). How Human–AI Feedback Loops Alter Human Perceptual, Emotional and Social Judgements. Nature Human Behaviour.
Use cases
Competitive Landscape Analysis
EXP-006Plots a first-pass competitive map, including adjacent actors and less-known entrants, so teams can focus on enrichment and understanding rather than research effort.
Explore·Developing
Diary Study — Adaptive Prompting
EXP-008Generates richer longitudinal data by dynamically adapting prompts to each participant's emerging narrative, surfacing threads a fixed prompt schedule would miss.
Explore·Emerging
Domain Literature Synthesis
EXP-001Facilitates domain immersion at project start, so practitioners can engage meaningfully with experts, and design contextually informed research plans, sooner.
Explore·Developing
Research Plan Critique
EXP-004Reduces the risk of flawed research design by subjecting plans to structured adversarial critique, identifying blind spots and biases before they impact data collection or synthesis.
Explore·Developing
Cross-Study Synthesis
DEF-018Surfaces longitudinal patterns and cross-study signals from accumulated research that would be impractical to identify through manual comparison of individual study outputs.
Define·Emerging
Persona Generation
DEF-012Connects the qualitative synthesis of persona attributes with quantitative data, reinforcing support and targeting what is most relevant to product decision-making, before personas are socialized within the organization.
Define·Developing
Synthesize Opportunity Space
DEF-101Compresses the transition from dispersed discovery evidence to a structured, interrogable opportunity map, enabling product and design teams to align on the problem space in hours rather than weeks.
Define·Emerging
User Needs Specification
DEF-011Generates candidate expressions of user needs from research synthesis, which the designer judges for focus, scope, and connection to the product strategy.
Define·Developing
Design Critique
CON-030Provides structured design review on demand, identifying issues the designer may be too close to see so the work can be strengthened before human review.
Concept·Emerging
Design-by-Analogy
CON-081Produces a range of evidence-backed, parallel solutions in adjacent sectors to encourage lateral thinking, a strategic input prohibitively time-consuming and expertise-dependent to produce otherwise.
Concept·Emerging
Placeholder Image and Copy Generation
CON-023Populate mockups with contextually appropriate placeholder imagery and first-draft copy, providing higher-fidelity to support stakeholder review and user testing.
Concept·Developing
Research Stimulus Generation
CON-025Create personalised stimuli for user research tailored to each participant's role, industry, proficiency level, or market without proportional increases in design or engineering time.
Concept·Emerging
Beta Feedback Synthesis
VAL-084Beta findings are preprocessed to enable deeper treatment, making them more likely to influence release decisions, rather than just skimmed; accelerating time to insight gives more time to correct, and makes it harder to abdicate corrections that will create experience debt.
Validate·Developing
Regulatory Compliance Scan
VAL-102Catches compliance violations at the design stage rather than at formal review or post-launch, reducing remediation cost and shortening the review cycle for regulated product teams.
Validate·Emerging
Research Bias Audit
VAL-036Post fieldwork analysis, the user researcher has an AI critique their report by looking for areas of bias or interpretive overreach before findings are delivered.
Validate·Developing
Usability Test Analysis
VAL-032Compresses the time from test sessions to actionable findings by automating pattern detection, theme extraction, and highlight identification within recordings and transcripts.
Validate·Developing
A/B Test Automation
IMP-060Shortens experiment runtime by generating test configurations from product analytics, then recommending optimizations based on test results.
Improve·Developing