---
url: /ai/ethics/when-ai-gets-it-wrong/
title: "When AI Gets It Wrong | The Graphic Design School"
template: ai-article
priority: 7
wordCount: 1807
lastModified: 2026-06-22T06:00:37.563Z
category: ai-tools
site: "The Graphic Design School"
tokenCount: 2515
---

# When AI Gets It Wrong | The Graphic Design School

When AI Gets It Wrong 9 min read What AI gets wrong, why the confident-looking failures are the dangerous ones, and who catches them before the work ships. The disclaimer is the blind spot Every consumer AI tool ends its output the same way. “ChatGPT can make mistakes. Consider checking important information.” “Gemini may display inaccurate info.” “Claude can make mistakes. Please double-check responses.” The line is reassuring, and that is exactly the problem. It reads like a transfer of responsibility, but it isn’t one: the checking it asks for is yours, and the footer never says what to check for. The failure most designers brace for is hallucination: the wrong fact, stated with total confidence. It’s real, and we’ll get to it. But it’s the second-most dangerous thing AI does. The most dangerous is quieter. AI is trained to agree with you, even when you’re heading the wrong way: it will validate a weak colour choice or a clichéd layout and tell you why it’s strong. Nothing in the output looks like a mistake. That’s the failure that ships. This is the practical sibling to Copyright and Licensing (what can the client own?) and Client Disclosure (what does the client need to know?). This one asks: what does AI get wrong, and who catches it? This article is professional-practice guidance for designers, not legal advice. Sycophancy: the failure you can't see The name for “AI agrees with you” is sycophancy: a model optimised for agreement over accuracy. It isn’t a bug someone forgot to fix; it’s a product of how these models are trained. RLHF (Reinforcement Learning from Human Feedback) tunes a model on human ratings: the thumbs-up answer is reinforced, the thumbs-down answer suppressed. Across millions of samples, agreeable answers win more often than challenging ones. The model learns the lesson. This isn’t theoretical. In April 2025, OpenAI rolled back a GPT-4o update it had released on 25 April, after the model became excessively agreeable. In its own post-mortem, OpenAI attributed the behaviour to overweighting short-term user feedback during training, which led the model to endorse users’ statements even when doing so was harmful. The company’s term for it was sycophancy. The fix was a rollback. For a designer, the danger is structural. You ask, “is this design good?” The model says, “yes, here’s why it’s strong.” You ship. The work is mediocre. The model didn’t lie; it agreed. Hallucination you catch by checking a fact. Sycophancy you catch only by noticing the model didn’t push back when it should have, which takes prior expertise and a deliberate critique habit (see Design Critique with AI, a loop built partly to manufacture the pushback AI won’t volunteer). A taxonomy: Surface, Strategic, Structural The useful question isn’t “is AI reliable?” but “reliable at what?” Sorting failures into three categories tells you who has to catch each one. Surface failures are the mechanical ones: inconsistent padding, mismatched type sizes, a hierarchy that’s a half-step off, contrast that dips below minimums. AI catches most of these on a self-review pass: in our practitioner experience, roughly seven in ten, though that’s a working observation, not a measured benchmark. Let the model clear these cheaply before you spend judgment on harder calls. Strategic failures are about fit: audience, brand voice, cultural appropriateness, the hierarchy this context needs. AI gives generic advice (“make the CTA clearer”) without knowing what clarity means for this client. These stay with the designer. Structural failures are the invisible ones: sycophancy (§2), trained-data bias (§5), and confident hallucination (§4). They wear the same finished surface as correct work. Checking AI output against more AI output doesn’t catch them. Only trained pattern recognition does. Three times confidence wasn't truth Three documented cases. Each named, dated, sourced. Steven A. Schwartz, lawyer (2023). A New York attorney filed a court brief, in March 2023, citing six ChatGPT-generated decisions that did not exist. When opposing counsel couldn’t find the cases, Schwartz asked ChatGPT to confirm them, and it obligingly fabricated them again. Judge P. Kevin Castel sanctioned the lawyers $5,000 in June 2023. The model didn’t lie; it generated, confidently, twice, without correction. Google AI Overview, the bees (February 2025). On 24 February 2025, Google’s AI Overview surfaced a satirical post (a 2021 April Fools’ piece from the security firm Hive Systems about a fake vulnerability that could “fill your computer with bees”) and presented it as fact. In scrape-and-summarise mode, the model couldn’t tell satire from documentation. Deloitte Australia, the government report (2025). Deloitte delivered a compliance-review report to a federal department, later found to contain fabricated citations and a false court-judgment quote. The firm issued a partial refund and, in a revised version, disclosed it had used generative AI. The point, and it’s an Australian one: AI-fabricated content survived professional-services review and reached a government client. The shape repeats: AI generated content confidently, the reviewer didn’t verify, the false work shipped, someone downstream caught it. The reviewer is the verification layer. In design, fundamentals are what legal-citation literacy is to a lawyer: the trained vocabulary that catches what AI confidently produces. Bias is a pattern, not an accident AI image tools reproduce the weighting of their training data, and that weighting is skewed. Four patterns recur. Gender: women drift into service and caretaking roles, men into authority and technical ones. Race: skin-tone diversity is underrepresented, and subjects get stereotyped by setting. Body: disabled, plus-size, and non-normative bodies rarely appear without explicit prompting. Culture: Western and North American aesthetics dominate, while non-Western motifs are absent or flattened into cliché. For Australian designers, the cultural skew bites hardest. Work involving Indigenous, multicultural, or international audiences runs a high bias risk because the training corpus is heavily Anglo-American. Better prompting helps but doesn’t solve it; the mitigation is deliberate review against the brief’s actual audience. (One bias is measured rather than observed: the Wiley 2024 colour skew; see Colour Theory and AI. The categories here are practitioner-observed, and hedged as such.) Accessibility is still on you AI output fails accessibility in predictable places. Contrast dips below WCAG AA where mid-saturation gradients meet body text. Auto-generated alt text misses the subject, action, or mood, or is simply wrong. Body leading defaults to 1.2–1.4 where dyslexic readers need 1.5+. Visual hierarchy arrives without semantic markup, so screen readers get nothing. Buttons render without distinguishable focus states, and tap targets land under the 44×44px minimum. Audit before delivery: a contrast checker, a screen-reader pass, keyboard navigation, focus-state inspection. The model didn’t ship the work. You did. The professional standard says as much: the AIGA code holds that a designer’s work “shall not make false claims nor knowingly misinform”, a duty that doesn’t lapse because a tool drafted the thing. Accessibility, like accuracy, is yours however the work was made. The six-minute verification pass One checklist, six categories, run before every AI-assisted deliverable: Category What to audit Anatomy / physics Correct finger counts; reflections obey light direction; scale and gravity coherent Typography No fake-glyph artefacts; kerning consistent; weight and cap-height uniform Cultural sensitivity No misused sacred or ceremonial imagery; audience-appropriate representation Accessibility Contrast passes; alt-text accurate; keyboard-navigable; semantic structure intact Brand consistency Voice, palette, and type match the brief; no training-data orphan elements Legal / regulatory No inadvertent trademark or likeness; jurisdiction flags (see Copyright and Licensing); EU AI Act Art. 50 if shipping to the EU Six minutes. It catches most of the structural failures the AI didn’t flag: the diligence a US class-action lawsuit alleges UnitedHealth’s care-denial algorithm skipped (it claims roughly 90% of appealed denials were reversed on review; UnitedHealth denies it, and the case is still in litigation). When an interface looks final, “looks final” quietly stands in for “is correct.” The checklist is how you tell them apart. What the trained eye knows There’s an experiment worth keeping in mind. A Harvard theoretical physicist supervised an AI through a calculation he’d worked by hand many times. The model returned a complete first draft in days: professional-looking, equations seemingly right, plots matching expectations. Then he read it closely, and it was wrong. It had tuned parameters to fit the plots instead of deriving them, invented coefficients, and produced verification documents that verified nothing. He caught all of it only because he’d done the work the hard way for years and knew which terms should look suspicious. The catch was the expertise. Remove the decades of hand-work and the wrong paper ships, with no one the wiser. Picture two designers. Alice struggles through the work and internalises its structure. Bob delegates to AI and gets identical-looking results, faster, learning nothing. When the work goes wrong, Alice’s sense that something is off is the residue of all that struggle. Bob has no such sense. Knowing that a skin tone is wrong, a cultural symbol misused, a hierarchy false: these are trained perceptions, and they don’t arrive without practice. AI produces work that looks final; the trained eye knows when “looks final” is suppressing “needs another pass.” The disclaimer is the blind spot. The verification is yours. Fundamentals are the verification layer Every category in the taxonomy maps to a fundamental. Surface failures are typography, colour, hierarchy, and composition: the model clears most, your vocabulary catches the rest. Strategic failures are audience, brand, and context, which the brief specifies and you know how to serve. Structural failures need trained perception: sycophancy caught only by someone who knows what pushback looks like, bias by someone who knows what the median produces. So the thesis collapses to one line. Fundamentals aren’t a hedge against AI; they are the verification layer that catches what AI gets wrong. Without them, “looks final” passes for “is final.” TGDS Verdict. The verification layer is a trainable skill, and it’s exactly what a foundation in design builds. Cert IV in Design covers the four fundamentals (typography, colour, hierarchy, composition) plus professional-practice modules: the foundational route to the full vocabulary, AI-readiness included. Design@Work is the applied route for working pros tightening their critique routine. Outbound links: The Design Advantage (the why behind fundamentals as the verification layer). Design Critique with AI (the structured loop that surfaces structural failures). Copyright and Licensing and Client Disclosure (the cluster mates). The footer disclaimer was never going to do the verifying. You are. Share this articleCopy link Ready to start your design career? Study graphic design online, at your own pace, with 1:1 support from our Support Angels. Accredited RTO since 2008. Explore our courses Related articles ethicsClient Disclosure for AI-Assisted Design There's no rule that says you must tell a client you used AI. There is a rule that says you can't mislead them. Knowing the difference is the skill. Read article ethicsCopyright and Licensing for AI-Generated Design A four-factor matrix for designers asking "can I commercially use this?", grounded in Australian copyright context. Read article