<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>LLM Reliability on RockB</title><link>https://baeseokjae.github.io/tags/llm-reliability/</link><description>Recent content in LLM Reliability on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Thu, 09 Jul 2026 12:00:00 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/llm-reliability/index.xml" rel="self" type="application/rss+xml"/><item><title>Claude Fable 5 Hallucination Real Example: The Invented Airline Case Study</title><link>https://baeseokjae.github.io/posts/claude-fable-5-hallucination-case-study-2026/</link><pubDate>Thu, 09 Jul 2026 12:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/claude-fable-5-hallucination-case-study-2026/</guid><description>A practical Claude Fable 5 hallucination case study using the Nordlys Air fictional airline demo.</description><content:encoded><![CDATA[<p>Claude Fable 5 did not prove that a real airline existed. In the HydePHP case, it generated Nordlys Air, a clearly fictional bush airline demo. The useful lesson is narrower and more practical: powerful models can create credible company-shaped details, so provenance has to travel with generated artifacts.</p>
<h2 id="did-claude-fable-5-hallucinate-an-airline">Did Claude Fable 5 hallucinate an airline?</h2>
<p>The clean answer is: not in the strongest sense of &ldquo;hallucination.&rdquo; The <a href="https://hydephp.com/posts/using-claude-fable-with-hydephp">HydePHP Nordlys Air demo</a> described Nordlys Air as fictional, and the generated site reportedly kept that boundary visible in the footer. Claude Fable 5 was asked to build a demo experience, and it produced a believable fictional airline with routes, aircraft profiles, a departures board, journal posts, and an operations manual.</p>
<p>That is different from Claude saying, &ldquo;Nordlys Air is a real airline operating these routes.&rdquo; I would not call the published demo a proven accidental hallucination.</p>
<p>But I would still use it as a strong hallucination case study. When building internal AI tools, I&rsquo;ve found that hallucination risk often starts one step before the obvious failure. The first output may be labeled fiction, synthetic data, or draft copy. The failure happens later, when someone reuses the asset, strips the label, copies the route list into a slide deck, or asks another model to summarize it as if it were factual.</p>
<p>In other words, the risk is not just &ldquo;the model invented something.&rdquo; The risk is &ldquo;the model invented something so well that the surrounding workflow treated it as verified.&rdquo;</p>
<p>If you are working on similar reliability problems, the same source-bound discipline applies to broader LLM workflows like <a href="/posts/rag-evaluation-checklist/">RAG evaluation checklists</a> and <a href="/posts/ai-agent-progress-reporting/">AI agent progress reporting</a>. The common failure mode is weak provenance, not weak prose.</p>
<h2 id="what-actually-happened-in-the-nordlys-air-demo">What actually happened in the Nordlys Air demo?</h2>
<p>The primary case source says Claude Fable 5 generated a HydePHP demo site for a fictional bush airline called Nordlys Air. The details were not vague. The generated output included:</p>
<table>
  <thead>
      <tr>
          <th>Artifact</th>
          <th>What Fable 5 generated</th>
          <th>Why it matters</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Routes</td>
          <td>Six Arctic routes represented as route and fleet data</td>
          <td>Structured facts look more trustworthy than loose prose</td>
      </tr>
      <tr>
          <td>UI</td>
          <td>A live split-flap departures board</td>
          <td>Realistic operational UI can imply real operations</td>
      </tr>
      <tr>
          <td>Fleet</td>
          <td>Aircraft profiles and original SVG schematics</td>
          <td>Visual specificity raises perceived credibility</td>
      </tr>
      <tr>
          <td>Content</td>
          <td>Flight journal posts in Markdown</td>
          <td>Narrative history can feel like company history</td>
      </tr>
      <tr>
          <td>Docs</td>
          <td>A crew operations manual</td>
          <td>Procedural writing can look authoritative</td>
      </tr>
      <tr>
          <td>Labeling</td>
          <td>A footer stating Nordlys Air is fictional</td>
          <td>The key mitigation in this case</td>
      </tr>
  </tbody>
</table>
<p>That last row is the difference between a useful demo and a dangerous artifact. The generated content had a visible fiction label. The HydePHP article also exposed implementation details, including the file tree and the Blade loop powering the board. That kind of transparency matters because it gives the reader a way to classify the output.</p>
<p>When I review AI-generated demos, I separate the output into three buckets:</p>
<ol>
<li>Verified facts about real tools, APIs, people, companies, dates, and systems.</li>
<li>Intentionally invented content that exists to make the demo feel complete.</li>
<li>Ambiguous claims that look factual but do not carry evidence.</li>
</ol>
<p>Nordlys Air belongs mostly in bucket two. The hallucination risk starts when bucket two gets copied into bucket one.</p>
<h2 id="why-does-a-fictional-demo-still-matter-for-hallucination-risk">Why does a fictional demo still matter for hallucination risk?</h2>
<p>Because fictional company data is one of the easiest ways to create believable false reality.</p>
<p>Developers tend to focus on obvious hallucinations: fake package names, imaginary API methods, invented legal citations, or bogus error messages. Those are serious, but they are not the only production risk. A model can also fabricate:</p>
<ul>
<li>Confidence: &ldquo;I completed the migration&rdquo; when no tool result supports it.</li>
<li>Provenance: &ldquo;According to the documentation&rdquo; without a real citation.</li>
<li>Entity existence: a company, route, regulation, benchmark, or product line.</li>
<li>Operational detail: prices, schedules, addresses, dates, model IDs, endpoints.</li>
<li>Context: a fictional demo described later as a case study about a real business.</li>
</ul>
<p>In practice, polished false context is more expensive than an obvious wrong answer. A fake airline with a route map, flight numbers, fleet notes, and an operations manual forces a human reviewer to verify many small claims. The model&rsquo;s fluency becomes a multiplier on review cost.</p>
<p>That is why I like this case better than a generic benchmark table. It shows the actual shape of the problem. Fable 5 can be strong at the coding task and still produce a large set of invented domain facts because the prompt invited a fictional world.</p>
<h2 id="what-did-fable-5-get-right-in-the-hydephp-work">What did Fable 5 get right in the HydePHP work?</h2>
<p>The HydePHP write-up says Fable 5 checked real Hyde v2 APIs against source before coding. That is the part many teams should copy. The model apparently grounded the framework work in the actual codebase instead of freehanding old API assumptions.</p>
<p>That distinction is important. A single output can contain both grounded engineering work and ungrounded narrative content.</p>
<table>
  <thead>
      <tr>
          <th>Output area</th>
          <th>Reliability posture</th>
          <th>Review method</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>HydePHP API usage</td>
          <td>Source-checkable</td>
          <td>Inspect code, docs, imports, build output</td>
      </tr>
      <tr>
          <td>Blade template loops</td>
          <td>Source-checkable</td>
          <td>Run the project and review templates</td>
      </tr>
      <tr>
          <td>Nordlys Air routes</td>
          <td>Synthetic unless sourced</td>
          <td>Require explicit fiction label or evidence</td>
      </tr>
      <tr>
          <td>Aircraft profiles</td>
          <td>Synthetic unless sourced</td>
          <td>Require citations, disclaimers, or generated-data flags</td>
      </tr>
      <tr>
          <td>Operations manual</td>
          <td>Synthetic unless sourced</td>
          <td>Treat as creative copy, not procedure</td>
      </tr>
  </tbody>
</table>
<p>I&rsquo;ve found that this mixed-output problem is where teams get sloppy. If the model correctly edits a React component, updates a Laravel route, and passes tests, reviewers start trusting the adjacent copy. But code correctness does not certify domain facts. A passing build does not prove a route exists, a company operates, or a manual describes real procedure.</p>
<p>The fix is not to ban rich generated demos. The fix is to make the provenance boundary explicit in the artifact itself.</p>
<h2 id="what-did-fable-5-invent-in-the-airline-example">What did Fable 5 invent in the airline example?</h2>
<p>Based on the brief, Fable 5 invented the airline concept and filled it with realistic detail: six Arctic routes, fleet data in YAML, journal posts in Markdown, a docs module shaped like an operations manual, and SVG aircraft schematics.</p>
<p>That is good demo generation. It is also exactly the kind of output that can become a hallucination if reused without context.</p>
<p>Here is the practical test I use:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>For every factual-looking noun in this output, ask:
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>1. Is this a real-world entity?
</span></span><span style="display:flex;"><span>2. If yes, what source verifies it?
</span></span><span style="display:flex;"><span>3. If no, where is it labeled fictional or synthetic?
</span></span><span style="display:flex;"><span>4. Will that label survive screenshots, summaries, exports, and copy-paste?
</span></span></code></pre></div><p>The fourth question is the one people skip. A footer disclaimer helps on the original web page. It does not necessarily survive a screenshot of the departures board, a cropped aircraft profile, or a second-order summary generated by another model.</p>
<p>For generated demos, I prefer redundant labeling:</p>
<ul>
<li>Put &ldquo;fictional demo data&rdquo; near the page footer.</li>
<li>Add comments in YAML, JSON, CSV, or Markdown files.</li>
<li>Add metadata fields such as <code>synthetic: true</code> or <code>source: &quot;generated demo data&quot;</code>.</li>
<li>Name fixtures clearly, for example <code>demo_airline_routes.yaml</code> instead of <code>routes.yaml</code>.</li>
<li>Keep real citations separate from generated examples.</li>
</ul>
<p>That may feel heavy for a demo. It is still cheaper than cleaning up a published artifact after someone mistakes synthetic content for research.</p>
<h2 id="how-do-fable-5-benchmarks-change-the-interpretation">How do Fable 5 benchmarks change the interpretation?</h2>
<p>They make the case more interesting, not less.</p>
<p>Anthropic launched Claude Fable 5 on June 9, 2026 as a Mythos-class model for demanding reasoning and long-horizon work. The launch materials and model documentation describe a 1 million token context window, up to 128,000 output tokens, the <code>claude-fable-5</code> API model ID, and pricing of $10 per million input tokens and $50 per million output tokens. Anthropic also said safeguards can route some risky requests to Claude Opus 4.8 and that those safeguards trigger in fewer than 5% of sessions on average.</p>
<p>Artificial Analysis ranked Claude Fable 5 first on its Intelligence Index with a score of 64.9. On AA-Omniscience, Fable 5 scored 40, seven points above the previous leader. But the same analysis said the result was driven by higher accuracy rather than a low hallucination rate. It also observed fallback routing in about 8% of Intelligence Index tasks and 9% of AA-Omniscience questions.</p>
<p>That is the lesson: benchmark leadership does not mean hallucination risk disappears.</p>
<table>
  <thead>
      <tr>
          <th>Signal</th>
          <th>What it says</th>
          <th>What it does not prove</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>#1 Intelligence Index score</td>
          <td>Strong broad capability</td>
          <td>Every generated claim is verified</td>
      </tr>
      <tr>
          <td>AA-Omniscience score of 40</td>
          <td>Better factual recall and calibration than many peers</td>
          <td>No entity fabrication in open-ended tasks</td>
      </tr>
      <tr>
          <td>1M token context</td>
          <td>Can inspect large projects and source material</td>
          <td>Will automatically cite the right source</td>
      </tr>
      <tr>
          <td>128K output limit</td>
          <td>Can produce large artifacts</td>
          <td>Long outputs are easier to audit</td>
      </tr>
      <tr>
          <td>Fallback routing</td>
          <td>Some risky sessions are handled differently</td>
          <td>Reliability is uniform across requests</td>
      </tr>
  </tbody>
</table>
<p>The 2026 Stanford AI Index also reports hallucination rates ranging from 22% to 94% across 26 top models in a new accuracy benchmark. Vectara&rsquo;s hallucination leaderboard, updated May 11, 2026, shows much lower rates for some summarization tasks, with top examples around 1.8%, 3.1%, and 3.3%. Those numbers can both be true because task design matters. Summarizing a supplied document is not the same as generating a full fictional airline website.</p>
<p>If you only read benchmark headlines, you miss the operational point: hallucination rate depends on task, prompt, evidence access, output format, and review workflow.</p>
<h2 id="what-should-developers-check-before-publishing-fable-5-output">What should developers check before publishing Fable 5 output?</h2>
<p>Use a checklist that matches the artifact, not a vague &ldquo;review the answer&rdquo; instruction. For a generated company, product, or case study, I would check these before publishing:</p>
<table>
  <thead>
      <tr>
          <th>Check</th>
          <th>Concrete question</th>
          <th>Failure example</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Entity existence</td>
          <td>Does this company, airline, route, product, or standard exist?</td>
          <td>Fictional company described as real</td>
      </tr>
      <tr>
          <td>Dates</td>
          <td>Are launch dates, outages, restrictions, and updates sourced?</td>
          <td>A July restoration mentioned in an April-dated article</td>
      </tr>
      <tr>
          <td>Names</td>
          <td>Are people, teams, agencies, and vendors real?</td>
          <td>Invented executive quote</td>
      </tr>
      <tr>
          <td>Routes and locations</td>
          <td>Are airports, service areas, or addresses verified?</td>
          <td>Impossible Arctic route treated as operational</td>
      </tr>
      <tr>
          <td>Prices</td>
          <td>Are API prices or fares current and sourced?</td>
          <td>Old model pricing copied into a new comparison</td>
      </tr>
      <tr>
          <td>Model IDs</td>
          <td>Does the API model name match official docs?</td>
          <td><code>claude-fable-v5</code> used instead of <code>claude-fable-5</code></td>
      </tr>
      <tr>
          <td>Screenshots</td>
          <td>Do screenshots preserve fiction labels?</td>
          <td>Cropped board loses disclaimer</td>
      </tr>
      <tr>
          <td>Citations</td>
          <td>Does each factual claim point to a source?</td>
          <td>&ldquo;According to benchmarks&rdquo; with no benchmark link</td>
      </tr>
      <tr>
          <td>Generated data</td>
          <td>Is synthetic content marked in files and UI?</td>
          <td>Demo YAML copied as production seed data</td>
      </tr>
      <tr>
          <td>Progress claims</td>
          <td>Are completed tasks backed by tool output?</td>
          <td>Agent claims tests passed without a test run</td>
      </tr>
  </tbody>
</table>
<p>I would apply the same rule to AI-generated technical articles. If a paragraph contains a factual claim, it needs either a source link, a reproducible command, or a label that says it is an example.</p>
<h2 id="how-should-you-prompt-fable-5-to-reduce-fabricated-facts">How should you prompt Fable 5 to reduce fabricated facts?</h2>
<p>Anthropic&rsquo;s hallucination-reduction guidance recommends allowing the model to say it does not know, grounding factual claims in direct quotes, and making outputs auditable with citations and source verification. The Fable 5 prompting guide also says progress claims in long autonomous runs should be audited against actual tool results before being reported.</p>
<p>For developer work, I use prompts that force separation between facts and generated material. For example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>You are writing a demo using Claude Fable 5.
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>Rules:
</span></span><span style="display:flex;"><span>- Separate verified facts from fictional demo data.
</span></span><span style="display:flex;"><span>- For every real-world claim, include a source URL or say &#34;source missing&#34;.
</span></span><span style="display:flex;"><span>- For every invented company, route, person, price, or date, label it synthetic.
</span></span><span style="display:flex;"><span>- Do not imply that fictional entities exist outside this demo.
</span></span><span style="display:flex;"><span>- Before final output, produce a table with:
</span></span><span style="display:flex;"><span>  claim, category, source, confidence, and publish risk.
</span></span></code></pre></div><p>For agentic coding tasks, I add a stricter progress-report rule:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>When reporting progress:
</span></span><span style="display:flex;"><span>- Only say a file was changed if a tool result confirms it.
</span></span><span style="display:flex;"><span>- Only say tests passed if the test command was run and returned success.
</span></span><span style="display:flex;"><span>- If a command was not run, say it was not run.
</span></span><span style="display:flex;"><span>- Do not infer completion from intention, plan, or partial output.
</span></span></code></pre></div><p>This sounds pedantic until you have an autonomous run that says &ldquo;implemented and verified&rdquo; after editing the wrong file. Fabricated progress is a hallucination too. It may not look like a fake airline, but it creates the same downstream risk: people make decisions based on unsupported claims.</p>
<p>For more on that workflow discipline, see my notes on <a href="/posts/source-grounded-ai-code-review/">source-grounded AI coding reviews</a> and <a href="/posts/llm-hallucination-benchmarks-2026/">LLM hallucination benchmarks in practice</a>.</p>
<h2 id="how-should-teams-classify-the-nordlys-air-case">How should teams classify the Nordlys Air case?</h2>
<p>I would classify it as &ldquo;labeled fiction with hallucination-adjacent risk.&rdquo;</p>
<p>That phrasing is less dramatic than &ldquo;Claude invented an airline and lied about it,&rdquo; but it is more accurate. The demo did not need Nordlys Air to be real. It needed Nordlys Air to be plausible enough to exercise HydePHP&rsquo;s content, layout, and documentation features. Claude delivered that.</p>
<p>The operational lesson is still sharp:</p>
<ul>
<li>Strong models can ground code while inventing domain content.</li>
<li>Realistic generated detail increases review burden.</li>
<li>Benchmarks do not remove the need for provenance.</li>
<li>Fiction labels need to survive reuse, summaries, screenshots, and exports.</li>
<li>Synthetic data should be marked in both UI and source files.</li>
</ul>
<p>This is also why the June 2026 Fable 5 suspension and July 1, 2026 redeployment matter as broader reliability context. Anthropic said access was suspended after US export controls on June 12, then restored globally for Fable 5 on July 1 after controls were lifted. Anthropic also described a new safety classifier that blocks the specific Amazon-reported cybersecurity bypass technique in over 99% of cases, while increasing false positives for some benign coding and debugging requests.</p>
<p>That is not directly about Nordlys Air. It does show that reliability is not one property. A model can be accurate on benchmarks, guarded in some risky sessions, occasionally over-refuse benign requests, and still generate fictional domain detail when asked to create a demo.</p>
<h2 id="what-is-the-practical-takeaway-from-this-claude-fable-5-case-study">What is the practical takeaway from this Claude Fable 5 case study?</h2>
<p>The takeaway is not &ldquo;do not use Claude Fable 5 for demos.&rdquo; I would use a model like this for demos precisely because it can create coherent UI, docs, data, and narrative in one pass. The takeaway is to make generated reality auditable.</p>
<p>For publishing workflows, I would require four gates:</p>
<ol>
<li>A fact/source pass for all real-world claims.</li>
<li>A synthetic-data pass for all invented entities.</li>
<li>A screenshot/export pass to confirm labels survive reuse.</li>
<li>A final provenance table for the editor or reviewer.</li>
</ol>
<p>The Nordlys Air example is useful because it shows both sides of the tool. Fable 5 appears capable of checking real framework APIs and producing a polished, multi-part demo. It also shows how easily a model can fill a blank business domain with realistic operational detail.</p>
<p>That is the balance teams need to internalize. Capability makes hallucination controls more important, not less important. The better the generated artifact looks, the more explicit its evidence trail needs to be.</p>
<h2 id="faq-what-should-developers-know-about-claude-fable-5-hallucinations">FAQ: What should developers know about Claude Fable 5 hallucinations?</h2>
<h3 id="did-claude-fable-5-really-invent-nordlys-air">Did Claude Fable 5 really invent Nordlys Air?</h3>
<p>Yes. In the HydePHP case, Claude Fable 5 generated Nordlys Air as a fictional bush airline demo with Arctic routes, aircraft profiles, a departures board, journal posts, and an operations manual. The important nuance is that the demo labeled the airline as fictional rather than presenting it as a verified real-world airline.</p>
<h3 id="is-the-nordlys-air-demo-proof-that-fable-5-has-a-high-hallucination-rate">Is the Nordlys Air demo proof that Fable 5 has a high hallucination rate?</h3>
<p>No. It is not a hallucination-rate measurement. It is a practical case study showing how realistic generated content can become risky if labels and provenance are lost. For rates and benchmarks, use dedicated evaluations such as AA-Omniscience, Stanford AI Index benchmarks, or task-specific tools like Vectara&rsquo;s summarization leaderboard.</p>
<h3 id="how-accurate-is-claude-fable-5-on-benchmarks">How accurate is Claude Fable 5 on benchmarks?</h3>
<p>The research brief cites Artificial Analysis ranking Claude Fable 5 first on its Intelligence Index with a score of 64.9. On AA-Omniscience, Fable 5 scored 40, seven points above the previous leader, but Artificial Analysis said that result was driven by accuracy rather than a low hallucination rate. That distinction matters for production use.</p>
<h3 id="what-is-the-safest-way-to-use-fable-5-for-generated-demos">What is the safest way to use Fable 5 for generated demos?</h3>
<p>Treat demos as mixed artifacts. Verify real framework, API, and product claims against sources. Label fictional companies, people, prices, routes, and procedures as synthetic. Keep those labels in the UI, source files, metadata, screenshots, and exports. Do not rely on one footer disclaimer if the content will be reused elsewhere.</p>
<h3 id="what-prompt-reduces-claude-fable-5-hallucinations-the-most">What prompt reduces Claude Fable 5 hallucinations the most?</h3>
<p>There is no single magic prompt, but the best pattern is source-bound generation: allow the model to say it does not know, require citations for factual claims, require labels for synthetic content, and audit progress claims against tool outputs. For agentic work, never accept &ldquo;done&rdquo; unless the tool transcript supports it.</p>
]]></content:encoded></item></channel></rss>