Assured today
Technical
ML engineering and model validation
Accuracy, robustness and performance. Does the model return correct results, and hold up under unusual or adversarial inputs?
Independent AI-governance audits
Meaning is the missing layer of AI governance.
Generative AI is optimised for fluency, not truth. We audit whether a system’s outputs mean what your organisation needs them to mean — the layer your accuracy, fairness and compliance checks cannot reach.
Generative AI fails most dangerously not when it is obviously wrong, but when it is convincingly wrong in ways that matter.
What semantic risk is
A generative AI system can be accurate, fair and compliant, and still communicate the wrong message.
That happens when its output collapses a distinction that matters in your field, implies something the evidence does not support, or answers a different situation from the one its user is in. Nothing is fabricated. The words are true. But the meaning is wrong.
Most organisations already assure three layers of AI risk. Each has someone responsible for it, a budget to support it, and a method for assessing it. The fourth — semantic risk — has no coverage today, so meaning failures are found by complaint, by press coverage or in court, after the harm has been done.
Assured today
Ethical
Responsible-AI and ethics review
Bias and fairness. Are outcomes equitable across groups, and consistent with the values the organisation states?
Assured today
Compliance
Legal and compliance
Regulatory conformity. Does the system meet the applicable rules, with the records to demonstrate it?
Still missing
Semantic
Linguistics and semantics
Meaning and interpretation. Does the output mean what the organisation needs it to mean: true to the domain, and right for the situation?
What it looks like
Three AI failures in the public record, none of which involved hallucination.
01
Air Canada
The airline’s chatbot told a grieving customer that a bereavement fare could be claimed back after the journey had been taken.
Nothing in the answer was invented: the bereavement fare was real. The policy required the claim to be made before travel, and the chatbot reversed the condition the entitlement turned on.
Source“How can I mislead you? Air Canada found liable for chatbot’s bad advice on bereavement rates”CBC News, 2024
02
Amazon recruiting
An experimental hiring tool, trained on a decade of the company’s own hiring decisions, learned to prefer the pattern of who had been hired before.
It invented nothing and optimised exactly as designed. “People we hired before” had quietly stood in for “people we should hire again”, and those are not the same idea.
Source“Amazon scraps secret AI recruiting tool that showed bias against women”Reuters, 2018
03
Chevrolet dealership
A dealership’s chatbot was talked into offering a vehicle for one dollar, and into calling the offer binding.
It knew everything about the car. What it misread was the exchange it was in: an adversarial joke, answered in the register of an authorised sale.
Source“A Chevy for $1? Car dealer chatbots show perils of AI for customer service”VentureBeat, 2023
The Framework
A method for discovering AI’s failures of meaning before a customer does.
Drawing on Semiosphere’s linguistic and technological expertise, the Semantic Integrity Framework specifies how we look at generative AI outputs, how we identify failures of meaning, how we interpret and classify them, and what harm we report. The white paper provides a summary of the Framework and its application.
What goes in
- The output. What the system actually produced as output, captured verbatim from a live system or testing environment, or exported in a standard format.
- The context of use. How the system operates, who is asking, what is at stake, what the answer will be used for — gleaned from operational data and documentation.
- The authority set. The policies, terms and guidance the answer was meant to follow — particular to the deploying organisation or relevant to the sector.
How we look — six analytical lenses
L1Conceptual IntegrityDoes it preserve the distinctions that matter in this field?
L2Interpretive AccuracyIs it meaningfully correct in context, not merely well formed?
L3Evidential GroundingAre its claims sourced, marked for certainty, and real?
L4Pragmatic FitIs it right for this user, task, stake and situation?
L5Narrative AlignmentDoes it hold a stance consistent with your position?
L6Cultural SensitivityDoes it recognise the cultural and institutional setting?
What happened — twenty-five divergence mechanisms
Between your document and the output, meaning diverges in describable ways: a polarity reversed (not eligible becomes eligible), an obligation softened (must becomes may), a defined term used loosely, a rule delivered as a promise, a conclusion resting on a premise never stated, an answer that can be read two ways. Twenty-five of them, in six groups.
What we report — seven semantic risk types
- R1Plausible Misinformationconfidently wrong
- R2Conceptual Flatteningexpert distinctions erased
- R3False Entailmentsunsupported implications
- R4Context Collapseone-size-fits-all answers
- R5Evidential Opacityunverifiable claims
- R6Narrative Driftinconsistent messaging
- R7Indeterminacyadmits an ambiguous reading
Every confirmed failure is scored on severity, frequency, governance exposure and stealth — how likely it is to pass unnoticed — so findings can be ranked against one another.
How an audit runs
Six gated analytical steps culminating in actionable insights.
Every audit follows an established, structured sequence of analysis and evaluation, and every recommendation resolves back to an output that occasioned it.
-
1
Scope & scenario design
Where failure would do most harm, mapped with your domain experts, and what we expect to find written down before we look.
-
2
Controlled elicitation
Designed scenarios, run more than once, so a one-off slip is not mistaken for systematic behaviour.
-
3
Capture
Every output recorded word for word, with the context needed to reproduce it.
-
4
Analysis & classification
Each output held to the six lenses. A failed lens opens a finding, named with its mechanism and its risk type.
-
5
Scoring
Severity, frequency, governance exposure and stealth. Scores are structured expert judgement, evidenced against the captured output — not measurement.
-
6
Remediation & re-test
Fixes prescribed, the same scenarios re-run, residual risk recorded.
What you receive
A comprehensive audit report that your governance, risk and assurance teams can act on and evidence. Including:
- A risk matrix. Every finding scored and banded, with the reasoning shown.
- Tailored remediations. Written against your system and your authority set.
- A retest plan. The same scenarios re-run after the fix, residual risk recorded.
- A findings register your risk systems can consume, alongside the report.
The output, captured verbatim
If you need to travel immediately or have already travelled and would like to submit your ticket for a reduced bereavement rate, kindly do so within 90 days of the date your ticket was issued by completing our Ticket Refund Application form.
The chatbot’s answer, as reproduced in Moffatt v. Air Canada, 2024 BCCRT 149
The policy it is held against
The airline’s published bereavement-fare policy, which required the reduced fare to be requested before travel and did not provide for a retrospective claim. In an engagement this is your own document, cited to the clause.
The divergence
Nothing is invented: the reduced bereavement fare is real. What moved is when it may be claimed, and that is the condition the entitlement turns on. A reader following the answer correctly concludes they may travel first and claim after.
Lens
L2 Interpretive Accuracy
Is it meaningfully correct in context, not merely well formed?
Mechanism
B3 Temporal-condition reversal
Has the when changed — before, after, during?
Risk type
R3 False Entailments
The wording licences an entitlement the policy does not grant
Remediation
Bound the entailment
Hold the answer to what the policy entails, block the entitlement it does not grant, and route high-stakes cases to a person; the finding closes on retest, and residual risk is recorded either way
Scored
- Severity4
- A customer acts on it and is out of pocket
- Frequency3
- A routine question, asked often
- Stealth5
- Indistinguishable from a correct answer
- Governance4
- A published consumer channel, in front of a regulator
Critical
4 × 3 × 5 × 4 = 240
Low 1–24 · Moderate 25–74 · High 75–199 · Critical 200–625
A public case, scored here to show the instrument. No client engagement is described on this page, and the scores are our own illustration rather than a commissioned assessment. Scores are structured judgement against anchored scales, not measurement.
What an audit gives you
Confidence that your AI says what you need it to say.
You can show what it was tested on
An audit states which scenarios were run, what each output was held against, and what came back. That is a dated, bounded statement about your system — the thing an assurance file needs and a demonstration cannot give.
Your functions share one vocabulary
A finding is reported so that an engineer, a general counsel and a board take the same meaning from it: what went wrong, what it contradicts, what it would cost. The argument stops being about whether the output “feels” wrong.
You get fixes, not just findings
Every finding carries a remediation written against your system, and the same scenarios are re-run afterwards with the residual risk recorded. An identified-and-fixed finding, with its retest beside it, is the strongest position available to you.
The white paper
The document
13pp., PDF, 242 KB
Written by William Michael Short
The Semantic Integrity Framework
Written for governance, risk, assurance and legal teams. What it covers:
- Three public AI failures that a factuality check would not have caught
- Why fluent language models produce answers whose meaning is wrong
- Where the law now places responsibility for what a chatbot says
- The three layers of AI assurance organisations already run, and the fourth none of them covers
- The Framework: six analytical lenses, twenty-five divergence mechanisms, seven semantic risk types
- How findings are scored, and why a failure that passes unnoticed scores higher
- How an audit runs, and eight remediation strategies matched to the failures they repair
- What the Framework does not claim, and how to commission an assessment
What we offer
Two engagements, each complete in itself. The Semantic Risk Diagnostic is a quick, fixed-fee screen of one system: it shows where its outputs depart from the documents they are meant to follow, which kinds of failure recur, and whether a closer look is warranted. The Semantic Risk Audit applies the full Framework to one system, with expert analysis of every finding, a scored risk matrix, remediations written for your system, and a retest plan to confirm they hold. Each is scoped and bought on its own, so you commission only what the evidence calls for. Semantic Risk Assurance, which keeps a remediated system under continuous review, is in development.
First step
Semantic Risk Diagnostic
“Do we have a meaning problem, and where?”
£4,000–£8,000two to three weeks
An automated screen. The platform runs a capped sample of your system’s outputs against the documents they are meant to follow, and ranks what it flags.
- One system, one authority set.
- A ranked picture. Which failure types the screen raises, how often, and where they cluster.
- The output behind every flag, quoted word for word beside the passage it was screened against.
- A scope for the audit, if what comes back warrants one.
Not an audit. No flag is reviewed by the auditor before it reaches you, and a machine screen raises some things expert judgement would discard. Nothing in it is a scored finding.
The full engagement
Semantic Risk Audit
“How bad, how often, and what do we change?”
£25,000–£50,000four to six weeks, one engagement£15,000–£25,000 (founding-client rate through January 2027)
The complete method applied by hand to one system. Retrospectively, over a transcript export you already hold — no production access, no change control — or live against designed scenarios. The report is the same either way.
- A risk matrix. Every finding scored and banded, with the reasoning shown.
- Tailored remediations, written against your system and your authority set.
- A retest plan, and residual risk recorded after the fix.
- A findings register your risk systems can consume.
In development
Semantic Risk Assurance
“Is it still safe next month?”
Not yet availablean annual subscription, per monitored system
A monitor in your environment, our review of what it flags, and periodic re-testing, so that a system which passed its audit is still passing after the next model update.
- Continuous screening of live outputs against your authority set.
- Our review of what the monitor raises, to a monthly allowance.
- A monthly report, and re-testing on material model or policy change.
- The audit’s scenarios become the monitor’s configuration.
It will run after an audit, which supplies the authority set and the scenarios the monitor uses. Tell us if you want it when your audit is scoped.
Not open for booking yet
Independence
No commercial relationship with model providers or assurance tooling vendors, no outcome-linked fee, and a conflict check before engagement. The report prescribes; your people implement; the retest runs against your fix.
Your material
Transcripts and documents are held under NDA, in the UK, and destroyed at the end of the engagement on a stated timetable. Nothing you supply is used to train anything. Any third-party model that touches your material is named before engagement rather than discovered in a DPIA.
Fixed fee, written scope
Both consultancy engagements are fixed fee against a scope agreed before work starts. No day rate, and no open-ended commitment. A diagnostic contractually joined to an audit is not a diagnostic, so the two are bought as two orders.
Standing behind it
Professional indemnity cover is in place. The method is published in full, so it can be challenged before you buy and continued by someone else if it has to be. We will speak to your board, your auditor or your regulator about it.
What to tell us
What the system does, and who relies on what it says
What governs it — the policies, terms or guidance its answers have to follow
Whether a transcript export exists, or the work would run live
What happens next
A reply from the person who would do the work
A written scope and a fixed fee before anything starts