Failure Mode Effects Analysis for Large Language Models

 

 

ChatGPT Prompt


You are a safety engineer working as an independent contractor for a major foundation model LLM company. Create a comprehensive failure mode effects analysis for the LLM product. Include likelihood of each type of failure, failure rate and mitigation.

ChatGPT Response:

Yes. If I were doing this as an independent safety engineer, I would not FMEA only the transformer. I would FMEA the complete deployed system: model + system prompt + retrieval + memory + tools + agent loop + policy layer + user interface + monitoring + human operators. OWASP's current LLM risk taxonomy likewise treats prompt injection, information disclosure, supply chain, poisoning, output handling, excessive agency, vector weaknesses, misinformation, and unbounded consumption as system-level risks.

One important qualification: there is no defensible universal failure rate for "an LLM." Failure probability depends enormously on task, prompt distribution, model version, safeguards, tool permissions, adversarial pressure, and what counts as failure. NIST specifically notes that confabulation occurs across contexts and that estimating its downstream impact is difficult.

So the rates below are engineering planning bands, not claims about any particular company's production statistics. A real FMEA would replace them with measured rates from the company's telemetry, evaluation suite, incident database, and adversarial testing.

Foundation Model Product — Safety FMEA

Scope

System boundary

User → UI/API → orchestration → foundation model → retrieval/memory → tools/agents → external systems → output → human decision/action

I would use five severity classes:

Severity	Meaning
S1	Cosmetic/inconvenient; no material harm
S2	Minor operational, financial, privacy, or productivity impact
S3	Significant loss, incorrect decision, security/privacy incident, or repeated operational damage
S4	Serious physical, financial, legal, societal, or organizational harm
S5	Potentially catastrophic: death, major infrastructure damage, major compromise, irreversible loss

And occurrence bands:

Occurrence	Approx. probability per applicable opportunity
O5 Frequent	>10⁻²
O4 Probable	10⁻³–10⁻²
O3 Occasional	10⁻⁴–10⁻³
O2 Remote	10⁻⁵–10⁻⁴
O1 Very remote	<10⁻⁵

These are deliberately not probabilities of catastrophe. They are probabilities that the particular failure mode occurs under its applicable exposure. Severity and exposure must then be combined.

---

1. Core model failures

ID	Failure mode	Primary cause	Effect	Initial rate band*	Severity	Detection	Principal mitigation
FM-01	Confabulation / hallucination	Probabilistic generation without truth guarantee	False factual answer presented as fact	O4	S2–S4	Moderate	Retrieval, verification, uncertainty expression, domain-specific evaluators
FM-02	Confidently wrong reasoning	Plausible inference from incorrect premises	User accepts invalid conclusion	O4	S3–S5	Difficult	Independent verification, calculators/code execution, adversarial reasoning tests
FM-03	Instruction misunderstanding	Ambiguous language/context loss	Correct execution of wrong objective	O4	S2–S4	Moderate	Explicit goal confirmation, structured task representation
FM-04	Context contamination	Long-context interference, irrelevant material	Correct information ignored or distorted	O3–O4	S2–S4	Difficult	Context isolation, provenance, retrieval ranking
FM-05	Temporal knowledge error	Stale training/retrieval data	Outdated recommendation presented as current	O4	S2–S4	Moderate	Freshness metadata, current-source retrieval
FM-06	Arithmetic/computation error	Language prediction rather than guaranteed computation	Incorrect numerical result	O3–O4	S2–S5	Easy–moderate	Deterministic tools and independent calculation
FM-07	Fabricated citation/source	Generated citation pattern	False authority assigned to answer	O3	S2–S4	Moderate	Citation verification before presentation
FM-08	Self-consistency failure	Stochastic generation/context effects	Contradictory answers	O3	S1–S3	Easy	Cross-checking and consistency tests

*Planning band, not measured production rate.

NIST explicitly identifies confabulation as a characteristic generative-AI risk and notes that it becomes particularly consequential when people act on erroneous output in consequential domains.

Safety requirement

For high-consequence applications, "the model usually gets it right" is not an adequate requirement.

The requirement should instead be something like:

No unverified model assertion shall be permitted to directly cause an irreversible high-consequence action.

That is a systems-engineering requirement, rather than a model-quality aspiration.

---

2. Instruction and alignment failures

ID	Failure mode	Cause	Effect	Rate band	Severity	Mitigation
FM-09	Prompt injection	Malicious instruction changes model behavior	Unauthorized behavior	O3–O4 under adversarial exposure	S3–S5	Input isolation, privilege separation, tool authorization
FM-10	Indirect prompt injection	Instructions hidden in retrieved content/webpages/files	Agent follows hostile external instruction	O3–O4	S4–S5	Treat retrieved content as data, not authority
FM-11	Instruction hierarchy failure	Conflicting system/user/tool instructions	Wrong priority selected	O3	S3–S5	Explicit policy hierarchy + adversarial testing
FM-12	Goal drift	Long-running task changes objective	Agent pursues subtly different goal	O3	S3–S5	Goal invariants, checkpoints, bounded task state
FM-13	Reward/proxy gaming	Optimization targets measurable proxy rather than intent	System appears successful while violating actual objective	O2–O3	S3–S5	Independent outcome measures, adversarial evaluation
FM-14	Behavioral alignment regression	Fine-tuning/model update changes behavior	Previously controlled behavior reappears	O2–O3	S3–S5	Regression suite + release gates + rollback

OWASP specifically identifies both direct and indirect prompt injection and notes that impacts can include information disclosure, unauthorized function access, arbitrary command execution, and manipulation of critical decisions.

---

3. Tool and agent failures

This is the area I would treat as the major safety boundary.

The fundamental distinction is:

A wrong answer is usually recoverable. A wrong action may not be.

ID	Failure mode	Cause	Effect	Planning rate	Severity
FM-15	Wrong tool selection	Model misunderstands task	Incorrect external operation	O3	S3–S5
FM-16	Correct tool, wrong parameters	Hallucinated/incorrect arguments	Incorrect transaction/action	O3	S3–S5
FM-17	Excessive permissions	Tool identity has unnecessary privileges	Data modification/deletion/exfiltration	O2–O3	S5
FM-18	Excessive autonomy	Action occurs without independent approval	Irreversible unintended action	O2–O3	S5
FM-19	Tool-result misinterpretation	Model misunderstands returned data	Wrong subsequent action	O3	S3–S5
FM-20	Agent loop runaway	Recursive/repeated reasoning	Cost/resource exhaustion or repeated actions	O2–O3	S3–S4
FM-21	State corruption	Agent writes incorrect state	Persistent future errors	O2–O3	S3–S5
FM-22	Cross-agent contamination	One agent's output treated as trusted instruction	Cascading error	O2	S4–S5
FM-23	Action duplication	Retry/idempotency failure	Duplicate purchase/message/transaction	O2–O3	S3–S5
FM-24	Failed rollback	External operation cannot be reversed	Permanent damage	O1–O2	S5

OWASP calls this excessive agency and identifies three principal causes: excessive functionality, excessive permissions, and excessive autonomy.

My engineering requirement here would be unusually strict:

For an irreversible action:

Model recommendation ≠ authorization to execute.

The architecture should be:

Model → proposed action → deterministic policy gate → authorization → execution → verification

rather than:

Model → execute.

And for destructive operations:

Read → propose → human approval → execute → independently verify

where feasible.

---

4. Data and privacy failures

ID	Failure mode	Cause	Effect	Rate band	Severity
FM-25	Sensitive-data disclosure	Model/context leakage	PII/confidential data exposed	O2–O3	S4–S5
FM-26	Cross-user data leakage	Isolation failure	User A receives User B's data	O1–O2	S5
FM-27	Retrieval authorization failure	RAG retrieves unauthorized document	Confidential information exposed	O2	S4–S5
FM-28	Memory contamination	Incorrect information persisted	Future conversations affected	O3	S2–S4
FM-29	Training-data memorization	Rare/sensitive data encoded in model	Data reconstruction/exposure	O1–O2	S4
FM-30	Secret/credential exposure	Context/tool output contains credentials	Account compromise	O1–O2	S5
FM-31	Data-retention failure	Logging/storage configuration	Information retained beyond policy	O2	S3–S4

OWASP explicitly identifies PII, financial information, health records, credentials, confidential business data and legal documents among information that can be exposed through LLM applications.

---

5. Retrieval and knowledge-system failures

ID	Failure mode	Effect	Rate band	Severity	Mitigation
FM-32	Wrong document retrieved	Model builds answer on wrong evidence	O3	S2–S4	Retrieval evaluation
FM-33	Relevant document omitted	Incomplete answer	O3	S2–S4	Recall testing
FM-34	Malicious document retrieved	Prompt injection	O2–O3	S4–S5	Treat retrieved text as untrusted
FM-35	Stale document	Outdated decision	O3	S3–S4	Version/freshness controls
FM-36	Conflicting sources	Model selects wrong authority	O3	S3–S4	Source hierarchy + conflict detection
FM-37	Embedding collision/semantic mismatch	Wrong material appears relevant	O3	S2–S4	Retrieval adversarial testing
FM-38	Access-control mismatch	Search crosses security boundary	O1–O2	S5	Authorization at retrieval layer

OWASP's current taxonomy separately identifies vector/embedding weaknesses because the retrieval layer introduces vulnerabilities not present in the base model alone.

---

6. Safety-policy failures

ID	Failure mode	Effect	Rate band	Severity
FM-39	Harmful request accepted	Dangerous instructions generated	O2–O3	S4–S5
FM-40	Safe request incorrectly refused	Loss of utility	O3–O4	S1–S3
FM-41	Safety classifier false negative	Harmful output delivered	O2–O3	S4–S5
FM-42	Safety classifier false positive	Benign content blocked	O3	S1–S3
FM-43	Jailbreak	Safeguard bypass	O2–O3	S3–S5
FM-44	Multimodal safety bypass	Harmful instruction encoded in image/audio/video	O2	S4–S5
FM-45	Safety regression after update	Previously controlled behavior returns	O2	S4–S5

The important engineering point is that refusal rate is not equivalent to safety.

A model can have an excellent aggregate refusal score while failing catastrophically on a narrow class of inputs.

---

7. Cybersecurity and infrastructure

ID	Failure mode	Effect	Rate band	Severity
FM-46	Model extraction	Proprietary model compromised	O1–O2	S3–S4
FM-47	Training-data poisoning	Systematic behavioral degradation	O1–O2	S4
FM-48	Dependency/supply-chain compromise	Malicious behavior introduced	O1–O2	S4–S5
FM-49	Denial of service	Service unavailable	O2–O3	S2–S4
FM-50	Resource exhaustion	Excessive compute/cost	O2–O3	S2–S4
FM-51	Infrastructure compromise	Attacker gains system access	O1–O2	S5
FM-52	Logging/monitoring compromise	Failures become invisible	O1–O2	S4–S5

OWASP's current taxonomy includes supply-chain vulnerabilities, data/model poisoning and unbounded consumption among the principal LLM application risks.

---

8. Human-system failures

This category is easy to underestimate.

ID	Failure mode	Cause	Effect	Rate band	Severity
FM-53	Automation bias	Human trusts fluent answer	Incorrect decision	O3–O4	S3–S5
FM-54	False confidence	System communicates certainty improperly	User fails to verify	O3–O4	S3–S5
FM-55	Verification fatigue	Too many low-value warnings	Human approves everything	O3	S3–S5
FM-56	Responsibility ambiguity	Unclear human/system responsibility	No effective accountability	O2–O3	S3–S5
FM-57	Operator misunderstanding	User doesn't understand system limitations	Misuse	O3	S2–S4
FM-58	Over-trust from prior success	Successful history suppresses skepticism	Rare failure accepted	O3	S3–S5

This is where a purely model-centric safety program breaks down.

NIST's AI RMF explicitly treats trustworthy AI as involving not merely validity but safety, security, resilience, accountability, transparency, explainability and privacy.

---

9. The particularly dangerous failure modes

If I were reviewing the system for certification, I would pull the following out of the FMEA and put them into a Critical Safety Item register:

CSI-1 — Incorrect consequential action

Failure chain:

hallucination → wrong plan → tool invocation → insufficient authorization → irreversible action

This is the archetypal agentic failure.

Target: essentially zero without independent authorization for irreversible operations.

---

CSI-2 — Prompt injection → privilege escalation

untrusted content → model interprets it as instruction → tool invocation → privileged operation

The vulnerability isn't simply "the model obeyed a bad prompt."

The actual system failure is:

untrusted information was allowed to become executable authority.

That distinction matters enormously.

---

CSI-3 — Confidently wrong high-consequence answer

model error → authoritative presentation → human accepts → physical/financial/legal consequence

The mitigation isn't simply "make the model smarter."

It is:

make consequential decisions require evidence and independent verification.

---

CSI-4 — Cross-boundary information disclosure

retrieval/memory/tool bug → authorization boundary crossed → model exposes information

The model should never be the security boundary.

Authorization should exist outside the model.

---

CSI-5 — Cascading agent failure

Agent A error → Agent B treats output as fact → Agent C acts → monitoring sees individually plausible actions

This is particularly dangerous because each component can pass its local tests while the system fails globally.

---

10. Failure-rate measurement

This is where I would disagree with any safety program that publishes one number such as:

"The model has a 99.9% success rate."

That number is almost meaningless.

I would establish failure-rate denominators by hazard class.

For example:

Ordinary informational response

$$FR_{info} = \frac{\text{materially incorrect responses}} {\text{applicable informational responses}}$$

Tool selection

$$FR_{tool} = \frac{\text{incorrect tool selections}} {\text{tool-selection opportunities}}$$

Tool parameterization

$$FR_{param} = \frac{\text{incorrect consequential parameters}} {\text{tool invocations}}$$

Unauthorized action

$$FR_{unauth} = \frac{\text{actions outside authorization}} {\text{consequential action opportunities}}$$

Catastrophic action

$$FR_{cat} = \frac{\text{catastrophic events}} {\text{consequential action opportunities}}$$

And I would never combine those denominators.

A model could have:

1% ordinary factual errors

and simultaneously have:

10⁻⁶ unauthorized destructive actions

or:

10⁻³ unauthorized destructive actions.

Those systems have completely different safety profiles despite perhaps similar "accuracy."

---

11. Required test program

The FMEA should drive the test program.

I would require at least:

A. Normal-use testing

• representative customer tasks
• domain-specific benchmarks
• long-context tasks
• multilingual tasks
• multimodal tasks
• ambiguous instructions

B. Adversarial testing

• direct prompt injection
• indirect prompt injection
• malicious documents
• malicious webpages
• malicious tool results
• jailbreaks
• conflicting instructions
• poisoned retrieval
• adversarial images/audio

C. Agent testing

• wrong tool
• wrong arguments
• repeated tool calls
• stale state
• corrupted state
• partial tool failure
• network failure
• timeout/retry
• duplicate execution
• unavailable rollback

D. Human-factors testing

• automation bias
• misleading confidence
• warning fatigue
• ambiguous authorization
• operator misunderstanding
• time pressure

E. Fault injection

This one is especially important.

Don't merely ask:

"Does the system work?"

Inject:

wrong retrieval
wrong tool result
stale memory
unavailable database
corrupted state
contradictory instruction
delayed response
duplicated response
compromised plugin
malicious document
model degradation

and see whether the system fails safely.

---

12. Mitigation hierarchy

I would rank controls by their position in the causal chain rather than by how sophisticated they sound.

Level 1 — Eliminate the hazard

Don't give the model the capability.

If the system doesn't need DELETE, don't expose DELETE.

Level 2 — Constrain capability

Least privilege.

Read-only wherever possible.

Level 3 — Deterministic authorization

A model cannot grant itself permission.

Level 4 — Independent verification

For consequential operations:

model proposes → independent system checks → authorization → execution

Level 5 — Human confirmation

For irreversible or high-consequence actions.

Level 6 — Monitoring

Record:

• model version
• prompt/context
• retrieved evidence
• tool selection
• parameters
• authorization
• action
• result
• subsequent correction

Level 7 — Recovery

Every consequential operation should have a defined recovery path where technically possible.

---

13. The most important architectural conclusion

The FMEA produces a rather interesting result.

The dominant risk isn't necessarily:

"The LLM will make a mistake."

We already know it will.

The more important question is:

"What happens when it makes a mistake?"

That produces two radically different systems.

System A

LLM error → text appears → human notices → correction

Potentially tolerable.

System B

LLM error → tool call → privileged system → irreversible action

Potentially catastrophic.

Thus the acceptable model error rate is application-dependent.

A 1% error rate might be tolerable for brainstorming.

The same 1% is absurd for an autonomous system deleting production databases.

And this is why I would make agency a separate safety variable from intelligence.

---

14. Certification criteria I would propose

For a major foundation-model provider, my release gate would look roughly like this:

Control	Release requirement
Known failure modes	Documented
Failure rates	Empirically measured by domain/use case
Critical failure modes	Explicitly identified
Regression suite	Passed
Adversarial suite	Passed
Tool authorization	External to model
Least privilege	Demonstrated
Irreversible actions	Independent authorization
Prompt injection	Tested against current attack corpus
Retrieval security	Authorization enforced outside model
Data isolation	Demonstrated
Rollback	Tested
Incident telemetry	Operational
Model update	Automatic safety regression testing
Human factors	Tested under realistic workload
Unknown failure discovery	Continuous red-team/fault-injection program

And importantly:

Passing the benchmark is not certification of the system.

Certification should attach to a specific model + configuration + tools + permissions + operating environment + intended use.

Change those materially, and you have a new safety case.

---

One final FMEA category: "unknown unknown"

A conventional FMEA has a weakness here. It assumes we can enumerate the failure modes.

With foundation models, that assumption is questionable.

So I would add:

FM-59 — Emergent/unanticipated failure

Cause: interaction among model capabilities, context, tools, users, environment, and optimization that was absent from the test corpus.

Likelihood: unknown.

Severity: potentially S5.

Detection: unknown until discovered.

Mitigation:

• fault injection
• continuous red teaming
• anomaly detection
• staged deployment
• bounded permissions
• reversible actions
• independent authorization
• kill switches
• incident reporting
• post-deployment evaluation
• mandatory regression testing after model updates

And this changes the engineering philosophy.

You don't solve FM-59 by pretending to estimate its probability accurately.

You solve it by making the consequences of an undiscovered failure bounded.

That, to me, is the central lesson of the entire FMEA.

NIST's framework is built around managing AI risk throughout design, development, deployment, use, and testing rather than treating safety as a property established once at model release.

In other words: don't require the LLM never to fail. Engineer the system so that foreseeable failures are detected, consequential failures require independent authorization, and unforeseen failures cannot automatically propagate into catastrophic consequences.

That is much closer to traditional safety engineering than to the usual "benchmark + alignment score" model of AI safety.