LLM API errors that report success and are actual failures. Silent content filters. Refusals riding on a 200. They’re quiet by construction, not by accident, and every uptime monitor ever built is structurally blind to them. In this failure class, it’s the quiet ones you have to worry about.
This piece first ran in the LinkedIn newsletter, Observability Next: Subscribe on LinkedIn
An LLM Failure Mode That Didn’t Exist Three Years Ago
A user asks your AI feature a question. The model refuses to answer. Your API call returned 200. Your dashboard is green. No alert fired, no error logged, nothing anywhere in your stack disagrees with the status code. The user got nothing, or got a refusal where their answer should be, and your monitoring watched the whole thing and called it healthy.
The short version: 200 OK is a lie.
This isn’t hypothetical and it isn’t rare. OpenAI returns finish_reason: content_filter on a successful 200 when moderation withholds or truncates content mid-generation. Anthropic returns stop_reason: refusal on a 200 whose entire payload is a refusal. And the cruelest one: on streaming calls, Anthropic’s overloaded_error can arrive as an event inside the stream, after the 200 is already on the wire, so every piece of fallback logic keyed to status codes never fires. All three are documented in the providers’ own docs and forums. All three return success.
There’s a loud cousin too. OpenAI’s invalid_prompt error, “your prompt was flagged as potentially violating our usage policy,” reads like a validation error. It’s a moderation verdict wearing a validation error’s clothes, and developers on OpenAI’s own forum have documented it firing on innocent prompts, correlating with prompt length rather than content, passing on one model and failing on another. Your error rate sees the 400. It cannot see that the reason attached is wrong, so the debugging session starts in the wrong place and stays there.
Why LLM Silent Errors are Structural, and Deadly
I want to be precise about the word silent here, because the industry has worn it out. Most people use “silent AI failure” to mean quality drift: the model got a little worse, the evals slipped, nobody noticed. That silence is statistical. This silence is structural. The transport layer said success, and everything downstream believed it. These failures are quiet by construction, not by accident. Moderation and capacity management run inside the provider’s inference pipeline, after your request was accepted. By the time the filter fires, the 200 is already sent. The failure lands as a field in the response body or an event in the stream. We’ve started calling that layer the quiet zone: the part of every response where failure signals exist but never propagate to your tooling, because nothing you run inspects it.
Regular readers will recognize the shape of this argument. This newsletter’s standing thesis is that observability was built by humans, for humans, to watch code written by humans. Here’s the corollary: it was also built to watch deterministic APIs. The entire monitoring stack rests on one contract, status code as the source of truth about request health. That contract held for twenty years. LLM APIs ended it. The same request can succeed, refuse, truncate, or die mid-stream, and in most of those cases HTTP reports success. Unknown unknowns again, in the most literal form yet: not failure modes you assessed and accepted, failure modes your instrumentation cannot represent.
First Catch the LLM error
Log the response body on every call, not just the non-2xx ones. That feels wasteful by deterministic-era instincts, and it’s exactly backwards now: for LLM APIs, the 200s are where the failures live. Silence isn’t the absence of signal. It’s a signal you’re not logging. Then watch rates, not instances. One refusal is the product working. A refusal spike is a prompt change, a model change, or a provider-side filter adjustment, and it has a cause you can find.
Secondly Mitigate Where Possible
Honestly: most of this class can’t be prevented. Moderation verdicts and capacity events happen on the provider’s side of the wall. You can handle refusal outcomes explicitly, consume streams defensively, and keep a failover posture for overload spikes. Every one of those reduces exposure. None eliminates it. The failures that remain will not announce themselves, which is why the durable posture is seeing, not hoping.
Get the Invisible LLM Errors Guide
We just published the full field guide: the complete taxonomy, a signature table covering every error string, and the detection posture, with a deep incident page for each failure. Ungated, no form: The Engineer’s Guide to Invisible LLM Failures https://www.controltheory.com/guides/invisible-llm-failures/
Bob Quillin is founder and CEO of ControlTheory, makers of Dstl8, runtime feedback for AI-generated code.
Sources for this issue: OpenAI developer documentation (error codes, finish_reason) and OpenAI community forum threads on invalid_prompt false positives (2025-2026). Anthropic documentation on streaming refusals and stop_reason. Microsoft Q&A threads on Azure OpenAI content filter behavior.
Table of Contents
Runtime Feedback for Agentic Engineers
Catch emergent patterns from AI-generated code fed directly back to your agent.
Backpress@controltheory.com
