LiteLLM Context Window Exceeded → Downstream MCP Tool Failures

A LiteLLM context window exceeded error occurs when a request to the LiteLLM proxy contains more tokens than the underlying LLM can process. The failure cascades downstream: AI agents calling MCP tools mid-orchestration lose the ability to process tool outputs, causing the tool layer to fail with errors that look unrelated to the LLM proxy.

LiteLLM

An open-source proxy that abstracts over LLM providers (OpenAI, Anthropic, Mistral, Bedrock, self-hosted models). Used by AI agent stacks for routing, fallback, and cost optimization.

Context window

The maximum number of tokens an LLM can process in a single API call. Varies by model — 32K, 128K, 200K, or 1M tokens are common.

MCP (Model Context Protocol)

An open standard for AI agents to call external tools. MCP servers expose tools; MCP clients (agents) invoke them.

context_length_exceeded

The specific error class returned by most LLM providers when a request exceeds the model’s context window.

What does this incident look like?

Two things happen at almost the same time:

Upstream: LiteLLM starts returning context_length_exceeded errors on a subset of requests. Error rate climbs from baseline (typically near 0%) to several percent. In observability data, you’ll see something like:

service: litellm
pod: litellm-5d7b9f8c2a-x4kp9
error_count: 21
error_rate: 4.7%
z_score: 10.9
window: 09:38 UTC – 09:48 UTC
error_class: context_length_exceeded

Downstream: Within minutes, the agent’s MCP tool-calling layer starts returning errors or timeouts. The MCP servers themselves are healthy — they respond normally to direct probes. But agents calling them via the orchestration loop start failing because the LLM that was supposed to consume the tool output can’t process the resulting context.

The two signal sources look independent to most monitoring tools. A traditional alerting setup will fire two unrelated alerts: “LiteLLM service error rate elevated” and “MCP tool call failure rate elevated.” An engineer paged for the tool failures will start investigating the tool layer — which is healthy — and lose 30 minutes before realizing the actual root cause is upstream.

Why does this attack work? — er, why does this cascade happen?

Three things need to be true for the cascade to fire, and they often are in AI agent stacks:

1. The agent accumulates context as it runs

Modern AI agents work in a loop: the LLM is invoked, decides to call a tool, the tool’s output is appended to the conversation context, and the loop continues. Each tool call adds tokens. A long-running agent task can easily accumulate 50K–200K tokens of conversation history, tool outputs, and intermediate reasoning.

2. LiteLLM proxies the request without truncating it

LiteLLM’s job is to abstract over LLM providers — it forwards the request as-is to the configured backend (OpenAI, Anthropic, Mistral, Bedrock, a self-hosted model, etc.). It doesn’t, by default, trim the conversation context to fit. If your context is over the model’s limit, you get back an error like:

{
  "error": {
    "type": "context_length_exceeded",
    "message": "This model's maximum context length is 32768 tokens.
                However, your messages resulted in 41284 tokens."
  }
}

3. Different models, different limits — and the limit is hit unevenly

Common context windows as of 2026 (varies by model version and provider):

If your LiteLLM config routes some requests to a 32K-window model (cost optimization, fallback, or A/B test), and your agent task pattern reaches 35K tokens of accumulated context — only those routes break. The 128K-window routes still succeed. This makes the failure look intermittent and traffic-pattern-dependent, which is the worst kind to debug.

The cascade in 5 steps

How severe is it?

Severity: Major — agent task failures are user-visible. The agent doesn’t complete the workflow, and depending on retry semantics, may either fail loudly or quietly skip work. Cost impact is also non-trivial: failed requests are usually billed at the prompt-token rate by the model provider, so context-overflow errors are paying for nothing.

Blast radius depends on your agent topology:

• Single user-facing agent: one user’s session breaks; they retry, may succeed, may not
• Batch-job agent: a chunk of records fails to process; backfill required
• Multi-agent system: one failed agent may block dependent agents — cascade compounds
• Long-running agent: work accumulated over hours may be lost if state isn’t persisted across retries

How to debug and remediate

1. Confirm the upstream cause first

If you’re seeing MCP tool failures, your first check should be whether your LLM proxy is throwing context errors in the same window. Pseudocode:

# Time-aligned correlation
SELECT
  bucket(time, '1m') AS minute,
  count(*) FILTER (WHERE service='litellm' AND error_class='context_length_exceeded') AS llm_errors,
  count(*) FILTER (WHERE service='mcp-gateway' AND status >= 500) AS tool_errors
FROM logs
WHERE time > now() - interval '1 hour'
GROUP BY minute
ORDER BY minute

If llm_errors and tool_errors rise and fall together, the tool errors are downstream effects. Investigating the tool layer in isolation will waste time.

2. Quick mitigations (deploy today)

• Truncate or summarize conversation history before passing to LiteLLM. Most agent frameworks support a sliding window strategy.
• Cap MCP tool output sizes. Tools that return large blobs (file contents, log dumps, search results) should truncate at the source — 10KB per response, not 1MB.
• Switch the affected route to a larger-context model. If LiteLLM is routing some traffic to a 32K-window model, move that route to a 128K+ model for now and re-evaluate cost.
• Set LiteLLM request_timeout aggressively so requests fail fast rather than hanging — this prevents downstream tool calls from queueing behind a stuck LLM call.

3. Structural fixes (deploy this sprint)

• Add context-pressure tracking to your agent loop. Log token count per LLM call. When you’re within 80% of the routed model’s window, log a warning and either truncate or escalate to a larger model.
• Implement summarization checkpoints. After every N tool calls (or M tokens), compress the conversation history into a summary message. This is the “Reflexion” pattern from agent research and the default in most agent SDKs.
• Persist agent state outside the context window. Tool outputs that the agent will need to reference repeatedly should be stored in a side cache and referenced by ID, not held in the conversation context.

4. Detection (so this doesn’t surprise you next time)

The single most useful alert here isn’t on LiteLLM errors alone or MCP errors alone — it’s on the correlation. Set up a query that fires when both rise together within a 5-minute window. Most observability tools won’t do this natively; you have to write it. Or use a tool that surfaces cross-service correlations automatically.

How Dstl8 detects this

Dstl8 surfaces the cascade in a single incident with both the upstream cause and the downstream impact named together. Here’s an actual detection from our own staging environment:

Three things to notice in that detection:

1. “Downstream impact on MCP tool calls observed” — Dstl8 connected the upstream error to the downstream effect automatically, in the same incident. No correlation query to write, no two alerts to triage.
2. Z-score 10.9 — the error rate jumped about 11 standard deviations above the historical baseline for this service. This is a statistically clean anomaly, not a threshold crossing.
3. 6 detection events over ~5 hours — Dstl8 caught a recurring pattern, not just a single spike. The cascade fired multiple times before the underlying context-management issue was fixed.

Related patterns

References

• LiteLLM project: github.com/BerriAI/litellm
• Anthropic Model Context Protocol spec: spec.modelcontextprotocol.io
• OpenAI context length documentation: model-specific in the API reference
• Anthropic context windows: docs.anthropic.com
• Agent design pattern (Reflexion + summarization): Yao et al., 2023