The Dstl8 Incident Library
A reference of real error patterns Dstl8 surfaces in production logs: what they look like, why they happen, how to fix them, and how to detect them automatically.
Category
Platform
Start here
Most distinctive entries
🚨 Security
PHP
pearcmd.php RCE Exploit AttemptRemote code execution via local file inclusion of PHP’s pearcmd utility. Includes real attack payloads, exploit chain breakdown, and remediation steps.
🟣 Networking
Calico
hostendpoints Watch ForbiddenOne missing RBAC verb halts host endpoint policy enforcement across twelve correlated services. Logged at WARNING, so no restart, no probe failure, nothing surfaces it.
🟥 Cascading Failures
LiteLLM Context Window to MCP Tool Failures
When your LLM proxy hits a token limit, the agent’s MCP tools start failing too. The two errors look unrelated, and most monitoring will not connect them.
All entries
29 entries
🚨 Security
PHP
pearcmd.php RCE Exploit AttemptRCE via local file inclusion of PHP’s pearcmd utility. Old vulnerability, still actively scanned.
🚨 Security
ThinkPHP RCE Attempt
Framework-specific RCE via
invokefunction. Frequently appears in same scan batch as pearcmd.
🚨 Security
Directory Traversal Attack Patterns
The
../ probe, the underlying technique behind most LFI-based RCE attacks. Detection and mitigation.
🟦 Code Regression
TypeError Cascade Across Services
One missing argument breaks a shared function called from four microservices. How Dstl8 correlates the pattern into a single root cause.
🟥 Cascading Failure
OTLP Exporter UNAVAILABLE to Cascading Service Failures
When the OTel collector hits memory pressure and your blocking exporters stall application threads, the telemetry layer becomes the outage.
🤖 LLM Errors
LiteLLM Context Window to MCP Tool Failures
When the LLM proxy hits a token limit, MCP tool calls downstream fail too. The cascade most monitors miss.
💾 Storage
S3 NoSuchKey Errors in Data Pipelines
Missing objects in your bucket cause silent pipeline failures. Why this happens (race conditions, deletes) and how to fail loudly.
⚙️ Infrastructure
Lambda Worker Deadline Reached
Function timeouts in long-running pipelines. The pattern that signals a backlog, and how to differentiate workload vs. infra fault.
🟪 Database
PostgreSQL
pg_stat_statements Does Not ExistWhen the query-stats extension isn’t installed in your production cluster. Why monitoring tools fail silently and how to fix it.
⚙️ Infrastructure
OTel Collector Pipeline Failures
When the receiver to processor to exporter chain breaks silently and your dashboards go blank. Backend auth, memory_limiter, processor ordering: the failure surface is wide.
⚙️ Infrastructure
OTel Cardinality Explosion
One innocent
user_id attribute can multiply your time-series count by a million. How to detect it, why it costs you, and where to bound it.
⚙️ Infrastructure
Datadog Agent Redis Authentication Failures
When credentials rotate but the agent doesn’t get the memo: observability blackout from a config-drift class of failure.
🟥 Cascading Failure
Search API Rate-Limiting to Vercel Anomalies
Upstream Google CSE rate limits surface as Vercel error spikes. The detection pattern for upstream to downstream causal chains.
🤖 LLM Errors
finish_reason: content_filter (200 OK)
OpenAI returns a clean 200 while silently withholding content. Your uptime monitor sees green. Why it’s invisible and how to detect it.
🤖 LLM Errors
stop_reason: refusal (200 OK)
A successful Anthropic response whose body is a refusal. Sometimes with no category at all. Why status-code monitoring can’t see it.
🤖 LLM Errors
overloaded_error mid-stream (529 after 200)
A 529 that arrives as a stream event after the 200, so status-code retry logic never fires. The stream just stops.
🤖 LLM Errors
invalid_prompt: “flagged as potentially violating”
Moderation dressed as a validation error. Fires on innocent prompts, correlates with length not content, and varies by model.
🤖 LLM Errors
Image content_policy_violation
Harmless images and screenshots rejected as unsafe. Baffling, seemingly random rejections in OCR and document pipelines.
🤖 LLM Errors
Azure content_filter across deployments
The same prompt trips the filter on one deployment, not another. Per-deployment blindness hides the inconsistency.
⚙️ Infrastructure
nginx connect() failed (111: Connection refused)
During a Kubernetes rollout, nginx routes traffic to pods that aren’t ready yet. A burst of 502s that self-resolves, unless it happens every deploy.
🟪 Database
ClickHouse
MaterializedView TTL FailingTTL on a materialized view is rejected outright. Retention silently stops applying while every query keeps working.
⚙️ Infrastructure
Flux
source-controller Helm Index Fetch FailureThe chart registry goes unreachable and HelmRelease upgrades stall. Git still reports success the whole time.
🟪 Database
MySQL InnoDB Cluster Instance Ejected, Read-Only
An expelled member goes read-only and rejects writes while the pod stays Ready and every probe passes.
🟣 Networking
Calico
hostendpoints Watch Forbidden (RBAC)One missing RBAC verb halts host endpoint policy enforcement. Logged at WARNING, so nothing ever surfaces it.
🟣 Networking
kube-apiserver
error dialing backendkubectl logs and exec return 500 while workloads run fine. The API server cannot reach the kubelet on 10250.
🟣 Networking
leader election lost Across Every Kubernetes ControllerA dozen controllers fail in the same second. Read alone each looks like a bug; together they name the real cause.
🟥 Cascading Failure
ingress-nginx
upstream timed out (504)504 after the read timeout expires. The access log tells you whether it is one bad endpoint or a saturated service.
💾 Storage
failed to acquire lease snapshot-controllerNormal at low rates, an incident at high ones. The same log line means opposite things depending on frequency.
🟣 Networking
MetalLB
speaker Gratuitous ARP FailureThe announcement interface is gone, so LoadBalancer traffic follows stale ARP to a node that no longer owns the IP.
+ More entries coming weekly. Submit a request or check back
No entries match your filters
Try selecting “All” or removing a filter to broaden your search.
See real detections in your own logs.
Dstl8 surfaces these patterns automatically: grouped, named by failure class, and explained in plain English. No alert fatigue, no manual correlation.