S3 NoSuchKey Errors in Data Pipelines

An S3 NoSuchKey error occurs when a service tries to read an object from a bucket using a key that doesn’t exist. In data pipelines, this almost always means a race condition (the object hasn’t been uploaded yet), an out-of-order delete, or a key-generation bug where the writer and reader disagree on the exact key format.

NoSuchKey

S3 API error returned when GetObject or HeadObject is called with a key that doesn’t exist. HTTP 404.

Object key

The full path within a bucket that identifies an object. Case-sensitive. users/123/file.json and Users/123/file.json are different keys.

Eventual consistency

The older S3 behavior where writes weren’t immediately readable from all regions. Eliminated by AWS in December 2020 — S3 is now strongly consistent globally.

Dead-letter queue (DLQ)

A secondary queue where unprocessable work items are sent for inspection. Prevents pipelines from silently dropping failed work.

What does this incident look like?

You’ll see entries in your application logs like:

ERROR  transcribe-pipeline   GetObject failed
  bucket: prod-audio-uploads
  key:    jobs/2026-05-30/job-71f4d29c/source.wav
  error:  NoSuchKey: The specified key does not exist.
  status: 404

One of these by itself is noise — keys get deleted, jobs get retried with stale references, a long tail of small bugs always exists. The pattern to watch for is a spike against your historical baseline. Real example from a Dstl8 customer (anonymized):

• Baseline NoSuchKey rate for the transcription service: ~0.05% of GetObject calls
• Sustained rate during incident: 1.5% — a 30× elevation
• Z-score: well outside historical bounds
• Downstream effect: “Transcription service S3 errors causing data loss” — a CRITICAL incident automatically raised by Dstl8 once the elevated rate was sustained for >10 minutes

The data-loss framing is what matters. NoSuchKey at 30× normal rate isn’t just “some files missing” — it usually means an entire batch or time window of work is failing silently and customers are losing their results.

Why does this happen?

Four common causes, in roughly descending order of frequency:

1. Race conditions in event-driven pipelines

The most common cause by far. A typical pattern:

1. Producer service starts uploading an object
2. Producer sends a message (SQS, Kafka, EventBridge) saying “object ready”
3. Consumer service receives the message, fires GetObject
4. The upload hasn’t finished yet → NoSuchKey

The race window is typically milliseconds for small objects, but can be seconds or longer for large files. If your producer sends the “ready” event before PutObject returns 200, you have this bug.

2. Out-of-order or unauthorized deletes

Common variants:

• Lifecycle rules deleting objects before downstream consumers have processed them (e.g., a 7-day TTL on raw inputs while a batch job runs weekly)
• Cleanup jobs running out of order — a worker deletes “processed” data before all consumers finish reading it
• Manual deletion by someone who didn’t realize the objects were still in use

3. Key-format mismatches

The writer and reader disagree on the exact key format. Common bugs:

• Case sensitivity: writer uses Users/, reader uses users/
• Trailing slashes or double slashes
• URL-encoded vs. raw special characters (%20 vs. + vs. space)
• Timezone in path: writer uses UTC date, reader uses local date
• Numeric ID padding: writer uses 00123, reader uses 123

These are particularly insidious because everything looks right in code review — the keys are similar enough that engineers don’t notice the difference until production traffic exposes the mismatch.

4. (Historical) S3 eventual consistency

Before December 2020, S3 was eventually consistent for some operations — a recently-written key might not be immediately readable. This is no longer a cause. S3 has been strongly consistent globally for years. If someone tells you “we’re seeing NoSuchKey because of S3 consistency,” they’re working from outdated knowledge.

How severe is it?

Severity is highly variable depending on what’s happening to the failed reads:

The most dangerous pattern is “silent drop.” Pipelines that catch NoSuchKey, log a warning, and continue with the next job are masking customer-impacting failures. Every NoSuchKey in the warning logs is a job that may have lost its data.

How to debug and remediate

1. Confirm the key actually doesn’t exist

Don’t trust your application’s error — verify directly against the bucket:

aws s3api head-object \
  --bucket prod-audio-uploads \
  --key jobs/2026-05-30/job-71f4d29c/source.wav

If this returns metadata, your application is failing because of IAM permissions, not a missing object. Different fix entirely — check your service’s IAM role for s3:GetObject on the bucket/prefix.

If this returns NoSuchKey from the CLI too, the key truly doesn’t exist.

2. Trace the upstream write

Search application logs for the key. There should be a PutObject success entry from your producer service. If there isn’t:

• The producer never tried to write (upstream bug)
• The producer tried but failed (check producer logs for errors)
• The producer used a different key than you’re looking up (key-format mismatch)

3. Check for race conditions

If both write and read happened in close succession:

# Pseudo-log search
PutObject  key=jobs/.../source.wav  result=success  ts=14:32:45.123
GetObject  key=jobs/.../source.wav  result=NoSuchKey  ts=14:32:45.118  ← 5ms BEFORE the write completed

If the GetObject timestamp is before the PutObject success timestamp, you have a race. The producer should wait for PutObject to return 200 before sending the “ready” event to consumers.

4. Audit for unauthorized deletes

Check CloudTrail or S3 access logs for DeleteObject events on the affected key or prefix. Look at:

• Lifecycle rules — are objects being expired before consumers are done?
• Cleanup jobs — is anything running that deletes “old” objects?
• Manual deletion — was someone cleaning up in the console?

Disable S3 lifecycle rules on production data buckets unless you’re absolutely sure no downstream consumer needs the objects.

5. Add retries with exponential backoff (for genuine races)

If you’ve confirmed a race condition and can’t fix the producer side immediately, add retries on the consumer:

for attempt in 1..5:
    try:
        obj = s3.get_object(Bucket=bucket, Key=key)
        return obj
    except NoSuchKey:
        if attempt == 5:
            raise
        sleep(0.1 * 2**attempt)  # 100ms, 200ms, 400ms, 800ms, 1600ms

Most AWS SDKs and pipeline frameworks support this natively — verify it’s enabled and the backoff is appropriate.

6. Route unrecoverable failures to a dead-letter queue

If retries exhaust, do not drop the work silently. Send the failed job to a DLQ with full context: the missing key, the timestamp, the upstream job ID, and a description of why it failed. This makes silent data loss visible and recoverable.

7. Alert on rate spikes, not absolute counts

Your normal NoSuchKey rate is almost never zero. Alert when the rate exceeds your historical baseline by a statistically significant margin (z-score > 3 or so), not when the absolute count crosses an arbitrary threshold. This is exactly the kind of pattern Dstl8 detects automatically.

How Dstl8 detects this

Dstl8 baselines your normal NoSuchKey rate per service and alerts on statistical anomalies. When the rate spikes well above baseline, Dstl8 surfaces it as a single incident with the impact framed as data loss, not just elevated error rate. Here’s an actual detection from a Dstl8 customer (anonymized):

Notice three things about how the incident is framed:

1. Severity is Critical, not Minor. Dstl8 correctly classifies “data loss” rather than just “error rate up.” A 1.5% NoSuchKey rate on transcription jobs means real customer files are vanishing — that’s not a Minor incident.
2. Root cause hypothesis included. “Likely race condition in upstream upload completion signaling” — the on-call doesn’t need to start from zero.
3. 30× baseline comparison. Not “high error rate” — a specific, statistically grounded multiplier that tells the engineer how far outside normal this is.

Related patterns

References

• AWS S3 strong consistency announcement (Dec 2020): aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
• S3 error responses reference: docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
• “Designing Data-Intensive Applications” — Martin Kleppmann, on race conditions in distributed pipelines
• AWS SDK retry behavior: docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html