Your Logs Lost the Request

Build a context-rich log event that keeps an upload's outcome, storage attempts, and service context together.

10:14:03 INFO upload started 10:14:04 WARN retrying storage request 10:14:05 ERROR upload failed

A customer says their document upload failed. These three lines look promising until you ask the obvious questions.

Which customer request? What document? How many times did storage retry? Did the request fail before or after the latest deployment?
Logging, as it's commonly practiced, is fundamentally broken. And slapping OpenTelemetry on your codebase won't magically fix it.

Let me show you what's wrong, and more importantly, how to fix it.

The problem is that useful context is scattered across individual log messages. When a request finishes, you need a record that brings its outcome and the relevant context together.

The Core Problem

A document upload passes through the gateway, authentication service, document service, and storage. Each component reports what it is doing:

10:14:03.101 DEBUG [gateway] Reading request body 10:14:03.122 INFO [auth] Checking access token 10:14:03.145 DEBUG [auth] Token accepted 10:14:03.166 INFO [gateway] Forwarding upload 10:14:03.184 INFO [documents] Starting upload 10:14:03.201 DEBUG [documents] Checking document type 10:14:03.223 DEBUG [documents] Checking file size 10:14:03.247 INFO [documents] Document validation passed 10:14:03.268 INFO [storage] Selecting storage region 10:14:03.291 DEBUG [storage] Opening storage connection 10:14:03.314 INFO [storage] Sending document, attempt 1 10:14:03.914 WARN [storage] Upload attempt timed out 10:14:03.938 INFO [storage] Retrying document upload 10:14:04.018 INFO [storage] Sending document, attempt 2 10:14:04.371 INFO [storage] Document stored: doc_789 10:14:04.402 INFO [documents] Upload completed 10:14:04.431 INFO [gateway] Response sent: 201 Created

That is 17 lines for one successful upload, including a timeout that recovered on retry. If 10,000 concurrent users each make one such request, that is 170,000 lines to collect and search. Concurrency alone does not tell you how many lines arrive per second; that depends on how quickly those requests happen.

Volume is only half the problem. The lines tell you what individual components did, but they don't consistently identify the request, the user, the document, and the final outcome. When one upload fails among thousands, stitching those details together becomes the work.

The old approach made sense when a request could be followed through one process on one server. In a distributed system, a request might pass through ten services, three databases, and two caches. Each service has its own view, and the useful facts are scattered across boundaries.

Searching for one user

Suppose you search for user-123. Different parts of a codebase might record the same identifier in different forms:

user-123 user_id=user-123 {"userId": "user-123"} [USER:user-123] processing user: user-123

A text search can find some of these, but inconsistent formats make structured filtering harder. More importantly, the storage service might know only doc_789. A user search will not reveal that line unless you first find the connection between the user and the document.

The application may have known more while handling the upload, but never recorded those details alongside the failure:

  • Document type: invoice
  • Document size: 12 MB
  • Storage error: STORAGE_TIMEOUT
  • Retry attempt: 3 of 3
  • Upload flow: new flow enabled

If those fields were never logged, searching harder will not bring them back. And if they were logged separately without a shared identifier, you still need to work out which lines belong to the same request.

Now you need another search. Then another. You are reconstructing a request from clues that were never collected in one place.

Logs are easy to write at the point where work happens. They are much harder to query when the facts you need are spread across separate lines.

OpenTelemetry Won't Save You

OpenTelemetry gives you a standard way to instrument and export logs, traces, and metrics. Its APIs, SDKs, and protocols help systems collect and exchange telemetry without tying instrumentation to one vendor's format.

That is useful. It does not automatically give you the context needed to diagnose a failed upload.

  1. It does not decide which application details matter. You still choose the fields that answer your team's questions.
  2. It does not know your business context. Unless your application records the document type, storage region, or enabled upload flow, the telemetry cannot include them.
  3. It does not repair a weak instrumentation strategy. Standardized telemetry can still contain little more than a timestamp and a generic error.

Consider a span with only the basics:

{ "span_name": "POST /documents", "duration_ms": 1840, "status": "ERROR", "error.type": "UploadError" }

You know the operation failed. You do not know whether it failed during validation or storage. Now add the context your application has available:

{ "span_name": "POST /documents", "duration_ms": 1840, "status": "ERROR", "document.type": "invoice", "storage.region": "ap-south-1", "storage.attempts": 3, "upload.new_flow_enabled": true, "error.type": "StorageTimeout" }

Now you can see where to start investigating and whether the new upload flow might be worth examining. Those fields do not establish the cause; they give you better questions to ask.

A span and a log event are different telemetry records. A span describes an operation over time. A context-rich log event captures the fields you need to query that operation. They can carry overlapping fields and be connected through trace context. With proper instrumentation and propagation, spans across services share a trace ID; logs must also carry that context if you want to correlate them. See the OpenTelemetry trace documentation.

The Fix: Context-Rich Events / Canonical Log Lines

Record the outcome of the request together with the context needed to understand it.
A context-rich log event is a structured event built as one service handles a request and emitted when that service finishes.

Build an event while the service handles the request, then emit it when the operation finishes. Include the outcome, timing, relevant identifiers, and the fields that help explain the result. Emit one such event for each service hop you want to investigate. Other diagnostic or audit logs can still serve their own purposes.

Stripe calls a related pattern a canonical log line: an information-rich line emitted at the end of a request or other unit of work, alongside ordinary logs.

Start with the questions you will ask

Suppose a customer reports a failed upload. You want to know:

  • Which request was affected?
  • Did the service reject the document or fail while storing it?
  • How long did the request take, and how many storage attempts occurred?
  • Was the failure associated with a particular release or storage region?

Those questions suggest fields. They do not require recording the document itself, the user's access token, or a full HTTP request body.

Here is a compact record that could support the investigation:

{ "request_id": "req_8a12", "service": "document-api", "deployment": "release-42", "status": 503, "duration_ms": 1840, "document_type": "invoice", "storage_region": "ap-south-1", "storage_attempts": 3, "error_code": "STORAGE_TIMEOUT" }

This request failed after three storage attempts and returned a 503. The record does not tell you why storage timed out. That might require a trace, storage metrics, or another service's records.

A good record narrows the investigation. It does not turn an error code into a root cause by sheer confidence.

With the request ID, you can find the event and see the outcome, region, duration, and number of attempts together. The storage timeout tells you where to investigate next. It does not tell you what caused the timeout.

Query the events

Once these fields are available as columns in a queryable events table, you can investigate patterns across requests. For example, this query groups uploads by storage region and calculates the share that returned a server error:

SELECT storage_region, COUNT(*) AS total, SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) AS errors, ROUND(100.0 * SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) / COUNT(*), 2) AS error_rate_pct FROM events GROUP BY storage_region;

For a dataset of 100 uploads, the results could look like this:

Storage regionTotalErrorsError rate
ap-south-13500.00%
us-east-13725.41%
eu-west-128310.71%

The last region deserves a closer look. The table alone does not prove a regional outage. Check the time window, request volume, error codes, and deployment before deciding what changed. These rates also assume every upload in the period is represented; biased sampling needs separate treatment.

See what each field adds

Here's how the request, document, storage, and error fields come together when an upload fails.

The request ID identifies the operation. The error shows that storage timed out, and the storage fields show that the service made three attempts. The document type gives you another way to investigate whether a particular class of document is involved.

A single event helps with one customer report. The same fields across many requests let you ask whether a problem is isolated or recurring. That is where consistent names matter. If one handler records storage_attempts and another records retries, somebody has to remember whether those numbers mean the same thing.

JSON is a container

Structured logging records named fields. JSON is one common way to represent them:

{"message":"upload failed"}

That is valid structured data. It still leaves you with the same question: which upload?

A context-rich log event is a choice about what belongs together. You collect context from the request and the work it triggers inside the service. Then you emit the result with a consistent shape.

Two terms come up when you query these records:

  • Cardinality is the number of distinct values in a field. Request IDs have many possible values. HTTP methods have relatively few.
  • Dimensionality is the number of fields you can use to filter, group, or otherwise explore the data. Region and document type are two examples.

High cardinality is useful when you need to find a specific request. It can also make storage and indexing expensive, depending on the backend. More dimensions can answer more questions, but each field needs a purpose and a privacy review.

The goal is useful context, not a competition to see how many keys fit in a JSON object.

Build the record as the request runs

At the beginning of a request, you know the route and request ID. You do not yet know whether storage will fail or how long it will take. Capture what you know at each step, then emit the record when the operation finishes.

The following TypeScript is a simplified service function. storage and logger represent application interfaces; the example leaves HTTP response mapping to the caller.

type UploadRecord = { request_id: string; service: string; document_type: string; size_bytes: number; storage_attempts: number; outcome?: "success" | "error"; error_code?: string; duration_ms?: number; }; async function uploadDocument(input: { requestId: string; documentType: string; bytes: Uint8Array; }) { const start = Date.now(); const record: UploadRecord = { request_id: input.requestId, service: "document-api", document_type: input.documentType, size_bytes: input.bytes.length, storage_attempts: 0, }; try { const result = await storage.put(input.bytes, { onAttempt: () => { record.storage_attempts += 1; }, }); record.outcome = "success"; return result; } catch (error) { record.outcome = "error"; record.error_code = classifyUploadError(error); throw error; } finally { record.duration_ms = Date.now() - start; try { logger.info(record); } catch { // Logging must not replace the upload's result or original error. } } }

The storage interface calls onAttempt for each actual attempt. That is what makes the count available even if the final attempt throws. If your storage client cannot expose attempts, do not pretend it can. Instrument the retry loop you control or omit the field.

The logging call is guarded so a synchronous logger failure does not replace the upload result. A real system still needs a way to monitor dropped logs. Silently losing records is a poor long-term observability strategy.

This also assumes the process reaches finally. A process crash or forced termination can prevent the final record from being emitted. For operations that need durable audit history, use a separate mechanism designed for that requirement.

Give the fields a stable meaning

A field is only useful across requests if it means the same thing each time.

Does storage_attempts include the first attempt, or only retries? Does duration_ms cover the entire HTTP request, or only the storage call? Does status describe the response from this service or the downstream storage provider?

Pick an answer and document it. For the example above, attempts include the initial call, and duration measures this function from entry to exit. An HTTP middleware could capture the response status and route around it, provided it records the status that was actually returned.

Use a small vocabulary for error codes. Raw exception messages can change between library versions and may contain sensitive data. A stable code such as STORAGE_TIMEOUT is easier to group by. Keep any detailed diagnostic information under an explicit retention and access policy.

If a request crosses three services, each service can emit its own record. Propagate a trace identifier or another correlation identifier so you can connect them. One service cannot reliably describe work it never observed.

Sampling: Keeping Costs Under Control

Capturing more context increases the amount of data you send and store. Sampling reduces that volume by retaining a subset of events. You might keep one in every ten successful requests, for example, instead of storing them all.

The question is which events you can afford to lose. A purely random sample might discard the one failed upload your customer asks about.

One approach is to decide what to keep after the request finishes, when its outcome and duration are known. For example:

  1. Keep errors. Retain events for server errors, exceptions, and other failures you want to investigate.
  2. Keep slow requests. Retain requests above a chosen latency threshold, such as a recently measured p99. Define how often that threshold is updated.
  3. Keep selected sessions. Retain internal test requests or sessions flagged for investigation, subject to your privacy rules.
  4. Sample the rest. Retain a random fraction of successful, fast requests, such as 1–5%.

These are policy choices, not guarantees. Even a rule to keep every error cannot recover an event dropped upstream, lost in the logging pipeline, or never emitted after a process crash.

Be careful with aggregate queries. If you keep all errors but only some successes, dividing the retained errors by all retained events will overstate the error rate. Use independent metrics or account for each event's sampling probability when estimating traffic rates.

Trace tail sampling is related. It decides whether to retain a trace after collecting most or all of its spans, so the decision can consider information across the request path. That is different from deciding whether to keep a single log event after one service finishes. The OpenTelemetry sampling guide explains the tradeoffs.

Sampling is only one part of cost control. Give each field a reason to exist, set retention deliberately, and leave raw documents, credentials, and unnecessary personal information out of the event.

The next failed upload

A customer reports another failed upload. You find the context-rich log event and see the service returned a 503 after three storage attempts. You have the request ID, the release, and the storage region. The next question is specific: what was happening on that storage path when the attempts timed out?

That is a better place to begin than three disconnected messages. Your logs still will not fix the upload. They can at least help you find the system that might.