Logs, Metrics and Traces: What Each One Is Actually For

The three signals are usually introduced as a set to collect, which skips the part that matters: they answer different questions, they scale differently, and using one where another belongs is how observability bills grow without the debugging getting easier.

For a separate people-operations application of the same measurement discipline, see workforce analytics software.

The distinction that matters

Metrics answer "is something wrong, and how much." Numeric, aggregated at collection, cheap and constant in cost regardless of traffic. They tell you the error rate rose. They cannot tell you which requests failed.

Traces answer "where did the time go in this request." A record of one request's path through the system, with timing per hop. They show that the 3-second response spent 2.8 seconds in one downstream call.

Logs answer "what exactly happened here." Discrete events with arbitrary detail. Cost scales with traffic, and they are the only signal that carries the specific values involved.

The practical workflow: metrics tell you there is a problem, traces tell you where, logs tell you what. Skipping a layer is what turns a ten-minute investigation into an afternoon.

Metrics: the two traps

Averages hide everything you need to see

A mean latency of 200ms is consistent with every request taking 200ms and with 95% taking 50ms while 5% take 3 seconds. Those are different systems, and only one of them has a problem.

Record percentiles, and record p99 specifically. The tail is where users notice, and it is where saturation shows up first.

Averaging percentiles across instances is invalid. The mean of ten instances' p99 values is not the p99 of the whole. Percentiles do not average. If your dashboard does this — and many do by default — the number displayed is not a percentile of anything. Aggregate from histograms, which merge correctly, rather than from precomputed quantiles.

Cardinality is the cost

Every unique combination of label values is a separate time series. A metric labelled with user ID or request path with identifiers produces an unbounded number of series, and this is the single most common cause of a monitoring system falling over.

Labels should be low cardinality and bounded: status class, endpoint template, region, instance. Not user ID, not full URL, not error message text.

Put the high-cardinality detail in traces and logs, where it belongs and where the cost model tolerates it.

Traces: sampling is where they go wrong

A trace records a request's journey: each span is one operation, with a start, a duration, and a parent. It shows you the shape of a request and where the time went.

Context propagation is the whole thing. If the trace context is not passed across a boundary — a queue, a thread pool, an HTTP client that was not instrumented — the trace breaks there and the downstream work appears as a separate, orphaned trace. Most disappointing trace setups are broken propagation rather than missing instrumentation.

Head-based sampling discards the traces you want. Deciding at the start of a request whether to sample it, at 1%, means you keep 1% of the slow ones and 1% of the errors. The interesting traces are rare by definition, and random sampling is precisely the wrong filter.

Tail-based sampling makes the decision after the request completes, when you know it was slow or failed. Keep all errors, all requests above a latency threshold, and a small random sample of the rest. This costs more to operate — the collector must buffer spans until the request finishes — and it is the difference between traces being useful during an incident and not.

Span attributes are where request-specific detail goes. Tenant, query shape, cache hit or miss, the size of the result. High cardinality is fine here.

Logs: structured or nearly useless

Log structured events, not sentences. {"event":"payment_failed","order_id":"...","provider":"...","code":"..."} is queryable. "Payment failed for order 12345 with provider Stripe" requires a regular expression, and the regular expression breaks when someone rewords the message.

Log at boundaries and decisions, not at every step. Entry and exit of a request, calls to external systems, decisions that affect the outcome, and errors with the context needed to act. Logging inside a hot loop is how you generate a terabyte a day and find nothing in it.

Include the trace and span ID in every log line. This is the single highest-value field and it is frequently missing. With it, you jump from a slow trace directly to the log lines emitted during it. Without it, you are correlating by timestamp and hoping.

Levels have meanings worth respecting. ERROR means something needs human attention. WARN means something unexpected that the system handled. INFO means a significant event. DEBUG means detail for investigation, off in production by default.

The most common failure is everything at INFO, which makes the level useless as a filter and makes ERROR alerting impossible.

Never log secrets, tokens, or personal data. Logs are copied, shipped to third parties, retained for months, and read by people who do not need the contents. Redact at the point of logging, not in the pipeline — a pipeline filter is one deployment away from being bypassed.

What to instrument first

If you are starting from nothing, in order:

1. Request rate, error rate, and latency distribution per endpoint. These three cover most of "is it working."

2. Saturation of every bounded resource. Connection pools, thread pools, queue depth, disk, memory. Queue depth and pool utilisation are the leading indicators — they move before latency does, which is what makes them worth alerting on.

3. Latency and error rate of every outbound dependency. Most incidents originate downstream, and this is what tells you it is not you.

4. Trace context propagation everywhere, before elaborate tracing. A broken chain makes the rest pointless.

5. Structured logs with trace IDs, at boundaries.

6. Business-level counters that would reveal a problem the technical metrics miss — orders per minute, signups per hour. A deploy that breaks a form produces perfect technical metrics and zero orders.

Alerting

Alert on symptoms, not causes. High CPU is not a problem if users are served correctly. Elevated error rate is a problem regardless of the cause.

Every alert should have an action. An alert nobody acts on trains people to ignore alerts, including the ones that matter.

Page on user impact; ticket everything else. The distinction is whether it needs someone awake.

Alert on the leading indicators too — queue depth rising, pool near exhaustion — because they give you time to act rather than notifying you of an outage in progress.

Controlling the cost

Observability bills grow faster than traffic if nothing constrains them.

Cardinality is the main lever for metrics. Audit label sets periodically; one bad label added in a routine change can multiply series count.

Sampling is the main lever for traces. Tail-based, keeping what is interesting.

Level and volume are the levers for logs. DEBUG off in production, no logging in hot loops, and short retention for high-volume low-value streams. Retention can differ per stream — errors for a year, access logs for a fortnight.

Ask what question each signal answers. Anything collected that has never been queried is a line item, not observability. An annual review of what has actually been used during incidents usually finds a lot to delete.

The summary

Metrics for detection, cheap and aggregated, percentiles rather than averages, cardinality controlled.

Traces for locating, with propagation working end to end and tail-based sampling so the interesting ones survive.

Logs for detail, structured, at boundaries, carrying trace IDs, without secrets.

And the field that ties them together is the trace ID. If your logs do not carry it, adding it is the highest-return change available in this whole area.