Measuring Performance So You Optimise the Right Thing

The expensive failure in performance work is not slow code. It is a week spent optimising something that contributed 2% of the total, chosen because it looked slow.

Technical benchmarks and team activity metrics require different definitions; organisations evaluating employee monitoring software should decide which operational question each metric is meant to answer.

Measurement is not a preliminary step. It is most of the work.

Start with the number that matters

Optimise the metric the user experiences. Usually end-to-end latency at a high percentile, occasionally throughput or cost per request.

Not average latency. A mean of 200ms is consistent with a well-behaved system and with one where 5% of requests take three seconds. Only one of those has a problem, and it is the one users are complaining about.

Not a microbenchmark. A function that runs 40% faster in isolation may contribute nothing measurable, because it was 0.3% of the request.

Set a target before you start. "Faster" is not a goal and has no stopping condition. "p99 under 300ms for this endpoint" is checkable, and it tells you when to stop — which matters, because performance work has no natural end.

Profile, do not guess

Intuition about where time goes is unreliable, including for people who wrote the code. The two useful modes:

Sampling profilers interrupt periodically and record the stack. Low overhead, safe in production, and they give a statistical picture. This is what you want for "where does the time go."

Instrumenting profilers record entry and exit of every call. Exact counts, high overhead, and the overhead distorts what it measures — small frequently-called functions look worse than they are.

Profile the real workload. A profile of a synthetic benchmark tells you about the benchmark. Production traffic has different data shapes, cache behaviour and concurrency.

Read profiles as flame graphs where you can, and read them by width, not by depth. A wide frame is where time went. A deep stack is just a call chain.

Distinguish self time from total time. A function with large total time and small self time is not slow — its callees are.

The traps that make benchmarks lie

Nearly every misleading performance result comes from one of these.

Warm-up. Managed runtimes interpret first and compile hot paths later. The first thousand iterations may be an order of magnitude slower than the steady state. Measuring without warm-up measures the compiler.

Dead code elimination. A benchmark computing a value nobody uses may have the computation removed entirely. A loop that "takes 0.2ns per iteration" was deleted. Consume the result in a way the optimiser cannot see through.

Caching at every layer. The second run hits the page cache, the query cache, the CPU cache. Decide deliberately whether you are measuring warm or cold, and be consistent.

Measuring on your laptop. Different CPU, no network, no contention, a fraction of the data, and thermal behaviour that differs under sustained load.

Coordinated omission. The subtle one, and it is why many load test results are wrong.

If your load generator sends a request, waits for the response, then sends the next, then during a stall it sends fewer requests. The slow period is under-sampled, and the reported percentiles are far better than reality. The correct approach sends at a fixed rate regardless of response times and records latency against the intended send time. Load generators differ in whether they do this; it is worth checking which yours does.

Averaging percentiles. The mean of ten instances' p99 is not the p99 of the whole. Percentiles do not average. Aggregate from histograms.

Too few runs. Variance between runs is often larger than the improvement being measured. Run enough times to see the distribution, and compare distributions rather than single numbers.

Where the time usually is

Before profiling, the priors are worth knowing, because they are stable across systems.

Waiting, not computing. Most request time in a typical service is spent waiting on the network, the database or the disk. CPU optimisation on a system that is I/O-bound produces nothing.

The N+1 query. One query to fetch a list, then one per item. A hundred round trips where two would do. The single most common performance defect in application code, and it is invisible in a profile of the application — it shows as time in the database client.

Serial work that could be concurrent. Three independent calls made one after another take the sum; made concurrently they take the maximum.

Data volume. Fetching a thousand rows to display ten, selecting every column to use two, returning a whole object graph to read one field.

Serialisation. Frequently a substantial share of request time in service-to-service architectures, and almost never where people look.

Lock contention. Which appears as time spent doing nothing and does not show up in a CPU profile at all — you need a lock or off-CPU profile to see it.

Latency, throughput, and the relationship

They are not the same and improving one can worsen the other.

Latency is per-operation. Throughput is per-unit-time. Batching improves throughput and worsens latency. More concurrency improves throughput until the system saturates, at which point latency rises sharply while throughput stops improving.

The knee of that curve is what you want to find. Load-test at increasing concurrency and plot both. Throughput plateaus and latency climbs at the same point — that is your capacity, and running above it means queuing.

Little's law is a useful sanity check: in a stable system, the number of requests in flight equals arrival rate multiplied by average time in the system. If your measurements do not roughly satisfy it, one of them is wrong.

Making a change defensibly

Measure the baseline properly, with enough runs to know the variance.

Change one thing.

Measure again under identical conditions, and compare distributions.

Verify the improvement in the end-to-end metric, not just in the component. A component twice as fast that contributed 3% gives you 1.5%.

Check what got worse. Memory, cost, complexity, and other endpoints. A cache that improves one path and evicts entries another path relied on is a net loss that a single-endpoint measurement will not reveal.

Write down the number. In the commit message or the pull request. "Improved performance" is unverifiable a year later, when someone is deciding whether the resulting complexity is still worth carrying.

Where to look first

In order of expected return:

1. Is it I/O bound or CPU bound? This determines everything else and it takes one look at a profile.

2. How many downstream calls per request? N+1 patterns and unnecessary serial calls.

3. How much data crosses each boundary? Rows, columns, payload size.

4. What is the p99 versus the median? A large gap means variance — contention, garbage collection, a slow path taken occasionally — which is a different investigation from uniform slowness.

5. Only then, the code.

The summary

Set a target, measure end-to-end, profile the real workload.

Most time is spent waiting, so look at I/O, call counts and data volume before algorithms.

Watch for coordinated omission and warm-up — they are why benchmark results and production behaviour disagree.

Compare distributions, not single numbers, and verify the improvement where the user experiences it.

And the discipline that matters most: stop when you hit the target. Performance work has no natural end, and the second week almost never returns what the first did.