Clocks in Distributed Systems: Why You Cannot Trust Timestamps

Ordering events by timestamp is the intuitive approach and it is wrong in ways that produce data loss rather than error messages. The failures are silent, intermittent, and nearly impossible to reproduce.

For a separate people-operations application of the same measurement discipline, see employee time clock software.

The underlying problem is simple to state: there is no shared "now" across machines, and the clocks each machine has are worse than people assume.

The two clocks on every machine

They are different things and confusing them is a common source of bugs.

The wall clock reports time of day. It is what System.currentTimeMillis(), time.time() and Instant.now() give you.

It is synchronised against external sources, which means it can jump — forwards or backwards. A correction can move it back several seconds. It is subject to leap second handling, daylight saving in local representations, and manual changes. It is meaningful across machines and unreliable for measuring intervals.

The monotonic clock counts elapsed time from an arbitrary origin. System.nanoTime(), time.monotonic(), clock_gettime(CLOCK_MONOTONIC).

It never goes backwards and it is not adjusted. It is meaningless across machines and it is the correct choice for measuring durations.

The rule: wall clock for "when did this happen," monotonic for "how long did this take." Measuring a timeout with a wall clock means a clock correction can make an operation appear to take negative time, or make a timeout fire immediately or never.

# wrong
start = time.time()
do_work()
elapsed = time.time() - start   # can be negative

# right
start = time.monotonic()
do_work()
elapsed = time.monotonic() - start

What goes wrong across machines

Even well-synchronised clocks disagree. NTP over a normal network typically keeps machines within some milliseconds of each other; under load, across regions, or with a misbehaving source, the divergence is larger and it is not bounded by anything you control.

Last-write-wins loses data. Two nodes accept concurrent writes to the same key and the one with the higher timestamp survives. If the node with the later write has a clock running behind, its write carries a lower timestamp and is silently discarded. No error, no conflict, just a value that quietly disappears.

Timestamp ordering does not give you causality. Event A caused event B, but B's host has a clock a few milliseconds behind, so B carries an earlier timestamp than A. Any consumer ordering by timestamp sees the effect before the cause.

Time-based pagination skips or duplicates rows. WHERE created_at > $last_seen misses rows that were assigned a timestamp before $last_seen but committed after your query read. Under clock skew across writers it is worse. Use a monotonically increasing identifier or a sequence, not a timestamp.

Expiry windows are not what you think. A token issued on a machine running fast and validated on one running slow is accepted after it should have expired, or rejected before. Clock skew tolerance in token validation exists precisely for this and it is a workaround rather than a solution.

What to use instead

Logical clocks

A Lamport clock is a counter per node. Increment on every local event; when sending a message, include your counter; on receipt, set your counter to max(local, received) + 1.

This gives you a total order consistent with causality: if A happened before B, A's counter is lower. It does not tell you the reverse — a lower counter does not mean A caused B, because concurrent events also get ordered.

Vector clocks carry a counter per node, so comparison distinguishes three cases: A before B, B before A, or A and B concurrent. That third case is the useful one — it identifies genuine conflicts that need resolving rather than silently picking a winner.

The cost is size: a vector grows with the number of nodes, and pruning it correctly is fiddly.

Sequence numbers

For a single writer or a single partition, a monotonic sequence is simpler than any clock and completely reliable. Database sequences, Kafka offsets, and per-partition counters all give you ordering without a clock at all.

If your ordering requirement fits inside one partition, use a sequence and stop reading. Most do.

Hybrid approaches

Hybrid logical clocks combine a physical timestamp with a logical counter. The physical part keeps timestamps close to real time and human-readable; the logical part guarantees monotonicity and causality even when the physical clock misbehaves. Used in several distributed databases and a reasonable default when you need both properties.

Bounded-uncertainty approaches treat time as an interval rather than a point: the system knows time is somewhere within a known error bound, and waits out the uncertainty before committing where ordering matters. This requires tightly controlled clock infrastructure — specialised hardware and a known error bound — and is not something you can retrofit onto ordinary servers.

Conflict resolution that does not silently lose data

If you accept concurrent writes, you need a merge strategy, and last-write-wins is the one that loses data.

Detect concurrency explicitly with vector clocks or version vectors, and surface the conflict.

Application-level merge where the data type allows it. Two edits to different fields of the same record can both be kept.

Conflict-free replicated data types where the semantics fit — counters, sets, registers with defined merge rules. They converge without coordination, and the constraint is that your data has to be expressible as one.

Keep both versions and let a human decide, for cases where automatic merging would be wrong. Unglamorous and correct.

If you do use last-write-wins, understand that you are choosing to discard data under skew, and make sure that is acceptable for the field in question. For a cache it may be. For an account balance it is not.

Practical checks

Search your code for wall-clock arithmetic. Any subtraction of two wall-clock readings is suspect. Timeouts, rate limiters, retry backoff and cache expiry are the usual offenders.

Check what happens when the clock moves backwards. A step correction of a few seconds is a normal event, not an exotic one. If your code assumes time only increases, it has a bug.

Never order events from different machines by timestamp unless you have accepted the failure mode explicitly.

Store timestamps in UTC, with an explicit type, and format for display only at the edge. Storing local time, or a string without an offset, causes a separate and equally durable family of bugs.

Monitor clock offset between your nodes. It is a cheap metric and a rising offset predicts a class of incident that is otherwise very hard to diagnose.

Test with skew. Deliberately offset a node's clock in a test environment and see what breaks. Almost nobody does this, and it finds real bugs immediately.

The summary

Monotonic clock for durations, wall clock for timestamps — and never the reverse.

Timestamps from different machines are not comparable for ordering. Use a sequence within a partition, or logical clocks across them.

Last-write-wins by timestamp discards data under clock skew, silently. If that is not acceptable, detect concurrency instead of resolving it by comparison.

The whole subject compresses into one sentence: time is not a total order, and treating it as one is a decision to lose data occasionally without being told.