DNS Is Usually the Problem

The joke exists because DNS sits under everything, caches at five independent layers with different rules, and fails in ways that present as an application problem.

For a separate people-operations application of the same measurement discipline, see how Microsoft Teams tracks activity.

The failures below all look like something else at first, which is exactly why they cost so much time.

The caches, in order

A lookup can be answered at any of these, and each has its own expiry rules.

The application or runtime. The JVM historically cached positive results indefinitely by default in some configurations, which is why a JVM process could keep hammering an IP address that had been decommissioned hours earlier. Most runtimes now respect TTL, and it remains worth verifying for yours.

The connection pool. This is the one people forget. A pooled connection was established to an IP address and it stays connected to that address regardless of what DNS now says. Changing a DNS record does not move existing connections.

The local resolver or system cache.

The recursive resolver your network points at, which honours the TTL but may clamp it.

Authoritative servers, which are the source of truth.

Consequence: the time for a change to take effect is not the TTL. It is the TTL plus every cache along the path, plus the lifetime of existing connections, plus any layer that ignores TTL entirely.

The failures worth recognising

The TTL nobody checked

A record with a 24-hour TTL is changed during a migration. Some clients follow within seconds; others are still resolving the old address the following day.

Lower the TTL before the change, not during it. Reduce it to a minute a day ahead, make the change, wait, then raise it again. Doing this after you have already made the change accomplishes nothing, because the old high TTL is already cached.

Connection pools pinned to a dead address

You fail over a database to a new address. DNS is updated. Applications keep failing, because their pooled connections point at the old IP and the pool only resolves when creating a new connection.

Fixes: a maximum connection lifetime in the pool so connections are recycled and re-resolved; or failing over by moving the address rather than changing the name; or a proxy layer that handles the failover so clients never see it.

Search domains multiplying lookups

In orchestrated environments, a short name is tried against a list of search domains before being tried as written. A configuration with several search domains and a threshold that causes an external name to be treated as relative means one lookup becomes several failed lookups first.

The symptom is latency on the first connection to an external service, appearing intermittently, and it is almost never diagnosed as DNS because the application eventually succeeds.

Fix: use fully qualified names with a trailing dot for external services, or adjust the resolver configuration so external names are not qualified first.

Negative caching

A failed lookup is cached too. If a name does not resolve during a brief outage, the failure is remembered — for a duration controlled by the zone's own settings, not by the record's TTL.

This is why a service can stay broken for minutes after the DNS problem is fixed, and why restarting the process "fixes" it.

One resolver, no redundancy

Every container on a node resolving through a single resolver instance means that instance is a single point of failure and a bottleneck. Under load, lookups time out, and a DNS timeout presents as a connection failure to the application.

Symptom: intermittent connection errors to services that are demonstrably healthy, correlated with load rather than with any particular target.

UDP truncation

DNS responses over UDP have a size limit; larger responses set a truncation flag and the client should retry over TCP. Some middleboxes block DNS over TCP, so the retry fails and the lookup appears to fail intermittently for records with many entries.

Rare, and extremely confusing when it happens.

Diagnosing

Ask the authoritative server directly, bypassing every cache:

dig +trace example.com
dig @ns1.example.com example.com

If the authoritative answer is correct and your application disagrees, the problem is caching, and now you know which side to look at.

Check the TTL on the answer:

dig example.com   # the number before IN A is the remaining TTL

A remaining TTL that keeps counting down and resetting tells you which cache you are hitting.

Resolve from inside the container, not from the host. The resolver configuration differs, and that difference is frequently the answer. See container isolation.

Check /etc/resolv.conf inside the container for search domains and options.

Time the lookup separately from the connection. Most clients report total connection time, which conflates them. If your tracing does not separate DNS resolution as its own span, add it — this alone identifies DNS problems that otherwise take days.

Preventing rather than diagnosing

Set a maximum connection lifetime in every pool. Not just for DNS — it also prevents connections outliving credentials and accumulating server-side state. This is the single most valuable setting in this article.

Lower TTLs before planned changes, well in advance.

Use fully qualified names for external dependencies.

Monitor DNS resolution time as its own metric. It should be sub-millisecond from cache. A rising trend is a leading indicator of an incident that will otherwise present as something unrelated.

Have more than one resolver, and check the failover works rather than assuming it.

Do not put logic in DNS that DNS is bad at. Weighted round-robin for load balancing is coarse, cached unpredictably, and ignores health. A load balancer or service mesh does this properly.

Watch expiry dates. Domain registration and the certificates on the endpoints it resolves to are separate expiries, both of which cause total outages, and both of which are entirely preventable with a calendar reminder.

The summary

DNS caches at five layers with different rules, so propagation time is not the TTL.

Connection pools do not re-resolve. Set a maximum connection lifetime and most DNS-related failover problems disappear.

Lower the TTL before the change, not after.

Instrument resolution time separately, because otherwise DNS failures present as connection failures and you look in the wrong place for a day.