Tests That Catch Regressions and Tests That Catch Typos

A test suite can have high coverage, run in ninety seconds, be entirely green, and have caught none of your last five production incidents.

For a separate people-operations perspective, this note covers how artificial activity patterns can be recognised.

That is not a paradox. Coverage measures which lines executed during the test run. It says nothing about whether anything meaningful was asserted, and nothing at all about the failure modes that actually break systems.

The question worth asking about a suite

Which of the last five incidents would this suite have caught?

Go and check. The answer is usually one or none, and it identifies the gap more precisely than any coverage report.

The incidents were probably: a schema migration that locked a table, a downstream dependency slowing down, a configuration difference between environments, a concurrency bug under load, an unhandled null in a field that is populated in production and empty in fixtures.

None of those are caught by unit tests, however many there are.

What each level actually verifies

Unit tests verify that a function does what its author thought. They catch typos, off-by-one errors, and regressions in logic when someone changes it later.

They are fast, they are cheap, and they cannot catch anything involving two components, real data shapes, concurrency, or the network. Which is where production failures come from.

Their real value is as a design signal: code that is hard to unit test is usually badly coupled, and the test difficulty is telling you something.

Integration tests verify that components work together — your code against a real database, a real queue, a real HTTP call to a stubbed service. These catch the mismatches that unit tests mock away: the query that is invalid SQL, the migration that does not apply, the serialisation format that does not round-trip.

This is where most of the return is, and it is where most suites are thinnest, because they are slower and more awkward to write.

Contract tests verify that a producer and a consumer agree about a message or an API. They catch the case where one team changes a field and another team breaks, without needing both systems running in one environment.

End-to-end tests verify that a user-visible flow works. Slow, flaky, expensive to maintain. A small number covering the paths that must never break is worth having; a large suite of them is a tax that people eventually route around.

Property-based tests generate inputs and assert invariants. They find edge cases nobody thought to write: empty collections, unicode, boundary numbers, orderings. For parsers, serialisers and anything with a round-trip property, they consistently find bugs that example-based tests do not.

The tests that would have caught the outage

Given the failure modes above, the high-return additions:

Run every operation twice. In integration tests, send each request or message a second time and assert it is a no-op. This finds idempotency bugs, which are otherwise found in production. See every request will be retried.

Test against production-scale data at least somewhere. Query plans change with table size. A migration that is instant on a thousand rows locks a table on ten million. This does not need to be in the fast suite — a nightly run against a restored production-shaped dataset catches an entire category.

Test with a slow dependency, not just a failing one. Inject latency and see what your timeouts, retries and pools do. Slow is harder than down and almost nobody tests it. See timeouts and retries.

Test concurrency deliberately. Two transactions with a chosen interleaving, asserting the invariant holds. One such test is worth a hundred sequential ones for finding isolation bugs.

Test the migration, forward and with old code running. A migration test that applies the change and then runs the previous version's queries against it catches the deploy-window breakage that expand-contract is meant to prevent.

Test configuration parsing and validation. Configuration errors cause a substantial share of incidents and are almost never tested. A test that loads each environment's config and asserts required fields exist and are well-formed is cheap.

Assertions, not execution

A test that calls a function and asserts nothing still counts toward coverage.

Assert on behaviour, not on implementation. A test asserting that a specific method was called on a mock breaks when the implementation changes and passes when the behaviour breaks. It is coupled to the wrong thing.

Assert the negative too. That an invalid input is rejected, that an unauthorised user gets nothing, that a duplicate does not create a second row.

A useful check: mutate the code and see if a test fails. Change a > to a >=, invert a condition, return a constant. If the suite stays green, that code is covered and unverified. Mutation testing tools automate this and are worth running once even if you do not adopt them permanently — the first run is usually sobering.

Flakiness is a correctness signal

A test that fails one run in fifty is usually telling you something true.

Common causes, in order: dependence on timing, dependence on ordering, shared state between tests, real clocks, real network, and genuine concurrency bugs in the code under test.

The last one is the reason not to just retry. Automatic retries on failure convert a real intermittent bug into a green build, and the bug ships.

Quarantine rather than retry. Move the flaky test out of the blocking suite, keep it running visibly, and fix it. A quarantine that never empties is its own signal.

A suite people do not trust is worse than a smaller one they do, because the response to a red build becomes "run it again" instead of "look at it."

Speed, and where it comes from

The suite has to be fast enough to run before every merge, or it stops gating anything.

Most slowness is setup, not assertions. Creating a database per test, starting containers per test, loading the full application context.

Share expensive setup across tests, isolate with transactions. Start the database once, run each test in a transaction, roll back. Orders of magnitude faster than recreating state, and the isolation is real.

Split by speed, not by type. A fast suite on every commit, a slower suite on merge, the slowest nightly. The category boundaries matter less than the time budget.

Parallelise, which requires tests not to share mutable state — and the work of making that true is worth doing anyway, because shared state is also the main cause of flakiness.

What not to bother with

A coverage target. Coverage is a diagnostic, not a goal. A mandated 80% produces tests written to reach 80%, which are the least valuable tests it is possible to write.

Unit tests for trivial code. A getter does not need a test. Neither does a function that only calls another function.

Mocking everything. A test where every dependency is a mock verifies that your mocks agree with each other.

Large end-to-end suites. Slow, flaky, and the maintenance eventually exceeds the value. Keep a few critical paths.

The summary

Coverage tells you which lines ran. It does not tell you what would be caught.

The useful question is which recent incidents the suite would have caught, and the answer usually points at integration, concurrency, scale and configuration rather than at more unit tests.

Send every request twice, test with a slow dependency, test at realistic data volume — three additions that between them cover a large share of what actually breaks.

Take flakiness seriously, because retrying is how a real bug becomes a green build.