NTP and time sync: the internet's heartbeat

When two machines both say “it is 12:00:00”, they are not saying the same thing. Distributed systems sort logs by time, expire caches by time, validate TLS certificates by time and order financial trades by time — all of it resting on the assumption that everyone agrees. And time is one of the things computers are worst at.

Image: ESO, CC BY 4.0, via Wikimedia Commons.

How wrong a quartz clock is

The crystal on your motherboard keeps time through the piezoelectric resonance of quartz, and it costs pennies. Its frequency drifts with temperature: an ordinary crystal can be off by 20 ppm across the operating range of a machine — 20 microseconds per second, or 1.7 seconds per day. A minute a month.

Image: Glosome, CC BY-SA 4.0, via Wikimedia Commons.

A caesium clock is a different category. The second is defined as 9,192,631,770 oscillations of the caesium-133 transition, so that frequency does not drift by definition. The catch is that you cannot put one in a laptop — a 1950s laboratory caesium clock filled a cabinet.

Image: Steve Jurvetson, CC BY 2.0, via Wikimedia Commons.

So the practical arrangement is: let the cheap crystal keep time, and let the expensive atomic clock correct it.

Four timestamps, one offset

NTP (Network Time Protocol) runs on UDP port 123, and its core is four numbers:

Symbol	Meaning
T1	client local time when the request is sent
T2	server time when the request is received
T3	server time when the reply is sent
T4	client local time when the reply arrives

Assuming the path is symmetric, the round trip is (T4 − T1) − (T3 − T2), and how far ahead the server is relative to the client is:

offset = ((T2 − T1) + (T3 − T4)) / 2

Add that offset to your clock and you are synchronised. Note that it computes a difference, never an absolute — NTP never asks what time it is, only whether you agree with everyone else.

One exchange only gives you the difference at that instant. So an NTP client does not sync once and leave; it samples continuously and fits a crystal’s frequency error and drift rate statistically, disciplining the clock into a smooth curve instead of yanking it around. This is also why ntpd rejects wildly outlying samples — it is protecting a curve that has already converged.

Image: Kristian Mollenborg, CC BY 2.0, via Wikimedia Commons.

Stratum is distance, not accuracy

NTP’s hierarchy is called stratum:

• stratum 0: the actual reference clocks — atomic clocks, GPS receivers. They do not serve time to the public;
• stratum 1: servers directly attached to a stratum 0 source;
• stratum 2: servers taking time from stratum 1, and so on up to 15; 16 means unusable.

The common misunderstanding is that a lower stratum number means better accuracy. It only means fewer hops from a reference clock. An overloaded, congested stratum 1 server can serve worse time than a healthy stratum 3.

Public NTP typically lands within a few milliseconds to a few tens of milliseconds; inside one facility, sub-millisecond. To go below a microsecond you switch to PTP (IEEE 1588), whose key difference is hardware timestamping in the network card, sidestepping the tens of microseconds of indeterminate delay inside the operating system.

Leap seconds: a patch that lasted fifty years

Atomic clocks tick with extreme regularity; the Earth’s rotation does not. To keep civil time aligned with the Sun’s position, we have inserted leap seconds since 1972 — 27 of them up to 2016.

The trouble is that a second cannot be inserted gracefully. A timestamp that goes backwards by one second can mean:

• log times run backwards and sorting breaks;
• a scheduled job fires twice for the same “instant”;
• cache entries with a TTL never expire.

At the 2012 leap second, Reddit, Mozilla, LinkedIn and others had outages. The industry invented workarounds: Google smears the second across 24 hours, Amazon uses its own time source.

In 2022 the CGPM decided to stop inserting leap seconds by 2035. At some point after that, UTC and UT1 will drift about a minute apart — and someone will have to come up with something else.

What happens when time is poisoned

Time sync is a chain of trust, and it is routinely ignored:

• NTP amplification: the old monlist command let a few dozen bytes of request return hundreds of times the response. In 2013–2014 it was used for DDoS attacks in the hundreds of Gbps.
• Forged time sources: if an attacker can answer your NTP requests, they can shift your clock. Certificate validity, token expiry and TOTP all break with it.
• The dependency is itself a single point: configuring one upstream server means handing time to that one machine.

The sane configuration is four or more unrelated upstreams — which is exactly what pool.ntp.org is for — letting NTP’s voting algorithm discard the outliers.

My take

Time sync is the classic thing that is invisible until it breaks, and then takes half a day to diagnose. What I do:

1. Every server gets four or more upstreams from different operators; never a single hard-coded host.
2. Alert on the offset to upstream, with a threshold around 100 ms. That is harmless for most workloads, but crossing it almost always means an upstream or network problem.
3. Where I need strict cross-machine ordering, I do not rely on the physical clock — I use logical clocks or version numbers.

A physical clock is only ever an approximation. Treating it as exact is the real bug.