Available for day contractsFrom 21st September I have availability for day and half day contracts. Please contact for more information.

Contact →
mikepreston.org

What Time Is It, Really?

A montage of various time related concepts and terms, including NTP, UTC, TAI, UT1, Zonefiles, a timezone map... and the obligatory analogue clock face.

It is one of the first questions a child learns to ask. It is one of the first things a computer needs to know. And it turns out that for computers — particularly for computers that need to agree with each other — it is a question with no clean answer.

Time feels like a solved problem. Your phone displays the correct time. Your laptop agrees with it. The timestamp on the email you just received matches your clock. Everything is synchronised, everything is consistent, and the whole apparatus hums along invisibly.

Until it doesn’t. Until a leap second brings down Reddit, LinkedIn, and the Linux kernel simultaneously. Until a one-line error in a time subtraction function causes Cloudflare’s DNS to stop resolving. Until someone deploys a distributed system that assumes clocks on different machines agree, and discovers at 2am that they don’t.

Time, it turns out, is a physical problem dressed up as a solved one. And the deeper you pull at the thread, the stranger it gets — all the way out to the Moon.

Three Kinds of Time

The first thing to establish is that “what time is it” doesn’t have a single answer. There are at least three different things the question might mean, and they don’t quite agree with each other.

UT1 (Universal Time 1) is astronomical time — time based on the Earth’s rotation relative to distant stars. It’s the closest thing to the intuitive notion of a day: the Earth rotates, the sun rises, it’s morning. The problem is that the Earth’s rotation is not constant. It’s slowing down, on average, due to tidal braking from the Moon. It speeds up and slows down irregularly due to atmospheric and oceanic effects, movements of molten rock in the mantle, and interactions with the Moon and Sun. UT1 is the time the planet is actually keeping, and it wanders.

TAI (International Atomic Time) is what you get when you average the output of around 450 atomic clocks distributed across more than 80 laboratories worldwide, all ticking at the frequency of caesium-133 transitions in a vacuum. It is extraordinarily stable — atomic clocks are accurate to within a second over millions of years — and entirely indifferent to what the Earth is doing. TAI doesn’t drift. It just counts SI seconds, relentlessly, without any reference to the sky.

UTC (Coordinated Universal Time) is the compromise between them. It uses TAI’s atomic precision but attempts to stay within 0.9 seconds of UT1 — close enough that noon UTC is still roughly when the sun is at its highest. The mechanism for keeping them aligned is the leap second: an occasional extra second inserted into UTC when UT1 is drifting too far behind. The International Earth Rotation and Reference Systems Service (IERS) monitors the gap and announces leap seconds roughly six months in advance, when they’re needed. Since 1972, 27 have been inserted as of 2026, the last in 2016.

This sounds reasonable in theory. In practice, it is the source of a remarkable amount of software grief.


The Leap Second Problem

Computers don’t like discontinuities. A great deal of software is built on the assumption that time moves forward monotonically — that the clock at moment B is always later than the clock at moment A. A leap second inserts an extra second into the sequence: 23:59:58, 23:59:59, 23:59:60, 00:00:00. The second labelled :60 doesn’t exist in normal time. Most systems have no idea what to do with it.

The responses have included: crashing, hanging, spinning in a busy wait loop consuming 100% CPU, corrupting timestamps in databases, and triggering cascading failures across distributed systems that disagreed about what time it was.

The leap second of 30 June 2012 is the most instructive example. When the second was inserted, several things happened at once. The Linux kernel had a bug in its high-resolution timer code that caused processes to enter a busy-wait loop on systems using the HRTIMER_NOHZ mode. Any process that called clock_gettime() during the leap second was affected. The result was systems running at 100% CPU, entirely unresponsive, for extended periods.

Reddit went down. LinkedIn went down. Foursquare went down. A significant chunk of the web running on Linux with the affected kernel version went down. The fix, in most cases, was a reboot — or, if you knew what you were doing, date -s "$(date)" to force the kernel to reinitialise the time subsystem.

Mozilla, Qantas, and various financial systems also reported issues. The total impact was substantial, and the cause was a single extra second that the software was not prepared to handle.

Smearing: The Distributed Systems Bodge

The industry’s response to the leap second problem, rather than fixing the software to handle discontinuities correctly, was largely to hide the discontinuity from the software.

Leap second smearing spreads the adjustment across a window of time — typically 24 hours — by running the clock very slightly slow (or fast) so that the cumulative effect absorbs the extra second without any visible jump. From the software’s perspective, time progresses monotonically. The leap second simply doesn’t appear.

Google implemented smearing for their internal infrastructure after painful experience with leap seconds. They published their approach, and it was widely adopted. AWS implemented their own variant. The NTP pool eventually added smearing support.

The problem is that different implementations smear differently. Google’s original approach used a linear smear over 20 hours. AWS uses a linear smear over 24 hours. Some systems don’t smear at all. During the smear window, different machines have genuinely different opinions about what time it is — by design — and those differences can be up to half a second.

For most applications, this is imperceptible. For applications that need to compare timestamps across systems — financial trading, distributed databases, anything with strong consistency requirements — a half-second of deliberate clock skew is not nothing.

The Cloudflare incident of 1 January 2017 is the canonical smearing cautionary tale. Cloudflare’s RRDNS implementation computed the time elapsed between two DNS queries by subtracting timestamps. Due to a leap second interaction, this subtraction occasionally produced a negative number. The code passed that negative number to a Golang time function that interpreted it as zero, causing certain DNS responses to be dropped. The result was a measurable increase in DNS resolution failures across Cloudflare’s network for a period around midnight UTC.

The bug wasn’t in the leap second handling directly — it was in a subtraction that assumed time differences would always be non-negative. Smearing would have prevented it. The smearing they were using didn’t cover the edge case that bit them.

The Abolition

In November 2022, the General Conference on Weights and Measures voted to abolish leap seconds by 2035. The accumulated difference between UTC and UT1 will be allowed to grow until it reaches a tolerance to be agreed — currently proposed as one minute — at which point a larger correction mechanism will be introduced.

The people who operate infrastructure were largely relieved. The astronomers and navigators who care about UTC tracking the actual position of the Earth were less happy. The compromise is that the larger correction, when it eventually becomes necessary, will be a leap minute — inserted far enough in advance, with enough warning, that software can be updated to handle it — in a century or two. The engineers have kicked the can down the road, effectively saying: we’ll deal with the big one when it comes; we’re done dealing with the small ones every few years.

The abolition doesn’t make leap seconds go away immediately. Systems will need to handle them until 2035, and legacy systems long after that. The 27 leap seconds already inserted are baked into the TAI-UTC offset. TAI will continue to be ahead of UTC by exactly 37 seconds (the current difference from when UTC was actually defined) until the next adjustment mechanism is defined.

Where Time Actually Comes From

Software clocks are terrible. They drift. A typical server CPU oscillator might gain or lose several seconds per day without correction. Network Time Protocol (NTP) is the mechanism that keeps them honest.

NTP works in a hierarchy called strata. At stratum 0 are the primary reference clocks — the actual hardware time sources. At stratum 1 are servers directly connected to those references. Stratum 2 servers synchronise from stratum 1, and so on. Your laptop is probably stratum 3 or 4, keeping time via a pool of public NTP servers that synchronise from GPS-disciplined reference clocks.

The interesting question is what’s at stratum 0. There are a few options.

GPS PPS (Pulse Per Second) is one of the most accurate freely available time sources on the planet. GPS satellites carry atomic clocks. The GPS signal encodes the current time. And crucially, GPS receivers can output a hardware pulse — one pulse per second, precisely aligned to the GPS epoch — that has sub-microsecond accuracy relative to UTC.

Cheap GPS modules with PPS output are available for a few pounds. A Raspberry Pi with a PPS-capable GPIO pin and the gpsd and chrony packages makes a perfectly respectable stratum 1 NTP server. This is a small but dedicated hobby community, and the results are surprisingly good — I myself have two.

There’s a wrinkle worth knowing: GPS time is not UTC. The GPS system doesn’t have leap seconds — it’s been running continuously since 1980 and has simply counted seconds since then. As of today, GPS time is 18 seconds ahead of UTC. The GPS receiver knows this because the satellites broadcast the current UTC-GPS offset in their signal. Your receiver’s firmware has a table of leap second corrections and applies them.

That table has a finite size and a finite update frequency, which is why some older GPS receivers started misbehaving after the GPS week number rolled over in 2019 — similar to Y2K, but for GPS. Receivers that hadn’t been updated couldn’t correctly interpret dates in the new epoch.

MSF is the UK’s longwave radio time signal, operated by the National Physical Laboratory (NPL). It transmits on 60kHz from Anthorn in Cumbria — not Rugby, despite the name “Rugby clock” persisting in common usage; the transmitter moved in 2007 and the name didn’t follow. The signal covers the UK and parts of northern Europe. Amplitude-modulated time codes encode the current time and date, readable by inexpensive radio receiver modules. Most radio-controlled clocks and watches, such as my Casio Pathfinder, in the UK use MSF.

The BBC’s six pips — the time signal broadcast before the news — were historically derived from NPL time standards and are still used as a cultural reference, though their broadcast accuracy via analogue radio has never been as precise as the name implies.

DCF77 is the continental European equivalent, transmitting from Mainflingen near Frankfurt on 77.5kHz. Stronger signal, wider coverage, similar modulation scheme.

PTP (Precision Time Protocol, IEEE 1588) is for applications where NTP’s microsecond-level accuracy isn’t sufficient. Financial trading systems, 5G base stations, power grid synchronisation, scientific instrumentation. PTP achieves sub-microsecond accuracy by using hardware timestamping in network interface cards rather than relying on the operating system scheduler, which introduces unpredictable jitter. The OS might add tens of milliseconds of uncertainty to a software timestamp; a hardware timestamp in the NIC captures the packet’s arrival time with nanosecond precision.

Building Your Own Stratum 1

This is an aside for the practically minded. A GPS PPS stratum 1 NTP server can be built for under £30:

A Raspberry Pi (any model with GPIO), although a Pi Zero doesn’t have physical ethernet, so is less useful.A GPS module with PPS output (u-blox NEO series modules are popular; available for £10-15, or less from your favourite Chinese market, but be aware of fakes)gpsd for GPS daemonchrony for NTP (preferred over ntpd on modern systems)A few lines of configuration to tell chrony about the PPS source

The result is a stratum 1 server accurate to within a few hundred nanoseconds of UTC, which is better than most commercial NTP servers you’d synchronise from. It’s a satisfying project and a genuine piece of infrastructure. It also makes an excellent illustration of the whole stack: GPS satellites carrying atomic clocks, a radio signal carrying time to a £10 module, a pulse-per-second hardware interrupt disciplining a software clock, which then serves NTP to your local network.

Timezones: A Brief History of Human Suffering

Before the railways, time was local. Every town kept its own time, set by the sun. Bristol was ten minutes behind London. Nobody particularly cared because nobody was moving fast enough for it to matter.

Railways made it matter. A train timetable covering a journey from London to Exeter needed a shared reference, and having every station on its own local solar time made timetables unworkable. Railway companies began standardising on a single time — typically the time at their headquarters or at Greenwich — and by 1880 Greenwich Mean Time (GMT) had become the legal time standard for Great Britain.

The international standardisation followed, unevenly, over the following decades. The result is the modern timezone system: a political map overlaid on the mathematical reality of the Earth’s rotation, with all the compromises and anomalies that political maps always contain.

China officially runs on a single timezone — UTC+8 — despite spanning roughly the same longitude range as Europe, which has five. Western China (Xinjiang, Tibet) observes what would geographically be UTC+5 or UTC+6. The unified timezone is a political statement as much as a practical one, and in Xinjiang the actual workday is shifted substantially later relative to the clock to accommodate the solar reality.

North Korea briefly ran on UTC+8:30, half an hour behind South Korea, introduced in 2015 to mark the 70th anniversary of liberation from Japan and explicitly described as moving away from Japanese Standard Time. They reverted to UTC+9 in 2018 as part of the diplomatic thaw with South Korea before the Singapore summit. A timezone change in service of geopolitics, then reversed in service of different geopolitics.

India runs on UTC+5:30, a half-hour offset chosen to be a compromise between the eastern and western extents of the country. Nepal runs on UTC+5:45, a quarter-hour offset from India. The Nepal offset is frequently described as existing specifically to be distinct from India, though the official explanation references the longitude of the meridian of **Gaurishankar, **a mountain peak 100km east of Kathmandu. The quarter-hour offset gives Nepal its own identity, but does add complications with many digital systems only able to deal with offsets to the nearest half hour.

Samoa skipped an entire day on 29 December 2011, jumping from UTC-11 to UTC+13 by simply not having a 29 December. The change moved Samoa to the same side of the international date line as Australia and New Zealand — its main trading partners — so business weeks would align. From the perspective of anyone in Samoa on Wednesday 28 December, Friday 30 December followed immediately. Thursday 29 December 2011 did not exist in Samoa.

Daylight Saving Time deserves its own article, or possibly its own support group. The twice-yearly clock change is a distributed systems event that breaks: scheduled jobs that run at the missing or duplicated hour, log files that have ambiguous timestamps, calendar events that move relative to wall clock time but not relative to UTC, meeting schedulers across timezones that disagree about when the change happens (different countries (or even states) change on different dates, and some don’t change at all). The amount of engineering time spent handling DST edge cases across the software industry is genuinely difficult to estimate but certainly enormous. Its continued existence is an ongoing mystery, only explained by international politics.

The IANA Timezone Database

The canonical reference for timezone rules worldwide is the IANA timezone database (also called the Olson database, after its original maintainer Arthur David Olson). It’s a set of files distributed with every Unix-like operating system and many other environments, encoding every current and historical timezone rule with their UTC offsets, DST transitions, and historical changes.

The database is maintained by a small group of volunteers on a mailing list. Its importance to global software infrastructure is difficult to overstate — every time a country changes its timezone or DST rules, an update to the IANA database has to follow for software to handle it correctly. Geopolitical disputes occasionally surface on the mailing list, as maintainers debate how to name timezones for disputed territories.

The files themselves are worth examining if you haven’t. The amount of historical complexity they encode — timezone changes during wartime, the Soviet Union’s various experiments with year-round DST, the United States’ mid-20th century timezone chaos before the Uniform Time Act — is a compressed history of the 20th century’s relationship with clock-setting.

The posix/ and right/ Problem

Most systems install the IANA timezone database in one of two flavours, and the difference is subtle enough that most engineers never notice it — until it matters.

posix/ timezones, which is what almost everything uses, treat Unix time as a count of SI seconds since the epoch (1 January 1970 00:00:00 UTC), with leap seconds silently omitted. When you ask for the Unix timestamp of a moment in time, you get a number that’s been computed as if leap seconds never happened. This is consistent, predictable, and wrong in a strict physical sense — but it’s what every programming language, every database, and every piece of infrastructure expects.

right/ timezones include leap seconds. Unix time in a right/ system is the true elapsed SI second count since the epoch. The current difference is 27 seconds — right/UTC is 27 seconds ahead of posix/UTC, because 27 leap seconds have been inserted since 1972.

The problem arrives when you mix systems. A timestamp generated on a right/ system compared to a timestamp generated on a posix/ system will show a 27-second discrepancy, even though both claim to be in UTC. This has caused real incidents in scientific computing and financial infrastructure where someone used right/ timezones for accuracy — the SI second count is genuinely more physically correct — and then compared timestamps with systems using posix/.

The practical advice is always to use posix/ unless you have a specific reason not to, document it if you do use right/, and never mix the two in the same system without explicit conversion.

What Time Is It on the Moon?

In April 2024, the White House directed NASA to establish a time standard for the Moon by the end of 2026. This sounds like a bureaucratic curiosity. It is actually a genuinely hard problem, and the solution involves revisiting every argument the timekeeping community has had for the last century.

The immediate practical motivation: as lunar activity increases — NASA’s Artemis programme, Chinese missions targeting 2030, commercial operators, multiple nations with overlapping activities — you need a shared time reference. Currently, lunar missions use the timezone of their mission control. Apollo used Central Time because Houston did. Chinese missions use China Standard Time. As soon as two missions from different countries need to coordinate on the surface, this stops working.

But there’s a deeper problem. Einstein’s general theory of relativity predicts that clocks run at different rates in different gravitational fields. A clock at the Moon’s surface, in a weaker gravitational field than Earth’s, ticks faster. The rate difference is approximately 56.02 microseconds per Earth day.

That sounds small. It accumulates. Since the J2000 epoch (1 January 2000), lunar clocks have already drifted ahead of Earth clocks by over half a second. For navigation — landing a spacecraft, docking, positioning — half a second of timing error translates to significant positional uncertainty, especially when you consider the high velocities often involved. For precision science, it’s not negligible. For a long-term lunar infrastructure with GPS-equivalent positioning, it’s a fundamental problem that has to be addressed in the time standard, not corrected for in every application separately.

In December 2025, researchers at the Purple Mountain Observatory in Nanjing published software to calculate Coordinated Lunar Time (LTC) accurate to 0.15 nanoseconds out to 2050, which suggests the mathematical framework is converging even if the political and organisational framework is still being assembled.

The solution — Coordinated Lunar Time — is being designed on the same principles as UTC but adapted for the lunar environment. A weighted average of atomic clocks on the lunar surface and in orbit, with a fixed, well-defined relationship to UTC that accounts for the relativistic offset. Crucially, it will not have leap seconds in the traditional sense. There’s no lunar rotation to track — the Moon’s rotation is tidally locked to its orbit, and “lunar day” means something entirely different from Earth day. The continuous relativistic drift is predictable and can be corrected mathematically rather than with discrete steps.

Which brings us back, neatly, to the posix/ versus right/ argument. LTC is essentially the question of how you build a time standard when the “astronomical time” analogue doesn’t map cleanly onto the physical environment — the UTC/TAI tension, made relativistic and extraterrestrial.

The same framework is expected to scale to Mars, where the relativistic offset and the length of the Martian day both complicate the picture further. We are, very slowly, building an interplanetary timekeeping system. The arguments being had now about LTC are the arguments that will shape how time works for human civilisation beyond Earth.

Why This Matters for Systems

Bringing it back to the practical: distributed systems are fundamentally dependent on time, and time is fundamentally unreliable.

Clock skew between machines causes: distributed database consistency failures, incorrect log correlation during incident investigation, authentication token validation errors (tokens that appear expired before they should, or valid after they shouldn’t be), cache invalidation bugs, and subtle ordering failures in event-driven systems.

The tools for dealing with this at the infrastructure level are: NTP with good upstream sources (use a local stratum 1 if you need accuracy; the public pool is fine for most purposes), monitoring clock offset as a first-class metric, using logical clocks (Lamport clocks, vector clocks) for ordering in systems that need it, and designing systems to be tolerant of bounded clock skew rather than assuming perfect synchronisation.

The tools for dealing with it at the code level are: always store and compare timestamps in UTC, never in local time; use monotonic clocks for measuring elapsed time (not wall clock time, which can jump); be explicit about timezone handling at every boundary; test with leap seconds if your software will be running around midnight on 31 December or 30 June. And if you’re writing software that will run on the Moon: you have a few years, but you should probably start thinking about it.

Time feels like a solved problem. It is, at best, a problem that’s been managed well enough that most people never notice the places where the management is held together with string. Pull on the string carefully.

Just… Leap seconds are typically added on the 30th of June or 31st of December. The last was 31st December 2016.

Moon escape velocity is approximately 2.4km/sec. At half a second error, you can be over 1km out…

Saturn and Jupiter with their deep gravity well, high rotational speeds, and many, many moons are going to be interesting challenges.

TLS for example will completely break on most systems if the clock skew is more than 300 seconds. This can complicate recovery on systems with bad CMOS batteries, or no clock at all.

The trauma is real, I have seen so many bugs with this specific issue it is not at all funny anymore. Use unix timestamps as a base type and derive what you need for display.

Modern languages aren't always safe here either. Python 3 supports timezone-aware datetime objects, but naive datetimes remain the default — datetime.now() returns local time with no timezone attached, and datetime.utcnow() returns UTC as a naive object, a distinction that's invisible until it isn't. Libraries like SQLite compound this: SQLite has no native datetime type, and SQLAlchemy's SQLite dialect returns naive datetimes by default. Your IDE's type checker may catch aware/naive mixing; your runtime will raise TypeError if you try to compare them directly — but only if you're comparing across the boundary. Two naive datetimes in different implicit timezones will compare silently and incorrectly.