What Telstra’s timing glitch tells us about critical infrastructure

I have spent the better part of three decades building clocks. Not the ones on your wall or your wrist, but ultra-precise clocks that are used by radar systems, navigation satellites and telecommunications networks.

When I read that Telstra’s national network had stopped working last week because of a breakdown in network timing, I saw this as a first public exposure of a serious vulnerability that we have been building into the complex networks on which we all depend everyday.

Modern telecommunications and computing networks depend on all nodes across the network sharing an extremely accurate sense of the current time. Not accurate to the minute or the second, but accurate to fractions of a microsecond – even down to billionths of a second (nanoseconds) in some extreme cases.

A cell tower handing a call to the next cell tower, a mobile phone authenticating to the network, a payment terminal talking to its bank: each of these interactions is checked against the time at both ends. If one end of the conversation thinks it is 2026 and the other thinks it is 2006, the security certificates the network uses to trust its own equipment become invalid. The network then does exactly what it was designed to do: it rejects the transaction.

Millions of Australians experienced that during the recent Telstra outage. Underneath, it was the network correctly defending itself against what it believed to be a security threat, generated by its own faulty sense of time.

A good analogy I would offer is that of an orchestra. If the members of the orchestra cannot see the universal timing set by the conductor’s baton, then what we hear is noise and not harmonious music. The individual musicians are all doing their jobs correctly but the slow drift of everyone’s internal clock destroys the beauty of the ensemble. In Telstra’s case, the reference did not drift by a small fraction of a beat — it drifted by nearly twenty years — but the result is just the same, a system that cannot work correctly.

Nothing I have written here is a criticism of Telstra. All operators of communications, computing and other critical infrastructure use comparable architecture and could see a comparable failure on a comparable morning. Telstra’s engineers restored service inside a day, and their executives spoke publicly about the cause while the analysis was still underway. That is the response you want when things go wrong at that scale. The question this event should raise for us is whether we take the class of failure it exposed as a curiosity or a call to action.

This is a story that emphasises how much of Australia’s day-to-day life now depends on assured timing, and how thin the layer that provides it has become.

Most modern operators, in Australia and elsewhere, take their time reference from one of a number of Global Navigation Satellite Systems, of which the USA’s Global Positioning System (GPS) is the most famous.

These GNSS are available almost everywhere, accurate to the nanosecond level and free. These satellite networks have been a spectacular development for our advanced societies transforming how we communicate, navigate and live. Every day they generate $2 billion in economic value, amounting to more than $1 trillion of value to human civilisation in aggregate.

However, the various GNSS are infrastructure that Australia does not own and cannot control. The satellites belong to other countries with their own sovereign priorities, and access could, in principle, be withdrawn at any time. When the GNSS are available and their signals are strong, this arrangement works well. When they are not, and there are more and more reasons for why that might not be, everything that depends on it is exposed. Compared to the US, Europe, China, Russia, Japan and India, we have no sovereign timing infrastructure of our own to fall back on.

As well as the sovereignty problem, major GPS jamming and spoofing incidents have been reported with growing frequency across Europe and the Middle East. A recent discovery has shown short-lived, mysterious and repeated continental-scale blackouts of GNSS signals across the entire European continent. Australian aircraft and ships operating in and near contested airspace are already seeing timing signals they cannot trust.

During the recent outage, the trigger happened to be an internal software update in a data centre. However, the outcome would be the same if external interference had produced a false timing signal on the GNSS signals arising at that centre. From the point of view of the phones, the payment terminals, the train radios and the emergency service switchboards, the two situations look identical. When the reference clock is wrong, everything that trusts the reference is wrong.

The good news is that the engineering to hold time independent of these GNSS networks already exists, and much of it has been built here.

I’ve made it my life’s work to develop precision clocks that can generate precision timing that are independent of GNSS signals. The earliest work led to a sapphire clock that delivered signals 1,000 times more perfect than anything that has gone before.

QuantX has made that device available to Defence’s Jindalee Operational Radar Network where its outstanding performance can deliver an upgrade to performance of the radar itself.

Another of our clock technologies was recently launched aboard a SpaceX Falcon 9 rocket. This is our first step towards a sovereign approach to delivering trusted time across our continent.

QuantX’s and Adelaide University’s atomic clocks have been part of international naval trials where they demonstrated the ability to maintain time when sattelite systems are no longer available.

I am writing to point out that the technical solution to the class of problem that stopped Telstra’s services is not a mystery, and Australia has been quietly building it. What has not yet been built is the procurement logic and the regulatory expectation that would put these tools inside the networks and critical infrastructure that most need them.

Critical infrastructure operators have created detailed plans to cope with interruptions to power and cyber. The brilliance of GPS and its cousins has perhaps created a false assumption that their timing signals will always be available and correct. That assumption held for a long time. It is holding less well every year. These events have highlighted these vulnerabilities and demonstrated a need for Australian priorities to be met by sovereign capabilities.