AWS re:Invent 2025 - It's About Time! Improving Distributed Systems with Amazon Time Sync (CMP409)

The Importance and Fundamentals of Clocks [00:04]

• Clocks are essential for building reliable distributed systems, not just for human observation (logs, metrics) but also for improving system performance by reducing communication and simplifying logic.
• Historically, distributed systems have not been designed to trust clocks, leading to complexity in areas like leader election and distributed locking.
• A fundamental computer clock can be built with a ticking oscillator and a counter, where time is calculated by cycles divided by frequency plus a constant.
• The accuracy of hardware clocks varies significantly, from small crystal oscillators to large, expensive atomic and optical clocks, with trade-offs in form factor, price, and accuracy.
• Even the most accurate clocks drift over time due to factors like manufacturing, aging, power supply variations, and temperature.

"Synchronized clocks are interesting because they can be used to improve the performance of a distributed system by reducing communication."

The Challenges of Clock Synchronization in Distributed Systems [08:39]

• Network variability is a primary challenge in clock synchronization; the fluctuating delays in communication make it difficult to distinguish actual clock drift from network noise.
• The network's asymmetry, where sending and receiving times can differ, creates ambiguity for a client trying to synchronize its clock, making it hard to differentiate clock error from network latency.
• To address these challenges, AWS developed the Amazon Time Sync service, initially as a link-local NTP endpoint for EC2 instances, offering sub-millisecond accuracy.
• The pursuit of nanosecond accuracy led AWS to invest in hardware solutions, embedding dedicated cables from GPS and redundant atomic hardware clocks directly into the infrastructure, bypassing the standard network.

"The network is part of the problem."

AWS Nitro System and Precision Time [13:02]

• AWS has integrated hardware-based time synchronization into its Nitro system to achieve nanosecond accuracy, bypassing traditional network paths for clock signals.
• This infrastructure includes dedicated cables from GPS and redundant atomic hardware clocks, delivering a reference signal directly to EC2 instances.
• The Nitro system features a hardware reference clock and supports both NTP and PTP (Precision Time Protocol) hardware clocks (PHC) directly within the instance, eliminating network communication for synchronization.
• This hardware-based approach provides sub-100 microsecond accuracy for NTP and around 20 microsecond accuracy for PTP at the software layer within instances, built across various instance types (Graviton, Intel, AMD, GPU).

"We really took the network out of network time synchronization and everything's hardware based and Nitro based."

Hardware Packet Timestamping for Enhanced Observability [16:40]

• Hardware packet timestamping, a feature launched by AWS, allows for capturing packet timestamps closer to the network wire, providing more accurate network performance insights.
• This feature, built into the ENA driver, offers nanosecond resolution on receive timestamps for network packets.
• It can be accessed via standard socket APIs or the DPDK toolkit, enabling applications to gain a more precise view of network latency and performance.
• Comparing user-space timestamping with hardware timestamping reveals a significant difference in roundtrip time measurement, highlighting the accuracy gained by the latter, which reduces the time spent within the operating system's network stack.

"If you can time some packet closer to the wire, you get a more accurate view about the network performance, which is very important if you are running a distributed workload."

NASDAQ's Use of Precision Time in Financial Markets [21:47]

• NASDAQ, a pioneer in electronic exchanges, is leveraging AWS's precision time synchronization for its market operations, aiming to run markets in the public cloud.
• The critical requirements for financial exchanges include ultra-low latency transactions (around 20 microseconds order-to-act), high resiliency, and regulatory oversight, handling billions of messages daily.
• To ensure deterministic and statistically fair markets, NASDAQ relies on physical characteristics of on-premise data centers for latency and network path predictability.
• In the cloud, with variable network paths, NASDAQ uses hardware timestamps to record transaction times at each point, enabling reordering of inbound transactions to maintain sequential processing and fairness, a technique for which they hold a patent.

"So, we put all these together, and I think you start to see the picture of why we've been very intentional over the past 10 to 15 years, continuing to move progressively to more and more real time and into the match."

The clockbound Software and Uncertainty Windows [44:00]

• Estimating clock error is crucial, and AWS introduced clockbound, an open-source software daemon, to simplify this by providing current time, the size of the uncertainty window (e.g., +/- 20 microseconds), and the clock's status (initialized, free running, synchronized) in a single operation.
• This addresses the complexity for customers in understanding and calculating clock error bounds due to oscillator drift and communication delays.
• clockbound supports PTP and NTP sources, scales to demanding workloads, and is built in Rust for performance and memory safety.
• The software provides an intuitive view of the uncertainty window, which grows linearly as the clock drifts and shrinks when updated, allowing applications to use local time with confidence.

"The tightest bound on the worst case clock error that is due to either oscillator drift and communication delays."

Innovations in Cloud-Native Time Synchronization and New AWS Time Daemon [55:57]

• AWS has released a new AWS Time Daemon, integrated into the clockbound software, offering a standalone time client and daemon that synchronizes the operating system clock and provides application-level time information.
• This new daemon is cloud-first designed, moving from historical feedback loop systems to a more cloud-friendly feed-forward approach for time synchronization.
• It incorporates built-in metrics and observability as first-class citizens, leveraging a shared memory segment for high-rate interprocess communication.
• The system achieves high accuracy, keeping workloads running smoothly with a window of uncertainty in the 20-26 microsecond range within applications, demonstrating significant improvements in accuracy and reliability.

"We've changed that around. We now have a system that feeds forward information the system which is much more friendly to cloud application."