Achieving Sub-ns Synchronization in Ethernet Networks

Time-sensitive networks (TSN) in Ethernet are improving computer-based measurements from sub-microsecond to sub-nanosecond precision. By using message-based protocols that synchronize clocks, it is possible to generate this required precision signaling locally. This solution is scalable from within the same node to multiple nodes throughout the world. To fulfill the more demanding needs of test and measurement applications, IEEE 1588 has been developed, which is able to provide sub-microsecond performance. 

Many of the research activities concerning IEEE 1588 have been targeted at Ethernet. Already, CERN is able to achieve sub-ns synchronization in the timing, trigger, and control system for the Large Hadron Collider, the biggest machine in the world based on Ethernet. This is an example of what TSN and standards such as IEEE 1588 and IEEE 802.1 will be bringing to COTS Ethernet. 

Time-Sensitive Networks

Measurement and automation systems involving multiple devices often require accurate timing to facilitate event synchronization and data correlation. To achieve this synchronization, devices in the system must either have direct access to timing signals from a common source, or the devices must synchronize their individual clocks to share a common time base. There are advantages and disadvantages to both methods of device synchronization. 

In a time-based synchronization scheme for data acquisition, we still have to provide the same timing signals as in a signal-based scheme, but the way in which we do so is different. All of the timing signals, triggers, and clocks are based off of a common time reference. Examples of time references are global positioning system (GPS), IRIG-B, 802.1, and IEEE 1588. Devices on the network, including switches and routers, can be synchronized precisely via the IEEE 1588 and IEEE 802.1 precision time protocol (PTP) standards.

Each subsystem may generate a local reference clock signal (local to the subsystem), which may be aligned and locked with respect to one or more similar respective reference clock signals of other subsystems, via a high-level PTP such as IEEE l588 or a GPS protocol. For instrumentation systems, each DAQ card within a given subsystem may generate a local sample clock (local to the DAQ card) based on the local reference signal and generate a local trigger clock (local to the DAQ card) based on the local sample clock. The trigger clocks may be synchronized with respect to each other, and each DAQ card then may use its trigger clock to synchronize any received trigger (or trigger pulse), resulting in received triggers being synchronized across all participating DAQ cards across all participating subsystems.

Sharing a common timing signal becomes infeasible when the distance between devices increases or when devices frequently change location. Even at moderate distances such as 50 meters, a common timing signal may require significant costs for cabling and configuration. Up to 100 meters, you can match trace length. Once you can’t, you lose sync at 1.5 ns/ft, so over 200 meters you can lose approximately 900 ns of sync. IEEE 1588 can maintain sync for 200 meters through only a single switch (single subnet) with what we have today.

Compensating for Path Delays

In general, distributed measurement and control systems often require their composite parts to be aligned to the same time base. One useful result of synchronization in these applications is the sharing of synchronized periodic signals, which can be used to take measurements at the same time or provide known relationships between control units in a distributed environment. In these situations, distributed clock synchronization becomes necessary. 

Using this approach, devices act on timing signals originating from a local clock which is synchronized to the other clocks in the system. Examples of distributed clock synchronization include devices synchronized to a GPS satellite, a PC’s internal clock synchronized to a network time protocol (NTP) time server, or a group of devices participating in the IEEE 1588 protocol. Instead of sharing timing signals directly, these devices periodically exchange information and adjust their local timing sources to match each other. 

The synchronization of distributed clocks requires a continuous process. A clock essentially is a two-part device consisting of a frequency source and an accumulator. In theory, if two clocks were set identically and their frequency sources ran at the exact same rate, they would remain synchronized indefinitely. In practice, however, clocks are set with limited precision, frequency sources run at slightly different rates, and the rate of a frequency source changes over time and temperature. 

Most modern electronic clocks use a crystal oscillator as a frequency source. The frequency of a crystal oscillator varies due to initial manufacturing tolerance, temperature and pressure changes, and aging. Because of these inherent instabilities, distributed clocks must continually be synchronized to match each other in frequency and phase. 

Ethernet—IEEE 1588 Synchronization

IEEE 1588 provides a standard protocol for synchronizing clocks connected via a multicast-capable network, such as Ethernet. Released as a standard in 2002, IEEE 1588 was designed to provide fault-tolerant synchronization among heterogeneous networked clocks requiring little network bandwidth overhead, processing power, and administrative setup. IEEE 1588 provides this by defining the PTP. IEEE 1588 is designed to fill a niche not well served by either of the two dominant protocols: NTP and GPS. IEEE 1588 is designed for local systems requiring accuracies beyond those attainable using NTP. It also is designed for applications where a GPS receiver at each node is too expensive or for which GPS signals are inaccessible.

Using Time-Based Synchronization

Slave clocks synchronize to the 1588 grandmaster by using bidirectional multicast communications. The grandmaster clock periodically issues a sync packet containing a timestamp of when the packet left the grandmaster clock. The grandmaster also may, optionally, issue a follow-up packet containing the timestamp for the sync packet. The use of a separate follow-up packet allows the grandmaster to accurately timestamp the sync packet on networks where the departure time of a packet cannot be known accurately beforehand. For example, the collision detection and random back-off mechanism of Ethernet communications prevents the exact transmission time of a packet from being known until the packet is completely sent without a collision being detected, at which time it is impossible to alter the packet’s content (Figure 1).

The master periodically broadcasts the current time as a message to the other clocks. Under IEEE 1588-2002, broadcasts are up to once per second. Under IEEE 1588-2008, up to 10 per second are permitted.

Each broadcast begins at time with a sync message sent by the master to all the clocks in the domain. A clock receiving this message takes note of the local time when this message is received.

By sending and receiving these synchronization packets, the slave clocks can accurately measure the offset between their local clock and the master’s clock. The slaves then can adjust their clocks by this offset to match the time of the master. IEEE 1588 does not include any standard implementation for adjusting a clock; it merely provides a standard protocol for exchanging these messages, allowing devices from different manufacturers and with different implementations to interoperate. 

Time-Based Synchronization 

Several factors affect the synchronization levels achievable using IEEE 1588 over Ethernet. During the time between synchronization packets, the individual clocks in a system will drift apart from each other due to frequency changes in their local timing source. This drift can be reduced by using higher stability timing sources and shortening the intervals between synchronization packets. Temperature-controlled crystal oscillators and oven-controlled crystal oscillators provide higher stability than standard crystal oscillators, and atomic clocks support still higher stability. In addition to stability, a clock’s resolution will affect the accuracy of the timestamps transmitted in the PTP synchronization messages. 

Devices that have a higher resolution clock are able to more accurately timestamp messages. Also, variations in network delay, caused by jitter introduced by intermediate networking devices such as hubs and switches, reduce the achievable synchronization level. IEEE 1588 provides an important alternative for systems requiring sub-microsecond synchronization in geographically distributed systems (Figure 2).

TSN and Deterministic Ethernet

Ethernet, Wi-Fi, and other IEEE 802-based network technologies have been very successful in a large number of connectivity applications, but until very recently, there was no way to provide critical time-sensitive services in those networks. The result has been a proliferation of specialized networks and connectivity systems for audio/video and real-time control applications. 

This lack of integration is in the process of being remedied by the creation of an IEEE 802 architecture for TSN. It specifies a profile for use of IEEE 1588 for time synchronization over a virtual bridged local area network defining how IEEE 802.3 (Ethernet) and IEEE 802.11 (Wi-Fi) can all be parts of the same timing domain based on three major advances:

• Universal time synchronization or time awareness by the network infrastructure enabling devices on the network, including switches and routers, to be synchronized precisely via the IEEE 1588 and IEEE 802.1 PTP standards.
• Time-sensitive queuing and forwarding in all devices to provide lower, and guaranteed, delays for time-sensitive data.
• Bandwidth and latency reservations so that the time-sensitive queues in the network do not overflow and packets are not dropped. 

CERN’s Timing Triggering and Control System

The Large Hadron Collider (LHC) at CERN is a complex of six circular and some linear accelerators which are interconnected. The biggest accelerator is 27 km long. All the devices which serve the accelerators (magnets, kickers, etc.) need to be precisely synchronized and controlled by a central control system. 

CERN adopted National Instruments’ RIO platforms to take advantage of the field-programmable gate arrays (FPGA) to serve as the timekeeper in the NI RIO platform and uses it to move blocks of graphite in place to absorb the protons that are not in the nominal path of the beam or, in other words, go astray. This process is commonly known as “collimation.” Since this is a 27-km tunnel, there are more than 100 of these collimators around the tunnel that have to be synchronized accurately and reliably. 

In a given collimator, the NI PXI chassis run LabVIEW Real-Time on the controller for reliability and LabVIEW FPGA on the reconfigurable I/O devices in the peripheral slots to perform the collimator control for approximately 600 stepper motors with millisecond synchronization over the 27 km of the LHC. The timekeeper on the FPGAs on these devices gives the level of control needed. 

White Rabbit 

A decision was made to base the new CERN timing system on PTP. The White Rabbit project is an Ethernet-based network with low-latency, deterministic data delivery and network-wide, transparent, high-accuracy timing distribution. The White Rabbit (WR) network extension is based on existing standards, namely Ethernet, Synchronous Ethernet (SyncE), and PTP. 

The approach aims for a general-purpose, fieldbus-like transmission system, which provides deterministic data and timing (sub-ns accuracy and ps jitter) to around 1,000 stations. It automatically compensates for fiber lengths in the order of 10 km (Figure 3).

The WR project focuses on an open design featuring:

• Sub-Nanosecond Accuracy: Synchronization of more than 1,000 nodes via fiber and copper connections up to 10 km apart. 
• Flexibility: Creating a scalable and modular platform with simple configuration and low maintenance requirements.
• Predictability and Reliability: Deterministic delivery of highest priority messages.
• Robustness: No losses of high-priority accelerator device control messages.

WR takes advantage of the latest developments for improving timing over Ethernet. SyncE is an ITU-T standard for computer networking that facilitates the transference of clock signals over the Ethernet physical layer. This signal then can be made traceable to an external clock. 

Network synchronization in WR is based on clock hierarchy with the highest accuracy clock at the top. Slave clocks are PLL-based and locked to an external reference that is being recovered by the data link. The PLL cleans the recovered clock by removing jitter generated from the clock recovery circuitry before being fed to the transmitting device.

The key ideas of the WR technology can be adapted and included into the next revision of PTP, consequently enabling the standard-compliant PTP devices to achieve high-accuracy synchronization using methods prototyped and tested in WR. In a WR timing network, a timing master will provide the master clock and the general timing to be used by all attached WR devices. It also is syntonized with an atomic clock and synchronized with a GPS receiver.

Components of a WR network are WR switches and WR nodes. Both components may be added dynamically to the network. Though conventional Gigabit Ethernet devices may be connected as well, only WR devices take part in the timing by using the timing information.

The switch is the core element of the WR network, implementing the standard IEEE 802.1x Ethernet bridge functionality and WR-specific extensions. The extensions are enabled only after a proper WR handshake has been performed. Therefore if a non-WR aware device is connected, it sees a standard 802.1x switch. 

Updating Standards

Upcoming changes to the Ethernet stand-ards can yield a giant leap in performance from nanoseconds to picoseconds. Currently, the P1588 working group is working on a new edition of IEEE 1588. The Project Authorization Request for revision of IEEE 1588-2008 was approved on June 4, 2013, with an expected completion date of Dec. 31, 2017. 

The standard specifies requirements for mapping the protocol to specific network implementations and defines such mappings, including User Datagram Protocol/Internet Protocol (IP versions 4 and 6), and layer-2 IEEE 802.3 Ethernet. 

The protocol enables heterogeneous systems that include clocks of various inherent precision, resolution, and stability to synchronize to a grandmaster clock. It supports synchronization in the sub-microsecond range with minimal network bandwidth and local clock computing resources. The protocol also enhances support for synchronization to better than 1 ns and specifies how corrections for path asymmetry are made if the asymmetry values are known. The grandmaster can be synchronized to a source of time external to the system if time traceable to international standards or other source of time is required.

These enhancements make the impossible possible for precision test and measurement applications. As they are incorporated by the makers of equipment, it should make it easier for end users. 

Additional Reading 

• Daniluk, G., “The White Rabbit Project: An Ethernet-based solution for sub-ns synchronization and deterministic delivery,” CERN, Jan. 30, 2014. 
• “GPS, IEEE 1588, and IRIG-B Timing and Synchronization Modules for PXI Express and PXI NI PXI-6683H, NI PXI-6683,” National Instruments.  
• “Introduction to Distributed Clock Synchronization and the IEEE 1588 Precision Time Protocol,” National Instruments, Oct. 29, 2013. 
• Losita, R., “CERN Uses NI LabVIEW Software and PXI Hardware to Control World’s Largest Particle Accelerator,” National Instruments.
• White Rabbit website. 

About the Authors

Kurt Veggeberg is the business development manager for NVH applications at National Instruments. He has been working at NI since 1985. Veggeberg holds a master’s degree in business administration from the University of Texas at Austin and a bachelor’s degree from Cornell University.
[email protected]

Ravichandran Raghavan is the product marketing manager for timing and synchronization products at National Instruments. He joined NI in 2011 after receiving his bachelor’s degree in electronics and communications engineering from Amrita University, India. [email protected]