PTP with SyncE in HFT Trading firm in their Network Design
I recently explored the HFT network after discussing with a friend who was working with one of the HFT firms how to synchronize the time on all the devices in their network, as it is critical that time delay can cause orders to be placed at wrong intervals due to incorrect time, causing serious issues.
HFT firms rely on the discrepancies between buying and selling prices in the market data. Order execution happens in nanoseconds.
In general, most firms use PTP for time synchronization from the grandmaster to the boundary clock toward the slave.
It seems beneficial to rely on PTP alone in an HFT network, but it might bring risk to them.
I was exploring SyncE along with PTP, which might be a good to avoid any risk of using only PTP in their network
Why recommending SyncE along with PTP? For that, you need to identify the possible issue with the PTP-only design
1. Packet Delay Variation (PDV)
PTP operates at Layer 3 and can also function at Layer 2. Its accuracy is influenced by network changes and packet timing, which can be delayed due to microbursts (PTP is subject to PDV), even on higher-capacity ports like 10G or 100G.
2. Oscillator Drift:
Even PTP packets are received at every interval, but the Node is not connected to the Master clock every time and it should rely on the local Hardware Oscillator.
Drift is not a one-time error; it's a slope. Accumulates slowly
Let's see how SyncE might enhance the HFT network
In HFT Network, enabling SyncE can act as a stabilizer during the time interval for receiving the PTP packets.
SyncE provides a constant and high-frequency reference directly from the physical layer. If the local oscillator is locked with SyncE signal, then it doesn't wander.
With SyncE, now the PTP algorithm works to correct the time offset, not the constant change in frequency.
What is the network device loss to the grandmaster? or continuous loss of PTP packets can be catastrophic.
Without Sync-E , if the node stops receiving the PTP packets then the local oscillator drift immediately can cause huge impact
With Sync-E, the frequency is continuously locked to the upstream source. The node can maintain the time perfectly for much longer time as it is sync with the other nodes in the network
In an HFT environment, SyncE ensures all the device are synchronized (running at same frequency)
Alone, PTP time accuracy can be within sub-microseconds, but PTP with SyncE time accuracy is nanoseconds.
Conclusion: PTP provides the time of day, and SyncE provides frequency stability.
Configuring a hybrid SyncE + PTP environment requires a 2-step approach: first, establishing the physical layer frequency (SyncE) end-to-end, and second, layering the packet-based time synchronization (PTP).
Configuration to enable SyncE and PTP on Cisco, Arista, Juniper and Nokia Devices
Cisco (IOS-XR / ASR 9000)
SyncE Configuration
frequency synchronization
source interface X/X/X
!
interface X/X/X
frequency synchronization
selection input
wait-to-restore 0
PTP Hybrid Configuration
ptp
clock
domain 0
profile g.8275.1
! Tells PTP to use the physical layer frequency
physical-layer-frequency
!
interface X/X/X
transport ethernet
port state master
Arista (EOS / 7130 or 7050 Series)
SyncE Frequency
frequency-spec
source interface Ethernet1
!
interface Ethernet1
switchport mode trunk
sync-e
PTP Hybrid Mode
ptp mode boundary
ptp profile <XXX>
! Ensures the local clock is locked to the SyncE source
ptp holdover-spec sync-e
!
interface Ethernet1
ptp enable
ptp role master
Juniper (Junos / MX Series)
SyncE
set chassis synchronization network-option option-1
set chassis synchronization interface ge-0/0/0 source-role primary
PTP Hybrid Configuration
set protocols ptp clock-mode boundary-clock
set protocols ptp profile X.X.X
# Link PTP to the SyncE frequency source
set protocols ptp hybrid-mode
set protocols ptp slave interface X.X.X
set protocols ptp master interface X.X.X
Nokia (SROS / 7750 SR)
SyncE
configure system synchronization
network-interface X/X
source-priority 1
no shutdown
exit
exit
PTP Hybrid Mode
configure system ptp
clock-type boundary-clock
profile <XXX>
# Enable Hybrid Mode to leverage SyncE frequency
hybrid
no shutdown
peer X.X.X.X
no shutdown
exit
exit
