Deploying SRv6 in Juniper: From Setup to Verification

I'm exploring the SRV6, and it's obvious that I am exploring now with Juniper after I tested SRV6 configurations and features in Nokia SROS.

In this blog post I will be covering the SRV6 enabling in Junos, and in upcoming blogs I will enable different services through the SRV6 tunnel and hurdles during the testing in detail.

I am using the below topology to enable SRv6 in Junos. IPv4 is not enabled; only IPv6 is configured and enabled ISIS over IPv6.

Loopback/locator IPv6 addresses

Let's configure the IPv6 address on respective interface as below

set interfaces ge-0/0/1 unit 0 family inet6 address 2001:db8:12::1/64

set interfaces ge-0/0/2 unit 0 family inet6 address 2001:db8:14::1/64

set interfaces lo0 unit 0 family inet6 address 2001:db8:0::1/128

Locator configuration on each router as per table

set routing-options source-packet-routing srv6 locator loc1 2001:db8:a1:1::/64 

Configure the flavour. How the IPV6 transit and endpoint and adjacency SID

Penultimate Segment Pop 

set protocols isis source-packet-routing srv6 locator loc1 end-sid 2001:db8:a1:1::0 flavor psp

Ultimate Segment Decapsulation

When a packet reaches its final SRv6 segment destination, if there are no more segments in the SRH, the USD instruction tells the router to pop the outer IPv6 header along with any remaining extension headers.

set protocols isis source-packet-routing srv6 locator loc1 end-sid 2001:db8:a1:1::0 flavor usd

Enable the Enhanced IP under the chassis

set chassis network-services enhanced-ip

Enable SRV6 in ISIS

set interfaces lo0 unit 0 family iso address 49.0001.0000.0000.0001.00

set interfaces ge-0/0/1 unit 0 family iso

set interfaces ge-0/0/2 unit 0 family iso

set protocols isis level 1 disable

set protocols isis interface ge-0/0/1.0

set protocols isis interface ge-0/0/2.0

set protocols isis interface lo0.0 passive

set protocols isis level 2 wide-metrics-only

Advertising SRV6 Locator in ISIS

set protocols isis source-packet-routing srv6

Verification:

IPv6 Locators are programmed in the inet6.0 table like other ipv6 loopback address and next hops that forward the traffic with next hop interface and Locator programmed in inet6.3 table and tunnel next hop tunnel SRV6. SRV6 traffic encoded with SRH header towards the destination.

show route table inet6.0 protocol isis   

2001:db8:a1:1::/64 *[IS-IS/18] 00:06:39, metric 0

                       Reject

2001:db8:a1:1::/128*[IS-IS/18] 00:06:39, metric 0

                       Receive

2001:db8:a1:2::/64 *[IS-IS/18] 00:01:57, metric 10

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0

2001:db8:a1:3::/64 *[IS-IS/18] 00:01:54, metric 20

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0

                       to fe80::5254:ff:feae:6d02 via ge-0/0/1.0

2001:db8:a2:5::/64 *[IS-IS/18] 00:01:50, metric 20

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0

2001:db8:a2:6::/64 *[IS-IS/18] 00:00:49, metric 30

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0

2001:db8:a3:7::/64 *[IS-IS/18] 00:01:44, metric 20

                    >  to fe80::5254:ff:feae:6d02 via ge-0/0/1.0

2001:db8:a3:8::/64 *[IS-IS/18] 00:02:03, metric 30

                    >  to fe80::5254:ff:feae:6d02 via ge-0/0/1.0

admin@R1# run show route table inet6.3 

inet6.3: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

2001:db8:a1:2::/64 *[SRV6-ISIS/14] 00:07:56, metric 10

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a1:2::

2001:db8:a1:2::/128*[SRV6-ISIS/14] 00:07:56, metric 10

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a1:2::

2001:db8:a1:3::/64 *[SRV6-ISIS/14] 00:07:53, metric 20

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a1:3::

                       to fe80::5254:ff:feae:6d02 via ge-0/0/1.0, SRV6-Tunnel, Dest: 2001:db8:a1:3::

2001:db8:a1:3::/128*[SRV6-ISIS/14] 00:07:53, metric 20

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a1:3::

                       to fe80::5254:ff:feae:6d02 via ge-0/0/1.0, SRV6-Tunnel, Dest: 2001:db8:a1:3::

2001:db8:a2:5::/64 *[SRV6-ISIS/14] 00:07:49, metric 20

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a2:5::

2001:db8:a2:5::/128*[SRV6-ISIS/14] 00:07:49, metric 20

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a2:5::

2001:db8:a2:6::/64 *[SRV6-ISIS/14] 00:06:48, metric 30

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a2:6::

2001:db8:a2:6::/128*[SRV6-ISIS/14] 00:06:48, metric 30

                    >  to fe80::5254:ff:fee8:e501 via ge-0/0/0.0, SRV6-Tunnel, Dest: 2001:db8:a2:6::

2001:db8:a3:7::/64 *[SRV6-ISIS/14] 00:07:43, metric 20

                    >  to fe80::5254:ff:feae:6d02 via ge-0/0/1.0, SRV6-Tunnel, Dest: 2001:db8:a3:7::

2001:db8:a3:7::/128*[SRV6-ISIS/14] 00:07:43, metric 20

                    >  to fe80::5254:ff:feae:6d02 via ge-0/0/1.0, SRV6-Tunnel, Dest: 2001:db8:a3:7::

2001:db8:a3:8::/64 *[SRV6-ISIS/14] 00:08:02, metric 30

                    >  to fe80::5254:ff:feae:6d02 via ge-0/0/1.0, SRV6-Tunnel, Dest: 2001:db8:a3:8::

2001:db8:a3:8::/128*[SRV6-ISIS/14] 00:08:02, metric 30

                    >  to fe80::5254:ff:feae:6d02 via ge-0/0/1.0, SRV6-Tunnel, Dest: 2001:db8:a3:8::

[edit]

admin@R1# run ping 2001:db8:a1:3:: 

PING6(56=40+8+8 bytes) 2001:db8:12::1 --> 2001:db8:a1:3::

16 bytes from 2001:db8:23::3, icmp_seq=0 hlim=63 time=796.705 ms

16 bytes from 2001:db8:23::3, icmp_seq=1 hlim=63 time=5.956 ms

16 bytes from 2001:db8:23::3, icmp_seq=2 hlim=63 time=205.600 ms

^C

--- 2001:db8:a1:3:: ping6 statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max/std-dev = 5.956/336.087/796.705/335.749 ms

admin@R1> ping srv6 isis 2001:db8:a3:7:: detail   

Request for icmp_seq=1, via interface 429, packet size 64 

Reply for icmp_seq=1 from 2001:db8:47::7 via interface 429 hops-limit: 63, trip-time: 9.497 ms

        Local transmit time: 2026-06-09 07:20:17 UTC 844.327 ms

        Remote receive time: 2026-06-09 07:20:17 UTC 974.926 ms

Request for icmp_seq=2, via interface 429, packet size 64 

Reply for icmp_seq=2 from 2001:db8:47::7 via interface 429 hops-limit: 63, trip-time: 67.630 ms

        Local transmit time: 2026-06-09 07:20:18 UTC 840.312 ms

        Remote receive time: 2026-06-09 07:20:19 UTC 6.273 ms

Request for icmp_seq=3, via interface 429, packet size 64 

Reply for icmp_seq=3 from 2001:db8:47::7 via interface 429 hops-limit: 63, trip-time: 38.220 ms

        Local transmit time: 2026-06-09 07:20:19 UTC 843.228 ms

        Remote receive time: 2026-06-09 07:20:19 UTC 983.084 ms

Request for icmp_seq=4, via interface 429, packet size 64 

Reply for icmp_seq=4 from 2001:db8:47::7 via interface 429 hops-limit: 63, trip-time: 10.654 ms

        Local transmit time: 2026-06-09 07:20:20 UTC 840.931 ms

        Remote receive time: 2026-06-09 07:20:20 UTC 971.686 ms

Request for icmp_seq=5, via interface 429, packet size 64 

Reply for icmp_seq=5 from 2001:db8:47::7 via interface 429 hops-limit: 63, trip-time: 5.468 ms

        Local transmit time: 2026-06-09 07:20:21 UTC 846.190 ms

        Remote receive time: 2026-06-09 07:20:21 UTC 976.553 ms

--- lsping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max/stddev = 5.468/26.294/67.630/23.712 ms

Segment Routing: Managing SRv6 Header Overhead with uSID and SRm6

In my last post, we looked at how to configure SRv6 on the Nokia 7750 SR. One big challenge with SRv6 is the header overhead. Since every "Segment Identifier" (SID) is a full 128-bit IPv6 address inside the SRH header, the header gets very large very quickly.

This extra weight often causes fragmentation. When a packet is too big for the MTU of a link, it has to be broken into pieces.

This is bad for two reasons:

1. It slows down the network because of the processing power needed for fragmentation/reassembly.

2. It's a security risk, as hackers often use fragmentation attacks to sneak past security filters.

We have two main options to make the IPv6 headers smaller:

1. Micro-SID (uSID)

Micro-SID is strongly recommended by Nokia and Cisco. Instead of giving every hop a full 128-bit address, uSID uses a "container" concept.

How it works: It takes one 128-bit address and divides it up. The first part (the Locator) identifies the network, and the rest of the bits are used for "mini-IDs" (16-bit length in FP-based platforms or 32-bit for other platforms in Nokia).

Ideally, 6 mini-SID hops are injected inside a single 128-bit address. This keeps the header small and prevents the packet from needing to be fragmented.

2. SRm6

This is a different approach, mostly recommended by Juniper. Juniper always does things differently!

SRm6 runs Segment Routing over "Native IPv6" without using the heavy SRv6 header. It uses much smaller IDs—only 16 or 32 bits per destination. This makes the packet look more like a standard IPv6 packet, which saves a lot of space.

---

Lab Example: SRH Header Capture

Here is a screenshot from my lab. You can see the SRH (Segment Routing Header) in the packet capture. Notice how much space the segment list takes up when we use standard 128-bit SIDs.

---

Final Thoughts

In my view, if you are working in a multi-vendor environment, sticking with uSID is the better path for SRv6. It’s becoming the industry standard for keeping the data plane efficient while ensuring compatibility between the big players like Nokia and Cisco.

Deploying Segment Routing SRV6 over IPv6 on Nokia 7750 Platforms

Nokia 7750 SR router support SRV6 on FP-based platforms on FP4 onwards. SR-OS does not support SRv6 for OSPFv3 with current support only for IS-IS in both Multi-Topology.

Steps to enable SRV6 over IPV6 network in Nokia 7750 SR

1. Forwarding Path Extension (FPE)

In General, Whenever IP encapsulated in IP on a FP based flatforms that need to be passed to FPE for processing multiple times, FPE is an extension of PXC  features that loop the ports logically for multiple loop itself allowing two time passes through FPE.

PXC configuration required only on originating/terminating Routers only.

Enabling the mac chip for multiple entry and exit. 

echo "Card Configuration"

#--------------------------------------------------

    card 1

        card-type iom-1

        mda 1

            xconnect

                mac 1 create

                    loopback 1 create 

                    exit

                exit

            exit

            no shutdown

        exit

        no shutdown

    exit

Enabling the port that will be used for looping the packets internally.

    port 1/1/c5/1

        ethernet

            dot1x

                tunneling             

            exit

            mode hybrid

            encap-type dot1q

        exit

        no shutdown

    exit

    port 1/1/c6/1

        ethernet

            dot1x

                tunneling

            exit

            mode hybrid

            encap-type dot1q

        exit

        no shutdown

    exit

Mac 1:

    port 1/1/m1/1

        no shutdown

    exit

Next step is configuring the PXC ports and assign the interface

    port-xc

        pxc 1 create

            no shutdown

            port 1/1/c5/1

        exit

        pxc 2 create

            no shutdown

            port 1/1/c6/1

        exit

    exit

    port pxc-1.a

        ethernet

        exit

        no shutdown

    exit

    port pxc-1.b

        ethernet

        exit

        no shutdown

    exit

    port pxc-2.a

        ethernet

        exit

        no shutdown

    exit

    port pxc-2.b

        ethernet

        exit

        no shutdown

    exit

Last step is enabling the Forwarding Path Extensions. FPC can be used for any other application so we are explicitly mention 

fpe 1 used for origination and fpe 2 used for termination of SRV6 tunnel

    fwd-path-ext

        fpe 1 create

            path pxc 1

            srv6 origination

                interface-a

                exit

                interface-b

                exit

            exit

        exit

        fpe 2 create

            path pxc 2

            srv6 termination

                interface-a

                exit

                interface-b

                exit

            exit

        exit

    exit

*A:PE5# show fwd-path-ext fpe 1 

=====================================================================

FPE Id: 1
=====================================================================
Description        : (Not Specified)
Multi-Path         : Disabled                          
Path               : pxc 1
Pw Port Extension  : Disabled                          Oper     : down
Sub Mgmt Extension : Disabled                          Oper     : N/A
Vxlan              : Disabled                          Oper     : down
Segment-Routing V6 : Enabled                           Oper     : up
SRv6 Type          : origination                       
If-A Qos Policy    : default                           
If-B MTU           : 9786 bytes                        Oper MTU : 8914 bytes
If-B Qos Policy    : default                           
Cups Steering      : Disabled                          Oper     : N/A
MVPN UMH Rate Mon  : Disabled                          
=====================================================================
*A:PE5# show fwd-path-ext fpe 2 
=====================================================================
FPE Id: 2
=====================================================================
Description        : (Not Specified)
Multi-Path         : Disabled                          
Path               : pxc 2
Pw Port Extension  : Disabled                          Oper     : down
Sub Mgmt Extension : Disabled                          Oper     : N/A
Vxlan              : Disabled                          Oper     : down
Segment-Routing V6 : Enabled                           Oper     : up
SRv6 Type          : termination                       
If-A Qos Policy    : default                           
If-B MTU           : 0 bytes                           Oper MTU : 8914 bytes
If-B Qos Policy    : default                           
Cups Steering      : Disabled                          Oper     : N/A
MVPN UMH Rate Mon  : Disabled                          
=====================================================================

Next step is to configuring the locator and assign the locator to the Base routing instanace

Locator enabled is 0 and 128, with Block-length of 48. we need to assign the originating and terminating fpe.

we need to configure the srh-mode for base routing instance, The srh-mode then defines the method of processing the SRH and can be either usp, psp, ps-usd, usp-usd, usp-usd, or psp-usp-usd, with the default being psp

Segment Routing v6 Configuration

       segment-routing

            segment-routing-v6

                origination-fpe 1

                source-address fd00::3

                locator "Algo0-locator"

                    block-length 48

                    termination-fpe 2

                    prefix

                        ip-prefix fd00:db8:1:3::/64

                    exit

                    static-function

                    exit

                    no shutdown

                exit

                locator "Algo128-locator"

                    block-length 48

                    termination-fpe 2

                    algorithm 128

                    prefix

                        ip-prefix fd00:db8:128:3::/64

                    exit

                    static-function

                    exit

                    no shutdown

                exit

                base-routing-instance

                    locator "Algo0-locator"

                        function

                            end-x-auto-allocate srh-mode usp protection protected

                            end 1

                                srh-mode usp

                            exit

                        exit

                    exit

                exit

            exit

Once the locator is configured then advertise them vis ISIS to all other routers in the network.

configure router isis 2

            segment-routing-v6

                locator "Algo0-locator"

                    level-capability level-2

                    exit              

                    level 2

                        metric 1

                    exit

                exit

                no shutdown

            exit

A:PE5# show router segment-routing-v6 local-sid 
=========================================================================
Segment Routing v6 Local SIDs
=========================================================================
SID                                               Type           Function
  Locator                                                        
  Context                                                        
-------------------------------------------------------------------------------
fd00:db8:1:5:0:1000::                             End            1
  Algo0-locator
  Base
fd00:db8:1:5:0:4000::                             End.X          4
  Algo0-locator
  None
fd00:db8:1:5:0:5000::                             End.X          5
  Algo0-locator
  None
-------------------------------------------------------------------------------
SIDs : 3
-------------------------------------------------------------------------------
=========================================================================

*A:PE5# show router segment-routing-v6 base-routing-instance 
=========================================================================
Segment Routing v6 Base Routing Instance
=========================================================================
Locator                                                           
 Type          Function  SID                                     Status/InstId
  SRH-mode     Oper Func Interface                               Protection
-------------------------------------------------------------------------------
Algo0-locator
 End           1         fd00:db8:1:5:0:1000::                   ok
  USP          1                                                 
-------------------------------------------------------------------------------
 Auto-allocated End.X:  USP Protected,
-------------------------------------------------------------------------------
 End.X         4         fd00:db8:1:5:0:4000::                   1
  USP                    PE1                                     Protected
  ISIS Level: L2 Mac Address: aa:c1:ab:46:a4:9a Nbr Sys Id: 0010.0000.0001
 End.X         5         fd00:db8:1:5:0:5000::                   1
  USP                    PE6                                     Protected
  ISIS Level: L2 Mac Address: aa:c1:ab:6e:ca:0b Nbr Sys Id: 0010.0000.0006
-------------------------------------------------------------------------------
=========================================================================
Legend: * - System allocated

*A:PE5# show router route-table ipv6 protocol srv6 
=========================================================================
IPv6 Route Table (Router: Base)
=========================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
fd00:db8:1:5::/64                             Local   SRV6      29d14h16m  3
       Black Hole                                                   0
fd00:db8:1:5:0:1000::/128                     Local   SRV6      29d14h14m  3
       Black Hole                                                   0
fd00:db8:128:5::/64                           Local   SRV6      29d14h17m  3
       Black Hole                                                   0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
=========================================================================

i could see that route is pointing towards blackhole but in PE3 its pointing to the PXC interface

*A:PE3>config>router>segment-routing>srv6# show router route-table ipv6 fd00:db8:1:3::/64 
=========================================================================
IPv6 Route Table (Router: Base)
=========================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
fd00:db8:1:3::/64                             Local   SRV6      11d16h21m  3
       fe80::201-"_tmnx_fpe_2.a"                                    0
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
=========================================================================
After checking the configuration identified on PE3 the PXC interface is shut, Enabled them now

*A:PE5>config>port-xc#  pxc 2 no shutdown 
*A:PE5>config>port-xc# pxc 1 no shutdown  
*A:PE5>config>port-xc# info 
----------------------------------------------
        pxc 1 create
            port 1/1/c5/1
            no shutdown
        exit
        pxc 2 create
            port 1/1/c6/1
            no shutdown
        exit
----------------------------------------------

After the configuration i could see that route is pointing now to pxc as expected

*A:PE5>config>port-xc# show router route-table ipv6 protocol srv6 
=========================================================================
IPv6 Route Table (Router: Base)
=========================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
fd00:db8:1:5::/64                             Local   SRV6      00h00m25s  3
       fe80::201-"_tmnx_fpe_2.a"                                    0
fd00:db8:1:5:0:1000::/128                     Local   SRV6      29d14h20m  3
       Black Hole                                                   0
fd00:db8:128:5::/64                           Local   SRV6      00h00m25s  3
       fe80::201-"_tmnx_fpe_2.a"                                    0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
=========================================================================
Route table entry for PE5 SRV6 Prefix
*A:PE3# show router route-table fd00:db8:1:5::/64 
=========================================================================
IPv6 Route Table (Router: Base)
=========================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
fd00:db8:1:5::/64                             Remote  ISIS(2)   27d00h05m  18
       fd00:db8:1:5::/64 (tunneled:SRV6-ISIS)                       41
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
=========================================================================
*A:PE5# show router route-table fd00:db8:1:3::/64 
=========================================================================
IPv6 Route Table (Router: Base)
=========================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
fd00:db8:1:3::/64                             Remote  ISIS(1)   11d16h40m  18
       fd00:db8:1:3::/64 (tunneled:SRV6-ISIS)                       41
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
=========================================================================
*A:PE3# 

Ping is working successfully between ALGO SRV6 Locator

*A:PE5# ping fd00:db8:1:3:0:1000:: 
PING fd00:db8:1:3:0:1000:: 56 data bytes
64 bytes from fd00:db8:1:3:0:1000:: icmp_seq=1 hlim=61 time=8.23ms.
64 bytes from fd00:db8:1:3:0:1000:: icmp_seq=2 hlim=61 time=2.22ms.
64 bytes from fd00:db8:1:3:0:1000:: icmp_seq=3 hlim=61 time=2.27ms.
64 bytes from fd00:db8:1:3:0:1000:: icmp_seq=4 hlim=61 time=2.37ms.
^C
ping aborted by user
---- fd00:db8:1:3:0:1000:: PING Statistics ----
4 packets transmitted, 4 packets received, 0.00% packet loss
round-trip min = 2.22ms, avg = 3.77ms, max = 8.23ms, stddev = 0.000ms
*A:PE5# 
*A:PE3# ping fd00:db8:1:5:0:1000:: 
PING fd00:db8:1:5:0:1000:: 56 data bytes
64 bytes from fd00:db8:1:5:0:1000:: icmp_seq=1 hlim=61 time=2.42ms.
64 bytes from fd00:db8:1:5:0:1000:: icmp_seq=2 hlim=61 time=2.29ms.
64 bytes from fd00:db8:1:5:0:1000:: icmp_seq=3 hlim=61 time=2.42ms.
^C
ping aborted by user
---- fd00:db8:1:5:0:1000:: PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 2.29ms, avg = 2.38ms, max = 2.42ms, stddev = 0.000ms
*A:PE3# 

Beyond PTP: Why Hybrid SyncE + PTP is the Competitive Edge in HFT Fabrics

PTP with SyncE in HFT Trading firm  in their Network Design

I recently explored the HFT network after discussing with a friend who was working with one of the HFT firms how to synchronize the time on all the devices in their network, as it is critical that time delay can cause orders to be placed at wrong intervals due to incorrect time, causing serious issues.

HFT firms rely on the discrepancies between buying and selling prices in the market data. Order execution happens in nanoseconds.

In general, most firms use PTP for time synchronization from the grandmaster to the boundary clock toward the slave.

It seems beneficial to rely on PTP alone in an HFT network, but it might bring risk to them.

I was exploring SyncE along with PTP, which might be a good to avoid any risk of using only PTP in their network

Why recommending SyncE along with PTP? For that, you need to identify the possible issue with the PTP-only design

1. Packet Delay Variation (PDV) 

PTP operates at Layer 3 and can also function at Layer 2. Its accuracy is influenced by network changes and packet timing, which can be delayed due to microbursts (PTP is subject to PDV), even on higher-capacity ports like 10G or 100G.

2. Oscillator Drift:

Even PTP packets are received at every interval, but the Node is not connected to the Master clock every time and it should rely on the local Hardware Oscillator.

Drift is not a one-time error; it's a slope. Accumulates slowly 

Let's see how SyncE might enhance the HFT network

In HFT Network, enabling SyncE can act as a stabilizer during the time interval for receiving the PTP packets.

SyncE provides a constant and high-frequency reference directly from the physical layer. If the local oscillator is locked with SyncE signal, then it doesn't wander. 

With SyncE, now the PTP algorithm works to correct the time offset, not the constant change in frequency.

What is the network device loss to the grandmaster? or continuous loss of PTP packets can be catastrophic. 

Without Sync-E , if the node stops receiving the PTP packets then the local oscillator drift immediately can cause huge impact

With Sync-E, the frequency is continuously locked to the upstream source. The node can maintain the time perfectly for much longer time as it is sync with the other nodes in the network

In an HFT environment, SyncE ensures all the device are synchronized (running at same frequency)

Alone, PTP time accuracy can be within sub-microseconds, but PTP with SyncE time accuracy is nanoseconds.

Conclusion: PTP provides the time of day, and SyncE provides frequency stability.

Configuring a hybrid SyncE + PTP environment requires a 2-step approach: first, establishing the physical layer frequency (SyncE) end-to-end, and second, layering the packet-based time synchronization (PTP).

Configuration to enable SyncE and PTP on Cisco, Arista, Juniper and Nokia Devices

Cisco (IOS-XR / ASR 9000)

SyncE Configuration

frequency synchronization

 source interface X/X/X

 !

interface X/X/X

 frequency synchronization

  selection input

  wait-to-restore 0

PTP Hybrid Configuration

ptp

 clock

  domain 0

  profile g.8275.1

  ! Tells PTP to use the physical layer frequency

  physical-layer-frequency 

 !

 interface X/X/X

  transport ethernet

  port state master

Arista (EOS / 7130 or 7050 Series)

SyncE Frequency

frequency-spec

 source interface Ethernet1

!

interface Ethernet1

 switchport mode trunk

 sync-e

 PTP Hybrid Mode

ptp mode boundary

ptp profile <XXX>

! Ensures the local clock is locked to the SyncE source

ptp holdover-spec sync-e 

!

interface Ethernet1

 ptp enable

 ptp role master

Juniper (Junos / MX Series)

SyncE

set chassis synchronization network-option option-1

set chassis synchronization interface ge-0/0/0 source-role primary

PTP Hybrid Configuration 

set protocols ptp clock-mode boundary-clock

set protocols ptp profile X.X.X

# Link PTP to the SyncE frequency source

set protocols ptp hybrid-mode

set protocols ptp slave interface X.X.X

set protocols ptp master interface X.X.X

Nokia (SROS / 7750 SR)

SyncE

configure system synchronization

    network-interface X/X

        source-priority 1

        no shutdown

    exit

exit

PTP Hybrid Mode 

configure system ptp

    clock-type boundary-clock

    profile <XXX>

    # Enable Hybrid Mode to leverage SyncE frequency

    hybrid

    no shutdown

    peer X.X.X.X

        no shutdown

    exit

exit

SR-MPLS Troubleshooting: Identifying and Fixing Controller-Injected Null EROs

Recently I faced the issue where SR-MPLS is down due to an empty ERO path for SR-TE LSP, and the path is managed by the controller.

Background:

Implemented SR-MPLS LSP for one of our customers. where the LSP paths are injected from the controller based on the constraint. We faced some issues with respect to the controller recently and fixed them after the upgrade. Post upgrade, we observed that one of the LSPs didn't come up.

Verification:

End-to-end path manually to see whether it meets the constraints mentioned for the path. Everything seems to be fine.

The team checked the controller logs; there are no errors but it is sending empty ERO to the router

As this is very critical, I did perform troubleshooting and am finally able to bring the LSP path with correct path details instead of the empty ERO from the controller:

LSP Configuration:

            lsp "srte-access1-pre-agg" sr-te
                to X.X.X.X
                path-computation-method pce
                max-sr-labels 4 additional-frr-labels 2
                pce-report enable
                pce-control
                bfd
                    failure-action failover-or-down
                exit
                path-profile 1 path-group 128946735
                primary "srte-p"
                    priority 1 1
                exit
                secondary "srte-s"
                    standby
                    priority 7 7
                exit
                no shutdown
            exit
            no shutdown

Troubleshooting Steps:

# show router mpls sr-te-lsp

=====================================================================

MPLS SR-TE LSPs (Originating)

=====================================================================

LSP Name                                            Tun     Protect   Adm  Opr

  To                                                Id      Path

---------------------------------------------------------------------

srte-access1-pre-agg                   4       N/A       Up   Dwn

  X.X.X.X

---------------------------------------------------------------------

LSPs : 1

=====================================================================

Enabled Debug for SR-TE LSP:

debug

    router "Base"

        mpls lsp "srte-access1-pre-agg"

            event

                iom

                lsp-setup

                xc

                frr

                mbb

                misc

                pcc

                te

            exit

        exit

    exit

config>log>log-id$ info

----------------------------------------------

            from debug-trace

            to session

            no shutdown

Debug Logs

Received PCC Reply for srte-access1-pre-agg::srte-p(LspId 62454)

User SR-TE, GenId 2, NumRequests 1

[Reply 1]

  LspType SR-TE, TunnelId 4, PathIdx 10

  LspId 62454, VrId 1, ExtTunnelId X.X.X.X

  LspName "srte-access1-pre-agg::srte-p"

  From X.X.X.X, To X.X.X.X

  Status Path not found, PCCHasLspState Yes

  ERO:

    No hops

  OperBandwidth 0Mbps, HopCnt 0, IgpMetric 0, TeMetric 0

From debug it's clear the controller is not able to calculate the path for the LSP. The next step removed the LSP control from PCC to PCE.

After configuring the local-cspf to calculate the path, LSP has come up. So it gave me an idea that there is no issue with the path or the CSPF configuration along the path.

PE>config>router>mpls>lsp# path-computation-method pce

=====================================================================

MPLS SR-TE LSPs (Originating)

=====================================================================

LSP Name                                            Tun     Protect   Adm  Opr

  To                                                Id      Path

---------------------------------------------------------------------

srte-access1-pre-agg                   4       N/A       Up   Up

  X.X.X.X

---------------------------------------------------------------------

LSPs : 1

=====================================================================

LSP Path Detail:

Primary         : srte-p

                                            Down Time: 0d 00:19:02

Bandwidth       : 0 Mbps

Standby         : srte-s

                                            Down Time: 0d 00:19:10

Bandwidth       : 0 Mbps

Rollback the configuration to PCC from PCE, then check again, but the LSP is again gone down with an empty ERO. Due to some reason, the controller is not able to calculate the path for the LSP. seems something is stuck but not able to figure out until this point

Next, path profiles are responsible for setting the constraint on the controller, so remove it, and the LSP has come up. So it's clear that something is wrong in controller in calculating the path, maybe due to something got stuck and its not able to remove it from its cache. 

PE>config>router>mpls>lsp# no path-profile 1

=============================================================

MPLS SR-TE LSPs (Originating)

=============================================================

LSP Name                                            Tun     Protect   Adm  Opr

  To                                                Id      Path

-------------------------------------------------------------

srte-access1-pre-agg                   4       N/A       Up   Up

  X.X.X.X

-------------------------------------------------------------

LSPs : 1

=============================================================

This time I enabled the path profile configuration on the router, but I did change the path group value, and then the LSP came up, and now the controller was able to calculate the path and send the details to the router.

Debug Logs:

"MPLS: SR-TE : PCC

Received PCC Reply for srte-access1-pre-agg (LspId XXX)

User SR-TE, GenId 2, NumRequests 1

[Reply 1]

  LspType SR-TE, TunnelId 4, PathIdx 9

  LspId XXX, VrId 1, ExtTunnelId XXXX

  LspName "srte-access1-pre-agg"

  From X.X.X.X, To X.X.X.X

  Status Path found, PCCHasLspState Yes

  ERO:

    [1] SIDType IPv4 Node, SID 23434343, RemoteAddr X.X.X.X, Loose

    [2] SIDType IPv4 Node, SID 23432434, RemoteAddr X.X.X.X, Loose

  OperBandwidth 0Mbps, HopCnt 2, IgpMetric XXX, TeMetric 0

"

LSP status:

show router mpls sr-te-lsp "srte-access1-pre-agg" path "srte-p"

=============================================================

MPLS SR-TE LSP srte-access1-pre-agg

Path srte-p

=============================================================

-------------------------------------------------------------

LSP Name    : srte-access1-pre-agg

From             : X.X.X.X

To               : X.X.X.X

Adm State        : Up                      Oper State        : Up

-------------------------------------------------------------

Path Name                        Type           Adm  Opr

-------------------------------------------------------------

srte-p

                                 Primary         Up   Up

=============================================================

This LSP was down before the controller upgrade, and this group value was copied somehow after the upgrade. Group value in the cache with the previous issue state causes the controller to send the empty ERO continuously. After reconfiguring with the new value, the old group value got cleared, and the LSP came up. After monitoring the LSP for some time, I changed the group value as per the agreed standard. 

Recommandation:

It's not recommended to perform the troubleshooting without the support team because, from my experience with various roles, it gave me enough confidence to do all the various ways of bringing up the LSP without causing the traffic impact in the network.

Implemeting SR-MPLS Tunnel over IPV6 Network Core in Nokia 7750 SR

SR MPLS over Ipv6 Core network same as IPV4 because it uses mpls label for to deliver the packets end to end. SR MPLS will be on top on IPV6 Core. We can only use IPV6 without requirement of MPLS that concept is SRV6, SRV6 uses IPv6 + SRH to delivery the packet end to end. We will explore more on SRV6 in our upcoming blog

Lets prepare the IPV6 address and prepare them to enable IPV6 end to end.

Configured the router's interface with IPV6 address

Enabling ISIS to advertise the IPV6 capabilities to the neighbour routers

 Enabling cspf for ipv6

/configure router isis 1 traffic-engineering-option ipv6

Enabling IPv6 routeing within mt0

/configure router isis 1 ipv6-routing native

/configure router

        segment-routing

            sr-mpls

                prefix-sids "system" 

                    ipv6-sid index 81

                    exit

                exit

 

ISIS Database:

 
*A:PE1# show router isis 1 database level 2 PE1.00-00        
 
===============================================================================
Rtr Base ISIS Instance 1 Database
===============================================================================
LSP ID                                  Sequence  Checksum Lifetime Attributes
-------------------------------------------------------------------------------
 
Displaying Level 2 database
-------------------------------------------------------------------------------
PE1.00-00                               0xcb1     0xcdce   1138     L1L2
Level (2) LSP Count : 1
===============================================================================
*A:PE1# show router isis 1 database level 2 PE1.00-00 detail
 
===============================================================================
Rtr Base ISIS Instance 1 Database (detail)
===============================================================================
 
Displaying Level 2 database
-------------------------------------------------------------------------------
LSP ID    : PE1.00-00                                   Level     : L2
Sequence  : 0xcb1                  Checksum  : 0xcdce   Lifetime  : 1135
Version   : 1                      Pkt Type  : 20       Pkt Ver   : 1
Attributes: L1L2                   Max Area  : 3        Alloc Len : 1492
SYS ID    : 0010.0000.0001         SysID Len : 6        Used Len  : 544
 
TLVs :
  Area Addresses:
    Area Address : (3) 49.0001
  Supp Protocols:
    Protocols     : IPv4
    Protocols     : IPv6
  IS-Hostname   : PE1
  Router ID   :
    Router ID   : 1.0.0.1
  TE Router ID v6  :
    Router ID   : fd00::1
  Router Cap : 1.0.0.1, D:0, S:0
    TE Node Cap : B E M  P
    SR Cap: IPv4 MPLS-IPv6
       SRGB Base:20000, Range:100
    SR Alg: metric based SPF, 129, 130
    Node MSD Cap: BMI : 12 ERLD : 15
    FAD Sub-Tlv:
        Flex-Algorithm   : 129
        Metric-Type      : delay
        Calculation-Type : 0
        Priority         : 100
        Flags: M
    FAD Sub-Tlv:
        Flex-Algorithm   : 130       
        Metric-Type      : igp
        Calculation-Type : 0
        Priority         : 100
        Flags: M
        Include Any Admin Group: 0x100000
  I/F Addresses :
    I/F Address   : 1.0.0.1
    I/F Address   : 192.1.1.1
    I/F Address   : 10.1.5.1
    I/F Address   : 10.1.21.1
    I/F Address   : 192.168.0.2
  I/F Addresses IPv6 :
    IPv6 Address    : fd00::1
    IPv6 Address    : fd00:10:1:2::1
    IPv6 Address    : fd00:10:1:11::1
    IPv6 Address    : fd00:10:5:1::1
  TE IS Nbrs   :
    Nbr   : PE5.01                             
    Default Metric  : 10
    Sub TLV Len     : 59
    IF Addr   : 10.1.5.1
    IPv6 Addr : fd00:10:5:1::1
    TE APP LINK ATTR    :
      SABML-flag:Non-Legacy SABM-flags: S 
        MaxLink BW: 10000000 kbps
    TE APP LINK ATTR    :
      SABML-flag:Non-Legacy SABM-flags:   X
        Admin Grp : 0x100000
    LAN-Adj-SID: 00 10 00 00 00 05  Flags:v4VL Weight:0 Label:524272
  TE IS Nbrs   :
    Nbr   : PE2.02                             
    Default Metric  : 10
    Sub TLV Len     : 61
    IF Addr   : 10.1.21.1
    IPv6 Addr : fd00:10:1:2::1
    TE APP LINK ATTR    :
      SABML-flag:Non-Legacy SABM-flags: S 
        MaxLink BW: 10000000 kbps    
    LAN-Adj-SID: 00 10 00 00 00 02  Flags:v4VL Weight:0 Label:524273
    LAN-Adj-SID: 00 10 00 00 00 02  Flags:v6VL Weight:0 Label:524286
  TE IS Nbrs   :
    Nbr   : ASBR1.03                           
    Default Metric  : 10
    Sub TLV Len     : 48
    IF Addr   : 192.1.1.1
    IPv6 Addr : fd00:10:1:11::1
    TE APP LINK ATTR    :
      SABML-flag:Non-Legacy SABM-flags: S 
        MaxLink BW: 10000000 kbps
    LAN-Adj-SID: 01 10 00 00 00 01  Flags:v4VL Weight:0 Label:524275
  TE IP Reach   :
    Default Metric  : 10
    Control Info:    , prefLen 24
    Prefix   : 192.168.0.0
    Default Metric  : 10
    Control Info:    , prefLen 24
    Prefix   : 192.1.1.0
    Default Metric  : 10
    Control Info:    , prefLen 24
    Prefix   : 10.1.5.0
    Default Metric  : 10
    Control Info:    , prefLen 24
    Prefix   : 10.1.21.0
    Default Metric  : 0
    Control Info:   S, prefLen 32
    Prefix   : 1.0.0.1
    Sub TLV   :
      Prefix-SID Index:11, Algo:0, Flags:NnP
  IPv6 Reach:
    Metric: ( IS) 0
    Prefix   : fd00::1/128
    Sub TLV   :
      Prefix-SID Index:71, Algo:0, Flags:NnP
    Metric: ( I ) 10
    Prefix   : fd00:10:1:2::/64
    Metric: ( I ) 10                 
    Prefix   : fd00:10:1:11::/64
    Metric: ( I ) 10
    Prefix   : fd00:10:5:1::/64
Level (2) LSP Count : 1

Enabling SR-TE LSP with IPv6

            lsp "tope3-ipv6" sr-te
                to fd00::3
                path-computation-method local-cspf
                metric-type te
                max-sr-labels 5 additional-frr-labels 2
                primary "local-cspf"
                exit
                no shutdown
            exit
            no shutdown

 

*A:PE5# show router mpls sr-te-lsp family ipv6 detail

=====================================================================

MPLS SR-TE LSPs (Originating) (Detail)

=====================================================================

Legend : + - Inherited

=====================================================================

Type : Originating

---------------------------------------------------------------------

LSP Name   : tope3-ipv6

LSP Type        : SrTeLsp                   LSP Tunnel ID        : 3

LSP Index       : 65538                     TTM Tunnel Id        : 655364

From            : fd00::5                  

To              : fd00::3

Adm State       : Up                        Oper State           : Up

LSP Up Time     : 0d 00:06:50               LSP Down Time        : 0d 00:00:00

Transitions     : 1                         Path Changes         : 1

Retry Limit     : 0                         Retry Timer          : 30 sec

Hop Limit       : 255                       Negotiated MTU       : 8902

PathCompMethod  : local-cspf               

FallbkPathComp  : not-applicable           

Metric          : N/A                       Metric Type          : te

Local Sr Protec*: preferred                 Label Stack Reduction: Disabled

Load Bal Wt     : N/A                       ClassForwarding      : Disabled

Include Grps    :                           Exclude Grps         :

None                                           None

Egress Stats    : Disabled                 

BFD Template    : None                      BFD Ping Intvl       : N/A

BFD Enable      : False                     BFD Failure-action   : None

WaitForUpTimer  : 4                        

ReturnPathLabel : None                     

BFD Source Addr : None

 

Revert Timer    : Disabled                  Next Revert In       : N/A

Entropy Label   : Enabled+                  Oper Entropy Label   : Enabled

Negotiated EL   : Disabled                  Override Tunnel ELC  : Disabled

VprnAutoBind    : Enabled                  

IGP Shortcut    : Enabled                   BGP Shortcut         : Enabled

IGP LFA         : Disabled                  IGP Rel Metric       : Disabled

AllowSrOverSrte : Disabled                 

BGPTransTun     : Enabled                  

Oper Metric     : 16777215                 

PCE Report      : Disabled+                

PCE Control     : Disabled                 

Max SR Labels   : 5                         Additional FRR Labels: 2

Binding SID     : 0                        

Path Profile    : None                     

Admin Tags      : None                     

Primary(a)      : local-cspf

                                            Up Time              : 0d 00:06:51

Bandwidth       : 0 Mbps                   

=====================================================================

* indicates that the corresponding row element may have been truncated.

*A:PE5#

 

*A:PE5# show router mpls sr-te-lsp family ipv6 path detail

 

=====================================================================

MPLS SR-TE LSP

Path  (Detail)

=====================================================================

Legend :

    S      - Strict                      L      - Loose

    A-SID  - Adjacency SID               N-SID  - Node SID

    +      - Inherited

=====================================================================

LSP SR-TE tope3-ipv6

Path  local-cspf

---------------------------------------------------------------------

LSP Name    : tope3-ipv6

Path LSP ID      : 13824                  

From             : fd00::5                

To               : fd00::3                

Admin State      : Up                      Oper State        : Up

Path Name   : local-cspf

Path Type        : Primary                

Path Admin       : Up                      Path Oper         : Up

Path Up Time     : 0d 00:07:11             Path Down Time    : 0d 00:00:00

Retry Limit      : 0                       Retry Timer       : 30 sec

Retry Attempt    : 0                       Next Retry In     : 0 sec

 

PathCompMethod   : local-cspf              OperPathCompMethod: local-cspf

MetricType       : te                      Oper MetricType   : te

LocalSrProt      : preferred               Oper LocalSrProt  : preferred

LabelStackRed    : Disabled                Oper LabelStackRed: Disabled

 

Bandwidth        : No Reservation          Oper Bandwidth    : 0 Mbps

Hop Limit        : 255                     Oper HopLimit     : 255

Setup Priority   : 7                       Oper SetupPriority: 7

Hold Priority    : 0                       Oper HoldPriority : 0

DelayMetricLimit : No limit                OperDelayMetricLim: N/A

Inter-area       : N/A                    

 

PCE Updt ID      : 0                       PCE Updt State    : None

PCE Upd Fail Code: noError

 

PCE Report       : Disabled+               Oper PCE Report   : Disabled

PCE Control      : Disabled                Oper PCE Control  : Disabled

 

Include Groups   :                         Oper IncludeGroups:

None                                           None

Exclude Groups   :                         Oper ExcludeGroups:

None                                           None

Last Resignal    : n/a                    

 

IGP/TE/Del Metric: 40                      Oper Metric       : 16777215

Oper MTU         : 8902                    Path Trans        : 1

Degraded         : False                  

Failure Code     : noError

Failure Node     : n/a

Explicit Hops    :                        

    No Hops Specified

Actual Hops      :                        

    fd00:10:5:1::1

    (fd00::1)(A-SID)

    Record Label        : 524281

    fd00:10:1:11::11

 -> (fd00::11)(A-SID)

    Record Label        : 524276

    fd00:10:13:11::13

 -> (fd00::13)(A-SID)

    Record Label        : 524285

    fd00:10:13:3::3

 -> (fd00::3)(A-SID)

    Record Label        : 524286

 

BFD Configuration and State

Template         : None                    Ping Interval     : N/A

Enable           : False                   State             : notApplicable

ReturnPathLabel  : None                   

BFD Source Addr  : None

WaitForUpTimer   : 4 sec                   OperWaitForUpTimer: 0 sec

WaitForUpTmLeft  : 0                      

StartFail Rsn    : N/A

 

=====================================================================