Why Does Adding a Third Spine Break ECMP in SONiC and SAI?
May 12, 2026
ONES & SONiC
TL;DR
ECMP worked perfectly with two spines, but failed when a third spine was introduced during lab validation. The issue traced back to how SAI handled next-hop group replacement during ECMP scale-up, causing group programming failures and orchestrator crashes.
Introduction
In modern data center networks, Equal-Cost Multi-Path (ECMP) routing is the backbone of efficient traffic distribution. It enables multiple parallel paths to carry traffic evenly, improving both resiliency and utilization.
Recently, during a lab validation, we ran into an unusual issue where ECMP stopped working when a third spine was added to the topology. Everything functioned flawlessly with two spines but as soon as the topology expanded, forwarding consistency broke and we saw a crash. What followed was a deep debugging exercise that revealed a subtle bug at the SAI (Switch Abstraction Interface) layer.
What Happens to Next-Hop Groups When the Topology Changes?
In SONiC, ECMP is implemented through next-hop groups. Each group represents a set of equal-cost forwarding paths.
When the network topology changes say, by adding a new spine SONiC dynamically updates these groups to include the additional paths.
leaf01 NOTICE swss#orchagent: :- addNextHop: Created next hop 169.254.0.1 on Ethernet0
leaf01 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group
169.254.0.1@Ethernet0,169.254.0.1@Ethernet16,169.254.0.1@Ethernet176For example:
- With 2 spines → ECMP group has 2 next hops.
- Add a 3rd spine → SONiC replaces the existing group with a new 3-path group.
This replacement process relies heavily on the SAI layer, which interfaces between SONiC and the underlying hardware SDK.
What Was the Network Topology Used During Testing?
The test setup was simple:
- Topology: 3 Spines + 4 Leafs
- Environment: Lab (SONiC-based fabric)
- Traffic: IPv4 routes advertised from spines to leafs via BGP EVPN
Why Did ECMP Fail After Adding the Third Spine?
After basic debugging on the BGP and control plane, next, we looked into the dataplane programming, focusing on how the ECMP groups were being handled inside the switch.
Using logs from swss and syncd, we noticed something interesting:
- When the third spine was added, SONiC attempted to replace the existing 2-path ECMP group with a new 3-path group.
- The operation sequence involved deleting the old next-hop group and creating a new one with an additional member.
- However, the SAI call for group creation failed it couldn’t add the 3rd member successfully.
leaf01 ERR swss#orchagent: :- remove: remove status: SAI_STATUS_OBJECT_IN_USE
leaf01 ERR swss#orchagent: :- removeNextHopGroup: Failed to remove next hop group 500000000072b, rv:-17
leaf01 ERR swss#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting
orchagent, SAI API: SAI_API_NEXT_HOP_GROUP, status: SAI_STATUS_OBJECT_IN_USEWhat Was the Root Cause Behind the ECMP Failure?
The root cause pointed to an SAI-level implementation issue in handling next-hop group updates.
When transitioning from 2 → 3 ECMP paths, SAI failed to allocate or program the new group entry properly, possibly due to resource handling or ordering logic in the underlying SDK.
Interestingly, this behavior was platform-specific, observed only on that platform when operating in the spine role. The same ECMP logic and test sequence worked flawlessly in a smaller 2-spine setup with the same software version.
This made the issue both rare and hard to reproduce, only surfacing in a specific scale-up condition.
What Key Lessons Did This ECMP and SAI Issue Reveal?
Incremental scale testing matters – Bugs like this often hide in transitional states not at full scale, but when scaling to a new level.
SAI behavior is critical for multi-vendor interoperability – SONiC’s abstraction model relies on consistent SAI implementations, and even minor inconsistencies can cascade into major dataplane issues.
Hardware abstraction is powerful and tricky – A small logic flaw in SAI can silently impact how routes or next-hops are programmed, despite control-plane stability.
Conclusion
Debugging this issue was a great reminder of how deep the interaction runs between software abstraction and hardware behavior in modern network operating systems.
What looked like a simple ECMP scale-up issue turned into an exploration of the fine line between SONiC orchestration and SAI implementation.
If you’ve come across similar ECMP or next-hop group anomalies in SONiC or SAI, we’d love to compare notes or collaborate on reproductions. Drop a comment or connect these are the kinds of challenges that make network engineering both humbling and fascinating.
Frequently Asked Questions
1. What is the Swan Principle and why does it matter for business leaders?
A swan looks calm and effortless on the water. What you do not see is the legs working hard underneath. The Swan Principle says the same thing happens in business. The visible outcome everyone admires is created by invisible preparation. Companies that look like they are gliding smoothly through change are the ones doing hard work before anyone else notices the wave coming.
2. Why is it risky for infrastructure leaders to wait before preparing for the AI wave?
When you are on top, you can absorb change on your own terms. Wait too long and you are forced to catch up on someone else's terms. AI is creating that wave right now. The companies that move smoothly into the AI era are not waiting for the water to get rough. They are preparing while the glide still looks smooth.
3. What hidden work should infrastructure leaders be investing in right now for AI readiness?
Five things come up directly: modernize the network, improve telemetry, automate operations, prepare teams, and build AI-ready infrastructure. None of these show up on a press release. All of them determine whether the organisation glides into the AI era or gets caught scrambling when the shift becomes impossible to ignore.
4. How does Dell moving early with the Dell AI Factory with NVIDIA connect to the Swan Principle?
Dell was already doing well. There was no obvious pressure to change. They moved anyway. Michael Dell put it simply: their job is to make AI more accessible. Investing early while things were still going smoothly is exactly what the Swan Principle describes. The work happened underneath while the surface still looked calm.
5. What question should executives be asking themselves about AI readiness today?
The book puts it plainly: what hidden work are you investing in today before the next wave makes it visible? Not what looks good in a board deck. What is actually being built underneath while the surface still looks calm. That is the question that separates companies that glide through the AI era from the ones that get forced to catch up on someone else's terms.
Learn more about real-world SONiC, SAI, and data center networking challenges by exploring similar deep-dive debugging scenarios and scale validation experiences.
