See What Gets Dropped: Introducing sFlow Mirror-On-Drop In SONiC 202611

1. The gap

Interface counters report that packets were discarded. They don’t tell you which packets were lost, where they were dropped, or why. In an AI factory, where distributed compute depends on network connectivity across large clusters, that lack of visibility makes troubleshooting packet loss harder. A counter can tell you that drops occurred; identifying the packets and the point of failure is what helps engineers determine the cause and the appropriate remediation.

Vendor-specific drop telemetry has existed for years — NVIDIA’s What Just Happened is the best-known example — but it’s proprietary to one ASIC family and one export path. SONiC runs on Marvell, Broadcom, Cisco, NVIDIA and Innovium silicon; a solution tied to one of them isn’t a SONiC solution.

Vendor drop telemetry also stops at the ASIC. A meaningful share of production packet loss happens after the packet has been delivered to the CPU — in the kernel’s receive path, in the backlog queue, in socket buffers — and no ASIC-side feature can see it.

SONiC 202611 ships an open-standard answer: per-packet drop notifications carried in sFlow, implemented through SAI, with the ASIC driver as the only platform-specific component.

2. What ships

Component	Where	Status
HLD	sonic-net/SONiC doc/sflow/sflow_hld.md v1.4	Merged via PR #1786
Orchestration	sfloworch in the swss container	Merged
Management daemon	sflowmgrd	Merged
Agent	InMon hsflowd, modules mod_sonic and mod_dropmon	Upstream
CLI, YANG, DB schema	sonic-utilities, sonic-buildimage	Merged
Wire format	sFlow Dropped Packet Notification Structures, sflow.org/sflow_drops.txt	Published spec
ASIC support	Marvell Teralynx	Validated

The sFlow v5 datagram carries two measurement types: sampled packet headers and interface counters. The sflow_drops extension adds a third — a discard record carrying the packet header, ingress interface and a drop reason, for every packet the device discards. It’s a notification, not a sample: the record exists because a drop occurred, not because a sampler fired.

Four drops, as a counter reports them and as a notification reports them:

What the counter shows	What the notification shows
leaf-2 Ethernet12 rx_discards +4,112 over 5 min	4,112 records: 10.1.4.7 → 10.1.9.2 udp/4791, ingress Ethernet12, group acl_drops, trap ingress_flow_action_drop, 02:14:07–02:19:03. The RoCEv2 flow a new ACL just blocked.
spine-1 Ethernet48 tx_discards +218	218 records: destinations in 10.1.9.0/24, ingress Ethernet3, group l3_drops, tv2rap blackhole_route. A withdrawn prefix, not congestion.
leaf-3 Ethernet7 rx_discards +1,020	1,020 records: VLAN tag 300, ingress Ethernet7, group l2_drops, trap ingress_vlan_filter. VLAN 300 is allowed on the far trunk and not on this one.
/proc/net/softnet_stat column 2, CPU 1, +37	37 records: 10.0.0.5 → 10.0.0.12 tcp/179, origin SW, reason cpu_backlog, symbol enqueue_to_backlog. BGP keepalives to one neighbour, dropped in the kernel at the moments the session reset.

The left column is the same number four times with a different label. The right column is four different problems, each with the flow, the place and the cause attached.

3. Data path

Two sources, one channel, one consumer.

Hardware drops. The ASIC raises a drop event to its driver with the packet and device-specific metadata. The driver encapsulates it as a generic netlink message on family NET_DM, multicast group events, and attaches standard attributes:

NET_DM_ATTR_IN_PORT             ifindex + netdev name
NET_DM_ATTR_ORIGIN              NET_DM_ORIGIN_HW
NET_DM_ATTR_HW_TRAP_NAME        e.g. acl_drop, blackhole_route
NET_DM_ATTR_HW_TRAP_GROUP_NAME  e.g. acl_drops, l3_drops
NET_DM_ATTR_TIMESTAMP
NET_DM_ATTR_PAYLOAD             truncated header
NET_DM_ATTR_ORIG_LEN, NET_DM_ATTR_TRUNC_LEN

The driver’s job is translation: device-specific drop metadata becomes the kernel’s standard attribute set. hsflowd never sees anything vendor-specific.

Software drops. The Linux drop_monitor module already publishes kernel-stack discards on the same family and group, with NET_DM_ATTR_ORIGIN = NET_DM_ORIGIN_SW plus NET_DM_ATTR_REASON, NET_DM_ATTR_PC and NET_DM_ATTR_SYMBOL identifying the drop site.

Consumer. hsflowd’s mod_dropmon opens one netlink socket on NET_DM/events, decodes both origins, and encodes each as an sflow_drops record. mod_sonic’s receive path injects it into the packet thread, where it’s batched into sFlow datagrams alongside flow samples and counters. Separate genetlink sockets are used for psample (flow samples) and NET_DM (drops).

3.1 Why one channel

Reusing the kernel’s NET_DM for hardware drops isn’t just convenient. Consider a packet with an expiring TTL. The ASIC traps it to the CPU; the kernel generates ICMP Time Exceeded and discards the original. If hardware drops had their own channel, the driver would have to decide whether a trap-to-CPU counts as a drop — and either double-report or miss it. With a unified channel the rule is simple: the driver reports packets the ASIC discards, the kernel reports packets it discards, and a trapped-then-delivered packet is reported exactly once, by whichever layer actually dropped it, tagged with its origin.

From the field: three weeks of BGP flaps, one invisible counter

Earlier this year we chased intermittent BGP session flaps across a production fabric for close to three weeks. Interface counters were clean. The ASIC reported no drops. Keepalives were leaving the peer and never reaching FRR.

The cause was softnet backlog drops — the kernel discarding packets at enqueue_to_backlog because the softirq path couldn’t keep up with control-plane traffic during bursts. The only evidence was the second column of /proc/net/softnet_stat, a per-CPU counter with no packet, no interface, no reason and no timestamp. It took weeks to connect that number to the flaps because nothing linked a dropped packet to the BGP session it belonged to.

Under this feature, those drops appear on the same NET_DM channel as hardware drops, tagged NET_DM_ORIGIN_SW, with the kernel drop reason (SKB_DROP_REASON_CPU_BACKLOG on current kernels), the drop site (NET_DM_ATTR_SYMBOL) and the packet header. A collector would have shown TCP/179 packets to a specific neighbour being discarded at a specific kernel function, at the timestamps the sessions reset. That’s a one-hour investigation, not a three-week one.

This is the case for the unified channel. A drop-visibility feature that only covers the ASIC would have reported nothing here — because the ASIC did nothing wrong.

4. Control plane

sflow drop-monitor-limit 50

1. CLI validates against STATE_DB (SWITCH_CAPABILITY|switch, MIRROR_ON_DROP_CAPABLE) and writes CONFIG_DB SFLOW|global, drop_monitor_limit.
2. sflowmgrd re-validates capability, writes APP_DB SFLOW_TABLE, and regenerates /etc/hsflowd.conf — the limit lands as dropmon.limit. Config changes restart hsflowd.
3. sfloworch consumes APP_DB and programs SAI.

4.1 SAI object model

On a capable platform sfloworch creates:

Object	Attributes	Purpose
TAM event	SAI_TAM_EVENT_TYPE_PACKET_DROP	Tell the ASIC to raise drop events
TAM object	attached via SAI_SWITCH_ATTR_TAM_OBJECT_ID	Bind the event to the switch
Hostif	SAI_HOSTIF_TYPE_GENETLINK, name NET_DM, mcgroup events	The delivery channel
User-defined trap	SAI_HOSTIF_USER_DEFINED_TRAP_TYPE_TAM, own trap group	Classify drop events as a trap class
Policer on the trap group	rate from drop_monitor_limit	Rate limit (6)
Hostif table entry	maps the UDT to the NET_DM hostif	Route trapped drops to the channel

4.2 Capability detection

The feature is gated at runtime. A platform is MOD-capable only if all of the following SAI capability queries succeed:

enum capability   SAI_TAM_EVENT_ATTR_TYPE                 contains SAI_TAM_EVENT_TYPE_PACKET_DROP
enum capability   SAI_HOSTIF_USER_DEFINED_TRAP_ATTR_TYPE  contains SAI_HOSTIF_USER_DEFINED_TRAP_TYPE_TAM
attr capability   SAI_SWITCH_ATTR_TAM_OBJECT_ID
attr capability   SAI_TAM_COLLECTOR_ATTR_HOSTIF_TRAP
attr capability   SAI_TAM_ATTR_EVENT_OBJECTS_LIST
attr capability   SAI_HOSTIF_USER_DEFINED_TRAP_ATTR_TRAP_GROUP

switchorch writes the result to STATE_DB:

"SWITCH_CAPABILITY|switch": { "value": { "MIRROR_ON_DROP_CAPABLE": "true" } }

On an unsupported platform the CLI rejects the config. There is no silent no-op.

5. Notification format

5.1 Drop reason vocabulary

Trap names and groups follow the Linux kernel’s devlink trap definitions, so a reason means the same thing on SONiC as on a Linux host and existing sflow_drops decoders need no changes. Representative entries:

Group	Trap names
l2_drops	source_mac_is_multicast, vlan_tag_mismatch, ingress_vlan_filter, ingress_spanning_tree_filter, port_list_is_empty, port_loopback_filter
l3_drops	blackhole_route, ttl_value_is_too_small, non_ip, ip_header_corrupted, unresolved_neigh, ipv4_lpm_unicast_miss, ipv6_lpm_unicast_miss, reject_route, blackhole_nexthop
tunnel_drops	decap_error, overlay_smac_is_mc
acl_drops	ingress_flow_action_drop, egress_flow_action_drop
buffer_drops	tail_drop, early_drop

Vendors may add names within the standard groups. The Teralynx implementation reports L3 drops as group l3_drops, trap user_l3 — the group carries the semantics a collector classifies on, the name carries the platform detail.

5.2 On the wire

mod_dropmon encodes each notification as an sflow_drops discard record: a drop reason, the input interface, the output interface where known, a running discard count, and the packet header as a standard sFlow flow record. The header is truncated per TRUNC_LEN; ORIG_LEN preserves the original size. A collector that already decodes sFlow flow samples decodes the header identically — the 5-tuple, VLAN, DSCP, and for RoCEv2 the UDP destination port 4791 and the BTH opcode are all available.

6. Rate limiting

drop-monitor-limit accepts 0–500 notifications per second per switch; 0 disables. The limit is enforced by the policer on the TAM trap group through the existing COPP path, not by hsflowd.

Two consequences worth being precise about:

• Notifications are not sampled. Below the limit, every drop produces a record. Above it, excess notifications are discarded by the policer. The collector sees a hard ceiling, not a statistical thinning, and shouldn’t scale counts upward as it would for flow samples.
• The limit protects the CPU path. Forwarding is unaffected either way.

Choosing a value: on a healthy fabric the steady-state drop rate is near zero, so even a low limit captures everything. During a burst, the first N drops per second are reported with full context — usually enough to identify the flow and reason — and the counter still records the total. Fifty is a reasonable starting point; raise it on CPU-rich platforms if post-incident analysis needs more depth.

7. Configuration and verification

sflow collector add ncp-collector 192.0.2.129 --vrf mgmt
sflow sample-direction both
sflow drop-monitor-limit 50
sflow enable

show sflow
  sFlow Admin State:              up
  sFlow Sample Direction:         both
  sFlow Drop Notification Limit:  50

To confirm the path end to end: verify STATE_DB shows MIRROR_ON_DROP_CAPABLE: true; confirm /etc/hsflowd.conf contains dropmon { limit = 50 } after sflowmgrd regenerates it; inject a known drop (an ACL deny on a test flow is the simplest); and decode the resulting sflow_drops record at the collector. sflowtool prints it in the clear.

8. Validated behaviour

On Marvell Teralynx 6512, the following drop classes have been exercised with notification, reason, header, switch and interface identity confirmed at the collector:

• Egress queue congestion (buffer_drops)
• ACL drops (acl_drops)
• Route drops (l3_drops)
• Buffer drops
• Policer drops

Validation used a four-switch fabric (two spine, two leaf, all Teralynx 6512) with an Ixia generator producing deterministic flows, so each injected fault — an ACL deny on one leaf, a withdrawn route at a spine, a VLAN missing from one trunk — produced loss on a known flow set that could be checked against the notifications. The three fault classes map to acl_drops, l3_drops and l2_drops respectively, and each was distinguishable from the others by trap group alone before any header inspection.

9. What the correlation buys you

Flow samples describe traffic that transited the fabric. Drop notifications describe traffic that didn’t. On a shared timeline:

• tail_drop on an egress queue, timed against sampled flow volume on that port, tells you whether ECN marking engaged early enough.
• Buffer drops with the packet’s priority field in the header tell you which PFC class exhausted headroom.
• acl_drops and l3_drops on a RoCEv2 flow (UDP/4791) separate policy and routing faults from congestion before anyone looks at a queue counter.

Network Copilot consumes the collector’s decoded records alongside interface counters and flow samples and answers the operational questions directly: which flows, which switch and port, which reason, whether the fault is isolated or fabric-wide. That’s the application layer; the point of this post is that the data now exists in an open format for any application to use.

10. Operational notes

• Software drops are on by default. mod_dropmon reports kernel drop_monitor events regardless of ASIC support, so the softnet, socket-buffer and netfilter drop classes are visible even on platforms that haven’t implemented the SAI path yet. If you take nothing else from this release, take that.
• Warm boot: sampling and notifications pause during a planned warm reboot; verify they resume.
• Config changes restart hsflowd. Expect a brief gap in all three sFlow measurement types on every change, not only drop-related ones.
• /etc/hsflowd.conf is generated. Don’t edit it by hand.
• Header truncation is set by the driver via TRUNC_LEN. Confirm it’s long enough to reach the BTH header on RoCEv2 (Ethernet + IPv4/v6 + UDP + 12 bytes) if you intend to classify by opcode.
• COPP interaction: the drop trap group’s policer is independent of the samplepacket trap group’s. Size both.
• CPU budget: each notification is a netlink message and an sFlow record. At 500/s on a modest control-plane CPU this is measurable; at 50/s it isn’t.

11. Enabling another ASIC

Everything from sfloworch upward is platform-independent and merged. A vendor adds support by:

1. Returning the enums and attributes in 4.2 from SAI capability queries.
2. Implementing the TAM packet-drop event so that discarded packets reach the driver with reason metadata.
3. Publishing each drop to genetlink NET_DM/events with the attribute set in 3, using devlink trap vocabulary for HW_TRAP_NAME and HW_TRAP_GROUP_NAME.
4. Honouring the policer on the TAM trap group.

Platforms that already implement devlink traps under upstream Linux have most of step 3 done. The Marvell implementation is the reference; the HLD is the specification; the mailing list and sonic-net/SONiC are where the work happens.

References

• HLD: sonic-net/SONiC/doc/sflow/sflow_hld.md, v1.4 — Madhu Paluru (Aviz Networks), Peter Phaal (InMon), Ravindranath C K (Marvell), Junying Yeh (Edgecore)
• PR: sonic-net/SONiC#1786, “Dropped Packet Notification (MOD) support”
• sFlow Dropped Packet Notification Structures: sflow.org/sflow_drops.txt
• InMon: SC24 dropped packet visibility
• host-sflow: github.com/sflow/host-sflow, src/Linux/mod_sonic.c, mod_dropmon.c
• Linux drop_monitor: include/uapi/linux/net_dropmon.h; trap vocabulary: Documentation/networking/devlink/devlink-trap.rst
• SAI TAM: opencomputeproject/SAI, doc/TAM
• Network Copilot - Agentic AI platform: https://aviznetworks.gitbook.io/ncp