Why Do AI Fabrics Need Two Operational Data Paths: Streaming Telemetry and Fast Notification?

TL;DR

AI fabrics need two operational data paths: streaming telemetry for continuous state, history and baselines, and fast notification for urgent events that need action now. A congestion event in an AI fabric can last two seconds and clear before a dashboard refreshes, while still stalling a training job or wasting GPU-hours. Correlating both paths with topology and policy turns a single alert into an evidence-backed incident with a next step and proof it worked. Aviz ONES correlates streaming telemetry with topology and device state today, and is building toward a fast notification lane in that same layer.

Introduction

Network monitoring used to have one job: show operators what the network looks like. Collect counters, graph utilization, alert on thresholds, investigate when something looks off. That model still works for most infrastructure.

AI fabrics break it.

A congestion event can last two seconds. A queue can build and clear before a dashboard refreshes. A single degraded link can stall a distributed training job, spike inference latency, or silently waste GPU-hours across a cluster, and be gone by the time anyone looks at a graph.

The fix isn't a faster dashboard. It's recognizing that AI fabric operations need two distinct data paths doing two distinct jobs:

1. Streaming telemetry: continuous state, history, and baselines.
2. Fast notification: urgent events that need action now, not context later.

One path tells you what the network looks like over time. The other tells you what just happened and what to do about it. Most platforms only really built the first one.

How is streaming telemetry different from fast notification?

Streaming telemetry is the operational record of the fabric, built for breadth and history. Fast notification is a separate lane for urgent events, built to get the right data to the right place fast. AI fabrics need both, correlated with topology and policy.

What does streaming telemetry solve for AI fabric monitoring, and what does it miss?

Streaming telemetry, delivered through mechanisms like gNMI Subscribe, gives AI fabric teams the operational record of the network: counters, queue statistics, optics health, routing state and historical baselines. What it misses is speed on urgent events. Because it optimizes for breadth and history, a microburst or queue breach can surface several dashboard refreshes too late.

Streaming telemetry, via gNMI's Subscribe RPC and similar mechanisms, was a genuine step up from polling. Instead of asking every device for everything on an interval, systems subscribe to state and get pushed updates. That's the right foundation for:

• interface counters, queue statistics, optics health
• routing state, device CPU/memory, link utilization
• packet drops, flow trends, server and NIC metrics
• historical baselines for capacity planning and anomaly detection

This is the operational record of the network, and every serious platform needs it. But it optimizes for breadth and history, not for the handful of seconds where something urgent is happening and someone needs to know immediately.

A microburst isn't just a utilization sample. A queue threshold breach isn't just another counter tick. Treating every operationally significant event as one more row in a bulk telemetry stream means it gets seen: eventually, averaged in, several dashboard refreshes later. For AI workloads, "eventually" is the whole problem.

Why are AI training and inference workloads so sensitive to network problems?

Traditional applications tolerate network noise. Distributed AI training and inference don't, because the network isn't underneath the application anymore. It's part of the application's performance envelope. Thousands of GPUs coordinate over tight collective-communication windows; a hiccup in one path doesn't degrade gracefully, it stalls the whole job.

Events that matter disproportionately here:

• queue buildup during collective communication (allreduce, etc.)
• packet drops causing retransmission or tail-latency spikes
• ECN/PFC-related congestion
• link flaps or partial degradation
• optical pre-failure signals
• path changes mid-job
• telemetry gaps from a collector or device going dark
• DCI degradation hitting distributed inference
• NIC issues that present as "network" problems

There's an active body of work in the IETF around exactly this gap. The Fast Network Notifications (FANN) effort, still in draft, argues that AI/ML workloads' bandwidth, latency, jitter, and loss requirements expose real limitations in how fast existing mechanisms disseminate operationally significant events. Their framing is useful even independent of where the draft ends up standards-wise: modern forwarding hardware can often detect a problem in microseconds to low milliseconds; getting that information to a system that can act on it typically takes far longer. That gap, not the absence of data, but the latency of getting the right data to the right place, is the actual operational problem.

What is fast notification in AI network operations?

Fast notification isn't a replacement for streaming telemetry. It's the complement.

Streaming telemetry answers: what is the current and historical state of the fabric? Fast notification answers: what urgent thing just happened, where, who needs to know, and what should happen next?

A notification that's actually useful carries more than "alert fired." It needs:

• event type and location
• severity and affected scope (device, link, queue, port, path, service)
• timestamp and confidence/source
• recommended consumer and required urgency
• a correlation key so downstream telemetry can be matched back to it

And the consumer isn't always a human. A network controller might reroute. A workload scheduler might avoid placing a job on an impaired segment. A remediation system might open an incident automatically. Treat this as its own operational lane, not a second alerting system bolted onto the first.

How do streaming telemetry and fast notification work together during an incident?

Fast notification flags the urgent event, and streaming telemetry plus topology supply the context: the trend, the affected workloads, the history and the policy. In the example below, one queue congestion alert on leaf switch L14 becomes a policy-approved reroute, with telemetry confirming recovery within 90 seconds.

Here's what that looks like end to end, on a single event:

A fast notification fires: queue congestion on port 12 of leaf-switch L14.

On its own, that's a blip. Correlated with streaming telemetry and topology, it becomes an incident:

• This queue had already been trending upward for the last six minutes.
• L14 connects to eight GPU nodes currently running two active training jobs.
• The same symptom appeared on a neighboring leaf twice in the last hour. This may not be isolated.
• Policy says: automatic reroute is allowed for this severity; a scheduler pause is not. That needs approval.
• After the reroute executes, telemetry confirms the queue depth returns to baseline within 90 seconds.

That's the difference between "a graph moved" and "here's what happened, what it affected, what we did, and proof it worked." Streaming telemetry alone gives you the first sentence. Only the correlation of both paths gets you the rest.

See how ONES correlates streaming telemetry with topology and device state for AI fabrics today: explore ONES for AI factory networking.

Why are dashboards not enough for AI network operations?

A dashboard is genuinely good at visibility. It's the wrong tool for the questions that matter most once something urgent happens:

• Which workload was actually affected?
• Was this a two-second blip or an ongoing condition?
• Did the network event precede the application slowdown, or follow it?
• Is this severe enough to act on automatically, or does it need a human?
• What's the safe remediation, and how do we verify it worked?

In AI environments the cost of not answering these fast is real money: wasted accelerator-hours, degraded user-facing inference, unnecessary application-level retries cascading through the system. That's why the next stage of AI network operations tools will be built around event-aware, topology-aware, policy-aware workflows, not bigger dashboards.

What architecture does an AI fabric monitoring platform need?

Don't build one pipeline and ask it to serve both jobs. A platform built for AI fabrics needs:

• high-volume streaming collectors for state and history
• a separate, low-latency ingestion path for urgent events
• normalized schemas across both, so they can be correlated at all
• topology and workload correlation
• time-windowed incident objects (not raw alert streams)
• policy-based remediation with clear auto/approve/block tiers
• verification after every action, and audit history for trust and compliance

This also changes what an "alert" is. It stops being a threshold crossing and becomes an evidence-backed incident: device state, topology context, recent telemetry, affected services, and a recommended next step, bundled together.

How does Aviz ONES support AI fabric observability today, and where is it headed?

Today, ONES correlates streaming telemetry (device, queue, optics, and flow data collected via gNMI and related sources) with topology and device state to give operators a real-time and historical view of the fabric.

The direction we're building toward is folding a fast-notification lane into that same correlation layer, so urgent events get matched against topology, active workloads, and remediation history automatically, rather than requiring an operator to manually connect an alert to a graph. Longer-term, that same foundation supports more automated workflows: detect, correlate, recommend, approve, execute, verify, roll back.

The goal isn't collecting more data. It's making sure the right data reaches the right decision at the right moment, whether that decision is made by a person or by the platform itself.

Conclusion

For teams running AI training and inference fabrics, one telemetry pipeline is not enough. Streaming telemetry shows what the network looks like over time, and fast notification shows what just happened and what to do about it. Correlating both with topology, workloads and policy is what turns an alert into an incident you can act on and verify.

If you are building GPU infrastructure as a neocloud or private AI cloud, see how Aviz ONES for AI factory networking correlates telemetry and topology across your fabric, or talk to the Aviz team about your AI fabric operations.

See how Aviz ONES correlates streaming telemetry with topology and device state to give your team a real-time and historical view of your AI fabric. Explore ONES for AI factory networking.