End-to-End Tenant Isolation: ONES Integrates with NVIDIA UFM

TL;DR

ONES Fabric Manager simplifies the automation of tenant isolation across Ethernet (North–South) and InfiniBand (East–West) fabrics, via integration with NVIDIA UFM. With automated handling of PKeys, VLAN/VNI provisioning, lifecycle management, and API/UI integration, operators are able to provision AI networking in a secure manner, all while significantly reducing errors and the burden of repetitive tasks.

Introduction

Modern Artificial Intelligence (AI) and High Performance Computing (HPC) workloads rely heavily on the network. Some AI training workloads can use hundreds of GPUs with multiple terabytes of RDMA (Remote Direct Memory Access) traffic. Every microsecond of latency can impact the outcome of such workloads. Innovations in high performance fabrics have dominated the industry while addressing the challenges of management and manual isolation of tenants and lifecycle/orchestration solutions. ONES bridges that gap for NVIDIA’s UFM (Unified Fabric Manager) integration by offering automation for both traffic planes.

Why Is AI Networking Becoming More Challenging?

High speed computing (HPC) and artificial intelligence (AI) workloads are highly dependent on networks. An AI training job can use a hundreds of GPUs and send thousands of terabytes worth of requests through Remote direct memory access (RDMA). Each microsecond matters. A lot of work has been completed on the performance of networks, especially in the East-West direction, with InfiniBand and NVLink, GPUDirect RDMA, and optimized networks. These things have made possible networks that can be used to process gradients at the maximum possible speed.

Up until now, the other side of the equation has lagged. In this area, some of the issues that lack automation include partitioning, tenant isolation, and lifecycle management of the East-West fabric. Meanwhile, Ethernet and the North-South side has had this automation for a while now.

Operators have been asking the same question: Why can’t I manage both planes of my AI fabric with the same level of automation? This question has been answered with the addition of the latest product to the ONES line.

What Is NVIDIA UFM?

NVIDIA UFM (Unified Fabric Manager) is the premier management platform for InfiniBand fabrics. These are the high speed fabrics that connect the largest AI training clusters and supercomputers. If we envisioned InfiniBand Fabrics as the ‘nervous system’ connecting thousands of GPUs, then we envision UFM as the ‘brain’ seeing, configuring, and maintaining the system.

Some key aspects of UFM:

• Complete Fabric Control & Visibility. UFM maintains a map for every connected host, switch, and port within the InfiniBand Fabric and stores their unique 64-bit GUIDs. This is the map of the physical connection and the definitive record of existence.
• PKey Domain Management. InfiniBand uses 16-bit PKeys, or ‘partitions’, in a similar manner to VLANs to separate traffic among groups of ports. UFM manages these partitions across each switch for you and allows isolation of traffic across 32,766 logical domains per fabric.
• Control over Performance and Congestion. UFM provides performance monitoring and link control in addition to advanced features to manage and control traffic congestion and routing behavior.
• Layout for RESTful API Calls. UFM exposes fabric control and partition management capabilities through a RESTful API. This API calls are authenticated and nowOnes automates these calls for you.
• UFM provides the foundation to manage InfiniBand fabrics and control at scale for AI clusters. What nowOnes adds is the advanced layers of AI cluster management, including self-service, coordination and resource lifecycle management at the cluster fabric layer.

What’s New with UFM Integration in ONES Fabric Manager?

ONES Fabric Manager can now automate the isolation of traffic across the East-West direction using NVIDIA UFM (Unified Fabric Manager) forInfiniBand networks to further simplify the management of hybrid networks. When combined with north-south isolation across Ethernet networks, ONES provides a unified management interface across a true hybrid network to deploy resources with one API or one click in the UI.

ONES uses infiniband partition keys (PKeys), partition keys for the isolation of virtual networks, to offer resource isolation to each of the tenants. When a tenant is created within the ONES platform, the PKey allocation for east-west GPU-to GPU traffic, the Virtual LAN (VLAN) or Virtual Network Identifier (VNI) allocation for north-south traffic, and the necessary wiring are done automatically with rollback, retry, and drift detection capabilities.

With this integration, the burden of managing partition key spreadsheets has been eliminated along with the frustrating partition drift issue where a solution that worked yesterday may have failed today.

Why Does the UFM Integration Matter?

A side-by-side look at the operational impact of enabling the UFM integration:

Aspect	Before UFM Integration	With UFM Integration
Network isolation	Manual partition setup	Automatic PKey allocation per tenant
Provisioning time	20–30 minutes	2–5 minutes
Error rate	15–20% (human error)	< 1% (validated automation)
Security posture	Shared fabric	Tenant-isolated partitions
Compliance	Non-compliant	GDPR / SOC 2 ready
Port flexibility	All ports or none	Per-port granular selection
Traffic plane coverage	Single plane	Dual-plane (N-S + E-W)

The impact is measurable: 10x faster provisioning, 20x fewer configuration errors, and a compliance story that actually holds up to an audit.

What Key Capabilities Does the Integration Provide?

Dual-Plane Tenant Isolation. Each tenant is assigned a VLAN/VNI and PKey - North-South and East-West - with just a single [PATCH /fabrics/{name}/tenants/{tenant}] API call.

Port-Level Granularity. AI servers with multiple InfiniBand ports can have some ports assigned to a specific tenant. Operators can do this on ONES so that on the same host, tenant-ai can use Port 1 and tenant-ml can use Port 2.

Background Fabric Sync. A job is scheduled to run every 30 minutes to check for changes in the fabric. An inventory is synced, and out of ONES control PKey assignments are detected so the fabric does not sync out of compliance.

Proactive Token Refresh. A background process automatically refreshes tokens 80% of the way through their Time-to-Live (TTL). This prevents 401s and failure during tenant provisioning due to requests timing out.

Fail-Safe Defaults. With setting ufm.enabled to false, graceful degradation will be enabled. In this situation, tenants can still perform CRUD operations on the North-South plane. If UFM goes offline, health endpoints will return a degraded state.

API + UI, Same Engine. It is completely the preference of the operator how they engage the integration. It can be engaged through a set of REST APIs, or through the UI flows hosted in ONES via the ONES UI.

Self-Healing & Drift Detection. Inactive background synchronization will not only refresh the inventory; it will check the live state of the UFM partition. It records any PKeys that are externally assigned to the fabric and logs these to prevent the fabric from drifting. Partitions created with other tools will be retained and not overwritten.

Resilient by Design. Circuit breakers allow UFM issues to be contained. You can also configure the failure mode via ufm.strict-isolation. UFM will fail quickly, or you can configure a degraded North-South plane with graceful failure for tenant operations, in addition to protected sync jobs in a 3+ instance HA deployment via ShedLock.

Automated Full Life Cycle. Operations including tenant creation, server placement, modification, and deletion affect both planes. The deletion of a tenant results in the removal of the VLAN and PKey.

How Can You Manage the Fabric Through API and UI?

Previous versions of the integration exposed everything via a REST API. While the REST API is still used and is the preferred option for use at the core of CI/CD, GitOps, and NCP orchestration, ONES now provides a UI for the entire UFM workflow meaning teams that use a console can now have the same functionality without the need for a single curl command. Registrations and validations of fabrics can be done through ONES. With an endpoint of UFM, ONES will validate the credentials and build the PKey pool. There’s no more manual juggling of tokens.

Visual selection of ports. The allocation view displays all discovered hosts and InfiniBand ports with their current state. Operators can assign ports by checking a box.

Dual-plane tenant visibility. A single view of a tenant shows what the tenant owns in both planes. This eliminates the need for using multiple tools.

UI status of the health of the system. Health endpoints exposed by the API are used to show the status of the health of the PKey pool and UFM connectivity and the status of degraded mode.

Health at a glance. The UI shows the status of the health of the system. Because both sides communicate with the same service layer, there is one source of truth. A tenant created through an API call appears the same in the UI, while a port that is clicked through is visible to any automation that queries the API. No drift exists between what a script does and what is shown in the console.

Example of this single source of truth. In the image on the left, a user selects 4 of the 8 GPU ports on node SA-ZT-NDR-01 for the Gold tenant using the ONES UI. In the image on the right, the result that ONES drove is shown beneath the NVIDIA UFM. It shows that that Gold tenant was given an exclusive access partition PKey 0x1 with limited 4 InfiniBand Port GUIDs and was isolated from the rest of the tenants on the fabric. The banner (Externally Managed Fabric) indicates that ONES detected that this partition was configured in UFM and exposes it in a read-only format, matching the external-PKey preservation.

How Does the Integration Work Under the Hood?

Each workflow in the integration is defined for a unified operator journey. There are five primary workflows constructed around the UFM fabric:

Bootstrap – Provision a UFM fabric and load a default allocation of 32,766 PKeys (0x0001 to 0x7FFE)

Fabric Discovery – Download the UFM hosts and system/port GUIDs and topology and cache for 15 minutes

Tenant Hybrid N-S + E-W Creation – In this case, configuration for the North-South is done first, and coordinated error handling is performed for the allocation of the UFM PKey for East-West

Server Allocation – Assign ports to the tenant for both the North-South and East-West

Tenant Deletion – Perform an atomic delete for the VLAN and PKey resources

Background Sync – Self-healing of inventory, detection of externally managed PKeys, and control plane drift

ONES provides production-ready patterns beneath the surface of the integration and include the use of circuit breakers to isolate UFM, compensating transactions for partial failure, and distributed locks for concurrent multiple instances safe use.

Who Benefits from the Integration?

AI/ML platform teams Self-service, per-tenant GPU fabrics with hard network isolation no more sharing the InfiniBand backbone on trust.	Network engineers Stop hand-editing PKey partitions and start reviewing automated audit logs.
Security & compliance leads Tenant isolation story backed by hardware-level PKey enforcement GDPR and SOC 2 ready.	Operators One UI, one API, one mental model for both traffic planes whichever way the team prefers to work.

What Is the Bigger Picture?

The ONES portfolio has always integrated heterogeneous, multi-vendor, multi-protocol fabrics into a single control plane via a set of easy-to-use APIs. We are now bringing this integration to the AI data center via UFM. In the AI data center, InfiniBand, RDMA, and GPU-to-GPU bandwidth limit how fast models can be trained. This is the first of many integrations that will allow ONES to simplify operator workflows as the AI infrastructure transitions to mixed Ethernet/IB fabrics, disaggregated GPU clusters, and RDMA spanning multiple sites.

Conclusion

The ONES portfolio unifies multi-vendor and multi-protocol AI networking under a single control plane. With the addition of NVIDIA UFM, new tenant isolation capabilities are now automated along with simplified operations and secure hybrid Ethernet and InfiniBand deployments.