AI scale-out networking is the network architecture used to connect multiple GPU servers, accelerator nodes and switches across a data center so they can operate together as a large distributed computing system.
Unlike traditional data center networking, where network traffic often connects users, applications, servers and storage systems, AI scale-out networking must support continuous high-volume communication between compute nodes during distributed training and inference.
As AI clusters grow from a few servers to hundreds, thousands or more GPU systems, the network becomes an essential part of overall computing performance. High bandwidth, predictable latency, congestion control, RDMA, efficient collective communication and high-density optical connectivity are therefore critical elements of AI scale-out infrastructure.
Scale-out networking connects multiple independent compute nodes so that workloads can be distributed across a larger pool of processing resources.
Instead of placing all computing resources inside a single system, scale-out architecture adds more servers or accelerator nodes and connects them through a high-performance network.
In AI infrastructure, this allows large models and datasets to be processed across many GPU systems simultaneously.
AI scale-out networking is a high-performance network fabric designed specifically for communication across multiple AI compute nodes.
It connects servers across racks and can extend across larger portions of a data center.
The network must support very large volumes of east-west traffic because GPUs in different servers continually exchange data during distributed workloads.
Modern AI workloads are highly distributed.
A single training job may use many GPUs that repeatedly exchange gradients, parameters, activations and other intermediate data.
If the network cannot deliver data efficiently, GPUs may spend time waiting for communication instead of performing computation.
| Feature | Traditional Data Center Network | AI Scale-Out Network |
|---|---|---|
| Main Traffic | User, application and storage traffic | Large-volume GPU-to-GPU communication |
| Traffic Pattern | Mixed and variable | Highly synchronized |
| Latency | Important | Highly important |
| Bandwidth | Application dependent | Extremely high |
| Congestion | General network concern | Critical for distributed workloads |
| RDMA | Optional | Commonly important |
| Network Fabric | Ethernet and IP | Ethernet/RoCE or InfiniBand depending on architecture |
| Optical Connectivity | Varies | High-density 400G, 800G and emerging 1.6T links |
Scale-up and scale-out solve different connectivity problems.
Scale-up networking connects processors or accelerators within a tightly coupled computing domain, such as multiple GPUs within a server or rack-level system.
Scale-out networking connects separate compute nodes across servers and racks.
Scale-up networking is designed for very high-bandwidth communication within a relatively small computing domain.
Its objective is to make multiple accelerators behave as a tightly connected computing resource.
Examples include dedicated accelerator interconnect technologies designed for GPU-to-GPU communication within a system or rack.
Scale-out networking extends communication across multiple servers.
Each server can contain multiple GPUs, but the network fabric connects those servers so distributed workloads can use the combined computing resources.
This architecture allows AI clusters to grow by adding more nodes.
| Category | Scale-Up | Scale-Out |
|---|---|---|
| Connection Scope | Within a computing domain | Across servers and racks |
| Typical Distance | Very short | Longer |
| Main Goal | High-bandwidth accelerator communication | Cluster expansion |
| Network Type | Dedicated accelerator interconnects | Ethernet/RoCE or InfiniBand |
| Optical Requirement | Depends on platform | Increasingly high |
Scale-out networking generally connects compute nodes across a data center fabric.
Scale-across networking extends connectivity beyond a single data center and can connect separate AI facilities into a larger distributed infrastructure.
These are different networking layers even though both involve distributing AI workloads.
A typical AI scale-out fabric contains GPU servers connected to top-of-rack or leaf switches, which connect upward through a larger switching fabric.
Depending on the architecture, the network can use leaf-spine, fat-tree or other high-radix topologies.
The design must provide sufficient path diversity and bandwidth for large-scale east-west communication.
Leaf-spine architecture provides predictable connectivity between servers connected to leaf switches and the upper-level spine switches.
Every leaf switch is connected to multiple spine switches, creating multiple network paths.
This topology is widely used in modern data center networks and can be adapted for AI scale-out fabrics.
A fat-tree provides a highly interconnected topology in which the aggregate bandwidth remains large as traffic moves toward the upper layers.
It is well suited to distributed computing because many nodes can communicate simultaneously without relying on a single narrow network path.
East-west traffic refers to traffic moving between servers inside the data center.
AI training generates substantial east-west traffic because GPUs in different servers repeatedly exchange data.
This makes internal fabric performance more important than simply providing high-bandwidth external network connectivity.
North-south traffic generally describes communication between the data center and external users, applications, storage systems or other networks.
AI scale-out traffic is primarily concerned with east-west GPU communication, although the complete AI data center also requires north-south networking for data ingestion, storage access and external services.
GPU-to-GPU communication is one of the fundamental requirements of distributed AI workloads.
When GPUs are located on different servers, their communication must pass through the scale-out network.
The network must therefore provide sufficient bandwidth and low, predictable communication delay.
Distributed training divides computation across multiple GPUs and servers.
During training, nodes can exchange gradients or other intermediate information at synchronization points.
As the number of GPUs increases, the volume and frequency of communication can also increase significantly.
AI frameworks frequently use collective communication operations to exchange information across many GPUs.
Common operations include all-reduce, all-gather, reduce-scatter and broadcast.
These operations can place significant bandwidth and latency demands on the scale-out network.
All-reduce combines data from multiple compute nodes and distributes the resulting value back to all participating nodes.
It is commonly used in distributed training to synchronize information between GPUs.
Efficient all-reduce performance therefore depends heavily on the network fabric.
AI workloads often operate in synchronized phases.
If one node experiences significantly more communication delay than the others, other GPUs may need to wait before the next computation phase can begin.
Reducing latency variation can therefore be as important as reducing average latency.
RDMA stands for Remote Direct Memory Access.
RDMA allows a network adapter to transfer data directly to or from memory on a remote system without requiring the CPU to handle every part of the data movement.
This can reduce CPU overhead and improve communication efficiency for distributed workloads.
AI workloads generate very large amounts of data movement between nodes.
Reducing CPU involvement in data transfers can help lower software overhead and improve communication efficiency.
RDMA is therefore an important technology in many AI scale-out networking architectures.
InfiniBand is a high-performance networking technology widely used for HPC and AI cluster interconnects.
It supports RDMA and is designed for low-latency, high-throughput communication.
InfiniBand can therefore provide the characteristics required by large distributed computing workloads.
Ethernet is another major technology used for AI scale-out networks.
AI-optimized Ethernet architectures can use technologies such as RDMA over Converged Ethernet, congestion control and specialized network adapters to support distributed GPU communication.
This allows existing Ethernet-based data center expertise and infrastructure to be extended into AI networking.
RoCE stands for RDMA over Converged Ethernet.
It enables RDMA communication over Ethernet networks.
RoCE is commonly associated with AI and high-performance data center networking because it combines Ethernet infrastructure with low-overhead RDMA data transfer.
| Feature | RoCE | InfiniBand |
|---|---|---|
| Underlying Network | Ethernet | InfiniBand |
| RDMA | Yes | Yes |
| Network Ecosystem | Ethernet-based | Dedicated fabric |
| Existing Ethernet Integration | High | Requires dedicated InfiniBand infrastructure |
| AI Applications | Yes | Yes |
| Congestion Management | Requires appropriate Ethernet configuration | Built around the InfiniBand fabric |
A standard Ethernet network is not automatically optimized for large-scale distributed AI.
AI workloads place unusual demands on synchronization, congestion behavior, latency variation and communication throughput.
An AI Ethernet fabric therefore requires careful design of switches, NICs, congestion control, routing, buffering, telemetry and software.
An AI NIC is a network interface controller or accelerator designed to support high-performance AI workloads.
It can provide high-speed connectivity, RDMA acceleration, congestion management and other functions needed for efficient GPU communication.
Some architectures also use DPUs or SuperNICs to offload infrastructure and networking functions from the host CPU.
DPU stands for Data Processing Unit.
A DPU can offload infrastructure, security, storage and networking functions from the host CPU.
In large AI data centers, DPUs can help isolate infrastructure processing from GPU workloads.
Scale-out switches provide the network fabric connecting GPU servers.
They must support large numbers of high-speed ports, high switching capacity, efficient buffering and congestion management.
As GPU cluster sizes increase, switch radix and aggregate switching capacity become increasingly important.
Radix describes the number of ports or network connections available on a switching device.
High-radix switches can connect more endpoints or reduce the number of switching stages required in a large fabric.
This can help improve network density and overall topology efficiency.
Oversubscription occurs when the aggregate bandwidth of downstream connections exceeds the capacity available toward upstream network paths.
AI training workloads can generate heavy simultaneous communication, making excessive oversubscription particularly undesirable in the scale-out fabric.
Network designers therefore pay close attention to bandwidth ratios throughout the topology.
Some AI networking architectures aim for very low packet loss because retransmissions can negatively affect distributed workload performance.
Techniques such as priority flow control, congestion notification, adaptive routing and carefully designed buffer management can be used in Ethernet-based AI networks.
The exact mechanisms depend on the network architecture.
Congestion occurs when network demand exceeds the capacity of a network path.
AI traffic can create synchronized bursts in which many GPU nodes send data simultaneously.
Efficient congestion control is therefore essential for maintaining predictable throughput and latency.
Adaptive routing can select among multiple network paths based on network conditions.
By avoiding congested paths, the network can improve load distribution and reduce communication delays.
This is especially valuable in large fabrics with many parallel network paths.
Telemetry provides visibility into network conditions such as congestion, packet behavior, latency and link utilization.
AI networks benefit from detailed telemetry because a problem affecting a small number of links can potentially reduce the efficiency of an entire distributed workload.
Optical communication is essential for connecting large AI clusters because copper electrical links are strongly limited by distance, loss and signal integrity at very high data rates.
Optical links allow high-speed connections to extend across racks and larger areas of the data center.
Optical transceivers convert high-speed electrical signals from switches or network adapters into optical signals for transmission through fiber.
As switch port speeds increase, optical modules become a major part of the bandwidth and power budget of an AI network.
400G optical transceivers can provide high-bandwidth connections between AI switches and compute nodes or between network switching layers.
Depending on the application, 400G modules can use parallel optics or single-mode WDM architectures.
800G optical transceivers are increasingly relevant to large AI cluster fabrics because they can provide twice the nominal aggregate bandwidth of a 400G interface.
Different 800G architectures can support short-reach parallel links or single-mode links using WDM.
The appropriate module depends on reach, fiber infrastructure, switch port architecture and network topology.
1.6T optical interfaces are being developed for next-generation high-bandwidth AI and data center networking.
They require higher electrical lane rates, advanced optical components and increasingly sophisticated thermal management.
As 1.6T deployment develops, optical transceiver power and front-panel density become important architectural considerations.
An optical engine contains the core optoelectronic transmit and receive functions of an optical system.
Depending on the architecture, it can include lasers, modulators, photodetectors, drivers, TIAs and optical coupling components.
Optical engines become particularly important in highly integrated AI networking architectures where optics need to be placed close to switching silicon.
| Category | Optical Engine | Optical Transceiver |
|---|---|---|
| Role | Core optoelectronic subsystem | Complete optical interface module |
| Typical Location | Inside or near system architecture | Often front-panel pluggable |
| Serviceability | System dependent | High for pluggable modules |
| ASIC Integration | Can be very close | Typically farther from ASIC |
| AI Application | CPO, NPO and integrated optics | 400G, 800G, 1.6T network links |
Co-Packaged Optics places optical engines close to the switching ASIC.
This can reduce the electrical distance between the SerDes and optical conversion point.
CPO is therefore being investigated for high-bandwidth switch systems where electrical channel loss, power and density become major limitations.
Near-Packaged Optics places optical engines near the switch ASIC without requiring exactly the same package-level integration as CPO.
NPO seeks to reduce electrical channel length while retaining more separation between the optical system and ASIC package.
Linear Pluggable Optics removes the conventional high-speed DSP from many optical module architectures.
This can reduce module power and processing latency, but it increases the requirements placed on the host electrical channel and SerDes.
LPO can therefore be relevant to AI scale-out systems where module power efficiency is a critical consideration.
DSP-based optical transceivers perform extensive signal processing within the module.
Functions can include equalization, signal recovery, FEC and other processing required by the particular architecture.
This can improve tolerance to difficult electrical channels but also increases module power and thermal load.
Single-mode and multimode fiber can both be used in AI data center networks depending on the transceiver architecture.
Multimode fiber is commonly associated with short-reach 850nm parallel optics, while single-mode fiber is widely used for longer-reach and WDM applications.
Fiber selection must match the optical module and required transmission distance.
Wavelength-division multiplexing allows multiple optical channels at different wavelengths to share the same fiber.
This can increase bandwidth density without requiring the same number of physical fibers.
WDM is particularly useful for high-speed single-mode optical transceivers.
Parallel optics transmits multiple optical lanes through multiple fiber paths.
This architecture is common for short-reach applications because multiple optical lanes can be implemented using arrays of VCSELs or other optical transmitters.
Parallel optics can be attractive where high port density and short-distance connectivity are required.
AI clusters require large numbers of high-speed network connections.
Depending on distance and interface type, these connections can use DAC, AOC, optical transceivers and structured fiber cabling.
Short links can use copper DAC or active optical cable, while longer or higher-density links commonly use optical transceivers and fiber infrastructure.
| Solution | Medium | Typical Application |
|---|---|---|
| DAC | Copper | Very short server or switch connections |
| AOC | Optical Fiber | Fixed high-speed short-reach links |
| Optical Transceiver | Fiber with pluggable modules | Flexible switch and server connectivity |
Latency affects how quickly data can move between GPUs located on different servers.
Low latency is particularly important for synchronization-heavy collective operations.
However, average latency alone does not determine AI network performance; jitter, throughput, congestion and collective communication efficiency also matter.
Bandwidth determines how much data the network can transport over a given period.
As model size and GPU counts increase, the aggregate bandwidth requirement of the scale-out fabric increases rapidly.
This is driving the adoption of 400G and 800G interfaces and the development of 1.6T-class optical connectivity.
Bisection bandwidth describes the total bandwidth available when a network is divided into two roughly equal groups and traffic crosses between them.
High bisection bandwidth is important in distributed AI systems because many nodes may communicate simultaneously across different parts of the fabric.
Low-oversubscription or non-blocking designs provide more bandwidth for simultaneous node-to-node communication.
AI workloads can benefit from high aggregate fabric capacity because synchronized communication can create large traffic bursts.
Actual topology requirements depend on workload characteristics and network architecture.
Blocking occurs when multiple communication flows compete for insufficient network capacity.
In an AI cluster, persistent or synchronized blocking can cause communication phases to take longer and reduce overall accelerator utilization.
Network topology, switch capacity and congestion control all influence blocking behavior.
GPU utilization reflects how effectively accelerators are kept busy with useful computation.
If the network delivers data too slowly, GPUs can spend time waiting for communication.
A well-designed scale-out fabric therefore contributes directly to efficient use of expensive accelerator resources.
Large AI clusters contain many switches, links, transceivers and network adapters.
The probability of encountering component or link failures increases as system scale grows.
High availability therefore requires path redundancy, fast failure detection, rerouting and effective monitoring.
Fault-tolerant AI fabrics can continue operating when individual links, switches or network interfaces experience problems.
Redundant topologies and dynamic routing can help isolate failures and maintain connectivity.
The exact recovery method depends on the network protocol and system architecture.
Managing a large AI fabric requires visibility across switches, NICs, optical links and compute nodes.
Important monitoring parameters include link status, utilization, latency, congestion, packet errors, optical power and temperature.
Automated monitoring becomes increasingly important as the cluster grows.
Optical transceivers can provide diagnostic information such as module temperature, supply voltage, transmit optical power and receive optical power.
This information can help identify fiber problems, excessive loss, overheating and other conditions that affect network reliability.
High-speed optical modules consume electrical power and generate heat.
As port speeds increase from 400G to 800G and beyond, the thermal contribution of networking equipment becomes an important part of rack-level design.
Switch cooling, module cooling, airflow and liquid-cooling infrastructure may all become relevant.
Optical module power is increasingly important because a large AI switch can contain many high-speed optical ports.
Even a small increase in power per module can result in a substantial increase in total switch power when multiplied across many ports.
This is driving interest in lower-power architectures such as LPO as well as more integrated optical approaches.
Front-panel space limits the number of traditional pluggable optical modules that can be installed in a switch.
As aggregate switch bandwidth increases, higher-capacity optical modules can help increase the bandwidth delivered by each port position.
This makes 800G and emerging 1.6T optical interfaces important for future high-density AI systems.
800G scale-out links can use several optical architectures depending on the required reach.
Short-reach solutions may use parallel single-mode or multimode optical lanes, while longer connections can use WDM-based single-mode designs.
Optical technology should be selected based on the complete network topology rather than aggregate bandwidth alone.
1.6T optical connectivity increases the bandwidth available per network interface but also raises challenges in SerDes speed, signal integrity, optical engine design, power consumption and thermal management.
These factors are driving multiple approaches including advanced pluggable modules, linear optics and integrated optical engines.
Silicon photonics can integrate optical functions such as waveguides, modulators, couplers and multiplexers on a photonic integrated circuit.
It can therefore provide a highly integrated platform for optical engines used in advanced AI networking systems.
VCSEL technology remains relevant to short-reach optical connectivity.
VCSEL arrays can provide multiple optical channels in compact parallel optical architectures and are widely associated with short-distance multimode links.
EML technology combines a laser with an electro-absorption modulator.
It is commonly used in high-speed single-mode and WDM optical systems where controlled modulation and low-chirp characteristics are important.
EML can therefore play an important role in selected high-speed AI networking modules.
A simplified AI scale-out architecture can be represented as:
GPU Server → AI NIC → Leaf Switch → Spine Switch → Leaf Switch → AI NIC → GPU Server.
Optical transceivers or active optical links provide the optical connections between the network devices, while RDMA and congestion-management technologies support efficient distributed communication.
| Layer | Main Function |
|---|---|
| GPU / Accelerator | Performs AI computation |
| AI NIC / SuperNIC | High-speed node connectivity and communication acceleration |
| Leaf Switch | Connects local GPU servers |
| Spine Switch | Provides high-capacity fabric connectivity |
| Optical Transceiver | Converts electrical signals to optical signals and back |
| Optical Fiber | Carries high-speed optical data between network nodes |
| Network Software | Controls communication, routing and workload networking |
Hardware alone does not determine scale-out performance.
AI frameworks, communication libraries, drivers and network software must efficiently use the underlying fabric.
Collective communication libraries can optimize operations such as all-reduce and all-gather according to the network topology.
NCCL is a communication library designed to facilitate collective communication between GPUs.
It can use available high-speed networking technologies and optimize communication patterns for distributed GPU workloads.
The effectiveness of the software layer depends on both the network hardware and system architecture.
GPU clusters also need access to training data, checkpoints, datasets and other storage resources.
Storage traffic and GPU-to-GPU scale-out traffic may use separate or shared network infrastructure depending on the data center architecture.
The network must therefore be designed to prevent storage traffic from negatively affecting latency-sensitive distributed compute traffic.
Some architectures use a common Ethernet-based infrastructure for multiple types of traffic.
This can simplify operations and reuse existing networking skills and infrastructure.
However, AI traffic places stricter requirements on congestion behavior and predictable performance than many traditional applications.
Large AI clusters can contain multiple workloads running simultaneously.
Network isolation and traffic management can help prevent one workload from consuming excessive bandwidth and affecting another workload.
Techniques can include virtual networks, quality-of-service mechanisms and traffic prioritization.
Scaling an AI network requires increasing compute nodes, network ports, switching capacity, optical bandwidth and software coordination together.
Adding GPUs without increasing network capacity can create a communication bottleneck.
Successful scale-out therefore requires coordinated scaling of compute and networking resources.
Major challenges include very high bandwidth requirements, synchronized traffic bursts, congestion, latency variation, power consumption, thermal density, optical module scalability and network reliability.
These challenges become more significant as GPU counts and model sizes increase.
Future AI clusters will require increasingly high-capacity optical links.
400G and 800G connectivity is already relevant to high-speed data center networks, while 1.6T-class interfaces are being developed for next-generation systems.
At the same time, optical engines, silicon photonics, LPO and CPO are providing different ways to manage power, density and electrical channel limitations.
| Category | AI Scale-Out Networking |
|---|---|
| Main Purpose | Connect distributed GPU servers into a large computing fabric |
| Traffic Type | High-volume east-west GPU communication |
| Key Technologies | Ethernet/RoCE, InfiniBand, RDMA and AI networking software |
| Important Metrics | Bandwidth, latency, jitter, congestion and reliability |
| Optical Connectivity | 400G, 800G and emerging 1.6T architectures |
| Optical Technologies | VCSEL, EML, silicon photonics and integrated optical engines |
| Advanced Architectures | LPO, NPO and CPO |
| Main Applications | AI training, distributed inference and high-performance computing |
AI scale-out networking is the high-performance communication fabric that connects distributed GPU servers so they can work together on large AI workloads. It is fundamentally different from simply adding more conventional network bandwidth because distributed AI workloads generate synchronized, high-volume east-west communication that places strong demands on latency, jitter, congestion control and throughput.
Ethernet with RDMA technologies such as RoCE and dedicated fabrics such as InfiniBand are important approaches to AI scale-out networking. At the physical layer, 400G, 800G and emerging 1.6T optical connectivity provide the bandwidth required as GPU clusters continue to grow.
Optical transceiver technology is becoming an increasingly important part of the overall AI network architecture. As bandwidth density and power requirements increase, solutions such as silicon photonics, optical engines, LPO, NPO and CPO provide different approaches to scaling optical connectivity.
Ultimately, AI scale-out networking is a system-level architecture. GPU performance, AI NICs, switches, optical transceivers, fiber, congestion control and communication software must operate together to keep distributed workloads running efficiently.
For any questions, please contact us by email or WhatsApp.
Email: sales@c-light.com
WhatsApp: +86 132 6656 7067
