Scaling AI: Why Data Infrastructure is the Real Bottleneck

Looking Below the Waterline: The AI Infrastructure Iceberg\n\nTo resolve these production issues, enterprise technology leaders must rethink the traditional model of AI infrastructure. Nirav Shah, senior vice president of product marketing at F5, compares modern AI systems to an iceberg.\n\nAbove the waterline sits everything that executives, application developers, and users see: the LLMs, cognitive search applications, orchestration frameworks (like LangChain or LlamaIndex), and increasingly expensive GPU clusters. This visible 10% of the architecture receives the vast majority of media attention and corporate investment.\n\nBelow the waterline, however, lies the remaining 90% of the infrastructure that determines whether those investments actually deliver business value: storage arrays, local and wide-area networks, traffic management configurations, security controls, and the specialized systems responsible for moving data between storage and compute.\n\nWhen data cannot move efficiently, securely, and consistently, even the most advanced GPU clusters sit idle. Because GPUs are among the most expensive assets in the modern data center, low utilization rates represent a direct drain on capital and operating budgets.\n\nFurthermore, data delivery failures can lead to system-wide outages. According to the Uptime Institute’s Annual Outage Analysis 2025, more than half of surveyed organizations stated that their most recent significant outage cost more than $100,000, and one in five reported costs exceeding $1 million. In an AI-driven business, where automated classification or customer service systems run continuously, the cost of data starvation or network disruption escalates rapidly.\n\n### Understanding the Mechanics of Data Starvation\n\nModern AI workloads—whether they involve training a custom model, fine-tuning an open-source model through Low-Rank Adaptation (LoRA), or running Retrieval-Augmented Generation (RAG)—depend on colossal volumes of unstructured data. This data is typically stored in Simple Storage Service (S3)-compatible object environments.\n\nAs Mark Menger, solutions architect at F5, notes: “The symptom looks like a compute problem. The root cause is often data starvation.” Unlike traditional enterprise workloads that query databases in structured, lightweight transactions, AI training and inference demand continuous, high-throughput streams of files, images, vector embeddings, and weight files. A slight network latency spike or packet drop that would go unnoticed in a traditional web application can cause the entire GPU pipeline to stall, leaving millions of dollars of compute hardware waiting for the next data batch.

The Architectural Shift: From Tight to Loose Coupling\n\nMany of today's network bottlenecks stem from legacy architectural assumptions. Historically, enterprises connected applications directly to storage systems. In small-scale or non-distributed applications, this direct coupling is straightforward and easy to configure. However, at the massive scale required for AI workloads, this direct relationship becomes a major architectural bottleneck.\n\nUnder a tightly coupled model, storage systems are suddenly forced to handle tasks far beyond simple data retrieval and storage. They must:\n1. Terminate Encrypted Connections: Handshake and decrypt incoming Transport Layer Security (TLS) calls.\n2. Manage Network Traffic: Route complex query structures and manage network congestion.\n3. Enforce Security Policies: Verify access control lists, validate OAuth tokens, and filter malicious payloads.\n4. Process API Requests: Handle high frequencies of S3 protocol variations from hundreds of active clients.\n\nEach of these tasks consumes CPU resources on the storage controllers. Every encrypted hand-shake requires cryptographic processing; every connection creates memory overhead. During high-performance AI operations, storage controllers spend substantial processing power managing network handshakes and translating security certificates rather than actually reading and writing data blocks to media.\n\n### Introducing the Application Delivery Controller (ADC)\n\nTo decouple compute from storage, leading enterprises are adopting a loosely coupled data delivery architecture. By inserting an Application Delivery Controller (ADC) as an intelligent control plane between compute nodes and storage clusters, organizations create a dedicated storage front door.\n\n\n[Compute Nodes / GPUs] ---> [Application Delivery Controller (ADC)] ---> [S3 Object Storage]\n (TLS/TLS Offloading, Traffic steering,\n Policy enforcement, Load balancing)\n\n\nThe ADC offloads TLS termination, certificate management, traffic shaping, policy enforcement, and protocol-aware S3 processing. Freeing storage controllers from networking and cryptographic management allows storage platforms to dedicate their compute cycles exclusively to what they were optimized to perform: high-efficiency data I/O.\n\nFurthermore, this loose coupling introduces operational flexibility. IT teams can upgrade, migrate, or expand storage arrays and disk systems behind the ADC without altering a single line of code or network configuration on the client-facing AI application.

Performance Testing and The Three Dimensions of Resilience\n\nFor decades, network architects viewed any additional layer in the data path with skepticism, operating under the assumption that adding a hop fundamentally increases latency. AI infrastructure, however, challenges this view.\n\nIndependent testing conducted by SecureIQLab evaluated the performance impact of placing an ADC in front of enterprise-grade, S3-compatible object storage. The findings revealed:\n* No Throughput Penalty: Under standard workloads, throughput remained within a variance range compared to direct, unmanaged node access.\n* Superior Performance Under Stress: Under simulated network congestion and high-concurrency client request loads, overall throughput was substantially higher when traffic was managed through the ADC control layer rather than routed directly.\n\nThe performance boost occurs because the ADC doesn't act as a passive "bump in the wire." Instead, it actively optimizes TCP parameters, manages keep-alive connections, buffers requests, offloads cryptography, and intelligently distributes client calls across healthy backend nodes. Rather than adding latency, the control plane streamlines incoming traffic to prevent storage controllers from becoming overwhelmed.\n\n### Real-world Case Study: Five-Fold Operations Latency Improvement\n\nA global financial services organization planning to scale its AI platform using Kubernetes-hosted microservices and S3 object storage ran into significant issues with its legacy setup, which used shared virtual load balancing. The setup created severe performance degradation and reliability drops as data volumes grew.\n\nRather than purchasing additional GPU clusters, the engineering team focused on the storage-to-compute boundary. They deployed dedicated, high-performance physical ADC hardware in front of their object storage clusters to manage traffic and optimize the S3 protocol.\n\nThe impact was immediate:\n* Object Operations: The organization achieved a 5x improvement in object creation, reading, and deletion speeds.\n* Latency Reduction: For specific critical operations—such as multi-part object updates and deletions—latency dropped by an order of magnitude.\n* Stability: The platform eliminated previous connection resets and retry storms without introducing any performance overhead compared to direct node routing.\n\n### The Three Pillars of Data Delivery Resilience\n\nDeploying an ADC or application delivery platform helps organizations achieve resilience across three distinct areas, as outlined by Mark Menger:\n\n1. Reachability: Ensures that LLMs and AI applications can consistently access active storage nodes. If a storage controller degrades, experiences a hardware failure, or undergoes a network drop, the ADC automatically and transparently redirects workloads to healthy nodes without disrupting ongoing training or inference jobs.\n2. Policy: Protects the dataset and storage arrays from self-inflicted damage. AI workloads can generate "thundering herd" scenarios, where hundreds of parallel workers attempt to access the exact same dataset or model weights simultaneously, creating traffic congestion. The ADC shapes this traffic, manages priority cues, and enforces rate limits.\n3. Delivery: Isolates application clients from backend storage changes. Whether IT teams are upgrading storage firmware, adding disk capacity, or migrating datasets on-premises or between clouds, the ADC handles client-side requests seamlessly, ensuring zero downtime for AI workloads.

The Next Frontier: From ADCs to ADSPs\n\nAs enterprise AI environments grow increasingly distributed—spanning on-premises datacenters, public cloud hyperscalers, and edge computing nodes—traditional traffic management is no longer sufficient. Organizations require unified observability, data protection, policy enforcement, and consistent management across environments.\n\nThis need is driving the evolution of application delivery controllers into Application Delivery and Security Platforms (ADSPs). An ADSP integrates load balancing, traffic engineering, comprehensive security controls (such as DDoS mitigation and API gateways), and deep telemetry into a single control plane.\n\n\n +---------------------------------------------+\n | Application Delivery & Security (ADSP) |\n | |\n | [Traffic Eng] [Security] [Observability] |\n +---------------------------------------------+\n / | \\\n / | \\\n [On-Prem] [Public Clouds] [Edge]\n\n\n“AI broke the model of solving delivery and security as separate problems,” says Nirav Shah. “When data is moving constantly between storage, compute, and applications across hybrid multi-cloud environments, you need one platform that delivers and protects that traffic at the same time.”\n\nThis transition mirrors the evolution of the early web. Two decades ago, simple load balancers redirected HTTP traffic; as applications grew more complex, they evolved into application delivery controllers with security capabilities. AI is driving a similar architectural shift. With data moving continuously between distributed databases, inference engines, vector stores, and custom user interfaces, organizations must secure and optimize that entire traffic flow as a unified system.\n\nFor more on securing these evolving AI architectures, see From Passive Walls to Active Intelligence: Transforming Cybersecurity Infrastructure.