The AI Scaling Problem Enterprises Aren’t Talking About

The biggest challenge for enterprises looking to scale AI is rarely just the technology itself. They have to contend with data quality and access, talent, governance, security, integration with legacy systems, and perhaps most importantly, proving that AI can deliver meaningful business value.

Infrastructure, meanwhile, can seem like a secondary concern—especially when pilots can be spun up quickly on rented GPUs or standard cloud environments. But that assumption may not hold once AI moves beyond experimentation.

As Sumed Marwaha, AHEAD India Managing Director, explains, the infrastructure gap becomes far more pronounced when successful pilots need to support larger user bases, higher concurrency and predictable performance.

Production AI demands a fundamentally different environment, one where compute, networking, storage, power and cooling are designed to work together rather than treated as separate technology layers.

Marwaha brings more than two decades of leadership across infrastructure, cloud and managed services, having led large-scale delivery operations and business growth at Dell, NTT DATA and Unisys across India and global markets.

In this conversation with The Left Shift, Marwaha discusses why infrastructure is often overlooked in the early stages of enterprise AI adoption, how legacy systems can complicate the transition to production, and why simply buying more GPUs is rarely the answer.

Edited Excerpts

1. What is the biggest infrastructure gap you see today between an enterprise successfully experimenting with AI and actually deploying AI at scale?

The biggest gap is the move from a working experiment to an integrated, production-grade environment. An organisation can often run a pilot successfully on a few rented GPUs or a standard cloud VM. Production AI is different. It needs infrastructure that can support high concurrency, predictable latency, continuous utilisation and significant data movement.

That typically means accelerated compute with high-bandwidth memory, low-latency GPU-to-GPU networking, storage that can feed many accelerators in parallel, higher rack power density, liquid cooling or major mechanical upgrades, and capacity planning for sustained inference rather than occasional experimentation. It also requires specialised scheduling so that expensive accelerators remain highly utilised.

In short, the gap is not one missing component. It is the absence of an integrated AI infrastructure cell in which compute, networking, storage, power, cooling and capacity management are engineered and operated as one system.

2. Many enterprises have moved from AI pilots to proofs of concept, but relatively few reach production. How much of that failure is an infrastructure problem versus a data, talent or organisational problem?

There is no reliable universal percentage because different studies define failure differently. Some projects are abandoned during the pilot because they do not demonstrate sufficient technical or business value. Others reach production but cannot scale, fail to deliver a meaningful return, or are deprioritised because of wider organisational priorities.

Infrastructure is usually not the first blocker. Early pilots can often run with limited, shared or rented capacity. The infrastructure challenge becomes much more important once a use case has demonstrated value and needs to support larger user populations, higher concurrency, predictable performance and stronger resilience.

At that point, organisations need to address accelerator availability, high-bandwidth memory, GPU scheduling, network fabrics, storage throughput, power and cooling. They also need to be clear about what they mean by infrastructure. Some organisations include data platforms and data quality within that definition, while others focus only on compute, networking and facilities.

The balance will change as more experiments move into production. Infrastructure may not be the leading cause of early project failure today, but integrated infrastructure will become a critical factor in scaling successful AI use cases.

3. What changes are enterprises discovering they need to make to their compute, storage and networking infrastructure when an AI workload moves from a small pilot to thousands or millions of users?

The infrastructure needs to become dedicated, repeatable and designed for sustained demand rather than occasional use. A pilot might use a shared pool of GPUs. A production environment typically needs dedicated and replicated accelerator pools, with the scheduling and resilience to support multiple workloads. It may also require high-bandwidth-memory-aware tiers, high-speed caching and parallel file storage so that data can be delivered to many accelerators at the same time.

Networking is another significant change. A standard Ethernet environment may need to be complemented or replaced by higher-speed fabrics that can support intensive traffic between accelerators, storage and enterprise systems.

The data centre itself may also need upgrades, including higher power density, liquid cooling and additional mechanical capacity. Global enterprises may require multi-zone availability and N+1 capacity for resilience.

The important point is that compute, storage, networking and facilities cannot be planned independently. They need to be designed as a production platform around the workload profile.

4. How should CIOs think about building infrastructure for AI when GPU demand, model architectures and workloads are changing so quickly? Is it better to build, buy or use cloud infrastructure?

CIOs should start with the workload and a reference architecture, rather than with the decision to buy GPUs or move everything to the cloud.

At AHEAD, we look at the requirements at each layer and then recommend the best-fit operating model. Public cloud is useful for experimentation, uncertain workloads and burst capacity, although costs can be difficult to predict and organisations may become dependent on a particular provider or technology. Managed private cloud can provide dedicated and secure capacity, but may offer less flexibility and a higher cost base.

Enterprise-owned infrastructure can make sense for high-volume, predictable workloads that require maximum control and security. The trade-offs include underutilisation, talent requirements and the risk of hardware becoming obsolete quickly.

For many organisations, a hybrid model will be the most practical approach because it combines different workload types and demand patterns. It does introduce architectural and operational complexity, so the integration model needs to be designed from the start.

The right answer is rarely build or buy in isolation. It is usually a combination based on workload, security, data sovereignty, utilisation, cost and the pace of change.

5. What role does legacy enterprise infrastructure play in slowing AI deployment? Are organisations trying to layer AI onto systems that were never designed to support AI workloads?

Legacy infrastructure can slow AI deployment because it was generally designed for reliable, predictable and controlled transaction processing. AI workloads are more distributed and variable, with demanding requirements for data movement, compute and rapid scaling.

That does not mean organisations need to replace everything. They need to assess the architecture as a whole and decide which components should be modernised, separated or extended.

For example, an AI programme may need a dedicated compute layer and a data lake for staging and processing data. The network may need to be upgraded to support high-volume accelerator traffic, while data centre facilities may need higher-density power and liquid cooling. Existing core systems can continue to provide stable transaction processing, with modern data, networking and AI layers built around them.

This is why many enterprises are adopting hybrid architectures. They are preserving the stability of legacy systems while modernising data movement and networking, and deploying purpose-built compute, storage and facilities for AI workloads.

The objective is not to force AI into infrastructure that was never designed for it. It is to create the right interfaces between the existing estate and the new AI platform.

6. AI agents are increasingly moving from experimentation toward autonomous, production workloads. How does agentic AI change the infrastructure requirements compared with traditional generative AI applications?

A generative AI application typically answers a request and completes. An agent keeps a task open over time and determines its next step. That difference has several infrastructure consequences.

Agent sessions become long-running and stateful, so memory, checkpointing and recovery need to be designed in from the outset. Orchestration and a tool gateway also become core infrastructure because every action an agent takes may reach an external system with its own rate limits, permissions and failure behaviour.

Identity requirements change as well. An agent needs its own credentials, a defined scope of permission, a managed lifecycle and an audit record showing what it did and under whose authority. This introduces machine-identity management at a scale many enterprises have not previously attempted.

Cost behaviour can also surprise organisations. A single user request may expand into many model calls, so demand is driven partly by machine-generated activity rather than predictable human usage patterns.

Guardrails therefore need to include limits on iteration and spend, together with checkpoints where a person approves an action before it affects money, customer records or production systems.

7. How important are observability, security and governance infrastructure when enterprises move AI into production? Are these being treated as afterthoughts during the experimentation phase?

They are essential, but they are often treated as afterthoughts during experimentation because the initial audience is small and the perceived risk is limited. The cost of that decision becomes clear when instrumentation and controls have to be retrofitted into a live service.

AI observability extends beyond uptime and latency. Organisations need visibility into the model version used, token consumption and cost, retrieval quality, response quality, hallucination indicators and model or data drift.

Security is equally important because AI expands the attack surface beyond servers and networks. Controls need to cover the data used for retrieval, model endpoints, secrets, application permissions and the identities used by agents. Organisations should extend their existing security controls where appropriate, while also adding protection for AI-specific data flows and runtime behaviour.

Governance should be enforced technically, not treated only as a set of policies. That includes approved-model controls, appropriate human intervention, access management, logging and audit trails.

Frameworks such as NIST and ISO can help map the controls, but they need to support a working operating model that defines accountability, performance measures and ownership.

8. From your conversations with enterprise technology leaders, what is the most common infrastructure mistake companies make when attempting to scale AI and what should they be doing differently?

The most common mistake is treating scale as a procurement exercise. A programme is approved, capacity is purchased based on the ambition of the programme, and the organisation then discovers that the real constraint lies elsewhere, often in data access, integration, governance or approvals. The infrastructure remains underutilised while the pilots are still blocked by issues that no purchase order can resolve.

A related mistake is defaulting to the most capable model for every task. Many requests could be handled by a smaller, less expensive model but are routed through a larger model without measuring the effect on quality, cost or business value.

The better approach is to define the outcomes first, create a reference architecture, assess the workload profile and benchmark the appropriate models and infrastructure. Data centre density and capacity should be planned upfront, while modularity should be built into the architecture so that new hardware and models can be adopted without a complete rebuild.

This makes cost and quality per transaction measurable from the first day of production. Capacity can then be purchased against observed demand rather than assumptions.

9. If you were advising a CIO today who has dozens of successful AI pilots but isn’t able to scale them, what are the first three infrastructure priorities you would ask them to address?

First, establish a reusable production reference architecture. The organisation should define a standard AI infrastructure pattern, or production pod, that can be reused across multiple projects. This avoids designing every deployment from scratch and improves utilisation across workloads.

Second, create a workload and capacity baseline. That should cover peak concurrency, input and output token usage, expected growth, burst demand, data sovereignty, latency requirements and resilience. Without this baseline, it is difficult to make sound decisions about infrastructure, cost or operating model.

Third, build the shared services needed to run AI safely and consistently. These should include end-to-end observability, model tracking, identity and access management, least-privilege controls for users and agents, service level objectives, quotas, disaster recovery and governance.

With these foundations in place, successful pilots have a clearer path into production. The organisation can scale through repeatable patterns, measured demand and shared controls rather than creating a separate infrastructure stack for every use case.