AI’s Thirst For Power Is Driving Us To The Edge: That’s A Good Thing
The narrative around AI infrastructure has been dominated by the topic of training: large, centralized clusters of GPUs, remote campuses demanding hundreds of megawatts of power and frontier models doubling in scale every few months. That story is real. But it is increasingly incomplete. We must now start shifting our focus as an industry to inference, running trained models in real time to power the applications that hundreds of millions of people use every day.
According to McKinsey, inference workloads are projected to grow at a 35% CAGR through 2030 and will surpass training to become the dominant AI workload within five years. This isn't an anticipated future demand. It’s the primary driver of new compute demand, and it requires a fundamentally different approach to infrastructure than the one we've been building.
Before I became the founder and CEO of SPAN, I trained as an engineer at Stanford and have spent my career focused on energy technology and systems. After five years at Tesla Energy, I became convinced that the limitations of an analog power distribution system were a key bottleneck for widespread electrification and deployment of clean energy technology.
That's what led me to found SPAN in 2018. The home electrical panel hadn't meaningfully changed in over 75 years, yet it sits at the center of every electrification decision a homeowner makes. By integrating sensing, controls and intelligence, we made the panel smart, and that unlocked the underutilized electrical capacity already available to enable the adoption of EVs, heat pumps and solar more quickly and affordably. Now that my company is expanding to unlock more of that capacity at the grid edge, I believe addressing the distinction between training and inference is crucial to making the most effective infrastructure investments.
Training workloads are latency-tolerant and tightly coupled, meaning they require large, contiguous GPU clusters running synchronously and need to be sited wherever power and land are available. Inference is fundamentally different: a constant stream of small, often batched requests, each increasingly latency-sensitive (especially for physical AI), that use the extraordinary models developed in training data centers.
Because these requests don’t require coordination across a cluster, the workload is distributable across smaller nodes. The infrastructure model that works for training isn’t purpose-built for inference, and yet that's largely what we're still trying to build. As I've seen in the industry, the demand for inference compute is growing at an unprecedented rate, and reimagining our systems, particularly our power delivery networks, is one of the most urgent challenges we as leaders can address. Furthermore, investment into infrastructure today underpins the speed of AI evolution and the economic progress it begets.
The infrastructure buildout wasn't designed to keep pace with AI compute demand. New data center capacity requires navigating utility interconnection queues, transmission upgrades, new substations and transformers and permitting processes that can stretch beyond three years. Even with the necessary capital, hardware and customers, a data center can still be years away from flipping the switch.
But there's another important nuance to consider: AI doesn't only have an energy problem—it also has a power problem. While frequently discussed interchangeably, energy and power are not the same thing. Energy is the resource; power is the ability to deliver it effectively, at the right time, in the right quantity and to the right place.
We can build more energy generation capacity and still fail to close the power gap if the delivery infrastructure can't keep pace. That distinction matters because it changes how to think about solutions. If energy supply were the only problem, the answer would be simple: Build more generation capacity. But if the problem is power capacity—and I've found it increasingly is—then better utilization of existing infrastructure becomes just as important as new supply. Building new capacity from scratch will lead to the same bottlenecks—the process is expensive, slow and grid-constrained. The better path is making better use of what already exists.
The solution to the AI compute problem isn't simply building more traditional data centers, just as the solution to the power problem isn't simply building more generation. In fact, large data center projects are also facing growing resistance from local communities.
This opposition has prompted some to propose creative alternatives, including space- and ocean-based computing. While there is vast potential for these approaches, space-based compute technologies will take considerable time, and oceanic approaches are still in their infancy. In the meantime, we will need any and all viable solutions to meet growing demand. There is a fundamental technology risk that will take time to address, and I believe a pragmatic, socially beneficial approach will help us meet demand through increased utilization in the very near term.
One additional solution to this problem involves leveraging untapped electrical capacity in homes and small businesses into a managed, distributed compute fleet, deployed node by node and coordinated as a computing cloud. This is something we're actively working on and implementing at SPAN.
Right now, we need solutions that can tap into the capacity we already have, that can avoid the growing friction around data center growth, that provide speed so that the companies demanding computing power can get it where they need it and that share the benefits of electrification with the public to benefit them and our grid. Leaders in this space need to consider those key tenets when addressing the power and energy problem in the coming weeks, months and years.
Like the Industrial Revolution, we’re in the middle of our own computing and power revolution, which is accelerating innovation in space, ocean waters and our own backyards. The grid edge has the capacity, proximity and scale we need to close the speed-to-power gap, and I believe we’re ready to unlock and access it.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?