The Artificial Investor - Issue 33: Is energy the next bottleneck for AI?

My name is Aris Xenofontos and I am an investor at Seaya Ventures. This is the weekly version of the Artificial Investor that covers the top AI developments of the last seven days.

This Week’s Story: Google signs deal to power its datacentres with nuclear energy

The most interesting story of last week was Google’s signing of the world’s first corporate agreement to purchase nuclear energy that will be deployed in 2030-35. This deal is important because it guarantees Google clean energy supply for its datacenters while not relying on America’s electricity grids. 

Why are AI energy deals like this one needed? Is this becoming an emerging trend for AI hyperscalers? Is energy the next bottleneck of AI? What are the opportunities for the Tech ecosystem?

If you prefer to listen to this issue, click here for the audio version.

Powered by ElevenLabs

🔌 Power up

Data centres require power primarily to run the hundreds of powerful servers, where all the data calculations (compute) take place, and for their cooling systems that are needed to ensure the systems don’t overheat. Additional power needs come from storage and connectivity equipment. Energy costs account for approximately 60%-70% of the total operational cost of a data centre.

However, data processing and storage are not new; they have been with us since Cloud computing started to commercialise in 2006. Have we not experienced a surge in power demand before? It appears that the answer is “no”. 

🐜 Doing more with less

Global datacentre power consumption has remained pretty much stable between 2010 and 2018 at about 200 TWh (Terrawatt hours) annually or about 1% of total electricity use. At the same time, datacentre demand grew significantly. On the consumer side, internet users doubled from 2 to more than 4 billion and annual data generation grew 16x from 2 to 33 zettabytes. On the business side, SaaS revenues grew 8x from 10.6 billion dollars to 87 billion dollars in the same period, which drove strong datacentre demand. 

How come global datacentre power consumption remained nearly stable while datacentre footprint grew at least 10x in the same 8-year period (i.e. 33% annually)? 

The answer is efficiency. Efficiency at all levels: i) energy efficiency of devices (servers and storage drives) driven by steady technological progress by manufacturers, ii) greater use of server virtualization software, which maximises utilisation by enabling multiple applications to run on a single server, and iii) migration of compute to large datacentres (also known as hyperscale facilities), which benefit from economies of scale, such as the use of ultra-efficient mass-scale cooling systems. For instance, Nvidia’s GPUs alone have consistently improved in energy efficiency by c.30% annually from 2011 to 2021 (measured in GFLOPS/Watt, which refers to the number of calculations made per unit of energy consumed). 

🥐 Hungrier than ever

So, what is different now? Why are the continuous energy efficiency gains of datacentres not sufficient? 

The answer is transformers, the technology that lies in the core of the Generative AI wave. Transformers are models that perform a significant number of computations on huge volumes of data. This is the case for both training the AI models and their post-training usage (called inference). For instance, it is estimated that training GPT-3 consumed 1,287 MWh (megawatt-hours). This is the equivalent of the annual energy consumption of about 120 average American homes or the equivalent of the energy used when watching 1,625,000 hours of Netflix. In terms of post-training usage (inference), a typical text generation task consumes 0.047 kWh, i.e. the equivalent of 9 minutes of watching Netflix, but generating an image consumes 60x the energy (2.907 kWh), which is the equivalent of charging a smartphone fully.  

Researchers have found that multi-purpose, generative architectures are orders of magnitude more energy-consuming vs. task-specific systems for a variety of tasks, including a complex algorithm like Google Search. 

What’s more, the more intelligent AI becomes, the more energy it consumes. As an example, GPT-4 consumed 40-50 times more energy than GPT-3 during training, despite being launched only about two years later. Specifically, it consumed 50,000 to 60,000 MWh, which is the equivalent of the annual energy consumption of 5,000 - 6,000 average US households. 

So, as AI-related hardware has become about 30% more efficient every year, training the latest generation GPT model has consumed 360% more energy on an annualised basis than its predecessor. 

It would then come to no surprise that datacentre electricity demand has nearly doubled from about 200 TWh to 400 TWh in the period 2018-2023, reaching a 2%-3% share of total electricity consumption. This has also led to datacenters becoming bigger. A typical hyperscaler datacentre currently has 50 MW capacity, with some Chinese centres reaching 150 MW and Microsoft’s Chicago facility reaching 198 MW. This is expected to increase as Google’s CEO confirmed the Big Tech is working on a 1 GW datacentre.

⚔️ Standing their ground

The Tech community has reacted. Many Tech companies, large and small, have been working on accelerating AI energy efficiency working on different areas: 

• Hardware advancements. BigTech companies, such as Nvidia and AMD, have been improving the energy consumption of their chips, while startups, such as Groq or Cerebras, have launched new chips optimised for AI use cases. 
• Model optimisation. An increasing number of research papers in relation to model optimisation have been published in the last few years, covering methods such as distillation, pruning, quantization, etc. Model optimisation inevitably results in energy efficiency. 
• Power management. Datacentres have been introducing data analytics and AI in power management, such as real-time usage metrics, remote control and monitoring, load distribution, etc.
• Cooling technologies. Large Cloud facilities have implemented various cooling techniques, such as submerging specific hardware components or even entire datacentres in special liquids.
• Edge Computing: Not necessarily an energy efficiency methodology; nevertheless, the growing trend of running computations locally on devices helps reduce energy consumption, as it leads to less data transfer and centralised processing.

Meanwhile, most hyperscalers have been taking their measures, securing power for their datacentres. Amazon signed an agreement with Dominion Energy, Virginia’s utility company, to explore the development of a small modular nuclear reactor, and, if all goes well, plans to invest more than 500 million dollars in the project. Google signed a deal with Kairos Power to buy 500 MW of energy from seven small nuclear reactors, with the first one due to be completed by 2030 and the remainder by 2035. Microsoft signed a 20-year deal with Constellation Energy to reopen the Three Mile Island nuclear plant in Pennsylvania. The AI challengers, OpenAI and Anthropic, have not signed such deals yet, but have been busy lobbying for energy production initiatives. 

♣️ Like a house of cards

According to the International Energy Agency, datacentre energy demand could more than double by 2026. Could this be a problem?

Yes, it can be. And the main reason is the challenges that electricity grids face:

• Age: Particularly in the US and Europe, the average age of regional grids is about 40 years. 
• Intermittency: The intermittent nature of solar and wind power makes balancing supply and demand more complex.
• Stability: The replacement of traditional high-inertia generation (e.g. steam turbines) with low-inertia renewables can lead to stability issues.
• Transmission challenges: Long-distance transmission lines are becoming increasingly important to connect renewable resources to areas of high demand. However, they are also more complex and less efficient.
• Security: Digitalisation of the energy sector increases vulnerability to cyber attacks.

Western policymakers are working on grid improvement initiatives, such as EU incentives for anticipatory investments and cross-border cost sharing, the Union’s stimulation of faster permitting for grid deployment, the US bipartisan Infrastructure Law, and implementing programs to enhance grid resilience against American extreme weather events. At the same time, many opportunities arise for Tech entrepreneurs and investors.  

🔮 Facing the future

Some thought leaders predict that in the long term energy will become abundant and almost free, driven by renewable energy sources (including nuclear power), better transmission materials and optimal energy storage. 

Nevertheless, in the near- to medium-term, grids are expected to come under a lot of pressure.  As a result, we see a big opportunity and are excited about the following trends:

• Smart grid management: Systems that aim to ensure efficient, reliable, and sustainable energy distribution and consumption. Their functionality consists of i) managing the variability and intermittency of renewable energies, ii) advanced and real-time smart metering, iii) automation and optimisation of the operations of electrical systems, iv) managing the integration of energy storage systems like batteries, etc.
• Energy storage: Systems such as grid-scale battery storage (that helps with renewable energy intermittency), distributed energy storage, hybrid energy storage that combines different battery technologies, alternative battery chemicals, energy storage as a service, etc.
• Distributed energy generation: This is a key trend and can play a big role in addressing challenges in problematic centralised grids and long transmission lines. The market is already worth more than 140 billion dollars. This trend is accelerated by increasing government support, the growth of the commercial segment (see Google’s, Microsoft’s and Amazon’s deals) and the expansion of the EV charging infrastructure (which is also another form of distributed energy source).

🍽️ Fun things to impress at the dinner table

Busted. These US university students were falsely accused of cheating by AI.

Down boy. This collar helps people understand what their dog tells them. 

Art attack. A portrait of the world’s first humanoid robot artist called Ai-Da is about to get auctioned soon, starting at 130,000 dollars. 

See you next week for more AI insights.