Are LLMs ready for an Upgrade?

On the topic of emerging architectures in Generative AI.

Read the entire article here.

For under $2 a week, read and access my best work. Upgrade access here.

There’s something to be said for finding emergent architectures or new ways of doing things. Take Groq’s Language Processing Unit (LPU) a specialized AI accelerator designed to optimize the performance of artificial intelligence workloads, particularly in inference tasks. Or Sandbox AQ’s Large Quantitative Models, LQMs.

Or take Sakana AI’s Neural Attention Memory Models (NAMMs) that they say are a new kind of neural memory system for Transformers that not only boost their performance and efficiency but are also transferable to other foundation models, without any additional training.

When we think of what’s ahead, Ilya Sutskever gave us a nice summary recently at NeurIPS, giving us additional context of the recent history of AI. If he confirmed that LLMs scaling has plateaued in terms of training data, this bottleneck allows for other innovations to take place. Deep learning never rests, even in pushing the boundaries of things like drug development and the future of biotechnology.

Broadcom believes hyperscalers will deploy 1,000,000 XPU clusters across a single fabric as soon as 2027, which is 10x that of which xAI has today in terms of datacenter AI compute. In just a few short years, Scaling laws with be powered by a greater focus on inference, memory, Agentic AI, and on the application layer with tremendously more capable GPUs and AI infrastructure. Broadcom itself is working with the likes of ByteDance, Apple and OpenAI on custom AI chips.

In the decade ahead we are somewhat likely to find new kinds of emergent architectures that expand what we are able to do with LLMs or perhaps will be more efficient and open up new avenues, even as small language models, and world models continue to expand our horizons.

AI pioneer Fei-Fei Li’s World Labs has raised $230 million to build “large world models” (LWMs) with a lot of experimentation at places like Google on world simulators. Google recently announced Genie 2, a foundation world model capable of generating an endless variety of action-controllable, playable 3D environments for training and evaluating embodied agents.

Heading into 2025 we don’t know exactly what to expect. We do know that capex is only going to increase and national competition could accelerate innovation. According to one estimate, the global AI market reached $196.63 billion in 2023 and could be worth $1.81 trillion by 2030. In Short, the AI market is expected to grow at a CAGR of 36.6% from 2024 to 2030 reaching the remarkable figure of 1,811.75 million USD.

Recently Omar Sanseviero of Hugging Face has released a book worth checking out if you dabble with LLMs, called Hands-On Generative AI with Transformers and Diffusion Models. It’s available at O’Reilly and Amazon. I’m not an affiliate, only an enthusiast. There are so many good books coming out around LLMs, those by Sebastian Raschka, PhD in my view are great.

I want to talk especially about AMD’s best on Liquid AI and Ayar Labs and what their architectures might mean for the future.

Upgrades at the Application Layer

"Like the human brain, which stopped growing in size while humanity advanced through tools and knowledge, might new progress come from agents and tools built on top of LLMs and new emergent capabilities and architectures?"

All of this also means new Human-AI interfaces. NotebookLM, the AI-powered research assistant from Google Labs, recently announced new look and an optimized interface with additional capabilities. You can now interact with AI hosts during Audio Overviews and manage content in a more intuitive way. NotebookLM Plus, a premium version with more features and usage limits, is now available for organizations and individuals.

Like several other AI Research agents, Deep Research compiles a detailed report that includes key findings and links to original sources. The advantage Deep Research has is all of Google’s search rankings to find the best information. Deep Research acts like a personal research assistant. Users can input specific questions or topics they want to explore, and the AI generates a multi-step research plan for approval. Once approved, it begins gathering relevant information from the internet. Google NBLM, GLA and DR might together form good AI toolset to complement Perplexity.

• I do recommend checking out Deep Research and comparing it to Perplexity and o1. To access DR make sure you are a Gemini Advanced subscriber,

Google also unveiled a new ways to do research with an AI Tool called Deep Research. In terms of emergent AI-human interfaces this has opened up an era of “Reasoning Models”. In late 2024 the o1 model turned Pro, but progress in Deep learning might come from unexpected sources, or fundamentally new architectures. It’s not clear if Agentic AI systems will be ready to offers consumers much in 2025. Although Google’s agentic era sounds a bit more legit than Microsoft’s Copilot Era. I’m frankly more excited looking ahead to some of the semiconductor innovations around new potential architectures.

Semiconductor Innovation in Emergent Architectures

This past week, AMD has taken a lead in investing in two pivotal startups. I was already fairly bullish on Etched. Recent funding in Liquid AI and Ayar Labs, really point to the potential for new architectures of how we think the future capabilities of AI.

To be honest, photonics is a pretty promising area for emergent architectures in semiconductors. I know TSMC and Nvidia are looking at this area pretty seriously. Can cutting-edge optical I/O solutions or photonic chip Quantum solutions be a new path forward?

Overall I consider who companies like Nvidia, AMD and TSMC back as investments in startups to be fairly relevant for our search for new approaches to the future of semiconductors. Part of my job as an emerging tech analyst means following Venture capital trends and startup news in this regard. This is why I pride myself on occasion, on covering AI startups that I find particularly innovative. Liquid AI is such a startup.

Read Blog

You will notice that many of the new cutting edge proposed architectures in AI are from the likes of MIT, and this is the case with Liquid AI’s rather unusual approach. Thinking about what comes next, I’m not actually sure Ilya is in the best position to know. Still some of his analogies are a bit interesting here, where Sutskever predicts current pre-training methods will end due to finite training data, comparing it to "fossil fuels" of AI. In his talk at NeurIPS in Vancouver recently there was one slide in particular we might want to pay attention to for our topic today; he draws parallels between AI scaling and evolutionary biology, suggesting the field might discover new approaches beyond current pre-training methods.

Even Meta and ByteDance are trying to think of new approaches to LLMs. Meta has developed a new AI architecture called Byte Latent Transformer (BLT) to solve a fundamental problem with today's language models: they can't reliably work with individual letters. Meanwhile ByteDance (the maker of TikTok) Hierarchical Large Language Model (HLLM) Architecture worth pondering.

What is Liquid AI?

Liquid AI is one of the biggest bets on an alternative emergent architecture. The Boston-based startup has secured $250 million in Series A funding led by AMD, valuing the company at $2.3 billion just a year after its founding. This is an enormous amount for a Series A I need to remind you.

Liquid Foundation Models

Learn About LFMs

The Series A round is being led by Advanced Micro Devices Inc., with participation from OSS Capital, Duke Capital Partners and PagsGroup, among other investors.

“In the coming years, two challenges facing AI will need to be overcome,” says Liquid AI CEO Ramin Hasani. “One is the energy cost. Another is making sure we humans stay in control.”

• Liquid Foundation Models (LFMs) are a new series of generative AI models developed by Liquid AI, an MIT spinout. These models are designed from first principles, utilizing liquid neural networks instead of traditional transformer architectures. This innovative approach allows LFMs to handle various types of sequential data, including text, audio, video, and signals, with enhanced efficiency and adaptability.
• Traditionally, developers have relied on neural networks loosely modeled off of the human brain to help AI systems learn through trial and error. Liquid AI is taking a different approach. Founded in 2023 by a group of AI researchers at the Massachusetts Institute of Technology, the startup is building a novel technology called liquid foundation models that instead uses mathematical techniques discovered by studying the architecture of a worm’s brain.
• By focusing on an organism — specifically, a small nematode worm called Caenorhabditis elegans that’s just 1 millimeter long — with far fewer neurons than the human brain, Liquid AI said it’s able to build an AI system that is more flexible. Its systems also require less data and computing power than the conventional transformer-based models that underpin chatbots and other popular AI tools.

Could a kind of neuromorphic computing inspired architecture that is more biology based as has inspired AI startup such as Sakana Labs be fruitful? You will notice that one is backed by Nvidia.

Try Liquid AI

Liquid AI says their 1B, 3B, and 40B LFMs achieve state-of-the-art performance in terms of quality at each scale, while maintaining a smaller memory footprint and more efficient inference. The startup says they have built a private, edge, and on-premise AI solutions for enterprises of any size.

Ilya as the founder of Safe Superintelligence Inc. (SSI) message for us for 2025 was simple: the AI industry is approaching a critical juncture where traditional training methods will no longer suffice. Overall his statements about AI agents and Superintelligence seem less reliable to me.

Liquid AI represents one of the most interesting more biological based approaches to LLMs I know of. Liquid neural networks consist of “neurons” governed by equations that predict each individual neuron’s behavior over time.

More Adaptable than traditional LLMs

The “liquid” bit in the term “liquid neural networks” refers to the architecture’s flexibility; inspired by the “brains” of roundworms, not only are liquid neural networks much smaller than traditional AI models, but they require far less computing power to run.

Companies like Nvidia and AMD have a lot riding on these potentially emergent architectures. As part of AMD’s investment, Liquid AI says it’ll work with the chipmaker to optimize its models for AMD’s GPUs, CPUs, and AI accelerators.

The startup was co-founded and led by Ramin Hasani, Mathias Lechner, Alexander Amini and Daniela Rus. The strategic partnership with AMD will help the Cambridge, Massachusetts-based startup to optimize LFMs with AMD's graphic, central and neural processing units.

A more Flexible Energy efficient Architecture

Traditional transformer-based models, like those behind ChatGPT, are resource-intensive and expensive to scale. Liquid AI’s “liquid foundation models” claim to do more with less, reducing the dependency on massive data centers and opening up on-device AI possibilities.

“Liquid AI’s unique approach to developing efficient AI models will push the boundaries of AI, making it far more accessible,” said Mathew Hein, Senior Vice President and Chief Strategy Officer of Corporate Development at AMD.”

Biologically Adaptive How?

Instead of mimicking the human brain like traditional neural networks, the company's technology draws inspiration from the Caenorhabditis elegans, a microscopic worm with a surprisingly efficient neural structure. This tiny creature, measuring just one millimeter in length, has become the blueprint for what the company calls "liquid foundation models."

"We have been proving the technology in the last year, making sure that an alternative structure to transformers can be scaled," said Ramin Hasani, Liquid AI's co-founder and CEO, in an interview with Bloomberg.

So these are some of the reasons I’m excited about Liquid AI, AMD and Nvidia’s decent picks in emergent architecture startups and how I’m trying to track some of these developments.

We are witnessing AI history and the unfoldment of deep learning structures in human society. Whatever you think about Generative AI, it’s not the end, it’s just the beginning.