Understanding Large Language Models and Their Implications: An Interview with OpenAI's CTO

In a recent video interview, Mira Murati, CTO at OpenAI, and Cristóbal Valenzuela, CEO of Runway, talked about the incredible potential of artificial intelligence (AI) and specifically, large language models like ChatGPT. They highlighted how these models are revolutionizing various aspects of life from storytelling to code writing. But what exactly are large language models? How do they function, and what sets them apart from other types of AI? They have succeeded in popularizing the underlying mechanisms of this technology. Here's a summary of the key points to bear in mind and my humble opinion.

Different Types of AI

AI can broadly be classified into two categories: Narrow AI and General AI. Narrow AI is designed for a specific task, such as facial recognition or image classification. In contrast, General AI aims to replicate human cognitive abilities, though this remains largely theoretical at the moment.

What is Generic AI?

The term "Generic AI" doesn't have a standard definition in the field of artificial intelligence, but it's often used colloquially to refer to a general-purpose AI system that can perform a wide range of tasks. In academic and industry contexts, the term that's more commonly used is "Artificial General Intelligence" (AGI). AGI aims to create machines capable of understanding, learning, and applying knowledge across a broad array of activities, much like a human being. Essentially, an AGI would be able to perform any intellectual task that a human can do, from understanding natural language and solving complex mathematical problems to displaying emotional intelligence.

Narrow AI: A Closer Look

On the other end of the spectrum, we have Narrow AI or "Weak AI." These are AI systems designed to perform a very specific task or a set of closely related tasks. They operate under limited pre-defined conditions or domains and are not capable of generalizing their understanding to perform other types of activities. Even if they are highly sophisticated and can outperform humans in their specialized domains, they lack the breadth of capabilities that characterize AGI.

Why is ChatGPT Considered Narrow AI?

ChatGPT, despite its impressive capabilities, is an example of Narrow AI. Here's why:

1. Task-Specific: ChatGPT is designed primarily for natural language understanding and generation. It can perform tasks like answering questions, generating text, and simulating conversation, but it cannot perform activities outside of this scope, such as image recognition or complex problem-solving in unrelated domains.
2. Limited Understanding: While it can parse and generate text based on the data it's been trained on, it doesn't "understand" in the way humans do. For example, it can't understand context beyond the text data it has been trained on, nor can it understand the world through sensory experiences.
3. No Transfer of Learning: ChatGPT can't transfer knowledge or skills from one domain to another. If trained to assist with customer service, for instance, it cannot then apply its "experience" to help with medical diagnosis.
4. Lacks General Intelligence: ChatGPT doesn't have the ability to learn new tasks autonomously or adapt to new types of challenges outside its training data. It can't develop common sense reasoning or emotional intelligence.

How Do Large Language Models Work?

Despite their complexity, large language models operate on relatively simple principles. They are rooted in statistics and use probabilities to predict the next sequence of text. For instance, if you were to train a large language model on all the plays written by Shakespeare (Mira Murati example), it would analyze the sequence of letters in those plays to predict what comes next based on a table of probabilities.

Initially, this approach may produce gibberish. The key to achieving coherence lies in training the model to consider a sequence of letters or even sentences, providing it a richer context. This is where neural networks come into play.

The Role of Neural Networks

A neural network is a computational model inspired by the human brain's neural structure. Instead of a simple table of probabilities, it utilizes a more complex system that can learn from its training data. In the context of large language models, these neural networks consider a broad sequence to predict the next best letter or token, improving the model's capability to produce meaningful text.

What Sets ChatGPT Apart according to Mira?

ChatGPT is unique in three significant ways:

1. It is trained on a massive dataset that includes virtually all the information available on the Internet.
2. Instead of focusing on just the 26 alphabets, it predicts tokens, which can be full words, parts of words, or even code.
3. It undergoes extensive human tuning to produce reasonable and safe content while mitigating the risk of generating biased or dangerous information.

Tokens: The Building Blocks of Intelligent Text Generation

Think of tokens in generative AI like the building blocks of a Lego castle. In the same way you'd piece together different Lego blocks to create structures, a generative AI uses tokens to construct sentences or paragraphs. A token can be as small as a single letter or as long as an entire word. Just like each Lego block has a specific shape and function, each token has specific linguistic properties that help the AI understand and generate text.

For example, let's say you want to generate the sentence "ChatGPT is cool." In this case, the tokens might be ["ChatGPT", " ", "is", " ", "cool", "."]. The AI would use its understanding of how these tokens usually appear in relation to one another to generate this specific sentence or to create similar sentences.

Why are tokens important? Well, imagine trying to build that Lego castle without having the right types of blocks. It would be incredibly difficult, if not impossible. Tokens give the AI the 'building materials' it needs to construct meaningful and accurate text. They allow it to break down complex tasks into manageable pieces, analyze those pieces, and then put them back together in a coherent way.

So, just like you can't build a Lego castle without individual blocks, a generative AI can't generate text without tokens. They're the essential building blocks that make the whole process work.

Shortcomings and Ethical Considerations

Despite the advanced capabilities, it's crucial to remember that large language models can and do get things wrong. They are not genuinely "intelligent" in the way humans are; they are probability machines that can produce amazing yet imperfect results.

They'll say it better than I can...

It's important to note that this system is still just using random probabilities to choose words. - Mira Murati, CTO at OpenAI

A large language model can produce unbelievable results that seem like magic, but because it's not actually magic, it can often get things wrong. - Cristóbal Valenzuela, CEO of Runway

The Future of AI

As both Mira and Cristóbal pointed out, the advancements in AI have far-reaching implications. From creating applications to discovering new drugs, the applications are endless. While debates continue on whether these models exhibit real intelligence, there is no denying their transformative power.

My Humble Opinion: The Evolving Landscape of Generative AI

Democratising AI through Chatbots

Generative AI, and particularly OpenAI's ChatGPT interface, has played a pivotal role in democratizing access to artificial intelligence. Whether you're brainstorming gift ideas or cooking up new recipes, ChatGPT makes it easier for everyone to understand the possible applications of AI. It's bringing the power of machine learning into everyday conversations, breaking down barriers and inspiring innovative uses.

The Future Lies in a Symphony of Narrow AIs

In my opinion, the next two years will be crucial for AI development, and success will come from combining various Narrow AIs, each tailored for specific tasks. The amalgamation of automated or AI-augmented tasks will open up new applications that we're just beginning to explore. For example, with tools like Zapier, ChatGPT, and Midjourney, one can automate the creation of an e-commerce site, manage SEO, grow an Instagram community, and automate sales and support. In the HR realm, a matching model can be combined with generative AI to offer transparent explanations, thus integrating two AIs with complementary skills.

Proceed with Caution

While I'm generally optimistic about the prospects of AI and the conveniences it will bring, I believe that a lack of understanding of its limitations can lead to disastrous outcomes. The phenomenon of AI 'hallucination' is one such limitation that I'll delve into in a future article. I've observed many software vendors in the HR industry indiscriminately applying generative AI to cash in on the hype. Instead, we should be using AI cautiously and for specific use-cases where it genuinely adds value.

The Trillion-Dollar Question

Will generative AI be a $4.4 trillion market, as projected by a McKinsey report that estimates an annual market growth of 34% over 10 years? Probably, but many who are not experts in the field don't understand the technical limitations, making some applications not yet viable. I see significant potential particularly in code generation and marketing, although current limitations exist even there.

Final Thoughts

The generative AI landscape is a burgeoning field with immense potential but also significant challenges. As we venture further into this exciting frontier, it's crucial to proceed with a blend of optimism and caution, ensuring that we harness AI's power responsibly and effectively.

Interested in diving deeper into the intricate world of artificial intelligence and its impact on the job market? Check out my book, 'Will AI Replace Me?' This comprehensive guide unpacks the myths and realities surrounding AI in the workplace, offering thought-provoking insights and practical advice. Whether you're a seasoned professional or a student entering the workforce, 'Will AI Replace Me?' provides the tools you need to navigate the changing landscape of employment in the AI era.