2024: 12 predictions for AI, including 6 moonshots

As the year draws to a close, it's time to reflect on the past and look ahead to the future. This year, I would like to focus on proposing some ambitious open challenges that could drive significant AI research and industrial innovation in 2024. Let's start:

1 - MLMs - Immerse Yourself in Multimodal Generation: The progression towards fully generative multimodal models is accelerating. 2022 marked a breakthrough in text generation, while 2023 witnessed the rise of Gemini-like models that encompass multimodal capabilities. By 2024, we envision a future where these models will seamlessly generate music, videos, text, and construct immersive narratives lasting several minutes, all at an accessible cost and with quality comparable to 4K cinema. Brace yourself Multimedia Large models are coming. likelihood 8/10.

2 - SLMs- Going beyond Search and Generative dichotomy: LLMs and search are two facets of a unified cognitive process. LLMs utilise search results as dynamic input for their prompts, employing a retrieval-augmented generation (RAG) mechanism. Additionally, they leverage search to validate their generated text. Despite this symbiotic relationship, LLMs and search remain distinct entities, with search acting as an external and resource-intensive scaffolding for LLMs. Is there a more intelligent approach that seamlessly integrates these two components into a unified system? The word is ready for Search large models or, shortly, SLMs. likelihood 8/10.

3 - RLMs - Relevancy is the king, hallucinations are bad: LLMs have been likened to dream machines which can hallucinate, and this capability it has been considered not a bug but a 'feature'. I disagree: while hallucinations can occasionally trigger serendipitous discoveries, it's crucial to distinguish between relevant and irrelevant information. We can expect to see an increasing incorporation of relevance signals into transformers, echoing the early search engines that began utilising link information such as PageRank to enhance the quality of results. For LLMs, the process would be analogous, with the only difference being that the generated information is not retrieved but created. The era of Relevant large models is upon us. likelihood 10/10.

4 - LinWindow - Going beyond quadratic context window: The transformer architecture's attention mechanism employs a context window, which inherently presents a quadratic computational complexity challenge. A larger context window would significantly enhance the ability to incorporate past chat histories and dynamically inject content at prompt time. While several approaches have been proposed to alleviate this complexity by employing approximation schemes, none have matched the performance of the quadratic attention mechanism. Is there a more intelligent alternative approach? (Mamba is a promising paper) In short, we need LinWindow. likelihood 6/10.

5 - AILF - AI Lingua Franca: As the field of AI continues to evolve at an unprecedented pace, we are witnessing a paradigm shift from siloed AI models to unified AI platforms. Much like Kubernetes emerged as the de facto standard for container orchestration, could a single AI platform emerge as the lingua franca of AI, facilitating seamless integration and collaboration across various AI applications and domains? likelihood 8/10.

6 - CAIO - Chief AI Officer (CAIO): The role of the CAIO will be rapidly gaining prominence as organisations recognise the transformative potential of AI. As AI becomes increasingly integrated into business operations, the need for a dedicated executive to oversee and guide AI adoption becomes more evident. The CAIO will serve as the organisation's chief strategist for AI, responsible for developing a comprehensive AI strategy that aligns with the company's overall business goals. They will also be responsible for overseeing the implementation and deployment of AI initiatives across the organisation, ensuring that AI is used effectively and responsibly. In addition, they will also play a critical role in managing the organisation's AI ethics and governance framework. likelihood 10/10.

7 - [Moonshot] InterAI - Models are connected everywhere: With the advent of Gemini, we've witnessed a surge in the development of AI models tailored for specific devices, ranging from massive cloud computing systems to the mobile devices held in our hands. The next stage in this evolution is to interconnect these devices, forming a network of intelligent AI entities that can collaborate and determine the most appropriate entity to provide a specific response in an economical manner. Imagine a federated AI system with routing and selection mechanisms, distributed and decentralised. In essence, InterAI is the future of the interNet. likelihood 4/10.

8 - [Moonshot] NextLM - Beyond Transformers and Diffusion: The transformer architecture, introduced in a groundbreaking 2017 paper from Google, reigns supreme in the realm of AI technology today. Gemini, Bard, PaLM, ChatGPT, Midjourney, GitHub Copilot, and other groundbreaking generative AI models and products are all built upon the foundation of transformers. Diffusion models, employed by Stability and Google ImageGen for image, video, and audio generation, represent another formidable approach. These two pillars form the bedrock of modern generative AI. Could 2024 witness the emergence of an entirely new GenAI paradigm? likelihood 3/10.

9 - [Moonshot] NextLearn: In 2022, I predicted the emergence of a novel learning algorithm, but that prediction did not materialize in 2023. However, Geoffrey Hinton's Forward-Forward algorithm presented a promising approach that deviates from the traditional backpropagation method by employing two forward passes, one with real data and the other with synthetic data generated by the network itself. While further research is warranted, Forward-Forward holds the potential for significant advancements in AI. More extensive research is required. likelihood 2/10.

10 - [Moonshot] FullReasoning - LLMs are proficient at generating hypotheses, but this only addresses one aspect of reasoning. The reasoning process encompasses at least three phases: hypothesis generation, hypothesis testing, and hypothesis refinement. During hypothesis generation, the creative phase unfolds, including the possibility of hallucinations. During hypothesis testing, the hypotheses are validated, and those that fail to hold up are discarded. Optionally, hypotheses are refined, and new ones emerge as a result of validation. Currently, language models are only capable of the first phase. Could we develop a system that can rapidly generate numerous hypotheses in an efficient manner, validate them, and then refine the results in a cost-effective manner? CoT, ToT, and implicit code execution represent initial steps in this direction. A substantial body of research is necessary. likelihood 2/10.

11 - [Moonshot] NextProcessor - The rapid advancement of AI has placed a significant strain on the current computing infrastructure, particularly GPUs and TPUs. As AI models become increasingly complex and data-intensive, these traditional hardware architectures are reaching their limits. To accommodate the growing demands of AI, a new paradigm of computation is emerging that transcends the capabilities of GPUs and TPUs. This emerging computational framework, often referred to as "post-Moore" computing, is characterized by a departure from the traditional von Neumann architecture, which has dominated computing for decades. Post-Moore computing embraces novel architectures and computational principles that aim to address the limitations of current hardware and enable the development of even more sophisticated AI models. The emergence of these groundbreaking computing paradigms holds immense potential to revolutionise the field of AI, enabling the development of AI systems that are far more powerful, versatile, and intelligent than anything we have witnessed to date. likelihood 3/10

12 - [Moonshot] QuanTransformer - The Transformer architecture, a breakthrough in AI, has transformed the way machines interact with and understand language. Could the merging of Transformer with Quantum Computing provide an even greater leap forward in our quest for artificial intelligence that can truly understand the world around us? QSAN is a baby step in that direction. likelihood 2/10.

As we look ahead to 2024, the field of AI stands poised to make significant strides, revolutionizing industries and shaping our world in profound ways. The above 12 predictions for AI in 2024, including 6 ambitious moonshot projects could push the boundaries of what we thought possible paving the way to more powerful AIs. What are your thoughts?