10 takeaways from interviewing AI researchers

Over the past few months, I recently had the opportunity of interviewing many leading researchers in AI from Stanford, Berkeley, Georgia Tech and other industries / executives. Their insights were fascinating and I thought I'd share 10 provocative takeaways that emerged from these conversations. The 3 themes that emerged were around the challenges with the architecture & models, expanding current AI capabilities, and predictions about the future of AI.

Architectural and Model Development Challenges: Are Compound Systems a Better Approach?

1. The inertia around transformer architectures is real. Unless a new company bets big, we'll likely see incremental improvements rather than architectural revolutions. This is partly due to the massive investment in optimizing transformers at every level, from chip design to software frameworks. Breaking this inertia would require not just a superior architecture, but a willingness to rebuild the entire AI infrastructure stack.
2. We don't use all parts of our brain for every task. Similarly, AI's future might lie in task-specific model routing rather than monolithic models. This approach, exemplified by systems like mixture-of-experts, could lead to more efficient and adaptable AI systems. By activating only the relevant parts of a model for a given task, we could create AI systems that are both more powerful and more computationally efficient.
3. Open-source models getting more specialized could be just as impactful as closed-source models getting more general. While companies like OpenAI and Google push for more general AI, there's a parallel opportunity in creating highly specialized open-source models for specific domains or tasks. This specialization could lead to more efficient, accurate, and accessible AI solutions for particular industries or problem types.
4. Grounding LLMs in the real world remains one of the hardest unsolved challenges in AI. We're still relying too heavily on language priors. This is particularly evident in tasks requiring real-world interaction or understanding, such as robotics or complex web navigation. Solving this challenge could lead to more reliable and versatile AI systems capable of true real-world problem-solving.

Expanding AI Capabilities and Applications: How can we work within the current limitations of AI?

1. There's more money (and immediate value) in improving LLM reasoning and planning capabilities than in pushing multimodal boundaries. While multimodal AI is exciting, the ability to reason, plan, and execute complex tasks has more direct business applications. Improvements in these areas could lead to AI systems that can handle more complex, multi-step problems, potentially revolutionizing fields like strategic planning and complex decision-making.
2. The future of AI lies in combining models with specialized tools, not just improving the models themselves. The real breakthroughs will come from AI systems that can effectively utilize a variety of external tools and data sources. This approach could lead to AI assistants that are not just knowledgeable, but truly capable of complex problem-solving by leveraging the right tools for each task. It's not just about making smarter models, but about creating smarter systems.
3. Closed-loop systems with teacher and student AIs might be the key to dramatically improving prompt engineering and AI instruction following. This approach involves using one AI to generate instructions or prompts, another to execute them, and a third to evaluate the results and provide feedback. Such systems could lead to rapid improvements in AI performance and adaptability, potentially automating much of the current manual work in prompt engineering and AI fine-tuning.

Future Directions: Structured data, complex tool use and more reliability for longer tasks?

1. The future of AI isn't just about language - it's about bridging the gap to private relational databases where the real value lies. Most organizations' most valuable data isn't in unstructured text, but in structured databases. Developing AI systems that can effectively work with this type of data, while maintaining privacy and security, could unlock enormous value in fields like supply chain, finance, healthcare, and business intelligence. Our portfolio company @Ikigai is doing exactly such.
2. The next frontier in AI is mastering multi-tab browsing and complex tool use. Current AI struggles with tasks that require switching between multiple contexts or using a variety of tools. Solving this challenge could lead to AI assistants capable of handling complex, multi-step tasks that mirror human workflow. This advancement would significantly expand the range of real-world problems AI could solve independently.
3. The real economic impact of AI will come from models that can reliably handle 30-45 minute complex tasks, not just quick interactions. Current AI excels at quick queries or simple tasks, but the next frontier is sustaining competence over longer, more complex operations. Achieving this could lead to AI systems capable of tasks like comprehensive data analysis, complex coding projects beyond entry level engineering, or detailed strategic planning & forecasting.