DAIR.AI

VideoRAG A framework that enhances RAG by leveraging video content as an external knowledge source. Unlike existing RAG approaches that primarily focus on text or images, VideoRAG dynamically retrieves relevant videos based on queries and incorporates both their visual and textual elements into the generation process. The framework utilizes Large Video Language Models (LVLMs) to process video content directly, enabling more effective capture of temporal dynamics, spatial details, and multimodal cues that static modalities often fail to convey. For videos lacking textual descriptions, they propose using automatic speech recognition to generate transcripts, ensuring both visual and textual modalities can be leveraged. The system achieves particularly strong results in domains requiring procedural knowledge or visual demonstrations, such as "Food & Entertaining" tasks. paper: https://lnkd.in/dBXPPh-T -- Learn how to build RAG systems here: https://lnkd.in/eEiYwhVx

LLMQuoter Enhances RAG capabilities using a "quote-first-then-answer" strategy. Adopts Llama-3B and finetunes with LoRA on a 15K sample subset of HotPotQA to enhance RAG by identifying key quotes before passing them to reasoning models. "This workflow reduces cognitive overhead and outperforms full context approaches like Retrieval-Augmented Fine-Tuning (RAFT), achieving over 20-point accuracy gains across both small and large language models." paper: https://lnkd.in/em3B-7mh -- Learn to build RAG systems in my new course: https://lnkd.in/eEiYwhVx

Here are the top ML Papers of the Week (Jan 6-12): - Search-o1 - Agent Laboratory - A Survey on LLMs - Towards System 2 Reasoning - Cache-Augmented Generation - Long Context vs. RAG for LLMs Read on for more:

Getting a lot of questions about where to learn how to build with AI. I got tired of pointing to outdated resources. This is why I built the AI Academy. Here is my recommended learning path: 1) Introduction to Prompt Engineering: Learn the best practices for prompting LLMs 2) Advanced Prompt Engineering: Apply more robust and advanced prompting techniques to your LLM applications 3) Introduction to RAG: Learn how to equip LLMs with external data sources and build more reliable AI applications 4) Introduction to AI Agents: Build agentic workflows for different domains. 5) Coding with AI: Push yourself by putting it all together using an AI coding assistant like Cursor. 6) Build and learn with a community: We have a very active community of learners so you can ask questions directly to me and other devs and professionals. Check out the AI Academy here: https://lnkd.in/embaNV_d

Large Reasoning Models meets Agentic RAG o1 like other standard LLMs often suffers from knowledge insufficiency. This work introduces Search-o1 to enhance large reasoning models (LRMs) with an agentic RAG mechanism and reason-in-documents module. What's new in Search-o1? It integrates an agentic search workflow into the reasoning process. This enables dynamic retrieval of external knowledge which helps with LRMs with knowledge gaps. What's the reason-in-documents module for? This helps to analyze and refine the retrieved information obtained from the search agent as it's typically verbose in nature. The refined docs are then injected into the reasoning chain. Results: The authors claim to observe good performance on complex reasoning tasks in science, maths, coding, and many QA benchmarks. My thoughts: The lack of complex knowledge understanding is something I have observed in my own experiments from models like o1 and Deepseek R1. This agentic search workflow can potentially help with further improving the reliability of LRMs. paper: https://lnkd.in/ejC5_FYf code: https://lnkd.in/eNiESK3y --- Learn how to build with AI agents and LLMs in my new courses: https://lnkd.in/eug3D2-h

Learning how to think with Meta Chain-of-Thought Proposes Meta Chain-of-Thought (Meta-CoT), which extends traditional Chain-of-Thought (CoT) by modeling the underlying reasoning required to arrive at a particular CoT. The argument is that CoT is naive and Meta-CoT gets closer to the cognitive process required for advanced problem-solving. This is a very detailed paper (~100 pages) presenting ideas and methods to achieve system 2 reasoning in LLMs. Lots of interesting discussion around scaling laws, verifier roles, iterative refinement, and the search for novel reasoning algorithms. paper: https://lnkd.in/e7bYiPeX ↓ Enjoy reading AI papers? Join 100K+ researchers and devs for our weekly summary of top AI papers: https://lnkd.in/e6ajg945

Agent Laboratory An approach that leverages LLM agents capable of completing the entire research process. Main findings: 1) Agents driven by o1-preview resulted in the best research outcomes 2) Generated machine learning code can achieve state-of-the-art performance compared to existing methods 3) Human feedback further improves the quality of research 4) Agent laboratory significantly reduces research expenses paper: https://lnkd.in/eW4kVS79 code: https://lnkd.in/eYUH4dMS --- Learn how to build AI Agents for different use cases here: https://lnkd.in/e5-c6f45

Agents Overview Great write-up on Agents by Chip. Here are my takeaways: 🤖 Agents Overview An AI agent is made up of both the environment it operates in (e.g., a game, the internet, or computer system) and the set of actions it can perform through its available tools. This dual definition is fundamental to understanding how agents work. 👨💻 Agent Example The figure shows an example of an agent built on top of GPT-4. The environment is the computer which has access to a terminal and filesystem. The set of action include navigate, searching files, viewing files, etc. 🧰 Importance of Tools Tools allow agents to both perceive their environment (through read actions) and modify it (through write actions). Adding appropriate tools can dramatically expand what an agent can do, from performing calculations to accessing real-time information. 💡 Tool Selection More tools give agents more capabilities but also make it harder for them to use them effectively. Finding the right tool inventory requires careful experimentation and analysis of usage patterns. 🧩 Planning Effective agents require robust planning capabilities to break down complex tasks into manageable steps. This planning should ideally be decoupled from execution to allow for validation before running potentially costly or time-consuming operations. 📍 Foundation Models Can Act as Planners While there's debate about whether LLMs can truly plan, they can be effective components of planning systems, especially when augmented with appropriate tools and reflection capabilities. ⛓️ Multi-Agent Systems Most practical agent implementations are multi-agent systems, with different components handling plan generation, validation, and execution. This separation of concerns allows for better specialization and error handling. 🎛️ Control Flows Agent plans can involve various control flows beyond simple sequential execution, including parallel execution, conditional statements, and loops. However, more complex control flows are harder to generate and execute correctly. 💭 Reflection and Error Correction While not strictly required, reflection capabilities (the ability to evaluate progress and correct mistakes) significantly improve agent performance. This can be implemented through self-critique or separate evaluation components. ❌ Failure Modes Agents can fail in multiple ways, including planning failures (invalid tools or parameters), tool execution failures (incorrect outputs), and efficiency failures (taking too long or using too many resources). 📈 Evaluation Proper agent evaluation needs to consider multiple metrics, including success rate, efficiency, cost, and time taken. This should be done across different tasks and compared against appropriate baselines. Full blog post: https://lnkd.in/egwCKD3J If you want to take it a step further, I would highly recommend my new course on AI agents: https://lnkd.in/e5-c6f45

Can LLMs generate good questions? Question generation with LLMs is useful for domains such as education, dialog systems, and model evaluation. This work systematically evaluates the quality of questions generated with LLMs. Here are the main findings: > There is a strong preference for asking about specific facts and figures in both LLaMA and GPT models. > The question lengths tend to be around 20 words but different LLMs tend to exhibit distinct preferences for length. > LLM-generated questions typically require significantly longer answers. > Human-generated questions tend to concentrate on the beginning of the context while LLM-generated questions exhibit a more balanced distribution, with a slight decrease in focus at both ends. These are all great insights to remember if you want to use LLMs to synthesize a question bank effectively. paper: https://lnkd.in/ehZ6B8Vq ↓ Enjoy reading AI papers? Join 100K+ researchers and devs for our weekly summary of top AI papers: https://lnkd.in/e6ajg945

LLMs for AGI Provides an in-depth discussion of foundational problems -- embodiment, symbol grounding, causality and memory -- that are required for LLMs to attain human-level general intelligence. A great read for everyone interested in AGI research. paper: https://lnkd.in/eBfPnHMZ -- Learn about LLMs in my new courses: https://lnkd.in/eug3D2-h