Galileo🔭

📊 Our Agent Leaderboard is 𝗹𝗶𝘃𝗲! We built a comprehensive benchmark of which LLMs work best for AI Agents 👀 After evaluating 17 leading LLMs across 14 diverse datasets, we're excited to share our findings about which models truly excel at tool-calling—and are ready to power AI agents to solve 𝘳𝘦𝘢𝘭-𝘸𝘰𝘳𝘭𝘥 𝘱𝘳𝘰𝘣𝘭𝘦𝘮𝘴 effectively. Key discoveries: 🏆 Google's 𝗚𝗲𝗺𝗶𝗻𝗶-𝟮.𝟬-𝗳𝗹𝗮𝘀𝗵 𝗱𝗼𝗺𝗶𝗻𝗮𝘁𝗲𝘀 with a 0.938 score at remarkably low cost 💸 The top 3 models span a 10𝘹 𝘱𝘳𝘪𝘤𝘦 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘤𝘦 with only 4% performance gap: 𝘀𝗼𝗺𝗲 𝗼𝗳 𝘆𝗼𝘂 𝗮𝗿𝗲 𝗼𝘃𝗲𝗿𝗽𝗮𝘆𝗶𝗻𝗴!  🛠 Mistral AI's mistral-small-2501 𝗹𝗲𝗮𝗱𝘀 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 options, matching GPT-4o-mini at 0.832 ❌ 𝗦𝘂𝗿𝗽𝗿𝗶𝘀𝗲 𝗳𝗮𝗶𝗹𝘂𝗿𝗲: DeepSeek AI V3 and R1 didn't make the rankings due to limited function calling support—making them ineffective for enabling AI agents to leverage tools Get more insights, dive into the full analysis and explore the interactive leaderboard on Hugging Face - https://lnkd.in/guG_N9kC 𝗔𝗻𝗱 𝗹𝗲𝘁 𝘂𝘀 𝗸𝗻𝗼𝘄: which LLM are you using for your AI agents? Are you getting the best value for your spend? 🤔 #AIAgents #MachineLearning #ArtificialIntelligence #LLMs #Gemini

If you’re building with AI, you know separating hype from reality is a full-time job. That’s why last week’s conversations—from meeting AI engineers in SF to testing real-world agentic workflows—were invaluable. 🔥 Energy-filled rooms for both of our Agentic Evaluation deep dives, whether it was a webinar with Roie Schwaber-Cohen, the talk from Atindriyo Sanyal and Erin Mikail Staples at AI Engineering Summit, or even our ChatGPT Roulette event in San Francisco with Jam and LaunchDarkly. From impromptu conversations to deep dives on AI evaluation, it was a week of: 🤖 Understanding what makes AI reliable (spoiler: it’s not just bigger models) 🔍 Measuring success beyond just “it works” 🤝 Connecting with the best minds pushing AI forward Next stop: HumanX Conference! Let’s talk AI evaluation, hallucination detection, and building trustworthy GenAI systems. #AI #GenAI #AIEngineering #AgenticEvaluations

LLM-as-judge approaches are the current standard, but has major limitations. In my recent interview with The New Stack, I discussed why enterprises need a more holistic approach to AI evaluation. Traditional evaluation methods face several critical challenges that include: • Position bias, verbosity bias, and self-enhancement bias limit accuracy • Rate limits and API restrictions severely impact application quality • Tracing errors through complex AI systems remains difficult • Open-source solutions tend to be "insufficient and myopic" At Galileo🔭, we've developed complementary solutions to address these limitations: 1️⃣ ChainPoll: Our agentic evaluation framework provides step-function improvements over basic LLM-as-judge approaches with customizable hallucination definitions. 2️⃣ Luna: Our suite of lightweight, fine-tuned evaluation models (440M parameters vs GPT-3.5's 175B) that outperforms both open-source alternatives and our own ChainPoll in hallucination detection benchmarks. The key insight? AI evaluation shouldn't just generate numbers. It should provide actionable, qualitative insights that integrate seamlessly into your development workflow with minimal code changes. What's exciting is how enterprises are implementing this tech. With 2 lines of code, developers at HP, Twilio, Reddit, and Comcast integrate evaluation guardrails that detect open-domain, closed-domain and custom hallucinations. We're driving the evolution from simple metrics to adaptive evaluation agents that improve your metric outcomes through human feedback. Thanks to Loraine Lawson and The New Stack for the opportunity to discuss this emerging field of AI agentic evaluation. https://lnkd.in/gNVgWFRF #AIEvaluation #AIAgents #GenerativeAI #TechLeadership

🔥 𝗝𝗨𝗦𝗧 𝗥𝗘𝗟𝗘𝗔𝗦𝗘𝗗: 𝗢𝘂𝗿 𝗟𝗮𝘁𝗲𝘀𝘁 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁 𝗟𝗲𝗮𝗱𝗲𝗿𝗯𝗼𝗮𝗿𝗱 𝗦𝗵𝗼𝘄𝘀 𝗦𝘂𝗿𝗽𝗿𝗶𝘀𝗶𝗻𝗴 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 We've just updated our AI Agent Leaderboard at Galileo, and the performance rankings challenge conventional wisdom about which models deliver the best value for AI agents. The headline finding: Gemini-2.0-flash-lite dominates with a 0.933 performance score, outperforming GPT-4.5 at a fraction of the cost. Three critical insights from our comprehensive evaluation: • 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲-𝘁𝗼-𝗖𝗼𝘀𝘁 𝗥𝗮𝘁𝗶𝗼: The top 3 models and GPT-4.5 span a staggering 1000x price difference while showing only a 2% performance gap. This raises important questions about cost efficiency in production AI agents. • 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲 𝗣𝗿𝗼𝗴𝗿𝗲𝘀𝘀: Mistral-small-2501 leads the open source category at 0.83, performing on par with GPT-4o-mini. This signals the growing maturity of open source models for tool-calling capabilities. • 𝗠𝗼𝗱𝗲𝗹 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗛𝗶𝗲𝗿𝗮𝗿𝗰𝗵𝘆: Claude-3.7-sonnet (0.953) > Gemini-2.0-flash (0.938) > GPT-4.5 preview (0.900) demonstrates a clear performance ranking across the major AI providers. Our evaluation covered 20 models across 14 diverse datasets, assessing real-world AI agent capabilities and tool selection quality. 𝚆̲𝚑̲𝚊̲𝚝̲'̲𝚜̲ ̲𝙽̲𝚎̲𝚡̲𝚝̲?̲ We're raising the bar. Our upcoming evaluations will incorporate more challenging metrics focused on real-world scenarios with additional complex and specific datasets. As AI agents grow more sophisticated, the foundation models powering them must improve in decision quality, goal alignment, and task completion—all while maintaining reasonable costs for builders. What other metrics or test cases would you like to see in our next evaluation? Check out the full updated leaderboard and methodology in the comments 👇 #AIAgents #GenAI #ArtificialIntelligence #LLM #ModelEvaluation #AgentEvaluation

𝗧𝗵𝗲 𝗔𝗜 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗖𝗵𝗲𝗮𝘁 𝗦𝗵𝗲𝗲𝘁: 𝗕𝗲𝘆𝗼𝗻𝗱 𝗧𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆 🌐 Technical leadership in the AI era demands more than technical skills—it requires a holistic approach. Galileo CEO Vikram Chatterji shared three critical capabilities that separate good from great AI teams on the Dev Interrupted Podcast: 🛠️ Continuous Building • Hands-on experimentation • Practical prototype development • Technical skill refinement 📚 Relentless Learning • Stay ahead of emerging trends • Embrace radical curiosity • Challenge existing assumptions 🤝 Strategic Community Engagement • Cross-pollinate insights • Share real-world experiences • Learn from collective intelligence Our evaluation platform isn't just a tool—it's a catalyst for transformative AI development. We help engineering leaders turn potential into performance. #AIInnovation #EngineeringLeadership #TechEvolution

Totally Agree with This Agents have huge potential but we also need to be very careful, they have lots of potential complications, so Only use AI Agents when truly need it, and focus on evaluation and monitoring, Many companies seem to be focusing on how "easy" is to build on their platforms but very few talk about the truly hard component make sure that they work every time (or most of the times at least)

We are falling into a dangerous trap with AI agents - treating them like magical solutions that will instantly solve complex business challenges. The reality is far more nuanced. Most leaders don't realize there's a critical difference between reactive AI assistants and truly autonomous AI agents. An assistant follows your lead, but an agent can proactively make decisions and take action on your behalf. This isn't just semantics - it's a fundamental shift in how intelligent automation works. Contrary to popular belief, AI agents aren't a recent invention born from large language models. They've been a concept in technology for decades, leveraging a rich ecosystem of AI techniques - from rule-based systems to advanced optimization and deep learning models. Today's LLM-powered excitement is just the latest chapter in a much longer story of intelligent automation. The key for enterprise leaders? Look beyond the hype. Understand the true capabilities and strategic potential of AI agents. They're not magic wands, but powerful tools that require careful, thoughtful implementation. 💡 Are you seeing beyond the AI agent illusion to real value? 🎧 Unpack the full insights in our latest podcast episode featuring Gartner's Haritha Khandabattu: https://lnkd.in/gyBj5_5k #AIAgents #EnterpriseAI #AIStrategy #TechInnovation

🔥 Who is building AI Agents out there? ⬇️ Have a look at Gemini 2.0 Flash, the most perfoming and cost-effective model for agent applications and real-world agentic scenarios according to Galileo🔭and Hugging Face!! More here 👉 https://lnkd.in/dn3Nbb-e

** Back from the Bay – Inspired by Galileo AI’s Leadership & Vision ** My recent trip to #SanFrancisco was nothing short of incredible — not just for the work we accomplished but for the amazing team and leadership I had the privilege to work with at Galileo🔭 What stood out most wasn’t just the depth of the sales messaging + methodologies we explored — it was the alignment, vision, and commitment of the Galileo leadership team. Yash Sheth, Brent Chalker, Jason Garoutte, and the entire leadership group are deeply invested in setting their team up for success — not just in driving revenue but in building a culture of excellence, customer impact, and continuous growth. From the very first session, it was clear that this team isn’t just chasing deals — they’re committed to solving real challenges for their customers, shaping the way #AI is adopted, and ensuring every interaction adds value. This wasn’t just a normal #SKO — it was the beginning of a journey toward sustained sales excellence. The Galileo team has the leadership, the drive, and the vision to create something truly special, and I’m excited to see how they continue to grow and scale. Looking forward to what’s ahead for Galileo AI! Massive Gratitude and a special shoutout to Brent Chalker — great to reunite and work together again after 10 years since our #RallySoftware days... your leadership is Inspiring!! #Leadership #SalesExcellence #SalesTraining #ChallengerSales #MEDDIC #RevenueGrowth #GalileoAI #CustomerImpact #SalesLeadership #SanFrancisco #Grateful