Comet

🛠️ Building a scalable Generative AI platform is challenging, but it doesn’t have to be. Join us and Amazon SageMaker for a technical session on: ✅ The importance of LLM observability in production ✅ How Comet’s Opik can track and monitor your LLMs ✅ Effortlessly setting up Comet within SageMaker AI Partner Apps 📅 Thursday, March 6th | 13:00 - 14:00 EST 🔗 Register: https://lnkd.in/dHBztcRm

🧑💻 Join a global community of developers on May 13-14th for Convergence 2025, a virtual conference dedicated to GenAI engineering. We'll explore: 🔎 The challenges of building and deploying LLM-based applications 💡 Advanced LLM evaluation techniques ⚖️ Responsible use of GenAI 🎟️ Register for free: https://lnkd.in/dfRWtwe2

⭐ Opik officially has 5,000 GitHub Stars! ⭐ Five months ago, we launched Opik out of a growing need from the community to be able to confidently test and trust their LLM applications. Since then, the adoption and engagement we’ve seen has been beyond what we could have imagined. 🚀 Opik trending on GitHub as the #2 top repo 📈 Tens of thousands of users 🤝 Contributions and callouts from users like Andreas Nigg, Jeremy Mumford, Carlos Kemeny, PhDx2, and Prakash Chaudhary 💡 Incredible projects powered by Opik, like Chia Jeng Yang’s PatientSeek, an open-source Med-Legal Deepseek reasoning model We're grateful for the entire community's contributions. Whether you’ve contributed code, shared feedback, or spread the word, we’re excited to keep building with you 🦉

🎉 Proud to be a community sponsor at the AI Tinkerers – NYC x OpenAI Hackathon this weekend! 📣 If you're attending, be sure to say hello to Claire L., who will be representing Comet and diving into our open‐source LLM Eval framework, Opik. Can't wait to see what teams build 🤖

Comet is now on Bluesky! 💙 As a team that's committed to investing in the open-source community, we're excited to join the conversation on Bluesky. 👋 Come say hi, give us a follow, and join us as we continue to build. Find us at https://lnkd.in/dDriMEY7 🚀

🚀 Opik Weekly Changelog 🚀 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁 𝗼𝗳 𝘁𝗵𝗲 𝘄𝗲𝗲𝗸: Multiple external contributions have been released this week ! Opik is gaining momentum even faster than I expected ! One of the best parts of working on Open-Source projects are community contributions, not only do they improve the overall features of the product but they also often improve the quality of the product significantly. From day one we decided to prioritize reviewing user contributions quickly and we couldn't be happier we did ! We also released: • Performance improvements for workspaces with over 100 million traces • Added support for cost tracking when using Gemini models • Added diffing of prompt versions • Improved support for Ragas metrics in `evaluate_*` functions in the SDK • Added support for Bedrock `invoke_agent` API And as always, thank you to all of Opik's external contributors including Jeremy Mumford, Rahul Kadam, Prakash Chaudhary, @demdecuong and @jeffy !

LLM-as-a-judge evaluators may seem simple on the surface, but implementing them in real-world applications is challenging. They excel at tasks that are difficult to quantify with traditional heuristic metrics like hallucination detection, creative generation, content moderation, and logical reasoning. But evaluating a model across multiple metrics often requires creating separate LLM-as-a-Judge pipelines for each metric and combining their outputs. G-Eval simplifies this process by consolidating evaluations into a single metric, effectively providing the model with a unified scorecard. 👉 Learn more about G-Eval, including how to use it out-of-the-box with Opik, in this new article from Abby Morgan: https://lnkd.in/eF-iBMzv #GenerativeAI #ArtificialIntelligence 📣 Also, a big shoutout to the authors of the original G-Eval paper: Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu, Yang Liu, Microsoft Cognitive Services Research

🔨 Building great open-source tools is hard. Building a scalable, open-source tool that can monitor production LLM workloads? That’s even harder. Our engineering team spent much of 2024 tackling this challenge. Andrés Cruz, Principal Engineer at Comet and lead engineer on the project, breaks down the key architectural decisions and trade-offs that shaped Opik in this blog post. 🔥 👀 Take an inside look at the project 👇

💡 Building LLM apps with Dify? Meet Opik 👋 🦉Opik, our open-source LLM evaluation tool, now integrates with Dify. While Dify makes it easy to build LLM-powered apps, Opik makes advanced evaluation possible with tools like trace annotation and log evaluation. Now you can trust your LLM outputs beyond “vibe-y” guesswork — all while staying in the tools you know and love. 📚 Learn more and get started here: https://lnkd.in/d8jNA2p3

AI systems aren't software pipelines—and that's the challenge. Non-deterministic models need observability to perform predictably. Some great thoughts from Aishwarya here. Proud to have leaders in the space recommending Opik, our open-source LLM evaluation framework.