LLMOps Space

Yesterday, Google DeepMind dropped a new benchmark for evaluating the 𝐟𝐚𝐜𝐭𝐮𝐚𝐥𝐢𝐭𝐲 𝐨𝐟 𝐋𝐋𝐌𝐬.📝 They're calling it "𝐅𝐀𝐂𝐓𝐒 𝐆𝐫𝐨𝐮𝐧𝐝𝐢𝐧𝐠", it evaluates model responses automatically using the LLM-as-a-Judge methodology, incorporating a combination of different LLM judges.🕵♂️ The FACTS Grounding dataset consists of 1719 examples, of which 860 are public and 859 are private. The examples include documents with up to a maximum of 32,000 tokens (roughly 20,000 words), covering domains - finance, technology, retail, medicine, and law. The user requests are similarly wide-ranging, including use cases for summarization, Q&A generation, and rewriting tasks. 🧪 FACTS Grounding evaluates model responses automatically using three frontier LLM judges — namely 𝐆𝐞𝐦𝐢𝐧𝐢 1.5 𝐏𝐫𝐨, 𝐆𝐏𝐓-4𝐨, 𝐚𝐧𝐝 𝐂𝐥𝐚𝐮𝐝𝐞 3.5 𝐒𝐨𝐧𝐧𝐞𝐭 to mitigate any potential bias of a judge giving higher scores to their own model. On their Kaggle leaderboard, the top 3 performing models are different versions of the Gemini. 🔥 𝐋𝐢𝐧𝐤 𝐭𝐨 𝐊𝐚𝐠𝐠𝐥𝐞 𝐥𝐞𝐚𝐝𝐞𝐫𝐛𝐨𝐚𝐫𝐝: https://lnkd.in/e4AWPMQj 𝐋𝐢𝐧𝐤 𝐭𝐨 𝐭𝐡𝐞 𝐏𝐚𝐩𝐞𝐫: https://lnkd.in/eXkjk7Wb Alon Jacovi, Connie T., Jon Lipovetz, Kate Olszewska, Lukas Haas, Gaurav Singh Tomar, Carl Saroufim, Doron Kukliansky, Zizhao Z., Dipanjan Das, Google DeepMind, Google for Developers ☝️𝐅𝐨𝐥𝐥𝐨𝐰 𝐦𝐞 𝐟𝐨𝐫 𝐜𝐨𝐧𝐭𝐞𝐧𝐭 𝐚𝐛𝐨𝐮𝐭 #machinelearning, #llms, and #mlops, as well as announcements about Deepchecks'#opensource releases & the community at LLMOps Space.

Meta introduced a new 𝐛𝐲𝐭𝐞-𝐥𝐞𝐯𝐞𝐥 LLM architecture that can match the performance tokenization-based LLM transformers. 💡 It's called 𝐁𝐲𝐭𝐞 𝐋𝐚𝐭𝐞𝐧𝐭 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫 (BLT), it encodes bytes into dynamically sized patches, which serve as the primary units of computation. In the paper, they presented byte-level models with support for up to 8𝐁 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬 𝐰𝐢𝐭𝐡 4𝐓 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐛𝐲𝐭𝐞𝐬. 🔥 Basically, BLT is composed of a large global 𝐚𝐮𝐭𝐨𝐫𝐞𝐠𝐫𝐞𝐬𝐬𝐢𝐯𝐞 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥 that operates on patch representations, along with two smaller local models that encode sequences of bytes into patches and decode patch representations back into bytes. 🤔 My take here is -- I’m not entirely sure how widely this new approach will be adopted by other research teams, but it’s definitely a fresh innovation of the type that could lead to “𝐦𝐮𝐭𝐚𝐭𝐢𝐨𝐧𝐬” towards the next generation of LLMs. Although of course most of these “mutations” don’t end up being promising enough =] Feel free to share your thoughts in the comments. 📝 𝐑𝐞𝐚𝐝 𝐭𝐡𝐞 𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞 𝐩𝐚𝐩𝐞𝐫 𝐡𝐞𝐫𝐞: https://lnkd.in/d7SvmUGk 👨💻 𝐆𝐢𝐭𝐇𝐮𝐛: https://lnkd.in/dSMPh-NC Srini Iyer, Yann LeCun, Artidoro Pagnoni, Pedro Rodriguez, John Nguyen, Gargi Ghosh, AI at Meta, Meta ☝️𝐅𝐨𝐥𝐥𝐨𝐰 𝐦𝐞 𝐟𝐨𝐫 𝐜𝐨𝐧𝐭𝐞𝐧𝐭 𝐚𝐛𝐨𝐮𝐭 #machinelearning, #llms, and #mlops, as well as announcements about Deepchecks'#opensource releases & the community at LLMOps Space.

Excited to share our new partnership with Amazon Web Services (AWS)! ❤️ Today, in the AWS re:Invent keynote, Swami Sivasubramanian announced their newest feature -- SageMaker Partner AI Apps. Deepchecks is one of the core parts of this launch, deeply integrated into Amazon SageMaker AI and Amazon SageMaker Unified Studio. ✅ By combining Deepchecks' expertise in LLM evaluation with Amazon SageMaker AI, we're enabling teams and enterprises to build a best-in-class LLMOps stack without worrying about data security or procurement. 🏁To get started, you can either search for Deepchecks within Amazon SageMaker AI or Amazon SageMaker Unified Studio, or reach out to us. ❤️ Huge kudos to Philip Tannor, Shir Chorev & the whole Deepchecks team! #LLMs #LLMOps #AWS #reInvent #SageMaker #Deepchecks LLMOps Space

🚀 I’m extremely excited to finally announce a 𝐦𝐞𝐠𝐚-𝐜𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧 between Amazon Web Services (AWS) (!) and Deepchecks around LLM Evaluation. 🕑 This was announced 𝐚 𝐟𝐞𝐰 𝐦𝐢𝐧𝐮𝐭𝐞𝐬 𝐚𝐠𝐨 during the AWS #reInvent keynote, as part of their announcement about their newest feature - 𝐒𝐚𝐠𝐞𝐌𝐚𝐤𝐞𝐫 𝐏𝐚𝐫𝐭𝐧𝐞𝐫 𝐀𝐈 𝐀𝐩𝐩𝐬. 🧠 Essentially, Deepchecks is now available 𝐧𝐚𝐭𝐢𝐯𝐞𝐥𝐲 𝐰𝐢𝐭𝐡𝐢𝐧 𝐒𝐚𝐠𝐞𝐌𝐚𝐤𝐞𝐫 - so companies building LLM Apps can use their existing AWS credits to get a fully managed, on-prem, version of Deepchecks LLM Evaluation. 🔥Over the last few months, our teams have collaborated closely to address the challenges in building robust LLM applications within AWS. By combining Deepchecks' expertise in LLM evaluation with Amazon SageMaker AI, we're enabling teams and enterprises to build a best-in-class LLMOps stack 𝐰𝐢𝐭𝐡𝐨𝐮𝐭 𝐰𝐨𝐫𝐫𝐲𝐢𝐧𝐠 𝐚𝐛𝐨𝐮𝐭 𝐝𝐚𝐭𝐚 𝐬𝐞𝐜𝐮𝐫𝐢𝐭𝐲, 𝐩𝐫𝐨𝐜𝐮𝐫𝐞𝐦𝐞𝐧𝐭, 𝐨𝐫 𝐈𝐓/ 𝐃𝐞𝐯𝐎𝐩𝐬 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬. Here's why this is a big deal: ✅ Data Locality within your AWS account, guaranteed by AWS. ✅ No Security Questionnaires or Architecture Discussions—AWS takes full responsibility. ✅ Zero-Time Procurement Process. Tools can be acquired with your AWS budget, including unused AWS credits. ✅ No Infrastructure Management. SageMaker AI handles setup, upgrades, and scaling. Here’s the detailed blog I wrote about the partnership: https://lnkd.in/dwGmdQBg 🏁To get started, you can either search for Deepchecks within Amazon SageMaker AI or Amazon SageMaker Unified Studio, or reach out via LinkedIn or email me at philip@deepchecks.com #Deepchecks #LLMs #AWS #reInvent #SageMaker 👏 Huge kudos to Swami Sivasubramanian Ankur Mehrotra Guy Maidan Mair Hasco Kelly Garcia Thomas Zerbach Sai Kumar Devulapalli Jia You Sumit Thakur Gwen Chen Sumit Thakur Eric Raúl Peña Vineet Sharma Rafe Gándola and many others, along with my amazing team at Deepchecks, that made this possible!

We're hosting a 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 𝐰𝐨𝐫𝐤𝐬𝐡𝐨𝐩 about "𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐧𝐠 𝐑𝐀𝐆 & 𝐋𝐋𝐌 𝐀𝐩𝐩𝐬".🚀 In this session, Shir Chorev (CTO & Co-Founder) & Nadav Barak (Head of AI) from our team, lead a hands-on workshop on evaluating RAG and LLM-based applications. 💡 We will give all attendees access to the Deepchecks system during the workshop. 🤓 We will cover methodologies for assessing initial experiments, comparing versions, and performing ongoing evaluations in production using Deepchecks LLM Evaluation. ✅ ( ❤️ 300+ people have signed up for this one already) 🚀 𝐑𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐡𝐞𝐫𝐞: https://lnkd.in/dscdK9HH 📆 𝐃𝐚𝐭𝐞 & 𝐓𝐢𝐦𝐞: Oct 8th, 2024 | 08:00 AM PST Feel free to drop your questions for speakers here in the comment section. 💬 #LLMOps #LLMs #AI #ML #GenAI #AISafety

😇Did you see OpenAI's announcement about #SearchGPT? OK, here is my bet - #SearchGPT won't be a big deal at first. They will see there are many missing features (like easy display of local businesses, easy transition to other websites with previews, etc), it will be a tiny improvement from ChatGPT. And won't really bite more into Google's user base than #ChatGPT already has... Then after a while, 1 of 2 things will happen. Either they will make major major changes to the experience, or they will take a step back and announce it didn't really work. Thoughts? ☝️𝐅𝐨𝐥𝐥𝐨𝐰 Philip Tannor 𝐟𝐨𝐫 𝐜𝐨𝐧𝐭𝐞𝐧𝐭 𝐚𝐛𝐨𝐮𝐭 #artificialintelligence , #llms 𝐚𝐧𝐝 #llmops, as well as announcements about Deepchecks' #opensource releases & the community at LLMOps Space.

The upcoming LLMOps Space event is about "𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐭𝐡𝐞 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞 𝐟𝐨𝐫 𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐨𝐟 𝐋𝐋𝐌𝐬". 🏗 In this session, Maksim Nekrashevich, ML & LLM Engineer from Nebius AI will discuss reinforcement learning with human feedback (RLHF), prompt tuning, and AI workflow management. 🚀 We will cover the key aspects of aligning LLMs and explore how to set up the necessary infrastructure to maintain a versatile alignment pipeline. ✅ 🚀 𝐑𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐡𝐞𝐫𝐞: https://lnkd.in/dM4sj9ur 📅 𝐃𝐚𝐭𝐞 & 𝐓𝐢𝐦𝐞: July 11th, 2024 | 8.00 AM PST | 5.00 PM CET 📣 The session will be hosted by Philip Tannor, CEO and Co-Founder at Deepchecks. #LLMOps #MLOps #LLMs #GenAI #AI #ML

🌌 Upcoming webinar: Taming AI or How we build the alignment pipeline https://lnkd.in/dvpFvsAw Speaking at the LLMOps Space community's webinar will be Maksim Nekrashevich, ML & LLM Engineer at Nebius AI. Accompanied by Philip Tannor, CEO and Co-Founder at Deepchecks, Maxim will discuss: - Incorporating LLMs into the data collection for supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to maximize efficiency. - Techniques for instilling desired behaviors in LLMs through the strategic use of prompt tuning. - An exploration of cutting-edge workflow management and how it facilitates rapid prototyping of highly-intensive distributed training procedures. When: Thursday, July 11 at 17:00 UTC+2 / 8:00 AM PST Where: Zoom Register: https://lnkd.in/dvpFvsAw #webinars #LLMs #alignment #SFT #RLHF #promptengineering

🔥 Two weeks ago we released #TextGrad, our new library for automated prompt optimization. The feedback we got since then has been amazing, with more than 600 stars on GitHub! ⭐ 📕 GitHub: https://lnkd.in/g9grWEwf 📕 Paper: https://lnkd.in/gYg3h7dP A short summary: 1️⃣ TextGrad is an "autograd for text" and provides an automated way to improve prompts with few lines of code. TextGrad's syntax is similar to PyTorch's, so it should all feel very familiar! 2️⃣ TextGrad implements an entire engine for backpropagation through text feedback provided by LLMs, strongly building on the gradient metaphor: We can optimize compound AI systems. 3️⃣ TextGrad can provide feedback on system prompts or coding solutions and optimize them! We have also applied TextGrad to molecule and treatment plan optimization! Amazing work led by Mert Yuksekgonul, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou

A general-purpose language model can only process relatively simple visual tasks such as answering basic questions about an image or generating short captions. 🤓 This is primarily due to the lack of access to detailed pixel-level information, object segmentation data, and other granular annotations that would allow the model to precisely understand and reason about the various elements, relationships, and context within an image. 👾 ✅ Fine-tuning LMMs on domain-specific data can significantly improve their performance for targeted tasks. Learn how to 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐞 𝐚𝐧𝐝 𝐝𝐞𝐩𝐥𝐨𝐲 𝐭𝐡𝐞 𝐋𝐋𝐚𝐕𝐀 𝐦𝐨𝐝𝐞𝐥 on Amazon SageMaker. 👩💻 𝐋𝐢𝐧𝐤 𝐭𝐨 𝐆𝐢𝐭𝐇𝐮𝐛: https://lnkd.in/d3VgZFyr 👉 𝐋𝐢𝐧𝐤 𝐭𝐨 𝐚𝐫𝐭𝐢𝐜𝐥𝐞: https://lnkd.in/gccN6ycM Changsha Ma, Alfred Shen, Amazon Web Services (AWS) #LLMOps #MLOps #LLMs #AWS #GenAI #LLaVa #AI #ML