lytix ai

Getting started with your LLM evaluations layer can feel like a chicken and egg problem - you don’t really know what to evaluate for before you launch, but do you want to launch without evals? And if you’re using techniques like LLM as Judge or Agent as Judge, how do you know if you’ve correctly instrumented your models to catch the errors you do know about? We've found that creating evaluation agents from a dataset of example events can be a great way to get started. The literature, and our experience building evaluations for customers, have found benefits to including high-signal events in your evaluators. We spun up a tactical deep dive on how to find high signal example events, and turn them into your first evaluation agents. https://lnkd.in/g_fjnfck

Excited to launch our weekly summary feature 🪩 Get a summary of your high level metrics + specific feedback on your workflows and custom evaluations right to your inbox every week 📨! Stay ahead of bugs or regressions with @lytixai

LLM developers deserve some automation too! 🚀🙏

Day 3 🚨 Dropping the next batch of companies from The #AIHot100 Market Report (East Coast)....curated by The AI Furnace 🧨🔥 ft. 100 of the hottest AI startups (pre-seed to series A) 🧯💥 ➡️ Colossyan - Create engaging learning experiences with AI video ➡️ Ohai.ai - Say ohai to an AI Assistant for parents ➡️ lytix ai - Datadog for LLMs ➡️ Wild Moose - Helping on-call devs tame production chaos with Generative AI ➡️ Alinea Invest - AI money manager app ➡️ Cosmos - A Pinterest alternative for creatives ➡️ Opkit - Automating Routine Calls for Healthcare Providers with AI ➡️ NuMind (YC S22) - LLM-powered custom NLP ➡️ Wondercraft - the AI audio studio for creatives ➡️ Spinach AI - The AI Project Manager that automates meeting notes, tasks, and tickets We curated the Hot 100 list from over thousands of submissions, with a committee of AI, technical, startup and industry experts and brought these founders together at The #AIHot100 Conference in NYC last week. Some hidden 💎s here - this is the culmination of one year of building grassroots community with The AI Furnace 🧨🔥, getting to know founders, seeing under the radar startups, and some who cannot demo due to being in stealth. The report has Sector View, Layer View, List View and more. 👉 If interested in seeing the full list, sign up for our newsletter to see the full PDF version https://lnkd.in/eGpzsTQr #AIHot100 #aifurnace Dominik Mate Kovacs, Sheila Lirio Marcelo, Sahil Sinha, Sid Premkumar, Yasmin Dunsky, Eve Halimi, Andy McCune, Sherwood Callaway, Etienne Bernard, Oskar Serrander, Josh Willis

🏆💯

🪩 We’ve pushed some awesome features over the past few weeks and wanted to take the time to showcase our favorite 🎉 https://lnkd.in/eiZspbft

🚀🚀🚀

Meta released a ~1 month update on the Llama family of models. TLDR - Llama 3.1 is growing like crazy 📈, and Zuck clearly has his eye on enterprise 👀 https://lnkd.in/gS-hk-fB 👉 Growth data According to Meta - 1. Llama models are approaching 350 million downloads to date (more than 10x the downloads compared to this time last year) 2. Usage is up too - token volume has doubled in just 3 months, and monthly token volume is up 10x since same time last year. 3. Conclusion: “Llama is now the leading open source model family.” - Meta Enterprise - I was surprised how explicitly Meta’s blog post calls out the various enterprise use cases they’ve seen. It’s honestly an impressive variety of companies - showcasing projects from companies like Shopify and Zoom, but also traditional players like Accenture, Goldman Sachs and AT&T. A go ahead from companies like Accenture and At&t can go a long way to onboarding fellow non-tech forward conglomerates. It’s clear Meta wants enterprises of all kinds, to feel comfortable playing with Llama (Why? Check out our post here on Meta’s overall strategy with Llama, and how they’re thinking about enterprise https://lnkd.in/gR2Q5h2T) Open Source - Finally, the blog post reaffirms Meta’s commitment to open source AI. While this is certainly an attractive angle for them, I think it’s worth pointing out a couple caveats: - When Meta started ‘open sourcing’ their models, via releasing the model weights with Llama 2, it’s not clear they meant to do so. Llama2’s model weights were leaked on 4chan after Meta released Llama2 to a limited set of parties. It seems to be only after seeing the engagement from the open source community, Meta leaned into the ‘open weights’ model (https://lnkd.in/gp7xpJBi) - What even is open source for LLMs? Critics of meta have pointed out that this isn’t truly open source in the ways we’re used to. While open model weights can make customizations (like finetuning) incredibly easy, we’re still not seeing the actual data or code for generating Llama 3.1. (Don’t take my word for it, her’s the Open Source Initiative’s response to Meta’s claims that Llama 2 was open source due to publicly released model weights https://lnkd.in/gfXQ_ipG)

This LORAX does NOT 🙅 speak for the trees 🌲🌳🌴- he speaks for finetuned models everywhere 🔊 LORAX, by Predibase, is an open sourced service (https://lnkd.in/guKYfYXU) that helps AI developers train and host their own suite of small language models. Specifically finetuned models can be more cost-effective and reliable than general purpose models, especially for teams exiting the prototyping stage. But the developer experience of actually training and deploying a finetuned model, has been fiddly and cumbersome thus far. LORAX solves this in a couple major ways 👇 First, a declarative framework for training, deploying, and calling finetuned models. This provides a far smoother developer experience for training, deploying and calling finetuned models, reducing the lines of code you need to train a smaller model from 20+ lines of code to just 3-4. Second, LORAX makes hosting suites of small models cheaper, faster, and more scalable - all with less developer hassle. All your small models are hosted on a single GPU (more cost-effective than dedicating a resources for each model). LORAX also makes your deployment more memory efficient and scalable (via a strategy they call “tiered weight caching”). And for folks curious how this looks under the hood 👀  1. As the name suggests, LORAX finetunes via LORA. 2. For the declarative framework, LORAX is built on Ludwig (https://lnkd.in/gmJHhbju, an open source framework developed by Uber for declarative AI/ML development). 3. And for more context on LORAX and Predibase: LoRAX: Serve 1000s of Fine-Tuned LLMs on a Single GPU @ The Linux Foundation https://lnkd.in/g9sKpG_s A more hands on demo: https://lnkd.in/g8Bab697 Blogs n docs: https://lnkd.in/gSPCP32b, https://lnkd.in/gh38ssg2, https://lnkd.in/guKYfYXU

Well RIP 🪦 my inbox from folks asking for the code we went through 😅 Check out our latest cookbook 👨🍳 to walk through safeguarding your workflows with Optimodel (https://lnkd.in/g8wchawU)