Arize AI

LLMs are changing the game, making it easier than ever to build amazing apps. But here’s the catch: getting started is simple—ensuring they actually work well in the real world? That’s the tricky part. Whether you’re tweaking prompts or steering your team toward cutting-edge solutions, nailing your evaluations is how you make sure your AI delivers. 🚀 Here are a few things we cover to help you get it right: ✔️ How LLM evaluations go beyond traditional testing like unit and integration testing ✔️ Smart ways to measure quality: relevance, hallucinations, latency, and more ✔️ Building datasets that you can actually trust ✔️ Dynamic, task-based methods for evaluating real-world performance ✔️ Using CI/CD pipelines to keep improving without breaking a sweat Dive in here: https://lnkd.in/graKa-xm

Learn how to instrument your LLM app, how to create multi-agent applications, or check out our agents series (now on-demand) for real-world examples of agents in production. All this and more, right this way. 👇

Bringing SF tech vibes to Europe All my Berlin homies, defo come out! I'll be helping my friend, and always well vibed Adam Chan, and I'll be hosting the Arize AI Phoenix lighting talk / quickstart challenge Many companies like Weaviate (our gracious host), deepset's Haystack, Neon AI, Jina AI, and AssemblyAI will be giving their talks as well What to expect at Hack Night? 💡 Casual, high-energy atmosphere with a mix of building and networking ⚡ Lightning talks from community members sharing their latest projects 🧮 Interactive problem-solving sessions and spontaneous collaborations 🍕 Snacks, drinks, and great conversations with fellow tech enthusiasts 🫶 A judgment-free zone for both beginners and experienced developers Hack Night is for everyone. Whether you're here to network or ready to build, you could bring your laptop, but it's not a must. Make sure to bring your ideas and enthusiasm! Will we see you in on December 12th, in Berlin? Sign up: https://lnkd.in/gvbEFEUq

Another great Arize AI meetup at the GitHub HQ in the books! 🎉 🚀 A big thank you to all our speakers: Lorenze Jay Hernandez from CrewAI, Ofer Mendelevitch from Vectara, and Laurie Voss from LlamaIndex! One of the highlights of these events for me is always hearing the creative ideas people are trying to build. In just a few minutes, I talked to people working on: 📚 Agent knowledgebase systems 🔬 ML models for drug discovery 🧑🏫 AI-powered elementary school learning assistants and many more The other striking piece is that many of these builders have limited or no technical background. The AI development assistant dream is real! We'll be returning to Github HQ in January, see you there!

🛝 Prompt Playground is out in Phoenix 6.0 Makes it easy to iterate on prompts, replay spans, and run experiments 🔗 Multi-provider support Day 1 support for   - 🧠 OpenAI   - 🌐 Azure OpenAI   - 🤖 Anthropic   - 🧪 Google AI Studio Let us know what providers you want to see next ⚡ Iterate fast and record your progress   - 🚀 Run up to four LLMs at once for rapid iteration   - 📝 Each invocation is recorded as a span that you can label, score, and add to a dataset 🛠️ Advanced tool calling UX Phoenix knows what to expect from each LLM provider   - ✍️ Guides your hand with auto-complete and syntax highlighting when defining tool schemas   - 🔄 Automatically translates from one provider's format to another 🔁 Span replay   - 📡 Instrument your application with OpenTelemetry and OpenInference to collect traces   - 🔍 Replay any span from your development or production data to recreate the LLM invocation, including model, messages, and parameters   - 🖍️ Annotate traces and add to datasets for future experiments 🧪 Datasets and Experiments   - 📊 Run up to four configurations over an entire dataset at once   - 🤖 Automatically record experiments, then evaluate using LLM-as-a-judge and code evaluators As always, Phoenix is fully open-source. Links in the comments below 👇 If you like what you see, leave us a ⭐ on GitHub 🙏 Huge shoutout to the Arize OSS team!

Women in AI 👉 Push the boundaries of AI with at this all-day RAG hackathon next month in Palo Alto. Chance to meet people, learn something new (+ build something, of course). Space is limited!

🚀 Announcing Arize Phoenix 6.0 🚀 Featuring Prompt Playground! 🛝 Prompt engineering and management is a huge part of developing with LLMs - but it's often a tedious and difficult process. Prompt Playground makes prompt iteration much easier by letting you test changes directly from spans and datasets, without flipping between platforms. 📊 Test and compare prompts, tool definitions, output schemas, and models, directly in the platform 🎥 Replay spans with adapted prompts 🚄 Run prompts over full datasets 🧪 Automatically trace results as experiments in Phoenix

Building and maintaining agents in production is tough, and trial and error is often the name of the game. Even after you launch, unexpected performance issues can send you back to the drawing board. In this blog, Sally-Ann DeLucia pulls back the curtain on how we iterate and improve our AI Assistant, Copilot. Learn how we used Arize and Phoenix to: ✔️ Identify and resolve issues in production ✔️ Iterate workflows with precision ✔️ Build a process that scales https://lnkd.in/gCXVKVEK

Join us next week (Dec 4) at GitHub HQ in SF for a bootcamp on multi-agent frameworks and evaluation techniques. Here's the schedule 👇 5:00 PM | Check-In 6:00 PM – 6:15PM | Automate the Boring Stuff with CrewAI. Lorenze Jay Hernandez on how to automate repetitive tasks and optimize agent workflows. 6:15 PM – 6:30 PM | Production-ready Agents through Evaluation. John Gilhuly will present on the latest techniques for evaluating and improving agents with Arize. 6:30 PM – 6:45 PM | Build an Agentic RAG Application in Minutes. Ofer Mendelevitch will show how you can gain insights into optimizing multi-agent systems with cutting-edge approaches from Vectara. 6:45 PM – 7:00 PM | Agentic RAG in 2024 with Laurie Voss 7:00 PM – 8:30 PM | Networking Wrap up the evening with casual networking, great food, and refreshing non-alcoholic beverages. Register here: https://lnkd.in/gbFn2WWC

Looking forward to AWS re:Invent 2024! 👋 Lots to do if you're headed that way next week, but here's where we'll be.. Dec 3, 1:30-2p: Swing by MongoDB's booth (823), where Jason Lopatecki will be giving a lightning talk on agent evaluation strategies. Trevor U. Samantha White & Greg Chase will also be there. Meet the team, grab some swag, and catch this and other talks. Dec 3, 6-8p: We're sponsoring an AI game night on with MongoDB, Fireworks AI, and Hasura at Sugarcane Restaurant in the Venetian. Great opportunity to network, share ideas, and have some fun. ♠️ Game night: https://lnkd.in/gMJiZWGs