Ragas

We are hiring for engineering roles remotely. We are building a small, slick team of passionate individuals to build a supercharged workflow for evaluating and optimizing LLM applications. Apply here: https://lnkd.in/gRiZShfs

🚀 Synthetic data is reshaping the way we train and evaluate AI models. But how do you tailor high-quality synthetic data to fit your unique needs?🤔 Our latest blog explores Synthetic Data Generation for : 👉 Synthetic Data for Pre-training 👉 Fine-tuning with Synthetic Data 👉 Model Alignment and Safety 👉 Evaluating LLM Applications https://lnkd.in/gFv5xqcU

There are 4 metrics you should use to evaluate your RAG pipeline. Here’s how to easily calculate them in Python, using the Ragas library — 1. Faithfulness: Measures how accurately the generated answer aligns with the given context, which indicates factual consistency. It is scored from 0 to 1, with higher values indicating better faithfulness. 2. Answer Relevance: Assesses how directly and appropriately the generated answer addresses the original question. It uses mean cosine similarity between the original question and questions generated from the answer, with higher scores indicating better relevance. 3. Context Precision: Evaluates whether all relevant items in the contexts are ranked higher. Scores range from 0 to 1, with higher values indicating better precision. 4. Context Recall: Measures how well the retrieved context matches the ground-truth answer. It ranges from 0 to 1, with higher scores indicating better alignment with the ground truth. Ragas makes it easy to evaluate your RAG pipeline on these metrics. You can create a dataset of QAs with their respective context, and choose multiple metrics to evaluate them on. Link to the library docs is in the comments. #AI #LLMs #RAG

Ragas today hits 500,000+ downloads per month. All credit to our awesome open-source community ❤️ Supercharge your LLM application evaluation with Ragas ⭐️ https://lnkd.in/drY7MQHW

Don't make the mistake 80% of AI engineers do when building RAG evaluations.   Many forget to measure the individual components of RAG and instead only focus on the output accuracy or relevance. To truly get consistent results with RAG, you need to evaluate these systems at multiple stages: 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐒𝐭𝐚𝐠𝐞: - Context Precision: What percentage of the retrieved documents are actually relevant to the query? - Context Recall: Out of all relevant documents, what percentage does the system successfully retrieve? If document ranking is important, consider metrics like: - NDCG (Normalized Discounted Cumulative Gain) - MRR (Mean Reciprocal Rank). 𝐎𝐯𝐞𝐫𝐚𝐥𝐥 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Evaluate the system end-to-end to ensure all components work harmoniously. Think of these questions when evaluating your RAG system:  - How scalable is it, both in terms of data storage and query traffic? - How much data can you process in bulk at once when indexing?  - What is query latency? and more... For a comprehensive evaluation framework, consider using ragas, an open source LLM evaluation library. This tool is specifically designed to assess both the retrieval and generation components of any RAG application. ... If you're eager to learn more about optimizing RAG systems, check out my course "RAG with Langchain" on the DataCamp's platform. You can take the first chapter for free here - https://lnkd.in/gnGXMkTe And if you want to gain full access to all of their courses, you can get 50% off within the next 36 hours! Use the link here: https://lnkd.in/guVFxeQq And start building RAG projects!

Aligning synthetically generated test data for RAG to what's observed in production can be hard. To solve this, we have found that using personas while generating Q&A can be a game changer. ⭐️ Personas indicate the role and a neat description of the set of users that are likely to interact with your knowledge base using RAG. Now generate much more aligned test sets using an automatic persona generator in Ragas. 👉🏽 Check out the tutorial: https://lnkd.in/gvcQrWB9

Weekly release update 📣 👉🏽 Ragas v0.2.5 release is out 🎉 👉🏽 Major Changes ► Added Batching support for evaluations, thanks to @ahgraber ► @llama_index support for the v0.2 version, thank @joaorura and @suekou for contributing fixes to make this happen. You can check out the docs → https://lnkd.in/eGeEAHNu ► Tracing and debugging evaluations with Ragas tutorial: https://lnkd.in/g6eExYMj ► Many improvements to the test generator. Added persona generator and various improvements to the generation algorithm to improve quality and diversity of queries. ► Many more bug fixes and documentation updates Huge thanks to our contributors—you’re the strength of our project. It’s a privilege to have you in our community ❤️

Evaluate tool calling AI agents built using Langraph (by LangChain ) with Ragas. In-depth tutorial covering building and evaluating a ReAct Agent for fetching metal prices. 👉🏽Tutorial covering: 1.⁠ ⁠Build a ReAct agent to fetch metal prices. 2.⁠ ⁠Set up an evaluation pipeline to track key performance metrics. 3.⁠ ⁠Run and assess the agent's effectiveness with different queries. 👉🏽 Check out the tutorial: https://lnkd.in/g4YtWYWj A complete beginner-friendly tutorial on evaluating Langraph agents by @sahusiddharth ❤️

Thrilled to announce that we earned the 𝐍𝐕𝐈𝐃𝐈𝐀 𝐍𝐈𝐌 𝐇𝐚𝐜𝐤𝐚𝐭𝐡𝐨𝐧 𝐖𝐢𝐧𝐧𝐞𝐫’𝐬 𝐂𝐞𝐫𝐭𝐢𝐟𝐢𝐜𝐚𝐭𝐞! 🚀 A huge thank you to my amazing team, 𝐂𝐔𝐃𝐀𝐈𝐧𝐬𝐢𝐠𝐡𝐭 𝐈𝐧𝐧𝐨𝐯𝐚𝐭𝐨𝐫𝐬. Your dedication, creativity, and teamwork were key in transforming ideas into impactful results! It was a wonderful experience developing the Hybrid RAG and GraphRAG applications within the NVIDIA cluster using NVIDIA models. During the hackathon, we explored Neo4j, Nemo GuardRails, ragas, NVIDIA microservices, and Streamlit. Thanks to my team member Karthik Elangovan, Priyanka Hundalekar, Venkatesh Lakshman, Velan and Suma. And also thanks to Yash G. for insightful mentorship from NVIDIA. #NVIDIA #NVIDIATech #OpenHackathons #Innovations #CUDAInsight #Cognizant #KnowledgeGraph #RAG #GraphRAG

A practical guide by Prasant Kumar on Evaluating RAG with Ragas and GPT-4o with LanceDB. 1. Extract content 2. Recursive Chunking 4. Embed Chunks with LanceDB Embedding API 5. Semantic search with Query, #LLama3 for resulting output using Ollama Simple Illustration 👉🏽 Check it out here: https://shorturl.at/HvmjK