Santosh Ananthraman’s Post

LLM Application Evaluations: Currently, everyone is so focused on evaluating new LLM model releases, yet hardly anybody seems concerned with evaluating the applications that we are building on top of them, or how to do so properly. In my experience, the few frameworks that allow you to evaluate your LLM application (RAG or otherwise), lock you into OpenAI flavored models for input/output judging. This lock in is not only due to incompatibility with other provider's APIs, but also the abstracted prompts used for evaluator instructions that are tailored to best fit GPT-4. There is no easy way to integrate your evaluation framework in an orchestrator agnostic way, creating a second layer of lock in. I can't see much of a reason for this as the orchestrator is not what's being evaluated; your contexts, inputs, and outputs are. Viewing of evaluation results depends on unstable instrumentation that is most likely not compatible with the application logic you have developed. Seeing a need in these areas, I have started building my company, GroundedAI to solve these issues. As a first step, I plan to fine tune and open source more efficient small language models as judges, starting with a toxicity judge you can try for yourself here: PEFT adapter: https://lnkd.in/dMmNSwBV Merged Model: https://lnkd.in/dugK2hEE I would love to hear your thoughts and feedback on what pain points you have around LLM application evaluation and how this process could be improved. #llms #evaluation #openai #datascience #rag #opensource #huggingface

For those who are interested in how LLMs are developed, have a read below. For others, do have a look at high level how the process works. Diagram flow included :) Read “Developing Large Language Models (LLMs): A Step-by-Step Guide from Concept to Deployment“ by Wasim Rajput on Medium: https://lnkd.in/gW9iVwfy

Don't expect an LLM to navigate your computer and do 'everyday tasks' just yet. This paper will allow us to know when we should start worrying. For now, we humans are still much better at the everyday than language models (but for how long?) #llm #vlm

Boost Inference Time and Quality - Implementing Speculative and Contrastive Decoding #optimize #LLM #genai implementations

🔨 "When You Have an LLM Hammer, Not Everything Is a Nail" In our excitement to harness the power of Large Language Models (LLMs), it’s easy to fall into the trap of treating every task as if it can be solved with this shiny new "hammer." However, not all problems are best tackled this way. LLMs, being probabilistic machines, interpret tasks based on numerous factors that can introduce variability, even if the instructions are crystal clear. For instance, filtering a large database might be done more efficiently with a simple code snippet rather than asking an LLM to identify key data points, where it might miss critical parameters. Relying on LLMs for every step of a project can soon become a more complex and labor-intensive process than using more straightforward, traditional tools. The key is not to abandon traditional approaches as a whole but to find ways to integrate LLMs as an additional tool in our arsenal. This balanced approach means evaluating workflows to identify where LLMs can genuinely add value and where conventional methods perform just as well or even better. Such thoughtful integration minimizes unnecessary complications, ensures more stable outcomes, and helps avoid the frustrations that come with over-reliance. By leveraging LLMs alongside proven methods, we can achieve a more efficient and effective process overall. Think carefully about your "process" and you will identify best possible use cases for the LLMs. #LLM #GenAI #HEOR #Process

People expect immediate transition from only-code to no-code solutions, but this expectation overlooks a critical challenge: achieving high accuracy in large language models (LLMs) solely using a unstructured information and huge context window is nearly impossible. Investing some coding into context filtering and processing will result in a close-to-expected model prediction results.

What's the best way to dive into LLMs? 🤔 This 90 min YouTube video by Jeremy Howard🌟 ✅ Foundational concepts that power LLMs ✅ Explore cutting-edge architectures shaping the future ✅ Master advanced strategies for model testing and optimization ✅ Gain hands-on tips for working with LLMs effectively 📺 Watch now: https: //https://lnkd.in/gRzvtVkX #AI #MachineLearning #DeepLearning #LanguageModels

Aligning Large Language Models with Diverse User Preferences Using Multifaceted System Messages: The JANUS Approach Quick read: https://lnkd.in/gQAsT95M Paper: https://lnkd.in/gkBAuWiC

Understanding how language model performance varies with scale is critical tobenchmark and algorithm development. Scaling laws are one approach to buildingthis understanding, but the requirement of training models across manydifferent scales has limited their use. We propose an alternative,observational approach that bypasses model training and instead builds scalinglaws from ~80 publically available models. Building a single scaling law frommultiple model families is challenging due to large variations in theirtraining compute efficiencies and capabilities. However, we show that thesevariations are consistent with a simple, generalized scaling law where languagemodel performance is a function of a low-dimensional capability space, andmodel families only vary in their efficiency in converting training compute tocapabilities. Using this approach, we show the surprising predictability ofcomplex scaling phenomena: we show that several emergent phenomena follow asmooth, sigmoidal behavior and are predictable from small models; we show thatthe agent performance of models such as GPT-4 can be precisely predicted fromsimpler non-agentic benchmarks; and we show how to predict the impact ofpost-training interventions like Chain-of-Thought and Self-Consistency aslanguage model capabilities continue to improve. #LanguageModels #ScalingLaws #ModelEfficiency #PerformancePrediction #EmergentPhenomena

𝐑𝐀𝐆-𝐅𝐥𝐨𝐰 : 𝐎𝐩𝐞𝐧-𝐒𝐨𝐮𝐫𝐜𝐞 𝐑𝐀𝐆 𝐄𝐧𝐠𝐢𝐧𝐞 RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data. 𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬 🍱 Template-based chunking 🌱 Grounded citations with reduced hallucinations 🍔 Compatibility with heterogeneous data sources 🛀 Automated and effortless RAG workflow RAGFlow details (in the comments) #rag #ragflow #nlproc #llms #generativeai #deeplearning #transformers