Toloka

Does your chatbot or AI agent need large-scale evaluation? We partnered with a team working on chatbot improvements to 𝘀𝗰𝗮𝗹𝗲 𝗵𝘂𝗺𝗮𝗻 𝗲𝘃𝗮𝗹 𝗯𝘆 𝟮𝟱𝟬𝘅 𝗮𝗻𝗱 𝗮𝗰𝗵𝗶𝗲𝘃𝗲 𝘀𝘁𝗮𝗯𝗹𝗲 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝘁 𝟴𝟱% 𝗽𝗿𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗮𝗻𝗱 𝗿𝗲𝗰𝗮𝗹𝗹.    Here’s how we did it: ✅ Rapidly built eval pipelines tailored to the client’s chatbot dialogs ✅ Onboarded and trained vetted experts to conduct evaluations ✅ Ran in-depth evaluation across 5 criteria: transparency, groundedness, accuracy, responsiveness, helpfulness ✅ Adjusted the data pipeline dynamically to meet specific needs ✨ Quality control is our superpower.  Read the case study: https://lnkd.in/gDzG8gSW #AIevaluation #Chatbots #CaseStudy

See you next week at #COLING2025! Our team will be participating in the following events along with our partners at MBZUAI (Mohamed bin Zayed University of Artificial Intelligence). Don’t miss out! January 19th @ 14:00 - Hands-on Tutorial: Labeling With LLMs and Human-in-the-Loop. Co-organizers Akim Tsvigun (University of Amsterdam and Nebius), Dominik Schlechtweg(University of Stuttgart), with Natalia Fedorova, Boris Obmoroshev, Sergei Tilga, Ekaterina Artemova, and Konstantin Chernyshev from Toloka. January 20th @ 11:00 - Ekaterina Artemova joins Panel Discussion 1 at SUMEval 2025: Challenges of Collecting Culturally Grounded Multilingual Data for Training and Evaluation of NLP Systems. January 20th @ 11:15 - Shared task at Gen AI detection workshop on Binary Multilingual Machine-Generated Text Detection (Human vs. Machine). Co-organizers: Prof. Preslav Nakov, Prof. Iryna Gurevych (MBZUAI), Prof. Nizar Habash (NYU Abu Dhabi) and Ekaterina Artemova (Toloka)   #AI #ML #AIContentDetection #COLING2025 #Partnership #GenerativeAI #Research

*2025 trends: AI Agents* AI agents are the buzzword of the moment—systems designed to make decisions and take actions independently. While we’re seeing early iterations, the industry is still figuring out what’s feasible and how best to approach building and applying agentic systems. In 2025, the industry will distill the concept of AI agents and define what they are capable of. At Toloka, we’re delivering data to enhance the capabilities of agents in real-world applications. Our current explorations are focused on three areas: effective methods for post-training, high-quality data collection, and safety evaluation and red teaming to identify and mitigate risks inherent in agent applications.

✨ 3 benchmarking initiatives from Toloka Research: math, science, and AI detection ✨ As AI models become more fluent, it’s getting harder to identify errors and biases. Our research team has released several domain-targeted benchmarks that match the practical demands of real-world AI evaluation. Read about our latest benchmarks: https://lnkd.in/gh_gpxA7 ➡️ University-level math (U-MATH and μ-MATH benchmarks) ➡️ Natural sciences / STEM ➡️ Artificial text detection (Beemo benchmark) But benchmarks are just a starting point for model evaluation. To pinpoint model weaknesses and areas for fine-tuning, our in-depth approach: - Leverages a global network of domain experts for deep insights. - Augments and assists human experts with fine-tuned LLMs for efficiency and precision. We can help you go beyond the basics of evaluation and fine-tuning. Connect with our team. #AIevaluation #MachineLearning #AIModels #AIbenchmarks

In the AI industry, precision in niche domains is overtaking broad knowledge as the benchmark for success. Check out Toloka's CEO Olga Megorskaya's latest article for Forbes to dive into how SFT is shaping the future of AI and making real-world applications more effective.

In the AI industry, precision in niche domains is overtaking broad knowledge as the benchmark for success. Check out Toloka's CEO Olga Megorskaya's latest article for Forbes to dive into how SFT is shaping the future of AI and making real-world applications more effective.

*2025 trends: Generative AI in the enterprise* As the new year gets rolling, our CEO, Olga Megorskaya, shared realistic expectations for GenAI adoption and how we're addressing it in 2025. The rapid evolution of frontier AI models presents a universal challenge: each new iteration, like the next GPT, raises the bar unpredictably, making it difficult for companies to determine where to invest. Legacy enterprises are still struggling to translate GenAI experimentation into real-world applications. Many view it as risky and lack reliable methods to evaluate solutions built on GenAI models. At Toloka, we’re devoting experiments and research to improving evaluation methods and benchmarks that can inform enterprise decisions and support GenAI adoption.

The truth about LLM math reasoning skills: The best problem solvers are not the best judges. 🔹 Our U-MATH benchmark ranks LLMs on university-level math 🔹 Our μ-MATH meta-benchmark ranks LLMs as judges of math solutions We tested large and small models to uncover their strengths and weaknesses. What we found: ➡️ Smaller specialized models like Qwen2.5-Math-7B outperform the larger LLaMA-3.1-70B in terms of judging skills. ➡️ Gemini’s performance as a judge is surprisingly poor, due to over-tuning. Do you know where your model stands? Download U-Math and μ-MATH on Hugging Face to test the LLMs you are using or building: https://lnkd.in/gVd8bA_t Learn more: https://lnkd.in/guXbUmXB

Attending #COLING2025 in Abu Dhabi? Join us for a Hands-on Tutorial: Labeling With LLMs and Human-in-the-Loop. You’ll learn how to speed up data annotation and reduce costs and human workload. 🔹 When to use synthetic training data, active learning, and hybrid labeling 🔹 Real-life case studies and practical applications 🔹 Best practices to optimize quality of the final dataset 🔹 Hands-on workshop to set up hybrid annotation Co-organizers: Akim Tsvigun (University of Amsterdam and Nebius), Dominik Schlechtweg (University of Stuttgart), with Natalia Fedorova, Boris Obmoroshev, Sergei Tilga, Ekaterina Artemova, and Konstantin Chernyshev from Toloka. Learn more: https://lnkd.in/ghc3vh-m 🗓️ January 19, 2025. See you there! #AI #ML #COLING2025 #annotation #DataLabeling

Highlights of 2024 from our parent company, Nebius Group — what an exciting year it's been!