Smriti Sharma’s Post

🤔 OpenAI's latest structured outputs feature is definitely a life-saver for building LLM-based apps, but it could come with some performance trade-offs. A new paper titled "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" highlights this concern. 📖 Some insights: ⛳ The paper examines the effects of structured generation, where LLMs produce content in standardized formats like JSON or XML, which is common in real-world applications for extracting key information. ⛳ It shows that while structured generation simplifies parsing and integration into applications, it also has a significant downside. Specifically, LLMs exhibit a notable decline in reasoning abilities when restricted to these formats, with stricter format constraints leading to greater performance degradation. ⛳ Looser format restrictions generally improve performance and reduce variance in reasoning tasks. Parsing errors, while not the primary cause of performance differences, can be mitigated through corrective prompting. 👉 I’m not sure if this applies to the latest OpenAI models since the authors only tested it on GPT-3.5 and a few other models that might not be fully optimized for structured outputs. But it’s definitely something to keep in mind and check if you’re planning to use this feature a lot. Link: https://lnkd.in/eHRURmSH

The best model for Codegen? GPT-4o vs 3.5 Sonnet vs everyone else TL;DR use Claude 3.5 Sonnet 20241022 OpenAI and Anthropic have traded the code generation crown a couple of times. Currently, Anthropic is in the lead. Anthropic's latest flagship model, Claude 3.5 Sonnet 20241022 tops most benchmarks and is widely anecdotally considered ‘better’. This extended their lead from the previous release of Claude 3.5 Sonnet. OpenAI’s latest models that focus on improved reasoning, the o1 family, have yet to be fully released and are missing some important API features like system messages and tool use. The full release of o1 is likely to happen before the end of 2024 and could challenge Claude 3.5. OpenAI’s next generation model, known as Orion or GPT-5, is rumoured to be months away. A jump from GPT-4 to 5 similar to that from 3.5 to 4 would reset the landscape and put OpenAI far ahead. These models will probably improve with time in a fairly linear fashion for at least the next couple of years. Code generation will benefit. There are a bunch of other options - Gemini, Llama etc. However closed source models with the most funding and compute capacity appear to have a strong lead. For high value use cases, like production code generation, the extra capability is worth the cost and closed source trade-offs. There are also several notable fine tunes and foundation models optimised specifically for code generation. In the long term, it's likely foundation models will outperform, especially in code generation, where their models have demonstrated exceptional generalisability. Progress is being made in areas other than raw code generation ability. OpenAI’s mini models and Google’s Flash models are around 30 times cheaper than their full-size counterparts. Groq runs open-source models on custom hardware, delivering extremely fast inference times. However, for code generation in a work setting, value exists mostly at the frontier.

📄 OpenAI's o1 still can't plan reliably - but is still a massive leap forward 🤔 Can OpenAI's o1 actually plan and reason, as claimed in its release? Researchers put it to the test using PlanBench, a planning benchmark that has stumped even the best language models. 🎯 The benchmark has interesting challenges: Blocksworld is similar to the well-known “towers of Hanoi”, where several specific steps are to be taken in succession to move blocks around. It has a more difficult version, called “Mystery blocksworld”, where some terms are replaced to obfuscate their meaning and prevent LLMs from imitating existing reasonings from its training corpus. 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: 🚀 On simple planning tasks, o1 nearly aced it - 97.8% accuracy vs 62.5% for the best language model 🧠 Unlike language models, o1 showed some ability to reason through obfuscated planning problems 📉 But performance drops sharply on longer/more complex plans 🙈 Still confidently gives wrong answers ~54% of the time on unsolvable problems! 💰 Costs skyrocket - researchers racked up a $1,897 bill in just a week of testing! ⏱️ Much slower than specialized planning algorithms like FastDownward, that gives 100% accuracy results The researchers conclude that while o1 is a big step forward, it's not yet reliable or efficient enough for real-world planning tasks. They suggest that hybrid approaches combining language models with specialized planners may be more promising for now. Read the paper 👉 https://lnkd.in/eRubBDsj

Hugging Face has done something great! OpenAI's o1 is a significant advancement in language models, but it's not yet reliable for planning tasks. PlanBench is a challenging benchmark for testing planning abilities. While o1 excels at simple planning tasks, it struggles with longer, more complex ones. Additionally, o1 often confidently gives wrong answers to unsolvable problems. Furthermore, o1 is significantly more expensive to use than specialized planning algorithms. Hybrid approaches combining language models with specialized planners may be more promising for real-world planning tasks. #AI #LLMs #Benchmark #Reasoning #HuggingFace #OpenAI #ChatGPT #Planning

absolutely right - very easy to get high scores just with prompt engineering on these weak benchmarks. I just tried Reflection 70B on a few reasoning examples and compared its responses to other models - in some cases such as as word puzzles (river crossing problems), Reflection just seemed to be really confused, wracked with self-doubt, and gave warbled answers. (edited - ) just tested it a bit more and it is sometimes better off in some simpler logical puzzles. I also tested with mate-in-one to mate-in-3 problems - some of them I devised myself. In these cases, Reflection70B verbosely showed its reasoning, but suggestions were wrong. Surprisingly, gpt3.5 actually did better than even Sonnet. Obviously, models are not consistent in reasoning - they may shine in a few places and actually are frustratingly bad in other places. A few puzzles here and there do not really test their reasoning capabilities efficiently. Need a better comprehensive benchmark with an uncontaminated testing set.

This is spot on! LLM benchmarks are easy to game. As mentioned, crowd sourced solutions like LMSys Chatbot or private solutions like Scale AI's are some of the few that can be trusted. I don't completely dismiss benchmarks but look at them with a very very sceptical eye. You can see this problem often when a random model goes to the top/near the top of the Huggingface Open LLM leaderboard, which unfortunately has become so polluted with models trained on the benchmarks that it's lost much of it's value.

With the rapid growth of artificial intelligence technology, converting spoken language into text has become an incredibly useful skill. OpenAI’s Whisper API is a powerful tool for doing just this—it can accurately turn your spoken words into written text. https://lnkd.in/gUTQB3Ah

Llama 3.1 405B matches or beats the Openai GPT-4o across many text benchmarks! New and improvements of 3.1:  - 8B, 70B & 405B versions as Instruct and Base with 128k context  - Multilingual, supports 8 languages, including English, German, French, and more.  - Trained on >15T Tokens & fine-tuned on 25M human and synthetic samples  - Commercial friendly license with allowance to use model outputs to improve other LLMs  - Quantized versions in FP8, AWQ, and GPTQ for efficient inference.  - Llama 3 405B matches and beast GPT-4o on many benchmarks  - 8B & 70B improved Coding and instruction, following up to 12%  - Supports Tool use and Function Calling Blog: https://lnkd.in/g9yTBFnv Model Collection: https://lnkd.in/g_bVRpmp

We put OpenAI's latest LLM, GPT-4o, to the test as a code review tool for developers. How will it fare with three different pieces of code? #LLMs #ChatGPT #AI by David Eastman