n2 Group

62% of developers are planning to deploy an LLM application to production within the next year, according to a recent survey. Here are 9 questions the STAC-AI inferencing benchmark and test harness can help you answer if you’re building an LLM-based system. 1. How quickly can your system load and get ready to use a large language model? 2. What’s the trade-off between how fast users send requests and how quickly the model responds? 3. Can the system keep up with how fast people read during real-time interactions? 4. How does the system's performance change with different context sizes? 5. What’s the highest number of requests the system can handle while maintaining an acceptable level of performance? 6. How does the system manage multiple users at once, and when should we add more GPUs to keep up with demand? 7. How closely do the model's responses match a reference set of answers? 8. How much text does the system produce for every dollar spent? 9. How does the system perform in terms of energy use, space, and cost efficiency? To demonstrate this, we ran the benchmark on a stack with: - Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct - 8 x NVIDIA A100-SXM4-80GB GPUs - 720 GiB of virtualized memory The benchmark uses LLMs to analyze financial data from quarterly and annual reports filed by publicly traded companies, illustrating how latency and throughput vary with request rates. This analysis raises important questions about user satisfaction at different times of day and the trade-offs between resource allocation and the cost of inference. The benchmark captures the following metrics plus a whole lot more: - Inferences per second - Words generated per second - Response smoothness - Energy efficiency (words per kWh) - Space efficiency (words per cubic foot of rack space) - Price Performance (words per USD) Here are just some of our findings from the Llama 3.1-8B-Instruct test: - The server loaded the model from storage into the GPUs and was ready for inference in 90 seconds. - A 26% reduction in the rate of prompts hitting the system reduced the median response time by 38%. - The system was 20x more efficient when processing a smaller-context data set than when processing a larger-context dataset. (EDGAR4a vs EDGAR5a datasets in STAC-AI) - At a peak request rate of 21.5 requests per second, the system achieved an output profile of 12.2 words per second, just below the typical maximum reading speed of 13 words per second for fast readers. We will share even more data from these tests at AI STAC conference in London on December 4th. If you’re interested in the finer details, we’ll be presenting these along with more information about STAC-AI at an AI workshop the following day. Register for either or both events if you want to learn more. They’re free to attend. 📅 Conference, December 4th 🔗 https://lnkd.in/eCwd4Q5a 📅 Workshop, December 5th 🔗 https://lnkd.in/eTKqfn6X

Great to see #ASReml featured in NC State Plant Breeding Consortium's upcoming workshop on computing skills in plant breeding! This is a fantastic opportunity for plant breeders and students to expand their data analysis skills. More details below... 🌱 #PlantBreeding #DataAnalysis

Object storage has gained traction for its scalability and integration with modern data pipelines. However, in typical applications, its latency—often reaching hundreds of milliseconds—limits its effectiveness for latency-sensitive AI in finance.   At the upcoming AI STAC events in London and New York, James Coomer, SVP Products at DDN, will argue that the problem isn’t object storage itself but rather how applications access it: RESTful APIs.   In a talk entitled: “Accelerating Large-Scale AI and Quantitative Trading Analytics with Low-Latency Object Storage”, James will advocate an alternative data path for object stores that reduces latency to sub-millisecond levels, making it viable for both model training and inference using performance-sensitive financial data like timeseries.   Join us to debate the role of object storage in HPC and AI today.   📍 London, December 4th 📍 New York, December 10th 👉 https://lnkd.in/eXcm-VsM

What a fantastic week the nAG team have had at #SC24! As the event closes we're reflecting on all the amazing conversations, inspiring innovation, and good times had at our booth. Co-exhibiting for the first time with n2 Group made it extra special. From demos to deep dives, we loved connecting with so many of you passionate about HPC, cloud, AI and related technologies. Thanks for stopping by, sharing ideas, and helping us push the boundaries of what's possible - until next year SC Conference Series! #hpc #cloudcomputing #ai #innovation #collaboration 

Can traders get quantitative insights from LLMs that they can stake their strategies on? Yes, but not out-of-the box. Jean-Philippe Rezler, Global Head of AI and Analytics at Opensee will provide his insights at the STAC Summit London on the limitations of off-the-shelf models when it comes to handling vast trading datasets with complex relational structures and propose a solution to overcome these challenges via a multi-agent architecture designed to ensure that quantitative analytics from LLMs can reliably be used for their strategies.   Register here to hear his talk:   🎙️ “Precision in chaos: Leveraging GenAI multi-agent systems for trade analytics” 📅 3 Dec 2024 🔗 Register: https://lnkd.in/eUNWFquj

X-ISS is in Atlanta for #SC24. Stop by and say hello to our team at booth #1944 and experience the expertise gained through our 20+ years in the world of HPC. You can also learn about ManagedHPC, our turnkey HPC Cluster Management solution, and how it can greatly improve your team's productivity and the utilization of an HPC environment. Worry-Free HPC Management with experienced HPC admins.   #SC24 #HPC #Supercomputing

Congratulations to #BioTeam Scientific Engineer Alex Oumantsev for receiving the prestigious National Eye Institute (NEI) Director’s Award for his outstanding accomplishments, which have brought special credit and distinction to both him and the #NEI. Read more about it https://lnkd.in/eiuvsNaS

nAG HPC Services are transforming high-performance computing with an innovative cloud-to-on-premise bursting approach. Recently implemented for a major aerospace organization, this solution optimizes cost and performance while protecting sensitive data. See how it was done https://lnkd.in/dfyFjEJ7 meet the team at #SC24 booth 2508 SC Conference Series Google Cloud #hpc #cloudcomputing #aerospace #innovation #casestudy #collaboration

nAG Principal Scientist and RWTH Aachen University Professor of Computer Science Uwe Naumann presents 'Matrix-Free Differential Inversion. A Lesson from Adjoint Algorithmic Differentiation' at QuantMinds today - reach out to Uwe or the nAG team at #QuantMinds to learn more or send us a note  #quantfinance #computation #algorithmicdifferentiation #data #innovation

AI - it's all the rage, but making your data and systems 'AI-Ready' is a critical step to get the most out of these new tools. Here are a few Do's and Don'ts to get things started. Visit us this week at booth 3946 at #SC24 in Atlanta to learn more about how BioTeam has been helping scientific research teams make their data and systems AI-ready. Swing by or arrange a meeting: https://hubs.ly/Q02PwMHg0 Do's and Don'ts for AI-Ready Data, see the attached PDF for more info! Do: * Use common data standards * Use standard APIs * Implement effective data management Don't: * Forget good metadata! * Skip on dataset documentation * Ignore dataset QA/QC