Levytation’s Post

We're thrilled that CRN has recognized us as one of the hottest products at #gtc24 this year! Read all about how we're beefing up GPU management => https://lnkd.in/gz5ivvC3 #aiinfrastructure #ai #gpu #gpucomputing #opensource #machinelearning

Interested in deploying Generative AI models in your apps quickly and easily? Come watch Samuel Kemp and Yufeng Li's session at Microsoft Build this Wednesday! https://lnkd.in/g-bhciBa

AlphaSignal, a newsletter for developers by developers. They claim to identify and summarize the top 1% news, papers, models, and repos in the AI industry. Can’t wait to read it all! Go subscribe!

B200 blows the H100 out of the water. B200 boasts 20 petaflops of AI compute compared to H100's 4 petaflops (at FP4 precision). That's a 4x improvement.

It took a while, but my conversation with my colleague Victor Jakubiuk and Ricardo Rocha from CERN is now available! We talked about how critical it is to efficiently use hardware for AI inference - and about the advantages of using general purpose CPUs instead of dedicated GPUs for "spikey" inference workloads. https://lnkd.in/eA3N-S-J (Note this is an updated video link compared to the post I made last week)

Muhammad Saad offers a comprehensive exploration of strategies to optimize LLM inference for limited GPU resources. Actionable techniques like vLLMs, quantization, FlashAttention, and CachedAttention are presented, which not only equips practitioners with the tools to deploy powerful LLMs on constrained hardware but also advances the conversation around accessibility and efficiency in AI deployment. https://lnkd.in/eaWs8hNx

🚀 Load Testing Self-Hosted LLMs: Key to Your Business Success! 🌟 As AI evolves, self-hosting Large Language Models

🚀 Unleashing the Power of LLaMA3 on Multi-Node GPU Kubernetes Clusters! 🚀 🌐 In the rapidly evolving landscape of AI and machine learning, deploying large language models (LLMs) like Llama3 efficiently is crucial. Whether you’re scaling your applications to handle massive traffic or optimizing resource utilization, the challenge is real—and so are the solutions. In my latest article, I dive into how to serve Llama3 over a multi-node GPU Kubernetes cluster with auto-scaling. Learn how to: 💡 Deploy Llama3 efficiently across multiple nodes using vLLM ⚙️ Implement horizontal auto-scaling for dynamic resource management Whether you’re an AI enthusiast, DevOps engineer, or simply curious about cutting-edge deployments, this guide will provide you with actionable insights and best practices. #AI #Kubernetes #Llama3 #GPU #MachineLearning #AutoScaling #DevOps #CloudComputing #NLP 👉 Read the full article here

Answer.AI, founded by Jeremy Howard (fast.ai) released an open source system that can train a 70b #LLM on a regular desktop computer with 2 standard gaming GPUs. This project combines 2 key pieces: 1- QLoRA (quantization + LoRA). Quantization allows to store neural network weights in 4 bits or less. LoRA -"Low Rank Adaptation"- does not train the whole model, but instead adds "adaptors", which are matrices smaller than 1% of the model. 2- FSDP (Fully Sharded Data Parallel) splits a large model parameters across multiple GPUs, allowing to overcome the RAM limitations of consumer GPUs (in contrast to DDP -"Distributed Data Parallel"- that keeps the full model on each GPU) Here is the article from Answer.AI which contains a link to their repo (AnswerDotAI) if you want to try FSDP/QLoRA.