Nyun AI’s Post

BERT is back in a new optimized look (ModernBERT). I still remember the old days in 2018. Yes, 6 years in the AI realm is considered old. When BERT model was considered a "large" language model with only 340M parameters. Nowadays Phi-4 model with 14B parameters is considered a "small" language model. I think that the AI world is still in need for these encoders as the BERT architecture despite the dominance of GPT-like decoders. If they are not used as the main models for the use case, the encoders are still an important component for retrieval stage in addition to the decoders for generation stage in any RAG system design. Anyway, welcome back BERT whether you are called large or small!! 🎊

WebNN is a web API designed to enable high-performance, hardware-accelerated neural network computations directly within web applications. #smalllanguageodel #slm #edgecomputing

Attention Transfer: A Novel Machine Learning Approach for Efficient Vision Transformer Pre-Training and Fine-Tuning Vision Transformers

AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card. No quantization, distillation, pruning or other model compression techniques that would result in degraded model performance are needed. Link : https://lnkd.in/dAUpt6FJ LLM-PlayLab : https://lnkd.in/ghhUPP5Q #llm #generativeai #llama3 #ai4good

I recently attended an enlightening lecture by Professor Tom Yeh on DeepSeek, which delved into its innovative architecture and practical applications. The session provided a comprehensive understanding of two key components: Multi-Head Latent Attention (MLA) and the Mixture-of-Experts (MoE) framework. Multi-Head Latent Attention (MLA): MLA is designed to optimize memory usage during inference by compressing the Key-Value (KV) cache into a latent vector. In traditional transformer models, the KV cache can become large and memory-intensive, especially with long sequences, limiting batch size and sequence length. MLA addresses this by significantly reducing the KV cache size, thereby lowering memory usage to just 5–13% of what the more common Multi-Head Attention architecture consumes. This compression allows for more efficient processing of long sequences without compromising performance. Mixture-of-Experts (MoE) in DeepSeek: DeepSeek's MoE architecture enhances model efficiency by activating only a subset of specialized "expert" networks during computation. This approach allows the model to scale effectively without a proportional increase in computational costs. DeepSeekMoE involves finely segmenting experts into smaller units and activating a flexible combination of them, enabling more precise handling of diverse tasks. Additionally, it isolates certain experts as shared ones to capture common knowledge and reduce redundancy. This structure ensures that each expert acquires focused, non-overlapping knowledge, leading to improved performance and efficiency. Real-Life Applications: The combination of MLA and MoE in DeepSeek's models offers several practical benefits: Efficient Resource Utilization: By reducing memory usage and computational demands, these models can be deployed on hardware with limited resources, making advanced AI applications more accessible. Scalability: The MoE architecture allows for scaling models to handle more complex tasks without a linear increase in computational costs, facilitating the development of more powerful AI systems. Specialized Task Handling: The expert specialization within the MoE framework enables the model to manage a wide range of tasks more effectively, from natural language processing to complex reasoning. These innovations position DeepSeek's models as efficient and scalable solutions for various AI applications, from chatbots to complex data analysis tools. For a more in-depth understanding, you might find this video helpful: How DeepSeek AI uses Mixture of Experts (MoE) to improve performance Spreadsheet : https://lnkd.in/d6aUrpHV https://lnkd.in/dgqEdgR3

🚀 Happy to share our latest work: FastDraft – a new method to train and align highly parameter-efficient draft models for large language models (LLM) inference with speculative decoding. ✨We showcase our method by training draft models for the popular LLMs Phi-3-mini and Llama-3.1-8B in under 24 hours on a single server with 8 Intel Gaudi-2 accelerators 🌟 Up to 3x bandwidth reduction, making LLM inference more power-efficient for AI-PCs and edge devices. 🌟 Running on the latest Intel® Core™ Ultra 200V Series (Lunar Lake) 🌟 2x average latency speedup of 2nd token generation compared to autoregressive generation in tasks like code completion and 1.5x for summarization, and instruction completion. 🌟 No accuracy loss 🌟 Works with quantization 📄 https://lnkd.in/djqF7idZ - will be presented at the 4th NeurIPS ENLSP 2024 Workshop Ofir Zafrir, Igor Margulis, Dorin Shteyman, Shira Guskin, Moshe Wasserblat, Robert Hallock #AIPC #LunarLake #LLM #SLM #efficientLLM #IntelLabs #SpeculativeDecoding #FastDraft

How to use the latest GPT-4o LLM to train specialized tinyML models to deploy on microcontroller-sized hardware at the edge.

GemFilter a new method accelerating LLM inference by 2.4 and reducing memory consumption by 30%. Available freely for lots of models. https://lnkd.in/epxegWeU

EuroLLM 9B Quantized to 4-bit: >95% accurate, 3x smaller I made 4-bit versions of EuroLLM 9B with AutoRound. The models can now run on a 12 GB GPU (or an 8 GB GPU for short sequences and if you quantize your KV cache). I used the GPTQ format for serialization so they are compatible with most inference frameworks: vLLM, TGI, Transformers, ... The quantization preserved >95% of the model's original accuracy on various benchmarks. Get the models: https://lnkd.in/ewJFaJbU Note: I only evaluated them for English tasks. Quantization might have destroyed the models for the other supported languages (unlikely but possible). Evaluate these models extensively for your target tasks and languages before deploying them.

Quantization reduces the precision of the numbers used to represent a model’s parameters, typically from 32-bit floating-point to 16-bit or even 8-bit integers. This process can significantly reduce memory usage and improve computational efficiency. To know more about quantization, do check the blog #quantization #llm #artificialIntelligence #machinelearning #deeplearning