Jinsung Choi’s Post

Interesting concept that builds on a framework already well known: CDN. Will watch this space closely to see how does an "AI edge server" look like.

These methods and techniques are ultimately bringing the AI interference capabilities down south to Telco Edge and Device Edge that too with a very cost effective and optimal resource way. Probably with time HPC platforms will be a must to have module to support MDN at the Device Edge.

Reducing the Size of AI Models Running large AI models on edge devices Image created using Pixlr AI models, particularly Large Language Models (LLMs), need large amounts of GPU memory. For example, in the case of the LLaMA 3.1 model, released in July 2024, the memory requirements are: The 8 billion parameter model needs 16 GB memory in 16-bit floating point weights The larger 405 billion parameter model needs 810 GB using 16-bit floats In a full-sized machine learning model, the weights are represented as 32-bit floating point numbers. Modern models have hundreds of millions to tens (or even hundreds) of billions of weights. Training and running such large models is very resource-intensive: It takes lots of compute (processing power). It requires large amounts of GPU memory. It consumes large amounts of energy, In particular, the biggest contributors to this energy consumption are: - Performing a large number of computations (matrix multiplications) using 32-bit floats - Data transfer — copying the model data from memory to the processing units. Being highly resource-intensive has two main drawbacks: Training: Models with large GPU requirements are expensive and slow to train. This limits new research and development to groups with big budgets. Inference: Large models need specialized (and expensive) hardware (dedicated GPU servers) to run. They cannot be run on consumer devices like regular laptops and mobile phones. Thus, end-users and personal devices must necessarily access AI models via a paid API service. This leads to a suboptimal user experience for both consumer apps and their developers: It introduces latency due to network access and server load. It also introduces budget constraints on developers building AI-based software. Being able to run AI models locally — on consumer devices, would mitigate these problems. Reducing the size of AI models is therefore an active area of research and development. This is the first of a series of articles discussing ways of reducing model size, in particular by a method called quantization. These articles are based on studying the original research papers. Throughout the series, you will find links to the PDFs of the reference papers. The current introductory article gives an overview of different approaches to reducing model size. It introduces quantization as the most promising method and as a subject of current research. Quantizing the Weights of AI Models illustrates the arithmetics of quantization using numerical examples. Quantizing Neural Network Models discusses the architecture and process of applying quantization to neural network models, including the basic mathematical principles. In particular, it focuses on how to train models to perform well during inference with quantized weights. Different Approaches to Quantization explains different types of quantization, such as quantizing to different precisions, the granularity of quantization, deterministic...

𝗧𝗵𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗼𝗳 𝗖𝗡𝗡𝘀: 𝗔 𝗟𝗮𝘆𝗲𝗿𝗲𝗱 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 𝘁𝗼 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 At the heart of a CNN is its layered architecture, designed to process and learn from image data. It starts with an input image, typically represented as a tensor—a high-dimensional matrix retaining the spatial and depth information of the image. This input then travels through a sequence of layers, each serving a distinct purpose: 𝗖𝗼𝗻𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿𝘀: These layers apply filters to the input image, extracting features like edges, textures, and patterns. By sliding these filters over the image, CNNs capture local dependencies and spatial hierarchies.    𝗥𝗲𝗟𝗨 𝗟𝗮𝘆𝗲𝗿𝘀: The Rectified Linear Unit (ReLU) introduces non-linearity into the model, enabling it to learn complex patterns. 𝗣𝗼𝗼𝗹𝗶𝗻𝗴 𝗟𝗮𝘆𝗲𝗿𝘀: These layers downsample the feature maps, reducing dimensionality and computation while retaining important information. Common pooling methods include max pooling and average pooling. 𝗙𝘂𝗹𝗹𝘆 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗲𝗱 𝗟𝗮𝘆𝗲𝗿𝘀: In these layers, neurons connect to all activations from the previous layer, integrating the learned features to make the final prediction. This is where the CNN transforms spatial features into a class score. 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗖𝗡𝗡𝘀: The Path to Optimization Training a CNN involves two main processes: the forward pass and the backward pass. 𝗙𝗼𝗿𝘄𝗮𝗿𝗱 𝗣𝗮𝘀𝘀: The input image passes through the network layer by layer, resulting in a prediction.    𝗕𝗮𝗰𝗸𝘄𝗮𝗿𝗱 𝗣𝗮𝘀𝘀: This involves backpropagation, where the error between the prediction and the ground truth is computed and propagated back through the network. The model’s parameters are adjusted using Stochastic Gradient Descent (SGD) to minimize this error. 𝗖𝗮𝘀𝗲 𝗦𝘁𝘂𝗱𝘆: 𝗧𝗵𝗲 𝗩𝗚𝗚-𝟭𝟲 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 One exemplary CNN architecture is VGG-16, known for its depth and small convolution filters. This network consists of 16 layers, including convolutional and fully connected layers. VGG-16 has demonstrated high accuracy in image classification tasks, contributing significantly to the field of deep learning. 𝗖𝗼𝗻𝗰𝗹𝘂𝘀𝗶𝗼𝗻 Convolutional Neural Networks have revolutionized how we approach and solve computer vision problems. By breaking down images into understandable patterns and learning from vast amounts of data, CNNs enable machines to interpret and interact with the visual world. Whether it's recognizing objects in images or powering advanced AI applications, CNNs are integral to the future of technology. Understanding the mathematical foundations of tensors, vector calculus, and gradient descent is essential for comprehending CNN operations. As we continue to innovate and refine these models, the potential applications of CNNs will only expand, making them a vital tool in any AI or machine learning toolkit. #ArtificialIntelligence #MachineLearning #ComputerVision #DeepLearning #ConvolutionalNeuralNetworks

What if the power of LLMs mostly came from good pattern matching? LLMs use sophisticated GenAI combined with pattern matching, but the quality of LLM output could be attributable more to pattern matching than AI. I've seen the same with synthetic data generation, with plenty of data transforms, normalization, regularization, feature engineering and standard ML to fix the poor performance of AI (GAN in this case), to the point that you could eliminate the entire AI part altogether and get better results: the reason why I created NoGAN which does a much better job much faster without deep neural networks. But when you charge by weight, you need AI to justify high costs and trillions of weights, you also need AI just for marketing purposes. I get better results with my LLMs with no neural network, no GPU. All open source and described in my new book, at https://lnkd.in/gGdM3pej Do you think LLMs could add more value, do a better job, with more exhaustive results, no hallucination, much faster, easily handling new prompt words not in your training set, at a far lower cost (no GPU) without sophisticated GenAI? With the price to pay being just loss in grammatical prowess and no cool English prose in output results? Some capabilities, like solving complex math problems, can be achieved with API calls, in this case Wolfram that has been doing a better job at that for 20 years without GenAI. In the end, the value is not the quality of the grammar, but whether or not you get sound answers to your questions. And even for grammar, you could use a large list of pre-made, customizable sentences that adapt to what is retrieved from your augmented corpus. On a specialized corpus, there are only so many prompts a user could ask, probably less than a million. Imagine a system where all potential prompts (cleaned prompts) - past and future - have an answer already in the system, possibly up to some words that need to be plugged in. Covering 99.9% of potential answers. In short, some brute force approach that requires far less computing power than current LLMs. Note that my xLLM also has a full UI, not just a "search box" where you enter your prompt.

DeBaTeR: A New AI Method that Leverages Time Information in Neural Graph Collaborative Filtering to Enhance both Denoising and Prediction Performance Recommender systems have been widely applied for studying user preferences; however, they face significant challenges in accurately capturing user preferences, particularly in the context of neural graph collaborative filtering. While these systems use interaction histories between users and items through Graph Neural Networks (GNNs) to mine latent information and capture high-order interactions, the quality of collected data poses a major obstacle. Read the full article: https://lnkd.in/e3tmduVe Paper: https://lnkd.in/e55v9nky

From ONNX to Static Embeddings: What Makes Sentence Transformers v3.2.0 a Game-Changer? Growing Need for Efficient AI Models There is an increasing demand for AI models that provide a good balance of accuracy, efficiency, and versatility. Many existing models face challenges in meeting these needs, especially in both small-scale and large-scale applications. This has led to the development of new, more efficient solutions for high-quality embeddings. Overview of Sentence Transformers v3.2.0 Sentence Transformers v3.2.0 is a major update focused on improving semantic search and representation learning. This release, the first in two years, includes features that enhance usability and scalability. Key improvements include: Better training and inference efficiency Support for more transformer models Increased stability for larger production environments Technical Enhancements This version introduces several important technical upgrades: Improved Memory Management: Handles large data batches more efficiently, speeding up training. Optimized GPU Utilization: Reduces inference time by up to 30%, making real-time applications more achievable. New Backends: ONNX and OpenVINO backends enhance model inference speed by 1.4x-3x, depending on precision. Expanded Compatibility: Works seamlessly with the Hugging Face Transformers library for easier access to pretrained models. New Pooling Strategies: Improve the quality of embeddings for tasks like clustering and semantic search. Introduction of Static Embeddings Static Embeddings offer a modern approach to traditional word embeddings. They allow for quick embedding generation without the need for neural networks. Key benefits include: Speed: Model2Vec distills Sentence Transformer models into static embeddings in seconds, achieving a 500x speed increase on CPU. Efficiency: Maintains reasonable accuracy (10-20% cost) while enabling fast searches. Performance and Applicability Sentence Transformers v3.2.0 shows significant improvements in speed and embedding quality, with: Up to 10% accuracy gains in semantic similarity tasks. 2x-3x speedups with ONNX and OpenVINO backends for real-time deployment. This makes it suitable for a variety of use cases, addressing the community’s need for more efficient and versatile solutions. Conclusion Sentence Transformers v3.2.0 enhances efficiency, memory usage, and model compatibility, making it versatile for various applications. Key improvements include: Pooling strategies for better embeddings GPU optimization for faster processing Integration with ONNX and OpenVINO backends Support for Hugging Face models Static Embeddings for scalable semantic tasks For more details, check out the https://lnkd.in/dGQm8RKs. Follow us on https://lnkd.in/e7mfJSf5, join our https... https://lnkd.in/dzNmTK99 https://lnkd.in/dr5FcgNJ

𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐌𝐋) 𝐯𝐬. 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐃𝐋): 𝐀 𝐒𝐢𝐦𝐩𝐥𝐞 𝐆𝐮𝐢𝐝𝐞 𝐟𝐨𝐫 𝐁𝐞𝐠𝐢𝐧𝐧𝐞𝐫𝐬 If you're new to Artificial Intelligence (AI), you've probably encountered the terms Machine Learning (ML) and Deep Learning (DL). While they are related, they differ in their workings and strengths. Here's a simplified breakdown: 1. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐌𝐋)? Machine Learning is a branch of AI where computers learn from data and make decisions without explicit programming. It's ideal for straightforward, well-defined problems. * 𝐇𝐨𝐰 𝐈𝐭 𝐖𝐨𝐫𝐤𝐬: ML uses algorithms to find patterns in data and make predictions. For example, it could analyze customer purchases to recommend products. * 𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬: • Works with smaller datasets. • Requires manual selection of important data points (feature selection). • Includes algorithms like Decision Trees, Random Forests, and Support Vector Machines. * 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬: ML is used in fraud detection, personalized recommendations, and predictive maintenance. 2. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐃𝐋)? Deep Learning is a more advanced subset of ML. It mimics the human brain using neural networks and excels at handling large, unstructured data like images, audio, or video. * 𝐇𝐨𝐰 𝐈𝐭 𝐖𝐨𝐫𝐤𝐬: DL uses multi-layered neural networks to process data. Each layer discovers deeper patterns, enabling complex decision-making. * 𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬: • Automatically identifies important features (no manual intervention needed). • Requires large datasets and powerful hardware, like GPUs. • Uses architectures like Convolutional Neural Networks (CNNs) for images and Recurrent Neural Networks (RNNs) for sequences. * 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬: DL powers technologies like self-driving cars, image recognition, and voice assistants. 3. 𝐊𝐞𝐲 𝐃𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞𝐬 𝐁𝐞𝐭𝐰𝐞𝐞𝐧 𝐌𝐋 𝐚𝐧𝐝 𝐃𝐋 • 𝐂𝐨𝐦𝐩𝐥𝐞𝐱𝐢𝐭𝐲: ML models are simpler, while DL models have more intricate layers. • 𝐃𝐚𝐭𝐚 𝐍𝐞𝐞𝐝𝐬: ML works well with smaller datasets; DL needs large-scale data. • 𝐅𝐞𝐚𝐭𝐮𝐫𝐞 𝐄𝐱𝐭𝐫𝐚𝐜𝐭𝐢𝐨𝐧: ML requires manual feature selection; DL does it automatically. • 𝐇𝐚𝐫𝐝𝐰𝐚𝐫𝐞: ML can run on regular computers, while DL needs powerful hardware (GPUs). 4. 𝐖𝐡𝐞𝐫𝐞 𝐭𝐨 𝐒𝐭𝐚𝐫𝐭? • 𝐅𝐨𝐫 𝐁𝐞𝐠𝐢𝐧𝐧𝐞𝐫𝐬: Start with ML. It’s simpler and covers a wide range of applications. • 𝐖𝐡𝐞𝐧 𝐭𝐨 𝐓𝐫𝐲 𝐃𝐋: Once you’re comfortable with ML, DL is the next step for more complex problems like image classification or speech recognition. 𝐅𝐢𝐧𝐚𝐥 𝐓𝐡𝐨𝐮𝐠𝐡𝐭𝐬: Both ML and DL are powerful tools in AI. ML suits simpler problems, while DL is essential for handling large, complex tasks. Choose the tool that best fits the problem you're tackling. #AI #MachineLearning #DeepLearning #DataScience #ArtificialIntelligence

ConceptDrift: An AI Method to Identify Biases Using a Weight-Space Approach Moving Beyond Traditional Data-Restricted Protocols Datasets and pre-trained models come with intrinsic biases. Most methods rely on spotting them by analyzing misclassified samples in a semi-automated human computer validation. Deep neural networks, typically fine-tuned foundational models, are widely used in sectors like healthcare, finance, and criminal justice, where biased predictions can have serious societal impacts. These models often function as black boxes, lacking transparency and interpretability, which can obscure potential biases introduced during fine-tuning. Such biases arise from datasets and can lead to harmful outcomes by reinforcing existing inequalities. Recent methods for addressing biases in subpopulation shift scenarios primarily analyze validation data without investigating the model’s internal decision-making processes. For example, B2T [13] highlights biases only within the validation set, overlooking whether these biases impact the model’s weights. This gap underscores the need to examine model decision pathways to understand if dataset biases influence predictions. Current methods for identifying biases often rely on analyzing misclassified samples through semi-automated human-computer validation. These methods provide explanations for wrong predictions but lack precision in highlighting unwanted correlations. Machine learning models often capture correlations or “shortcuts” that may solve a task but are not essential, leading to biases that hinder generalization outside the training distribution. Prior methods such as B2T, SpLiCE, and Lg have identified dataset biases through data analysis. Techniques for debiasing have become essential to ensure fairness and accuracy, with approaches like group-balanced subsampling, reweighting, and data augmentation commonly used. In cases without annotations, some methods involve training and refining a biased model based on its misclassifications to reduce bias. Fairness research in machine learning is extensive, aiming to create ethical, equitable outcomes across subpopulations while overlapping with domain generalization and worst-group performance improvements. Interpretability is crucial in fairness, as understanding model decisions aids in bias mitigation. Invariant learning approaches enhance robustness to distributional changes by enforcing that models remain consistent across environments; however, in cases without predefined environments, data subsets can be created to challenge invariant constraints, using algorithms like groupDRO to improve distributional robustness. A team of researchers from the University of Bucharest, Romania Institute for Logic and Data Science, and the University of Montreal, have come up with ConceptDrift, a novel method designed to identify concepts critical to a model’s decision-making process. ConceptDrift is the first to employ a weight-space approach to...

🏁 Unlocking the Power of Siamese Networks in AI Have you ever wondered how AI determines whether two things are similar, even when they've never been seen before? Enter the Siamese Model, a deep learning architecture designed to compare and understand relationships between two inputs. Let's explore how it works, its potential, and how it can transform industries. How It Works A Siamese model uses two identical neural networks that share weights and parameters. These networks process two inputs (e.g., images, texts) independently and produce embeddings (feature vectors) for each input. These embeddings are then compared using metrics like Euclidean distance or cosine similarity to determine how "similar" the two inputs are. For instance, in facial recognition, the model can tell if two images belong to the same person by comparing their embeddings. What You Need To build and deploy a Siamese model, you need: 1. High-quality labeled data: Examples of matching and non-matching pairs. 2. A base architecture: Typically a Convolutional Neural Network (CNN) for images or a Recurrent Neural Network (RNN) for text. 3. Loss function: Use contrastive loss or triplet loss to optimize the model. 4. A deployment framework: Options include TensorFlow Serving, PyTorch TorchServe, or ONNX for real-time inference. Applications in Production Siamese models can be applied to: E-commerce: Product matching and duplicate detection. Healthcare: Medical image comparison, such as detecting changes in X-rays or MRIs. Authentication: Face verification for secure logins or ID validation. Content moderation: Detecting near-duplicate media to prevent plagiarism or misinformation. Risks and Challenges 1. Data Bias: Training data must be diverse; otherwise, the model may perform poorly on underrepresented groups. 2. Scalability: Comparing embeddings can become computationally expensive with large datasets. 3. Overfitting: Risk of overfitting if the model memorizes specific pairs instead of learning generalized features. 4. Interpretability: Understanding why the model predicts certain similarities can be non-intuitive. Benefits 1. Generalization: Once trained, the model can compare unseen pairs without retraining. 2. Efficiency: Generates embeddings that can be reused, saving computation time. 3. Versatility: Works with various data types—images, text, audio, and more. Scaling Siamese Models 1. Use vector databases like FAISS or Pinecone to store embeddings for efficient similarity searches. 2. Leverage distributed systems like Apache Kafka or Kubernetes to handle high traffic in production. 3. Optimize inference by quantizing the model or deploying on edge devices for real-time comparison. #deeplearning #ai #machinelearning #ml #siamese #CNN #RNN #Keras #Tensorflow