Bernardo Brito’s Post

💡 Important Lessons from AWS Lambda Layer Size Limitations Recently, I encountered a challenge while trying to process Google Sheets data using AWS Lambda. Here's what I learned that could save you time: 🚫 The Issue: Trying to package Google Sheets API dependencies in Lambda layers resulted in: "Layers consume more than the available size of 262144000 bytes" (250MB limit). 🔍 Key Takeaways: 1. Lambda Layer Limitations: - Each layer is limited to 250MB unzipped - Combined layers cannot exceed 250MB - This includes ALL dependencies 2. When NOT to Use Lambda: - Heavy data processing tasks requiring large dependencies - Applications needing full Google API clients - Projects with multiple large external libraries 3. Better Alternatives: - Amazon ECS (Elastic Container Service) - AWS Fargate - Amazon EC2 for complete control - AWS Batch for batch processing 🎯 Pro Tip: Before building on Lambda, check your dependencies' sizes. Sometimes, what seems like a simple serverless task might be better suited for containerized services. #AWS #CloudComputing #ServerlessComputing #TechLessons #CloudArchitecture #AWSLambda #DataEngineering

Data Engineering using AWS EMR - Here is a rundown of how I achieved it. 1. Data Storage: Stored raw data in Amazon S3. 2. Network Setup: Created a Virtual Private Cloud (VPC) for secure networking. 3. Cluster Creation: Set up an Amazon EMR cluster. 4. Secure Access: Used CLI to SSH into the EMR cluster. 5. Data Transformation: Wrote Spark jobs to transform CSV data into Parquet format. Output Storage: Saved the transformed data back to S3. This hands-on experience with EMR, VPCs, and Spark showcases the power of scalable data processing on AWS. 💡🔧 #AWS #EMR #BigData #CloudComputing #DataTransformation #Spark #S3 #TechProjects

Quantizing vector embeddings can greatly reduce size of embeddings by 4-32x and greatly increase speed and reduce latency of vector searches.

🌊☁️Real-time data ingestion using AWS Kinesis ( Intermediate Tutorial ) 🌊☁️ ! The Full detailed video is available in Full HD quality exclusively on my YouTube channel : https://lnkd.in/gf63Ky32 Please feel free to REPOST and share this video with your network if you found this post helpful. Thank you. 🙏 #aws #awscertified

Automate Vulnerability Reporting with Amazon EventBridge, Lambda, Glue, DynamoDB and QuickSight https://lnkd.in/dR34VeTX #amazon #aws #lambda #glue #dynamodb #quicksight #devsecops #terraform

AWS Bedrock now supports compression for embedding models from Cohere 👊 in the image below, you can see example how to generate embeddings for batch of strings in three different formats: - float32 (which is the default) - int8 - binary You can choose only one of course Now why this is important, this means when storing these embedding in Vectore Database, it will take less memory and disk usage This leads to faster, more scalable, and cost-effective RAG usage for enterprises dealing with TBs of data. Amazon Web Services (AWS) #aws #ml #bedrock #cohere #embedding

ExitCertified is at the Amazon Web Services (AWS) #reinvent24 conference. Learning all about the new features in the AWS Data Stack.

As a machine learning engineer, leveraging the power of cloud computing can significantly enhance your productivity and project capabilities. Amazon Web Services (AWS) is a leading cloud service provider that offers a comprehensive suite of tools and services tailored for machine learning professionals. This podcast will walk you through the essentials of AWS, from understanding cloud computing foundations to preparing for certification exams. https://lnkd.in/dUv5GSYu

I know that there are plenty of lists with re:invent upsates, but here is my subjective overview of the (p)re:invent updates that potentially are going to change or affect the way I build on AWS with list of interesting related extra materials. https://lnkd.in/dKCZhaVz #aws #reinvent #reinventathome

Happy to inform that I have completed my course in AWS DEVGENAI #jijothomas #jijothomas.in #jijothomasai