Atika Rifda Nafi'ah’s Post

𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁 𝗶𝘀𝘀𝘂𝗲 𝘄𝗶𝘁𝗵 𝗮 𝗹𝗲𝗴𝗮𝗰𝘆 𝗱𝗮𝘁𝗮 𝘀𝘁𝗮𝗰𝗸 𝗮𝗻𝗱 𝗘𝗧𝗟? 💥 When answering that question, we should keep in mind that the stack defines not just the technology but also the methodology of data transformations. On a legacy stack, you get ETL — Extract, Transform, Load — where you extract the data, do most of the transformations and only then load it into the warehouse. This entails various issues around scaling, cost optimisation and the way the business accesses the data. The last one is extremely important because if end users have any questions or need more context, it is next to impossible to give them an answer as anything important happens before the warehouse. This is why, when there is a migration, you also see a change in methodology from ETL to ELT (Extract, LOAD, Transform). With a modern data stack comes ELT, where you do the vast bulk of transformations on top of the new data warehouse. Often, this methodology change is the very reason you do a migration, as it solves the major problem of how business users leverage the data to get insights. 🎧 Listen to the rest of this session with Solutions Architecture Manager Mike Burke (dbt Labs) and data engineering consultant Samy Doreau: https://lnkd.in/dy-QyC5m You will see what a successful migration looks like and get insights into accelerating the process. If you are planning a migration but struggling with uncertainty, give us a shout. We are here to help. #DataMigration #ELT #ETL #ModernDataStack #LegacyDataStack

Data engineering must set and communicate a responsible data governance model. In any big data platform, the following should be clearly defined: a. Who should have access? b. How is PII handled? c. How can we educate end-users on our data catalog? One solution to handle these scenarios is to use a data catalog and a data loss prevention API for guarding PII. 1. Simplify data discovery at any scale: A data catalog makes all the metadata of your datasets available for users to search, flag sensitive columns, and more. 2. Unified view for all datasets: With many different datasets and tables, each with varying access levels for different users, a data catalog provides a single, unified user experience for quickly discovering these datasets. 3. Data governance foundation: A data catalog helps to manage, identify, and protect sensitive data. #dataplatformengineering #bigdataengineering #platform #architecture

I am honored to deliver a keynote on a governance framework for data management at the Data Governance Conference 2024 in Vienna organized by ADV: https://lnkd.in/e_Ky-aha I just returned from #EDW2024. I've seen that many companies have been doing a lot in different areas of data management. The key focuses are implementing data catalogs, data architecture (especially data mesh), semantic modeling, data quality, etc. However, implementing a governance framework remains challenging, especially when organizations redesign their data architectures. In the keynote, I will demonstrate an approach to implementing a governance framework that should meet an organization's goals, needs, and resources. #data #datamanagement #framework #datagovernance

I've seen firsthand how a well-structured data lake can transform business operations. Today, I'm sharing the industry-standard approach that's proven to deliver results: 🥉 Bronze (Landing) Layer - Raw data ingestion - Preserve original format - Enable data lineage and auditing 🥈 Silver (Transformed) Layer - Cleansed and conformed data - Implement data quality rules - Optimize for processing efficiency 🥇 Gold (Consumption) Layer - Business-ready datasets - Aggregated and denormalized views - Optimised for analytics and ML Why this matters: ✅ Improved data governance ✅ Enhanced data quality ✅ Faster time-to-insight ✅ Scalable architecture Have you implemented this approach? What challenges did you face? Follow me Sampath Vijayan for more insights and tips! #DataEngineering #BigData #DataLake #Deltalake #lakehouse #s3 #Adlsgen2 #gcs #databricks #BestPractices

Data lakehouses are the foundation for modern data architectures, unifying vast, diverse datasets. They also expose organizations to substantial risks when data quality is left unchecked. Data inconsistencies, schema misalignment, and unvalidated stale data disrupt downstream data-driven implementations. Poor-quality data in the lakehouse introduces costly reprocessing and undermines the reliability of insights, impacting business decisions. Maxim Lukichev, CTO and co-founder of Telmai, explains how the Medallion Architecture mitigates these data quality risks by structuring lakehouse data into Bronze, Silver, and Gold layers. Max outlines the common issues that can plague your data from pre-ingestion stages to analytics-ready stages. He also dives into data quality checks tailored for each stage—from early-stage validation and anomaly detection in the Bronze layer to advanced profiling and filtering in the Silver and Gold stages. Click on the link in the comment section to learn about how proactive data quality monitoring can transform your lakehouse into a reliable source for data-driven decisions. #dataquality #datalakehouse #data #dataobservability

Understand the power of Data Lakehouse Architecture for 𝗙𝗥𝗘𝗘 here... 🚨𝗢𝗹𝗱 𝘄𝗮𝘆 • Complicated ETL processes for data integration. • Silos of data storage, separating structured and unstructured data. • High data storage and management costs in traditional warehouses. • Limited scalability and delayed access to real-time insights. ✅𝗡𝗲𝘄 𝗪𝗮𝘆 • Streamlined data ingestion and processing with integrated SQL capabilities. • Unified storage layer accommodating both structured and unstructured data. • Cost-effective storage by combining benefits of data lakes and warehouses. • Real-time analytics and high-performance queries with SQL integration. The shift? Unified Analytics and Real-Time Insights > Siloed and Delayed Data Processing Leveraging SQL to manage data in a data lakehouse architecture transforms how businesses handle data. Learn more about it here for FREE... https://lnkd.in/ggc98c_i Join DataGeeks Community here... https://lnkd.in/gU5NkCqi do follow Ajay Kadiyala✅ #DataLakehouse #DataManagement #RealTimeAnalytics #DataStrategy #SQLIntegration

"Logical Model: The Backbone of Data Architecture A well-structured Logical Model is crucial for ensuring data consistency, integrity, and efficient data flow in any OLTP system. It serves as the blueprint for your data warehouse, guiding data integration, transformation, and analytics efforts. The lack of accurancy might cause problem of Relational integrity. In addition it likely to provoke trouble the whole process in the Data archtecture. Ligia P. #DataScience #DataEngineering #DataAnalytics #BI"

Ready to optimize and upgrade your data lake or lakehouse in 2025? Aqfer CEO Daniel Jaye outlines critical elements of their success in this latest post: Metadata & Provenance. These elements transform chaotic data repositories into reliable, query-ready ecosystems, enabling seamless workflows and compliance with strict regulations. Read more: https://lnkd.in/ea2YyE9E #datalake #datalakehouse #datastorage #datamanagement #igdata #metadata #provenance

🚀 Say hello to smarter data management with Index Lifecycle Management (ILM) in Elasticsearch! 💡 ILM automates the complex process of managing your indices through various stages—hot, warm, cold, and delete—tailored to your specific needs. 🌐 This game-changer not only boosts performance and ensures compliance but also optimizes storage resources, cutting down on costs. 📊 Ready to revolutionize your data management? #Elasticsearch #DataManagement #TechInnovation #ILM #StorageOptimization #BigData #MachineLearning #CloudComputing #DataScience

Moving data is expensive and time-consuming. Therefore, businesses are shifting from developing ETL or ELT (Extract, Transform, and Load or Extract, Load, and Transform) procedures to focusing on ET (Extraction and Transformation) processes. This article explains how to use edgeTI to create a Data Mesh layer without transferring or migrating the source data. #datamesh #awscloud #data #datawarehouse #datagovernance #awscommunitybuilder #enterprisearchitect #solutionarchitect #dataarchitect #analytics #edgeti ***If you are not a "medium" member, click this link to read the whole story=> https://lnkd.in/ggvS-Zgv https://lnkd.in/gSZrQjk3