Ai+ Training’s Post

Ranging from experimentation platforms to enhanced ETL models and more, here are some more sessions coming to the 2024 Data Engineering Summit. #DataScience #AI #ArtificialIntelligence https://hubs.li/Q02qTWrs0

Ranging from experimentation platforms to enhanced ETL models and more, here are some more sessions coming to the 2024 Data Engineering Summit. #DataScience #AI #ArtificialIntelligence https://hubs.li/Q02qBt0Y0

Do you know where your data comes from? Apache Airflow does and it’s getting updated to advance data orchestration #AI #DataInfrastructure

👨🚀 Modern Data Engineering in the LLM Era 💡 Article by: Pavan Emani ➡ Link to Article: https://lnkd.in/ee6a4ySA - - - - - - - - - - - - - - - - - - - - - - - - - - - 💡 Follow for more great content! ✅ ✍️ Want to contribute? Connect with us. 🔗 Let's create together! ✅ - - - - - - - - - - - - - - - - - - - - - - - - - - - #dataengineering #AI #datascience #softwareengineering

4 Stages of Data Modernization for AI (Concluding) How each stage of next-gen data engineering supports today’s AI.   Summary: Changing needs of modern AI application forces us to re-look at how we do data engineering. Data engineering today must be re-shaped to enable knowledge creation & reasoning engines, without giving up the operational and semantic needs of traditional insight generation. From part 2 - the structure of this data engineering shift is a set of stages with each stage addressing some specific needs/gaps of AI enablement: 1.     Trusted Actionable Insights 2.     Traditional ML for qoq revenue/profitability 3.     LLM-apps, Vision-products etc. 4.     Multi-component inference, Agents & Systems Intelligence & Ops artifacts that are added to each stage of Data Engineering Each stage needs specific add-on components to enable the kind of semantic intelligence and operational effectiveness necessary for the modern array of AI apps. These add on mechanisms, when aggregated, is called Data-Intelligence-Ops. (see Graphic in attached paper). Formally, DataIntelligenceOps is an abstract set of operations meant to increase (a) semantic intelligence (b) operational intelligence & (c) governance abilities of data. It builds on top of existing investments in data-lakes, cloud-EDW, dbt automation, ELT, feature-stores etc. The main architectural artifacts are: · Semantic Intelligence Enhancements: a broad set of components for complex data products, which can be aggregated or configured through a low code IDE. · Connected DataOps: a connected DataOps architecture that “causally” ties together observability, lineage, storage/gov/sec-Ops, programmable pipelines, data contracts – to create an embedding layer for the above intelligence enhancements. Implemented as a full-featured knowledge graph that captures data platform wide meta-data. · Governance as Code enablement: Governance DAGs embeddable within pipelines allow for governance simplification, as well as policy implementations to be seamlessly executed. The effect of DataIntelligenceOps is to enhance the “intelligence” of a firm’s data, thus facilitating today’s AI apps. Parting thoughts:   1.     AI apps are rapidly increasing in complexity and capability – so, old boundaries of data engineering do not apply. 2.     The way to enable this AI led shift is to move to a modern style of data engineering that systematically adds semantic and operational value in 4 different stages of maturity. 3.     In many cases firms will choose to skip a stage to move faster and nothing prevents that. 4.     Existing building blocks such as ingestion mechanisms, pipeline tools, cloud EDW etc., remain unaffected – this is not a rip n’ replace design. 5.     Data engineering must now support knowledge enoblement, reasoning engines & qoq AI ROI. Paper - https://lnkd.in/gqG25drN

Generative AI: A Game-Changer for Data Engineering in 2024 ? The world of data engineering is constantly evolving, and 2024 promises exciting new possibilities. Generative AI, with its ability to create realistic data and text formats, is poised to be a game-changer. By leveraging this innovative technology alongside advancements like vector databases, data engineers can unlock the true potential of data. Ready to explore the future of data engineering? Check out my full blog post for a deeper dive into the trends shaping the landscape in 2024: 👇 👇 https://lnkd.in/g64UwC3a

4 Stages of Data Modernization for AI (Part-1) Part 1 – Defining the need for a shift in Data Engineering - Is your firm’s data platform, data engineering and DataOps capable of supporting successful AI? - Expanding the scope of data engineering to meet the needs of modern Knowledge & Reasoning Apps   Summary: Changing needs of modern AI application forces us to re-look at how we do data engineering. Data engineering today must be re-shaped to enable knowledge creation & reasoning engines, without giving up the operational and semantic needs of traditional insight generation. A 4-stage data engineering structure, with each stage solving specific AI enablement issues may be the answer. This structure is called DataIntelligenceOps. 2 Key insights drive the thinking behind this: 1.     The existing boundaries that define data engineering do not meet the needs of modern AI apps – these boundaries need to be expanded. 2.     Adding semantic richness and operational value to data is necessary to enable the potential ROI value of AI apps. Preface: Defining the Shift Most industries are building knowledge creation, insight generation and reasoning engine applications; these application categories were seldom seen even a few years ago. Some examples of such apps include: o  Plan & solve applications to automatically research any area of technology/science. o  Robotaxis o  Full length movie, advertisement w/ story-line & image-sequence generation o  ++ (see attached graphic)   As mentioned before the applications above throw up interesting artifacts which have minimal footprint (if any) in today’s data engineering. Examples of such artifacts include: ·       Cross inference step evaluation & verification data sets ·       Reasoning based function/tool calling. ·       Plan & solve agencies deployed as service-as-software. ·       ++ (see attached graphic) The key message that comes across from these shifts is that “don’t be limited by existing definitions of data engineering”. Leading Question: In all the above example apps, even if the industry moves to much better capabilities such as a few shot synthetic data generation or multi-modal Knowledge Graph embeddings, expanded data engineering with all its implications will remain a key enabler. So, the question is: with this significant shift in application needs, why do we think that data engineering of old will meet the needs of these new categories of applications? And, if there is a shift necessary, what should it look like! The simple response is that old style data engineering does not meet these emerging needs as the list of emerging artifacts suggest; new thinking is necessary. (to be continued)

Data contracts define a link between the data producer and one or more consumers of the data. It also links a logical world, dear to architects, and an implementation world, loved by engineers. https://hubs.li/Q02nqXqV0 #DataScience #AI #ArtificialIntelligence

@_odsc Data contracts define a link between the data producer and one or more consumers of the data. It also links a logical world, dear to architects, and an implementation world, loved by engineers. #DataScience #AI #ArtificialIntelligence https://hubs.li/Q02mzC180

Data contracts define a link between the data producer and one or more consumers of the data. It also links a logical world, dear to architects, and an implementation world, loved by engineers. #DataScience #AI #ArtificialIntelligence https://hubs.li/Q02nYJMt0