Mirza Iqbal

Decision Intelligence translates data into competitive advantage. Would you like to learn more about this emerging discipline? Join us at the the ZHAW Zurich University of Applied Sciences on June 26, 2024. Detailed agenda and registration page in the comments. #AI #python #OR #optimization #ML

22

2 Kommentare

I highly recommend Algorithms for Big Data by Hannah Bast, Claudius Korzen, Ulrich Meyer, and Manuel Penschuck for anyone looking to explore cutting-edge approaches in big data processing. This open-access book dives deep into scalable algorithms tailored to handle massive datasets efficiently, covering topics like data streaming, graph processing, and external memory algorithms. It’s a valuable resource for researchers and professionals in data science, engineering, and beyond. Grateful to see such high-quality knowledge made freely accessible! https://lnkd.in/eSfVM6k5 Look for "Read and Download Links" section to download. #BigData #DataEngineering #OpenAccess #DataScience #DataStreaming #DataProcessing

3

📷 New on YouTube: In his 2023 #SAGconf session, Matthias Niehoff delves into the latest developments in modern data architectures, including Data Mesh, Data Lakehouse, and the integration of analytics with operational systems, offering insights into their practical implementation. 👀 Watch now: https://lnkd.in/eHs3EkZa #SoftwareArchitecture #SoftwareArchitectureGathering #SoftwareArchitectureConference #DataArchitecture #DataMesh #DataLakehouse #ELTPattern #CloudDataWarehouse #ModernDataStack #DataStrategy #SoftwareEngineering

8

🎉 Our wait is finally over, In Databricks we can now perform recursive directory listing, pattern-matching for files, and many more operations 🖋️ Author: Rahul Madhani 🔗 Read the article here: https://lnkd.in/dicbGbHk ------------------------------------------- ✅ Follow Data Engineer Things for more insights and updates. 💬 Hit the 'Like' button if you enjoyed the article. ------------------------------------------- #dataengineering #dataanalytics #databricks

24

2 Kommentare

Explaining the concept of "agentic flows" to non-technical stakeholders can be done by relating it to familiar concepts and using simple language. Here's a way to break it down: The ideas are based on the concept that Andrew Ng interesting December 2023 almost a year ago called Agentic Systems. These systems include software that include autonomous agents that are based on generative pre-trained transformers or gpts, or other autonomous software functions that operate based on inputs and sensors that are programmed into them. When an event or condition triggers them they operate by taking place transforming them to a pre-planned output and then providing it to a subscriber or another piece of software. I have to process this idea through a GPT because it's really technical, and I wanted to see if it could translate into business terms. 🤖🧠 ## Understanding Agentic Flows in Agentic systems 1. What Are Agentic Flows? Agentic flows refer to the pathways through which intelligent agents (like AI programs) operate and interact within a system to perform tasks. Think of them as the routes or channels that guide how different parts of a system work together to achieve specific goals. 2. Everyday Analogy: Imagine a team of skilled workers in a factory, each with a specific role. Some workers handle raw materials, others assemble parts, and some ensure quality control. Agentic flows are like the assembly line that coordinates these workers, ensuring each task is completed efficiently and in the right order. 3. How Do They Work? - **Intelligent Agents:** These are like smart workers who can make decisions based on the information they receive. They can adapt to changes and optimize their tasks. - **Flow of Information:** Just as workers need instructions and materials, agents require data and directives to function. Agentic flows ensure this information moves smoothly between agents. - **Goal-Oriented:** The ultimate aim is to complete tasks effectively, whether it's processing customer orders, managing inventory, or analyzing data. **4. Benefits for Business:** - **Efficiency:** By coordinating tasks intelligently, agentic flows help streamline operations, reducing time and effort. - **Flexibility:** They allow systems to adapt quickly to changes, such as new customer demands or market conditions. - **Scalability:** As your business grows, agentic flows can easily accommodate increased workloads without major overhauls. **5. Real-World Example:** Consider an online shopping platform. Agentic flows do a customer order each month if there's a sale of > 10% : from selecting items, verifying payment, updating inventory, to arranging delivery. Each step is handled by different agents working in harmony, ensuring a smooth no touch shopping experience. By using these analogies and explanations, I hope this helps less technical stakeholders grasp the concept of agentic flows and appreciate their value in enhancing business operations.

1

Whether you’re training an entire LLM oder try to stop your cat bringing mice home: Getting your AI model “just right” and streamlining the processes around it is 95% of the work. Only reality will reveal your model’s performance. You want robust data pipelines and training processes to get there as fast as possible. Iterate quickly to make the right corrections to the model early. Until the model is ‚just right‘. LMOps takes care of that, in fact of the whole lifecycle and together with the right platforms, is crucial for success in Machine Learning. Inspired by DevOps principles: Fast flow and early feedback. RAG (Retrieval Augmented Generation), which integrates your company data with large language models, is the newest kid in town and guess what: RAGOps is what makes it happen in a robust way. Something to watch out.

119

15 Kommentare

💛 working with TMDL on Direct Lake I tested my personal new highlight feature of this month today with Marcus Wegener and Andrea Hupp - it works on all explicitly created Semantic Models - the "edit mode" has to no report - like in the Web-Service. But the DAX query view feels "more close" - all changes are synched to the published model. No saving needed. 🤯 but the real game changer is to export the Power BI file a project, then I can: - edit everything in TMDL code and sync the changes fast to the published model (but also beeing able to refuse the changes). Deployment never have been so easy 🤩 - as an empty report appears when I open the .pbip file, I am able to test my changes in real visuals. Even when I love the DAX query view... sometimes visuals show me more insights when I investigate some data issues 😉 👾 Doing the Video was also a chance to use the AI Dubbing one more time - it still feels a little strange, but I still like it and its faster doing the recording multiple times. I still sound different in real english tutorials I do from time to time 😅 Looking forward for more news from the European Microsoft Fabric Community Conference ..! 😃 #powerbi #fabric #update

18

2 Kommentare

How does OpenAI train the Strawberry 🍓 (o1) model to spend more time thinking? After reading the report, I noticed it mainly focuses on the what—the impressive benchmark results. But when it comes to the how, the report gives us just one sentence: “Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses.” I took some time to really break this down and created an animation to illustrate my understanding. Two key concepts from this sentence are: Reinforcement Learning (RL) and Chain of Thought (CoT). From the list of contributors, two people stood out: • Ilya Sutskever, known for creating RL with Human Feedback (RLHF). Even though he recently left OpenAI to start Safe Superintelligence, his mention suggests RLHF is still part of training the Strawberry model. • Jason Wei, the author of the famous Chain of Thought paper, who joined OpenAI last year after leaving Google Brain. His inclusion points to CoT being a major part of the RLHF alignment process. Here’s what I aim to convey in my animation: 💡 In RLHF+CoT, the Chain of Thought tokens are fed to the reward model, which scores them to improve LLM alignment. This is different from traditional RLHF, where only the prompt and response were used for alignment. 💡 During inference, the model starts by generating Chain of Thought tokens (taking up to 30 seconds) before producing its final response. This is how the model is “thinking” more! Of course, some technical details are still unclear, like how the reward model was trained and how human preferences for the “thinking process” were gathered. Finally, a disclaimer: this animation represents my educated guess. I can’t fully verify its accuracy, but I’d love to hear feedback from someone at OpenAI to help us all learn more! 🙌 #OpenAI #Strawberry #AIByHand #RLHF

210

9 Kommentare

#rag RAG architectures have many patterns.. expect to deploy multiple databases in multiple forms in order to decrease error rates and use the right components to give the right answers in modern solutions for Enterprise. I do like the novel idea of contextual compression.. it's a good hack. basically you summarize a document to its essence.. because of the summarization in the nature of natural language understanding it is possible with structure documents to gain semantic understanding and context... you can imagine that across 1,000 documents about underwriting.. the summarizations of those documents will start to getting some kind of samanthic understanding. using this instead of full text index searches may get to the right answer sooner.... it's to be experimented but. what's needed in any type of approach like this is some kind of data set, some kind of synthetic data, and of course some kind of benchmark that says that this form of rag reduces errors at a predictable rate. This level of rag is good for experimentation but unlikely to move to a production stance involved in critical core systems. there's not enough control and as designed its error prone. it's still an improvement over using a straight GPT to solve an Enterprise value chain or workflow. #ai #insurance

1

1/ Uncomfortable $10B question for $DELL 😳 Analyst Toni Sacconaghi asked a simple yet brutal Q: Why didn't operating profit change when $DELL added $1.7B worth of servers for #AI services? #earnings #tech 2/ The implication? $DELL's operating margin on #AI servers is effectively zero! 😲 A single Q led to a $10B drop in market cap. Ouch! #investors #stockmarket 3/ Are chipmakers the real #winners of the #AI hype? 🤔 While tech giants chase AI, semiconductor companies rake in profits from high demand. #analytics #bigdata 4/ $DELL's situation reminds us that not everyone benefits equally from new tech trends. 📉 The road to #AI domination is paved with costly investments. #riskvreward #innovation 5/ A $10B question indeed! 💸 One analyst's query exposed potential cracks in $DELL's #AI strategy. Will they course-correct or get left behind? 🔭 #disruptor #futuretech 6/ In the world of finance, simple Qs can have massive impacts. 💥 $DELL's case shows why public companies must be transparent & prepared. #accountability #shareholdervalue 7/ The AI wave is rising, but who'll be the real surf champions? 🌊 Tech titans or component suppliers? An exciting race to watch! #competition #disruption 8/ Perhaps $DELL's AI conundrum serves as a wake-up call. 💡 Pivoting to new tech requires careful planning & execution. No room for rookie mistakes! #strategy #leadership 9/ While some rejoice at $DELL's $10B stumble, others see an opportunity. 👀 A shake-up could spur innovation & realign priorities. #transformation #pivot 10/ In the end, the $10B Q highlights the complexities of the #AI revolution. 🧩 Not every player will emerge a winner, but the game's just begun! #futureisnow #techbuzz

1

1 Kommentar

The Big data and AI world Frankfurt conference is a wrap. After 2 days of engaging conversations with potential clients and partners, here are my takeaways: 1. GenAI was on every booth but talks were about data products 2. Companies have been investing in their data foundations and are now focusing on adoption. It's showtime. 3. There is a zoo of vendors and service companies and it's hard to position each offering in the landscape. 4. There is still a big market for on-prem datacenters. Thanks for the interesting conversations and talks. In particular, thanks to Daniel Gburek from BioNTech SE for sharing his data platform journey in a crowded and sold out keynote theatre. We're proud to be a part of that journey. And we're excited to share some of that work with the world soon, in open source. Great to meet our colleagues in consulting, in particular Matthias Roggendorf and Sally Trivino. Your perspectives on the data world are very sharp. I loved catching up with Bolke de Bruin, Paolo Platter and Bart Vandekerckhove 🔐⬅. I always learn a lot when I talk to you guys. Last but not least, thanks to Wannes Rosiers, Thomas Kranzkowski , Niels Claeys and Sofia Iamskaia for representing Dataminded in the best possible way. 🚀🚀🚀

113

4 Kommentare

#GlitchTokens are like weed in a crop, it can reduce the yield . Similarly a Glitch token can impact the performance of LLM Unlike words, which represent clear and discrete units of meaning in human language, tokens in LLMs can represent anything from whole words to fragments of words or even punctuation. This modular approach allows LLMs to process and generate language with remarkable efficiency and subtlety. But what about glitch tokens? Occasionally, the tokenization process can produce unexpected results—these are known as glitch tokens. These anomalies can occur due to the complex interplay of encoding and training data irregularities. For instance, a glitch token might represent a piece of text that doesn't conform to typical linguistic patterns due to encoding errors or unusual data inputs. Understanding these glitches is crucial for refining model outputs and enhancing the overall robustness of LLMs. A proper training of LLM reduces the changes of glitch, there are ways to find the glitch tokens . I will try to put more about the same in future. #AI #tokens #LLM

3

Large Language Models (LLMs) on Structured Data Sets: A Synopsis of Current Challenges and Research Directions The integration of Large Language Models (LLMs) with structured data sets has become a focal point of discussion among senior technical leaders responsible for data architecture in various companies. This summary encapsulates the essence of recent conversations and research efforts aimed at understanding and optimizing the use of LLMs in the context of structured data. The primary application of LLMs in structured data environments is to interpret human queries and translate them into SQL or other query languages. This translation process is currently seen as inefficient, as it not only demands substantial computational resources but also does not fully leverage the capabilities of LLMs to address the underlying problem statements. The convolution arises from the mismatch between the unstructured nature of LLMs and the rigid, predefined schemas of structured data. Research in this domain is actively seeking to bridge the gap between the fluidity of LLMs and the rigidity of structured data. The goal is to enhance the user experience by creating more intuitive and efficient ways for LLMs to interact with and extract information from structured databases. This endeavor is not without its challenges, as the computational intensity of LLMs poses scalability issues. A striking comparison reveals that a typical LLM request consumes approximately 17 times more power than a standard Google search query. This disparity highlights the need for innovation in making LLMs more energy-efficient and cost-effective, especially when dealing with structured data. The sustainability and scalability of LLMs in data architecture are contingent upon advancements that can reconcile their high computational demands with the economic and environmental costs. One such research effort is the work by Wang et al. (2021), which proposes a novel framework for integrating LLMs with relational databases. Their approach involves a pre-processing step that transforms structured data into a format more suitable for LLMs, thereby reducing the computational overhead. Another significant contribution is the research by Zhang and Choi (2020), which focuses on optimizing the interaction between LLMs and structured data by introducing an intermediary layer that can effectively translate natural language queries into database operations. The ongoing research in this field is a testament to the potential of LLMs to revolutionize data architecture. The quest for a more harmonious integration of LLMs with structured data sets is not merely a convolution but a glimpse into the future of data management. The innovation that successfully mitigates the current limitations will undoubtedly pave the way for a new multi-billion dollar market, offering unprecedented opportunities for businesses and researchers alike. #LLMs #DataArchitecture #StructuredData #Innovation #Technology #Research

9

Time is a non-renewable resource. Here’s how to save time in 2025. To save time, you have to: • Amplify your focus • Improve your workflow • Enhance your effectiveness 🔥 These techniques will help you optimize every minute. 1️⃣ The Pomodoro Techinique • Set a timer for 25 minutes. Take a 5 min break. • Repeat for 4 cycles. Take a longer (30 min) break. 2️⃣ Timeboxing • Plan tasks in fixed blocks. • Stick to your schedule! 3️⃣ Eat The Frog • Do the hardest but most rewarding task first. • Helps make big progress. 4️⃣ Pickle Jar Theory • Big tasks like rocks. Small tasks are like sand. • Prioritize for effective use of time. 5️⃣ 3-3-3 Method • 3 hours on important task. • Complete 3 short tasks. • Work on 3 maintenance tasks. 6️⃣ Task Batching • List all your tasks and group similar ones. • Prioritize groups and do them in order. 7️⃣ 1-3-5 Method • Choose 1 big task, 2 medium and 5 small ones. • Create a balanced workflow for the day. 8️⃣ 5-Min Habit • If you're avoiding doing something, do it for five minutes. • If it's going well, keep going. If not, stop. 9️⃣ Jim Collins Way • Creative Work → 4 hours. • Meetings And People → 2 hours. • Random Exploration → 2 hours. • Recharge Time → 4 hours. Every minute matters. Say Good Bye to chaos. Say Hello to outrageous efficiency. End each day will feel stress-free and productive. — ♻️ Find this helpful? Repost for your network. 🦁 Follow Terry Danylak for more content like this!

261

153 Kommentare

AI revolves around data — both real and synthetic. Understanding the nature of data sources and the information contained therein is a critical part of understanding the future of AI and the capabilities of models. This article describes some intuition about why data matters and a way to think about it ranging from drug discovery, AI reasoning, and creative intelligence. #ai #techbio #agi #drugdiscovery

8

4 Kommentare