MinusX’s Post

Dramatically Accelerate Your Work With DataWalk!! Analysts, data scientists, and many other users often spend a significant portion of their time trying to locate and prepare the data they want, before they can actually do anything with it. DataWalk dramatically accelerates efficiency and time-to-results by having all of your desired data pre-connected and in the DataWalk repository. Unlike other Enterprise-class systems, in DataWalk your data is automatically connected after ingest – for example automatically connecting a person’s phone number with other people who may share that phone number - further accelerating efficiency by offloading users from the need to create such connections manually.

Iceberg made table optimization more standard but it’s still hard to get right. At a recent face-to-face event I spoke with a data engineer about the challenges of implementing an Iceberg lakehouse. From her experience, while the initial Iceberg lakehouse implementation wasn't overly challenging, she found the continuous optimization of data to be more complex than she originally anticipated. In particular: - Knowing when to run optimization tasks: right after the ingest/transform job? on a separate schedule, say every 10 minutes or 1 hour? - What should the minimum file size or minimum number of files be to trigger compaction? - What is the ideal target file size, per table and per partition? - How many files to rewrite and commit concurrently? - Should rows be sorted and how, z-order or binpack? This conversation sparked an idea, and long story short, Upsolver enhanced its original table optimizer and developed a smarter table optimizer that adapts to all of these changing variables and automatically determines how to best optimize your tables. We call this Adaptive Iceberg Optimizer. This experience is why I value community events so much. The talks are insightful, but the real gold lies in the opportunity to discuss challenges, share experiences, and connect with peers. It's a chance to help each other grow and innovate. That's why we're launching a series of face-to-face events to tackle Iceberg challenges: Chill Data Summit. Starting with the Bay Area, this event will focus on improving your Iceberg skills, networking, and learning from each other. We’re bringing together the best minds in the industry to give you an edge. This is your opportunity to learn and become an Iceberg expert.   Check out the agenda and register for the event here: https://lnkd.in/eAWWRtZk

🚀 Master Your Data with New Relic's Updated Query Interface! 🛠️✨ Explore powerful new features that make finding insights easier and more intuitive. Even I've picked up a tip or two! #DataAnalytics #TechTips #NewRelic

📦 Broadcast Join: It's like sending a small package to all your friends. The smaller table is sent to every node, making it efficient for joining small tables with larger ones. 🔀 Shuffle Hash Join: Picture sorting and grouping people based on their characteristics before pairing them up. It's efficient for joining large datasets where data is shuffled across nodes based on a common key. 🔄 Sort Merge Join: Imagine two lists of names sorted alphabetically, and you're merging them into one sorted list. This join method is effective when both datasets are sorted based on the join key.

Let's discover together how to get a quick grasp of any DataFrame with 2 simple commands👇🏻 1️⃣ .𝗱𝗲𝘀𝗰𝗿𝗶𝗯𝗲() When you're knee-deep in numbers, `.describe()` is your trusty sidekick. This gem provides a statistical summary at lightning speed—mean, median, max, you name it! It's like having X-ray vision for your data! 🚀 2️⃣ .𝗾𝘂𝗲𝗿𝘆() Need to filter data based on certain conditions? .query() is here to rescue! This function selects rows using a SQL-like query string, helping you dive deep into specific data aspects. Did you like this post? Then join my freshly started DataBites newsletter to get all my content right to your mail every week! 🧩 👉🏻 https://lnkd.in/dA8xuFJ5

𝘚𝘦𝘤𝘰𝘯𝘥𝘢𝘳𝘺 𝘋𝘢𝘵𝘢 📽 𝐏𝐨𝐢𝐧𝐭𝐬 𝐢𝐧 𝐭𝐡𝐞 𝐕𝐢𝐝𝐞𝐨 1. What is Metadata? 2. What is Primary Data? 3. What is Secondary Data? 4. Isn't that just Metadata? 5. Where and How can Secondary Data be Used? 6. Demo 💻 𝐃𝐞𝐦𝐨 𝐰𝐢𝐭𝐡 𝐭𝐡𝐞 𝐖𝐡𝐢𝐭𝐞 𝐂𝐫𝐨𝐰 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 1. Looking at Primary Data in the Data Collector 2. Looking at the metadata of a data file from the file processing audit trail file 3. Looking at metadata and linking primary data to an experiment in the Observation Manager 4. Looking at Secondary Data and equipment in the Crow's Nest 𝑳𝒆𝒕 𝒎𝒆 𝒌𝒏𝒐𝒘 𝒚𝒐𝒖𝒓 𝒕𝒉𝒐𝒖𝒈𝒉𝒕𝒔. Thanks for watching

Let's discover together how to get a quick grasp of any DataFrame with 2 simple commands👇🏻 1️⃣ .𝗱𝗲𝘀𝗰𝗿𝗶𝗯𝗲() When you're knee-deep in numbers, `.describe()` is your trusty sidekick. This gem provides a statistical summary at lightning speed—mean, median, max, you name it! It's like having X-ray vision for your data! 🚀 2️⃣ .𝗾𝘂𝗲𝗿𝘆() Need to filter data based on certain conditions? .query() is here to rescue! This function selects rows using a SQL-like query string, helping you dive deep into specific data aspects. Did you like this post? Then join my freshly started DataBites newsletter to get all my content right to your mail every week! 🧩 👉🏻 https://lnkd.in/dA8xuFJ5 Don't forget to follow me to get more content like this! 🤓

Put your data under a microscope and see it in a new light. #melissadata profiling tools allow you to research and monitor data quality to deliver more accurate and trusted data. Click here: https://lnkd.in/gzh6P6NZ #dataquality #DataProfiling #dataresearch #datamonitoring #DataTools #highqualitydata

With #Gemini ✨ in Google Sheets, you can: 📋 Create tables. 🔣 Create formulas. 📝 Summarize your files from Drive and emails from Gmail. Get your #workspace today! Connect with our experts at business[at]versupinfotech[dot]com or drop us a message VERSUP InfoTech #GoogleWorkspace #GooglePartner #VERSUP

Data comes in different shapes and sizes. Sometimes, we want to shape the structure of this data to one that fits your needs. With JSON-data, you can achieve this using JOLT. In this series, Morad Aoulad Abdenabi explains what JOLT is and how it can help you when working with JSON data.