Kensu

"You don't have a Data Quality problem." As an evangelist and pioneer in the data observability category, I often surprise people when I say that "there is no data quality problem", and thus data observability is not a solution for this. I don't mean there is no data quality issues, I mean this is not the problem. In fact, all data have data quality issues. A.L.L. O.F. T.H.E.M. However they are not statistically significant relatively to: - The amount of data. - The amount of data sources. - The amount of usages. Therefore, if you are hunting data quality issues using so called "advanced techniques", you'll find them, oh yeah, a LOAD of them.  And, this will scare you to hell, so you will likely be tempted to hide it. Well, because, at the end of the day, data has always been used and this discovery doesn't align with the experience on the field. Right? Here is a positive message to the whole data community. Look at this from the "MOST" lens:  - Most of the time - Most of the data is good enough for  - Most of the usages If you're okay with this, then what is "the problem"? The problem is like Einstein (never!!) said: 9x10=91. It takes only one error to create distrust. And, of course, the solution cannot be to chase all errors, or, say, avoid and clean all (=100%) bad data. The solution is to *be* trustable, to make errors acceptable, because resolving is a well-understood and smooth process. That's part of the job, yup. Yes, they'll be complains, nothing can be perfect, we're all grown-up people we can deal with that (like we've all "accepted" and dealt with software bugs...) FWIW, here are some of the reasoning that lead me to bring up what is called today "data observability" and why I say "there is no data quality problem". More on this in my book: Fundamentals of Data Observability at O'Reilly https://lnkd.in/e7FT99Wh #dataquality #dataobservability #problemsolving

👊 Data quality punchline: “Garbage in, garbage out is an excuse, not an explanation!" Quote from the book “Fundamentals of Data Observability” by Andy Petrella. ✅ I recommend this book, which gives a deep insight into the more and more popular concept of data observability. Beyond data observability, taking a step back, I really enjoyed this book which gives a view of what modern data management should be: not only a matter of processes and governance (although these dimensions remain essential), but also a matter of cutting-edge technology and automation. Well done Andy! 🙌 #datamanagement #dataquality #dataobservability