How do you deal with uncertainty and ambiguity in your data modeling and data visualization projects?

Data modeling has multiple layer to it, if we are clear about our business needs and requirements, we will not get into uncertainty and ambiguity state during conceptual and logical phase of the model, these are the characteristics of physical model along with implementation details. Whenever we end up into this ambiguity sate, please trace back to the logical aspect along with initiating the conversation with the SMEs, we will always get to the right direction. Involving both the business owners and stakeholders in every stage helps us to navigate through these aspects with ease.

I guess there are a whole lot of people that don't understand that the "uncertainty" and "ambiguity" might actually be providing you with the correct answer of "at this point, we don't know" or "this observation shows that we may actually be doing something wrong in the <somename> project (or whatever).

By clearly defining the context and purpose, data analysts and data modelers can: 1-Identify the Scope and Requirements: Determine the boundaries of the project, including the data sources, the types of data to be analyzed, and the specific questions to be answered. This helps focus the effort on relevant information. 2-Define Success Criteria: Establish clear criteria for evaluating the success of the project. This involves defining the desired outcomes, metrics, and benchmarks that will indicate whether the project has achieved its goals. 3-Understand User Needs and Expectations: Identify the stakeholders, users, or audiences of the data. Comprehend their needs, expectations, and preferences for data presentation and interpretation.

The key is to understand what you're trying to do with the data. For most BI purposes, a normal Kimball-star schema will work fine. If you require "graphy" algorithms to process certain KPI's, (e.g. breadth-first or depth-first search related metrics), then a graph DB is helpful. Other NoSQL DB's are fine for storing unstructured data, but often the interesting KPI's concerning that data (e.g. a sentiment quotient calculated from free-text) are structured data and belong back in your Kimball-star along with all of the metadata to point to your original unstructured data back in your NoSQL DB. The point is that you shouldn't just use some shiny new tech because you read an article about it. Use tech if it helps solve a specific problem.

Parte importante da solução é a granularidade dos dados. A depender da informação demandada, da forma que será usada e do público consumidor, podemos detalhar ou sumarizar os dados no nível que mais ajude o usuário.

Os pontos de atenção do projeto devem ser documentados e comunicados, assim como o escopo, os requisitos, as regras de negócio e de cálculo alinhados. Recomenda-se criar no ETL alertas de qualidade técnicos e de negócio que monitorem falhas e instabilidades.

Young professionals sometimes hesitate to communicate uncertainties, ambiguities, or erroneous data, often blaming themselves for these anomalies. It's essential to address this issue by employing monitoring tools, and data quality checks throughout the whole data pipeline, from generation to the final product like a Power BI report. This approach not only prevents the hiding of uncertainties but also improves overall data quality.

Evaluate if the data uncertainty or ambiguity from the data model by its definition or the data visualization how the user experiences; like color, charts or information structure; are designed: the problem may come from one of them or only from one, and improvement must be done in the right place. Some recommendations: > Evaluate what is the purpose and scope of the data model. > Have updated data quality checks on your data model and data quality metrics to detect faster problem of the ambiguity/uncertainty aspect. > Define and manage metrics with connection of the data visualization metrics: color for user type, number of click to travel on dashboard, relation sheet-charts to give right information on one place.

Ambiguity and uncertainty are eventually eliminated when the data model better supports the needs of the visualization project; so iteration and feedback from users helps improve how the visualization project addresses their needs. Like any data project, this one has a specific purpose and scope; You must verify that this is always respected in all iterations and feedbacks: it will always help to mature the data model.

Well, I would love to add some experience-based suggestions; Handling uncertainty in data projects: Understand the data's origin, limitations, and uncertainties. Communicate them transparently in visualizations. Conduct sensitivity analysis to assess uncertainties' impact. Use probabilistic modelling for more accurate predictions. Visual cues like error bars help represent uncertainty. Take an iterative approach, gradually refining models. Document assumptions, steps, and decisions made. Embrace uncertainty as a natural part of data work.