¿Cómo se pueden evitar las pérdidas de tiempo habituales en la arquitectura de datos?

One of the first things to do is to define the terms you are going to be using with your stakeholders. If this is not done you won`t be getting quality information and the clarity is lost already at the beginning. In addition make sure that all stakeholders understand the data models in the origin systems that will feed information to data architecture. In many cases even the date fields in the origin systems are misunderstood for their functionality and purpose. The good foundation is everything.

I believe the biggest challenge is the lack of data governance and poor data. We also should ensure we do not Over-engineer and create complications. As a leader vision plays a pivotal role in defining the roadmap.

Generally, data architecture and governance endeavors are typically abstract unless definitive goals and cascaded objectives are defined for them. To engender clarity in data architecture, it is essential to treat it a a project or to deploy project management techniques. In that manner, the goal of the data architecture or data governance initiative is set early in the day. Due yo the multiplicity of data, unlike other technology artifacts, this helps to eliminate distractions and concentrate attention on the set goals and objectives. This enables clarity, which reduces time wasted on executing data architecture or governance initiatives

A mind-set shift in how data is viewed would be an essential starting point. Traditional application centric approaches don't have enterprise harmonization scalability for the purposes of answering compelling business questions. Very few enterprises have a single data model. They have one data model per application, and they have thousands of applications (including those they bought and those they built). Very few organizations have applications based on an extensible model. Most data models are very rigid. An extensible model, by contrast is one that is designed and implemented such that changes can be added to the model even while the application is in use. This be the key to future proofing your architecture ...become data centric

A common pitfall in data architecture is neglecting or underestimating the importance of business requirements. It is crucial for a data architect to have a comprehensive understanding of the current business needs before beginning the design process. While grasping the existing requirements is relatively straightforward, the challenge lies in anticipating the future evolution of these requirements. This foresight is essential to ensure that the proposed architecture is adaptable to future changes. To avoid significant time and financial waste, the architect must have a thorough understanding of the business context surrounding the required architecture.

The day job of data architecture is the documentation of the data landscape of the enterprise, this should be seen as a mandatory activity, built into the operating model of the data architecture practice. These activities should also combine and connect to all other architecture disciplines to support the documentation of the enterprise architecture. This should be repository driven, accessible and quality assured. There are no shortcuts here, do it and succeed or neglect it and fail.

Documentation should as much as possible follow two simple rules. First, it should be automated. You shouldn't have a problem finding the latest version of your data model or data architecture, because you should proactively document your systems using whatever tools and systems are necessary to keep your documentation up to date. Second, your documentation should be used. It's all well and good to keep a dusty picture of the 1000-table data model on your cube wall. But documentation should be used by your coworkers to navigate your systems. Strive to tailor and integrate your documentation into company workflows, so people can identify gaps and you always have an incentive to keep your maps up to date.

Documentation can include a data catalog and glossary to assist in disambiguation across dependent systems. This can be considered as part of a reference architecture, helping stakeholders to build consensus across decision making matrices. Enabling alignment requires base agreements to ensure clarity and understanding of documentation.

Documentation is often overlooked but crucial for data analysis and projects. In large organizations, gathering information about data can take up 60% or more of the total project time due to tribal knowledge and inadequate documentation. Even when documentation exists, if it's not easily accessible or interpretable, it's useless. To address this issue, tooling such as data catalogs and processes are needed to create and make documentation available. The manual task of documentation can be improved with Gen AI and ML tooling to automate generation, allowing data architects and SMEs to review and approve it. By prioritizing documentation, we can save time and ensure that data is easily accessible and usable for everyone in the organization.

Tedious Perception: Make sure to address the perception that documentation is tedious by emphasizing its long-term benefits(it's a hard job but a must). Robust documentation reduces ambiguity, accelerates bringing new team members up to speed, and serves as highly valuable reference for troubleshooting and future enhancements.

Continuous Review and Refactoring: It's super helpful to implement a continuous review and refactoring process for your data architecture. Regularly assess the existing solutions, identifying areas of suboptimal and outdated design. Proactive refactoring helps address technical debt before it goes to a level where it's hard to solve or would need a complete separate project which would never kick off.

Addressing technical debt in data architecture involves consistent maintenance and proactive strategies. Regularly allocate time for refactoring and improving existing data structures. Prioritize resolving critical technical debt to prevent it from accumulating. Establish clear documentation to enhance understanding and minimize future technical debt. Implement robust testing practices to catch issues early, and involve the team in discussions about the trade-offs between short-term solutions and long-term maintainability.

Tech debt is - unavoidable because what is current today is dated tomorrow - necessary to deliver “good enough” instead of perfect. - on the backlog to address down the road - creeping in due to inefficiencies that were once acceptable Tech debt when managed diligently can be a buffer for a data architecture to remain a strategic differentiator for a company as the ecosystem evolves and the company adapts to the market.

What i noticed in large-scale organizations : - Each team often develops its own solutions and methodologies, leading to a mix of tools that may not integrate well, resulting in suboptimal maintenance and follow-up during the project's lifecycle. - This scenario turns data governance into a daunting task, complicating the management and utilization of information. Consequently, launching new projects becomes increasingly difficult, not due to the inherent challenges of the project implementation itself, but rather due to the complexities involved in gathering and organizing the existing data within the company.

Technical debt stems from prioritizing quick solutions over quality in data architecture, impacting system performance, reliability, and security. Adhering to best practices and regular architecture reviews helps prevent and resolve these issues for a stronger data infrastructure.

I normally try to write long answers, this time I'll be succinct - You can almost always solve problems with batch processing. Its very rare that the complexity from streaming is worth it. If you cannot explain how something works to a non technical friend, you do not understand it. Go back, re-design and learn.

One thing I always keep in mind is KISS principle (Keep It Simple Stupid). Many times we overthink on multipliers and the solution is no more relevant to original question. If you think the problem is complex then breakdown the problem to multiple problems and arrive at a simple solution. Simple solution translates to cheaper maintenance and better readability.

It seems that, due to lack of clarity of the actual problem, business analyst make things complex whereas there could be a straight and easy way to solve the problem. To avoid these kind of complex issues, it is neccessary to deep dive into analysis, and mark the information in form of "Must have" , "Good to have", "Not Needed", This is general practice we used to avoid unnecessary complexity.

One thing I've found helpful here is to constantly ask the question of WHY. WHY are we doing this or adding this feature. Is there a business need and can this need be met with existing or simpler architecture. The main thing I've found is that you need to be always asking this question, even to yourself. The hard part is doing it in a way that doesn't come across as being negative or adverse to building the solutions.

Referring the the section of the YAGNI principle the article manages: Looks all nice and fine on paper, but there are IT systems in even "new" tech. stacks which might have decades of a history. So eventually, you will need to overthink what you are doing - that is: not only IT but also business needs to consider if that is still a good practice or there needs to be a considerably long roadmap for re-designing.

👊 This point is based in self-control and enviroment-aware from one-self. Remember that not all things need to be distractions and interruptions; because one call or email can give you more context that you need or must to know; but you must stop irrelevant things to avoid away from doing your task: this is part of one-self when one is working.

The biggest distractions and interruptions happen when at meetings discussions start circling. Yes, sometimes some earlier made decisions need to be reviewed and amended. But only sometimes. To avoid wasting of time ALL architecturally significant decisions MUST be captured along with date, person who made either decision or proposed solution, and whether it has been accepted as consensus or by authority. At least keep track of ADR (Architecture Decision Record) on dedicated confluence page, or use appropriate tool. The second way to reduce wasting of time - is to establish culture of agenda preparation and "reserved 5 minutes". I.e. discussions are to be stopped minutes before deadline, to summarize outcome and set action items.

I've observed that distractions often arise when there isn't a dedicated team solely focused on data architecture. When individuals are multitasking and sharing roles with data engineers while also shouldering responsibilities for daily business operations, it can hinder the focused and strategic approach required for effective data architecture design.

En este caso, más allá de las interrupciones y de las distracciones, el punto crítico es la necesidad de perseguir a los stakeholders. Muchas veces, sucede que quién debe proporcionar el detalle desde el punto de vista de negocio, tiene una agenda incompatible con la del resto de la organización, lo que origina enormes desviaciones.

I find that distractions and interruptions are the norm. When I need to focus, I come in early, stay late, or work on a Saturday to get the job done. As a manager and task lead, I try to take those distractions and interruptions so my team can stay focused. I ask my team members to have requests come to me, so I can give them the mental and temporal space to work on our scheduled/assigned tasks. I set these boundaries with my co-leads and we work together to reduce the distractions. Most of the time this is successful and is fully supported by project management. This also works with the daily stand-ups we can identify and plan how to address the day-to-day to maximize productivity.

🔥 There are a truth in software world can be applied to architecture; like any part of data engineering world, in my opinion: "premature optimization is the root of all evil" ✨ You architecture must be funcional and focus to support business with minimum required things, no perfect. Part of organizational madurity is evolve your data architecture component and system while business teams is maduring too.

A data architect’s bit is really the data models (conceptual, logical, and physical). Since you’re mostly talking with software developers, UI/UX designers or scrum masters/project managers, all you need to do is to access the project documentation. If none is provided, then you might need to manage your stakeholders towards realizing the importance of having them. Encourage the adoption of project methodologies. Agile methodology for example, helps break down complex systems into small manageable builds (working software) that incrementally scales to a Minimum Viable Product(MVP). Make use of UML tools to produce simple diagrams. This helps stakeholders relate more with your project outcomes, and keeps an eye on the ball.

Placing excessive emphasis on perfection in every facet of data architecture has the potential to cause delays. Instead, the focus should be on adopting an iterative approach that incorporates feedback loops, allowing continuous enhancements over time. When crafting data models for the vital process automation and modernization initiative at a prominent bank, our architecture team faced initial challenges in foreseeing and accommodating all attributes and dependencies for the involved systems and entities. The swift adoption of the AGILE approach, marked by iterative releases and a continuous cycle of feedback, reflection, and improvements, proved instrumental in overcoming bottlenecks and ensuring a smoother delivery process.

Procrastination and perfectionism, although opposite, share the risk of delaying projects. Introducing agile methods, such as Scrum, can be an effective way to combat procrastination by establishing defined work cycles. Additionally, fostering a mindset of embracing failures and embracing iterativity can help mitigate the detrimental effects of perfectionism. Recognizing that absolute perfection is often unattainable and that the continuous pursuit of improvement is more valuable can free teams from the trap of crippling perfectionism.

To overcome procrastination, we need to understand the objective, key results, priority, timelines, and potential impact of not meeting them. If this understanding doesn't provide clarity and motivation, we should question if we are a fit for the project. Perfectionism can also be a roadblock. As an IC, I found success by defining the scope of the MVP inclusive of quality expectations based on project objectives, timelines, and constraints, and seeking buy-in from stakeholders. This alignment and commitment, along with regular check-ins, empowered stakeholders to hold me accountable.

In my experience, defining scope is critical to avoid wasting time. A data architect can’t define an architecture that will solve every business problem and as a consequence clarity of scope is critical. This will help manage stakeholders expectation, communicate successes, manage roadmaps, and enhance focus on effective solution delivery.

From my research, 75%+ of data projects fail due to lack of organizational alignment and support from leadership. Having a clear vision and strong investment in a data project is essential to its success. In my experience, when a data project struggles, it is because the organization has not yet fully understood the crucial way in which good quality data and clear requirements for that data powers the business in every way. And that understanding must be fundamentally understood and valued both from the bottom up and especially from the top down.

One of the things that I have learnt during my career is that it is not important to shine as the best performer of the month/year, best team and so on. It is important to be able to work seamlessly through the ways of working in the organisation and be able to remove hurdles. Doing the best data architecture is a personal satisfaction. It might not satisfy the goals of the organisation. So, it is always important to define the end goal of data architecture. The end goal needs to be well thought about and data architecture, in itself, can be handled in an agile way by taking small and manageable parts and show that the end goal (in that perspective) can be achieved.

Communication is the key… one should communicate regularly with the stakeholders and solicit their feedback on regular basis and take inputs to ensure that they are on the same page and that the work meets their needs and expectations.

Mucho ojo a poner por delante la tecnología y las últimas tendencias. Hay ocasiones en las que ser un poco más conservador te puede ayudar a no ir de PoC en PoC desviando el foco de lo realmente importante. Con eso no digo que no haya que estar pendiente de las últimas novedades, sino que hay que evaluar muy conscientemente si el beneficio a obtener justifica o no el cambio. Desde mi punto de vista, sólo si permite hacer el trabajo más simple, mantenible y eficiente, hay que lanzarse a ello. "Los experimentos siempre con gaseosa"