Como você gerencia as alterações do modelo de dados em diferentes ambientes e plataformas?

Data model changes should adhere to the same disciplined approach used in application code changes. Begining with a Data Modeling solution, such as SqlDBM. Here, all model changes are made and documented. Once these changes are finalized, Data Definition Language (DDL) scripts are generated and deployed into a code repository. This allows for transition into a (CI/CD) process, where changes are rigorously tested across different environments. This ensures consistency, reduces errors, and enhances collaboration between teams. My experience has shown that bypassing these steps and implementing schema changes without following a proper development lifecycle often leads to significant challenges.

1. Version Control System (VCS): • Utilize a robust VCS like Git to track changes to your data model. • Maintain separate branches for each environment (e.g., development, testing, production) to isolate changes.

I would like to add, along with the version control system (VCS) we should have proper ticket / task in the agile board like ADO or JIRA. It will ensure traceability and governance of the model.

A version control system (VCS) is essential for managing data model changes across environments. By tracking modifications, VCS enables: Collaboration: Multiple teams can work on data models simultaneously without conflicts. Reproducibility: Data models can be recreated consistently across environments. Rollback: Previous versions can be restored in case of issues. Auditing: Change history is tracked for accountability and compliance. Efficiency: Automated deployment and testing are facilitated.

Other than the points listed by the author, I think we can also look into the following: 1. Validate your data model for the right datatype each time a new set of business data comes in 2. Data capacity planning and the right retention approach helps in maintaining the per row size of data in each table 3. Data model maintenance should also maintain a track of which attributes are getting obsolete so that they can be partitioned into a different physical table

Standards and conventions are not mere guidelines; they are foundational for your model. Would you design a house without the proper scale, annotation, and consistencies? Setting up your model with standards is the first step to creating a design with harmony. Naming conventions add polish but are primarily for end-users' consumption. Why force a business user to decipher various ways of abbreviating terminology? Consistency ensures wider understanding and adoption. Notes, comments, and keys for reference also assist users with less experience in reading and navigating the diagrams. Setting all of this up from the beginning will ensure you can manage and maintain your model for many releases to come.

In my experience data modeling will keep on changing till final design is not endorsed. There are document versions maintained for changes in the data modeling. Data modeling help you to track object build vs object version maintained in the modeling repository. Metadata version can be tracked as per modeling versions. As we keep modifying objects as per data modeling, you can store it and also move it across environments through DevOps pipelines built using tools like Github, Gitlab etc. To migrate code across say DR or Preprod these pipelines storing data model specific object version are very reliable and we can manage cutover very quickly across complex landscapes.

In my experience, organizations had enterprise architecture team, consists of Database architects and DBAs, which needed to approve the model after making several iterations of review. There can be suggestions to de-normalize the models (for better performance), to follow naming standards & conventions, and to create possible indexes, from the team. My point is, the process (or lifecycle) is mandatory for data models.

A well-defined data model lifecycle ensures consistency and control over data model changes across diverse environments and platforms. Key benefits include: Standardization: Enforces consistent data definitions and structures. Version Control: Tracks data model changes, enabling rollback and impact analysis. Collaboration: Facilitates teamwork and communication among data stakeholders. Quality Assurance: Improves data quality through rigorous testing and validation. Efficiency: Streamlines deployment and reduces errors by automating processes. Compliance: Aligns data models with regulatory requirements and industry standards.

Before apply any change in your data modeling; like remove columns, new focus of use of tables or segment yout table into small tables; check how it affect data team and IT team in its processes and include them in testing, deployment, monitoring and evaluation step. How data modeling is designed; based in approach 3NF, dimensional, data vault, anchor and so on; can help to implement those change and define how interact with data team and IT team.

You can use a version control system like repository too; such that: - You can have several stored changes in your data modelings in simultaneously in different branches. - You can control and tracking changes was testing, approvated, merged and applied to production; and just store right data model. - You can easier apply it in CI/CD pipeline into repository to automatic your testing and monitoring operations by any stored changes.

Beyond basic storage, a data model repository should support integration with data lineage and impact analysis tools. Understanding how changes in one part of the model affect others is crucial for maintaining data integrity across platforms. A repository should facilitate tracking dependencies between models and allow visualization of data flows and transformations. Integrating the repository with other systems, such as BI tools or data catalogs, creates a unified view of data assets, improving governance and decision-making.

Some points you must know to get automation process: - Use synthetic data to create new content in tables inside dataset: doesn't hurt nobody and nothing; but it must be closer real world reflect. - Get test environment with prod mode data in minor scale: easier to test and apply all changes prod environment. - Get ready your filling scripts when you fill new columns or old columns with new data: it must be modular to easy adaption in data pipeline. Note: test it in test enviroment, then it's ok to apply in prod enviroment. - Apply unit tests in your support queries to check performance of database to acchieve SLA metrics. - Define and apply a deployment strategy to extend new changes: don't want all costumers faced with unexpected issues.

Automation and rigorous testing are indispensable for effective data model management across diverse environments and platforms. By automating the deployment process, organizations can significantly reduce errors, improve efficiency, and ensure consistency. Comprehensive testing verifies data integrity, schema compatibility, and performance implications, enabling early identification and mitigation of potential issues. Moreover, version control for data models provides a robust mechanism for tracking changes and facilitating rollbacks when necessary. These practices collectively contribute to enhanced data quality, reliability, and overall system performance.

When you're planning any change in data modeling, you must talk all needed stakeholders to move the change to prod system: they will help your change don't (or very little) impact negatively in systems related with data modelings. Define a process that involve them: how they communicate to coordinate works, what they will do based in your change that impact in their systems and how they will test and deploy it when change is ready.

Foster a culture of transparency and shared responsibility by setting up regular sync meetings and updates focused on data model changes. Use visual collaboration tools like whiteboards or modeling diagrams to facilitate discussions. Create and maintain documentation that is accessible and understandable to non-technical stakeholders, ensuring that business users can participate in the decision-making process. Implement feedback loops that allow stakeholders to provide input on data model changes, ensuring that the models align with business goals and user needs.

When managing data model changes across different environments, it's vital to consider the impact of these changes on data analytics and reporting processes. Changes in data models can affect the accuracy and performance of analytical queries and reports. Establish a dedicated analytics team that works closely with data modelers to understand the implications of model changes on analytics outcomes. This collaboration ensures that data model changes support analytical goals and do not disrupt business intelligence operations.