Building Interoperability between Enterprise Architecture IT Portfolio Management and Visual Modeling Platform

Introduction

I was recently involved in a pilot project related to the integration between

• An enterprise solution for Enterprise Architecture management, IT portfolio management and IT planning, realized by mean of SoftwareAG's Alfabet
• A collaborative modeling platform supporting solution architects, with standardized usage of ArchiMate models in order to produce Architecture Dossiers documents, realized by mean of Sparx' Enterprise Architect with ProCloud Server

It constitutes a perfect textbook case concerning the current actual practices for the establishment of Interoperability between applications within the enterprise, and the more theoretical approach I've been developing for years though my research activities.

It also allowed me to emphasis on some aspects of interest I didn't address in the past, and which can enrich the framework I've been developing through this research.

In this article, I describe roughly the project, the encountered difficulties and the lessons learnt.

Description of the project

Motivation

The motivation of the project was to simplify the life of Solutions or Applications architects in order to make them more effective and quick, and in order to improve the quality of the produced Architecture Dossiers (AD), i.e. making Architecture Dossiers content more aligned with what should be required for producing an accurate document for a more effective process, relying on the legacy Enterprise Architecture Portfolio management application and on the legacy collaborative visual modeling platform.

As is situation

In the AS IS initial solutions, the AD was based on a standardized document definition, coming with an Office Document Template to be fed with the relevant information to be in the document in order to support the solution development process. The architectural descriptions to be provided in the document were not constrained in terms of diagrams to be used.

To be situation

In the TO BE targeted situation, a simplified ARD document was to be proposed in alignment with the concerned stakeholders. A relevant subpart of the content was to be built from ArchiMate models and views (diagrams) specified by some dedicated viewpoints, the other parts being fed as usual in the document. The process was to be supported by Enterprise Architect features for modeling, model checking and automated generation of documents. It was also intended to push some content from the Enterprise Architecture Portfolio Management application within a dedicated model repository in EA Production Server, translated in ArchiMate in order to be depict the actual environment where each application under development will have to be deployed, integrated and used. The main idea was to reuse the legacy repository, and to be able when accurate to publish some views directly based on this model.

About building the interoperability

I was involved initially as an expert, knowing pretty well ArchiMate, and being able to contribute to an accurate mapping of legacy data with ArchiMate. Due to some resource constraints, I was also involved in the realization and I add to specify and to develop a part of the realization interface between the two platforms, so integration, and building the interoperability

Underestimated Integration Effort

The integration part was largely underestimated in terms of complexity and required work. In particular, each platform came with its own complexity and constraints concerning what was initially envisaged by Product Owners.

About expression of needs

Indeed, the expression of needs was made relying on what can be seen on user interface of the legacy application realized by mean of Alfabet. It means it relies on some views of the customized data model which are not necessarily reflecting what is actually stored (semantic contain, data structure) and in the underlying database structure as described in Alfabet Guidelines.

Concerning ArchiMate models in EA, it was also based on the ArchiMate language, but without considering how ArchiMate models are actually realized by EA and what is the underlying data structure.

About the implementation of the data exchange chain

Alfabet comes with export facilities allowing to create export extracting tables from SQL queries on the Alfabet relational database (we choose CSV syntax in place of XML syntax for these tables). As well known and used by involved participant in the project in charge of the application, we decided to use it as the first element of the tool chain implement the exchange between applications.

EA comes with many import and integration facilities. However, as envisaged in the business story, the idea was to obtain an ArchiMate Model derived from managed database of portfolio in a way supporting daily update. Also, as the elements of the model have be referenced by the different project diagrams, it should be ensured that the update process will not change their Global Unique Identifiers, in order not to have to daily rebuild the diagrams.

The different import functionalities were assessed in order to find if an accurate one exist, supporting functional requirements without implying extra development. The only way identified in order to perform it was XMI import with preservation of the GUIDs, which can be launched only from the user interface, and is not available through APIs which can be invoked remotely or though EA scripting. So at this stage and for the targeted MVP, the only reasonable solution implies having a human in the loop which will have to import XMI file(s) containing the derived model of the extracted content from Alfabet translated in ArchiMate.

XMI is the XML Model Interchange standardized format for UML specified by the Object Management Group, and it is based on object and component paradigms, which are far away from what can be find in a relational database. As a consequence, complex transformation are to be realized which can be easily performed with native export capabilities of Alfabet. So it was decided to realize the transformation related to both structural and semantic relying on a scripting language, initially Javascript, but due to some operational constraints, powershell was finally chosen as provided by default with Windows and as used by the team providing the operational support of the service related to alfabet.

Defining the accurate XMI to be produced

For defining the accurate XMI content for obtaining exactly the expected result in terms of expected ArchiMate model in EA, and due to the very weak documentation concerning how XMI import and export are realized, it was necessary to analyze, making successive export of targeted model, what should be the XMI file to be produced. If XMI is standardized, the way it is used by each tool is specific, as well as some extensions used by those tools. Also, ArchiMate support realization by any UML modeling platform is usually based on a specific profile for each product solution, which is exchangeable but not shared with the others. So an important work was required to defined exactly what should be put in the XMI file and how it should be structured.

Defining the accurate XMI imports for continuous regular update

Another point to be addressed: XMI import is about managed packaged. When importing an XMI with the update of a legacy package, all the legacy elements which are not in the imported model will be removed, the legacy elements with some changed tagged values will be updated and the new elements will be added.

It implies to import all the legacy elements which were not changed in the source. In addition, EA maintains the consistency of the models. In particular, it ensures that all the ingoing references from the packages which are imported packages dependent are maintained and not removed.

Implied cost

Maintaining such a consistency is very costly in terms of processing and requires to rely on transactions and process which are not very will documented. It was not identified during the initial feasibility study.

The consequence is that importing relationships and having many dependencies with what is imported will make the manual import process very long and slow, requiring to adapt the exchange strategy.

Raised questions

It also raised the several questions

• Which content of an enterprise IT portfolio database inside a collaborative modeling environment supporting design processes and why? It is to be considered that such design processes are run by each project related to a single application or solution.
• What does it imply in terms of data flows to be implemented, but also in terms of collaboration between design processes and IT portfolio management processes?Due to lack of experience (what can we do, what should we do), the intended collaboration was not precisely formalized. It means that after the return on experience using the deployed MVP, some business process reengineering may be needed, with a different data sharing strategy. It will also have to rely on the feedback from users who will have to gain more experience on the model driven practices. We are just at the beginning of the journey. But some lessons learnt can already be shared
• Which kinds of data are to be considered?

Kind of data to be considered in Alfabet and EA

An enterprise repository for the whole enterprise, such as Alfabet supports the realization, is to be exhaustive and performant. For this, they rely on relational databases where only the most relevant data are stores, avoiding when possible to deal with n to n relationships which require extra tables, but also many operations consisting in joining tables which are very resource consuming and slow. The diagrams are not stored, even if some visual representation can be created on the fly. So we can have some visualization services, but the diagrams are not managed and stored in the database.

ArchiMate language is mainly based on entities and binary relationships, so on graphs. This is reflected by the Open Exchange Format for ArchiMate specified by the Open Group. When realized on top of UML platforms by mean of profile, as done by EA but also other UML platform providers, the underlying paradigms, as supported by XMI, is the object and component ones. Also, as being a visual modeling language, UML not only store data for the models but also for the diagrams which are also managed and serialized. This allows to versioned them and to perform change analysis and comparison not only on data of the models, but also on the diagrams, being for the represented model elements but also for all the visual representation of the diagram elements: shape, size, color, etc. As a consequence, a visual model contains quite more information than what is usually considered in a solution like Alfabet, as it doesn't support the same needs. The diagram is a key artefact for visual modeling.

Impacts of project mode for models life duration

Other thing to consider: a project related to one application component or for a set of application component is finite in time and resources. The modeling artefacts produced to support their activities will be released at the end of a project and become dead data, which be not subject to change anymore. However, applications as defined in the AD are still to be deployed, run, operated, used and continuously adapted to their environment following processes such as those defined by ITIL. What if some changes occur? Will the application models to be changed and updated? Who will ensure it, and according to which processes? How the consistency with the IT portfolios will be ensured? And what if not considering only two applications as we did, but also all the other enterprise referential of the enterprise, distributed and often siloed in several applications?

Ambiguity created by Consulting companies

Observing how consulting companies such as Gartner Group, Forrester or IDC have been describing those kinds of application, we can observed some shifting concerning "Enterprise Architecture Governance", starting with EA modeling solutions and considering more today to Alfabet like solutions. In the meantimes, it seems that EA modeling solutions disappeared from the business landscape.

Before 2020, Magic Quadrant about EA tools included a solution such as Sparx EA. In 2019, the solution was indicated as a niche player. It disappeared from the radars since 2020, with new kind of solutions such as Software AG ones, considering mainly Alfabet and not other solutions such as ARIS. Considering how different solutions as Alfabet and Sparx EA are, it seems to reflect the shift in terms of definition of Enterprise Architecture tools as indicated earlier, even if some are considering that it is due to the fact that such solution are low ends of the market. It seems to favorise very expensive solutions which are not relying on any open visual modeling standards, with clear limitations in terms of engineering and end to end process.

Conclusion

Removing silos for end to end collaboration in such distributed system is quite challenging and complex, as the simple described case illustrate it. It is not only a question of technologies, but people, processes and data are to be considered holistically.

This is also important to point out the main differences between applications such as Alfabet and those aiming at ArchiMate modeling. The main differences lie in the use of standardized visual languages for precise modeling, support for collaborative modeling with change tracking, granularity and detail in modeling capabilities, and alignment with industry standards and best practices. While applications like Alfabet excel in governance and decision support for application portfolio management, dedicated modeling tools are better suited for creating precise architectural models collaboratively with track changes functionality.

In terms of implementation of data exchange, sharing and aggregation between these tools, it implies to deal with some complexity related to the way data are structured, in order to support data processing and pragmatic usage of produced information.

Coincidentally, Archimate aims to help us to support it.

Next articles will illustrate it, but also come back to some of the encountered issues for more detailed description.

Don't hesitate to indicate what you would like to see developed further.

Let's like and share the article if you think it creates value ;)