Today, I want to talk about application integration and how COZYROC has extended SSIS to make it convenient. An ETL platform like SSIS has been designed for very efficient processing of flat, tabular format of data. For application integration you need a system to be able to process hierarchical data layouts, typically encoded in XML or JSON payloads. Systems like Microsoft BizTalk were designed precisely for that requirement, but because of their "one-message-at-time" design, these systems were known to be slow in processing hundreds of thousands of records. The proposed solution for that technical deficiency was to purchase multiple licenses, making the end solution expensive.
Is it possible to process hierarchical data in SSIS? On first look, it appears possible. The data flow components in SSIS like XML Source can work with multiple outputs, where each output represents an individual element of the hierarchy. However, in the SSIS data flow there is no straightforward way to identify which children rows (a sequence) belong to which parent row.
To solve the challenge, COZYROC came up with the concept of "end of sequence marker" in the form of an empty data flow row appearing in each child output. The empty row is a generic concept, allowing easy grouping of children rows belonging to a parent row. It is consistently implemented in all COZYROC components working with hierarchical data. In addition, we have also implemented a data flow component called Query, very similar to the standard SSIS Lookup component. The Query component can retrieve multiple output rows for a specific input row. With that component, you can retrieve your hierarchical data from a database.
In summary, the SSIS platform is designed to process large amounts of data by offering batch-processing capability. By including COZYROC in your toolkit, you can implement any required application integration and enjoy all these performance benefits.
Do you like what COZYROC has created? Please repost to help us bring even more magic to SSIS.
---
COZYROC is the leading provider of SSIS technology, providing more than 200 additional components. COZYROC's components are completely free for testing and development from Visual Studio.
#ETL #SSIS #DataEngineering #SQLServer #COZYROC
Azure, Power BI, SSIS, SQL, SSRS, ETL Developer
2moI'm interested