The Eternal Battle - Data vs App

If you look at the root of most data quality issues it is normally embedded within apps - either poor data design (how many app developers have formal training in data design) or app defects (either design or execution). The ugly thing is that many of these defects are not detected until the data has moved well downstream, and when the train is moving down the tracks at 100mph it is hard to stop it and reverse it back to the station.

Another problem has to be that the two teams tend to be very territorial. The data team gets to known as picky (they whine about any quality issue) and the app team thinks they are above any data issue (we are moving the company forward, you just cleanse the data). So we end up with limited cooperation between the two teams and no consolidated roadmap to fix any data issues. The data team ends up in an endless cycle of rinse and repeat for cleaning data, and any new app changes can even screw with that.

It is easy to get in a little hidden battle of wills - where each team advances on each other teams territory. Often this results in creative little digs offering to help by screwing with your reputation and competence.

What we should do is just bite the bullet.

1. Formalize the data design using actual data designers - not app designers or database designers.

2. Open joint defects for identified data quality issues. Assign a roadmap to remediate and correct going forward. Put this effort on the timeline.

3. Be transparent. Data quality issues affect the whole company. Hiding them or obfuscating them will eventually catch up with you. Then, you could be looking for a new job. Data quality issues can screw with corporate numbers - do you really want to explain that you knew the data was incorrect but you didn't think to share that. Scapegoats are always at the bottom of the decision chain.

So avoid the pissing contest and don't start lobbing defects across the fence. It won't end nicely. This is where you need good leadership who can bridge this divide and work out a way for everybody to work together. It is to everybody's advantage. The app team will learn a lot about data design just from being involved in the process, and the data team can dream about getting out of the endless cycle of data cleansing.

Be brave, be friendly, be smart.