Consistency paves the path for automation

About a year ago, a mid-sized public organization talked to us about their data maturity level. The company was challenged by the increasing demand for data, both for internal reporting and application integration, as for external public organizations and policy makers. Our suggestion was to start with an audit on key domains such as data governance and solution architecture.

Upon analyzing the existing environment, the typical challenges in classical BI and data warehousing emerged: unclear business requirements, no or limited governance, limited scalable architecture, increasing pressure on the BI team, lots of operational overhead and so on. To face these challenges, we recommended to re-use the existing building blocks of skills, software and infrastructure to create two strong pillars: one for governance and one for architecture.

We were tasked to create a roadmap to implement this vision and assist in executing its fundamentals. So, a new data warehouse was set up using the Sparkle best practices centered on agility and automation: Data Vault 2.0 methodology, Microsoft technology, Vaultspeed data warehouse automation and Attunity integration. But how to combine data governance, automation and agility with constantly changing business requirements?

Well in our case, before starting developing, we documented our design principles, naming conventions and best practices for two key components in the standard Sparkle architecture: the business data vault and the presentation layer. Three important principles were the following:

  1. The presentation layer would be entirely virtualized.
  2. For every dimension we would make a PIT table and for every fact there would be a Bridge table.
  3. All business rules would be maintained in the business data vault using calculated satellites, links and links on satellites. Business rules were only allowed in the presentation layer by exception.

These principles proved to very useful throughout the last year. New people joining the team were easily up-to-speed because of the consistency and best-practices for Data Vault 2.0. Small changes in the business requirements were delivered within a sprint because changing database views is easier then changing persisted tables.

And the path for full automation was paved. In September 2019 the new version of Vaultspeed was launched, introducing business data vault automation with PITs generated by Vaultspeed. Since our implementation was compliant with the Data Vault 2.0 methodology, the migration to Vaultspeed required little rework, allowing a smooth and efficient transition.

So, investing in thinking about design principles, best-practices and methodology will result in benefits in the semi-long-run. Needless to say, in the data industry, there’s always plenty of work to be done. That’s why we so strongly believe in agility, automation and governance, so that you can focus on creating value from your data with the limited time you have.

Koen Moermans

https://www.sparkle.ms

Geef een reactie

Het e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *