In the past we have talked about the role of governance, and in Monday’s post we talked about SOA – Top-Down vs Bottom-Up. It had me thinking about how to manage things like:
- Enterprise Platforms (BEA, Websphere, and the like)
- Data Warehouses (Federated, Enterprise and Data Mart)
- Integration (SOA, Web Services, EAI and EII …)
And one of the key roles of governance and stewardship when architecting and owning these types of platforms is to set guiding principles. The purpose of these is to ensure that we are getting the business and technology benefit of choosing these platforms.
So taking Enterprise Data Warehousing as an example … one of the traps that seems to be always an issue, is the focus on evolving and growing the EDW on the chosen platform as opposed to allowing orphan operational data stores or data marts to spring up. We all know this creates data duplication, leading to data inaccuracy, complex reporting among many other issues. With the advent of Enterprise Information Integration layers (Ipedo or AquaLogic) this can be handled, but all of a sudden…oops you are now federated. So where in the EDW you may have the information stored once (Teradata) twice to thrice (IBM or Oracle) you have it once in the EDW = $X + Other data store = $Y +… and this can get out of hand making the cost of your so called EDW seem really expensive.
Another example in EDW where guiding principles are necessary are on design principles:
- How do you handle extract – one per source system (don’t overhandle the data)
- Centralized ETL Development in ETL staging using one tool and one methodology or the same thoughts on ELT
- Integrated Data Quality Monitoring (how, when and where to assess, monitor and maintain data quality)
- Data Marts or Operational Data Stores
- Views or direct to the EDW to get the data
- Can Business Intelligence go directly to the operational source system, or only through the EDW
- How do we enforce these rules as Architecture only sets direction and does not have a compliance slush fund (these are guiding principals and we are not the fun police)
When guiding principles are not set, how are the business and IT going to know when to go from the top or from the bottom. Going from the bottom when you should design at the top can lead to a spaghetti of loosly integrated multi-copied data elements that should have been a 3rd normal form data store.