One of the sayings we have heard form business users in the data warehousing world is "deliver value in three month or forget it." Our experience indicates that to bring in a new subject area and produce answers to questions in three months is complex. The traditional development methodologies can make this quite near impossible. Particularly if you are introducing a new subject area to a existing data warehouse. The infrastructure that needs to be put in place to maintain the integrity of the warehouse adds to the complexity of achieving the goal. However we take these issues as a challenge and use many different innovative approaches to create rapid results.
One approach we have taken is to introduce the new subject area into a working data store (WDS) which resides on the EDW platform but does not have to be integrated with the EDW proper. We create a working prototype which answers the business question in a faster way. The data quality and other issues are identified and addressed. We derive a production ready system. Once the prototype is approved the next step is to integrate the WDS into the production data warehouse. We ensure that the WDS has a limited life span and must be integrated into the EDW if it is to continue to be used. If the WDS system demonstrates the ROI that is expected then the expenditure is justified. If the return is not there, the system will be discontinued.
By building a nearly real production conditions it is possible to:
- Leverage production data volumes
- Make tactical adjustments as issues are identified
- Identify logic gaps before our ETL coding gets deeply committed
- Test the soundness of the business case
The business users must be made aware that the system has been built is a prototype and that the WDS is not a production system. When the pilot is complete, the funding organization needs to be prepared to resolve the cost issues of promoting the project deliverables to production.
The WDS approach as design or development prototype can be used to address many of the pre-production issues. If the basic business case is not sound, the WDS will reveal the issues early which saves considerable amount of money and effort.
This is the second blog on Rapid BI. I invite comments, observations and other experiences.