Data quality is always a challenge for data warehousing work. How to create confidence in the answer provided for business questions? I think we could learn some lessons from our financial people. How do financial records have the reputation for being so accurate?
I think the financial people work very hard at developing a set of rules that governs how finances are reported. However we have some very important organizations that created big surprises recently and their books were audited. Enron, Worldcom, and Nortel are famous examples. These organizations flouted the rules to the maximum and destroyed our confidence in the rules. These examples pointed out that a set of balanced books is not sufficient and new rules were introduced to prevent blatant abuses.
Although financial reports have a good reputation, the answers provided on financial statements depends on the approach you take. It may surprise you that they are rarely dead right. We must look behind the numbers and understand the rules that were used to produce the numbers. Things can be presented in many different ways showing quite diffferent results.
In the reporting world of data warehousing, we often have difficulty reconciling the financial reports with operational reports. Revenue and sales are one of the classic problems but there are many. I would guess that one of the issues is that the rules for non-financial item are not rigorous and the same discipline does not exist. We should not be surprised that developing rules to consistently provide quality answer is a challenge.
The challenge of producing good cross-functional reports also should make us look more carefully at some of the reports people get from spreadsheets. The rules and formulas used in a spreadsheet are usually created by the author and often not tested and check by others. Often spreadsheets are not created with discipline and rules. Again the lesson of having rules and procedures is very important. However the real answer for data quality is use the data warehouse to produce reports and outlaw spreadsheets. Unfortunately the largest user of spreadsheets are the financial people. I will write more about spreadsheets in a future blog.
I think we can learn some important lesson from the people in finance about how we do all our reporting. The development of discipline in procedures will really help move us to a more confidence in the numbers.
One thing about the financial reporting is that if the same rules are followed every time then we can compare periods with some confidence. Similarly in other reporting if we are rigorous in our rules, period to period comparison can be very helpful.
However even with all that rigor, we have a problem because things are changing very rapidly and our rules and procedures are having difficulty keeping pace. I think we need help from some good financial people to help us create more discipline in all our reporting. The experience they have in reconciling conflicting numbers and achieving closure would help all the business analyst.
In summary, data quality is a challenge for everybody and requires rules and discipline to create confidence in the answers. Even with the excellent reputation in financial reporting, financial people have their challenges. We can learn some lessons about rules and discipline from our financial friends.
I would welcome some comment here.