Recently I have been doing some research on the characteristics of successful data warehousing services. I have learned some very interesting things and the challenges that organizations face. The interesting thing is how important this area is for a successful business. The stories of the leaders are legendary and the skills necessary to use good business intelligence are in great demand. However the challenge of people offering business intelligence services is significant.
The challenge of every organization is to find a way of sorting out truth from fiction. Most organizations have a tremendous amount of information in the systems that run their business and do their financial reporting. However the information is often not available in a form that can be used by decision makers. The challenge is to collect the data into a data warehouse which can produce some good reports and help people make good business decisions.
The challenge in many organizations is the people in IT have a vision of what is possible with good decision making tools but the decision makers are really not aware of what is possible. Some business sectors, like finance and retail, have made great strides in changing the direction of their organization with good business intelligence. Many others have had difficulty getting started.
A typical sign of an organization which is moving towards better business intelligence is the use of very complex spreadsheets for management reporting. Usually the data for these sheets are entered manually for other sources and then manipulated to give some impressive and important reports. Most organizations have at least one horror story of a decision which was made because of errors in entering the data or an error in the spreadsheet. Most organizations do not really have much of a choice because to get started in data warehousing can be quite intimidating. I do not think it has to be difficult but needs to be carefully planned.
One of the difficulty many organizations face when trying to get started with a data warehouse is they set too ambitious target at the beginning. Do not bring in all the data at the beginning. Set up a structure that can grow but start small. Many people think that the first step is huge, I believe a approach that leads to early short term benefits is really important. Often the grand design is easier to imagine but with some effort and creativity you can create some early wins. I think the business people must demand early results and be impatient. These demands can create some design challenges but lead to better learning and business results. The penalty of not moving forward to make good decisions from good information can be very serious for an organization.
The challenge of the IT leader is to move with the readiness of the organization and at the same time take leadership, very delicate balance. The ideal circumstance is a business leader who is demanding better information and an IT leader who sees a vision of data warehousing that meets immediate needs and builds for the future.
The difficulty is the justification of the expenditure. With operational systems, the justification is related to fairly tangible return on investment. How can one predict the value to better information to make better decisions? I think the metaphor of mining is a great one. Mining companies spend a large investments looking for new ore bodies. Management is always looking for better business intelligence to make better decisions. The search is very important because in the rapidly changing business environment, being blind or basing decisions on multiple versions of the facts is very risky. Not every search will lead to gold but without the search the business is blind. After the search is successful, we now want more and more. One of the most interesting things that the data warehousing project finds frequently is that the sources of data is very bad. From the business point of view, basing decisions on bad data is very risky. Looking at the data outside of the operational framework often reveals weaknesses that do not necessarily affect the operational systems. Always be skeptical about all information and always give it the test "Does this make sense?"
