Flexibility BI/DW Systems for Business

In the fast paced business world we live in there is a big push to be flexible. Fleetness and flexibility are the means to success in the new millennium. If your business is going to survive it must proactively change before the market, rather that reactively chase after the competition. A business’s survival is contingent upon its ability to adjust quickly to market forces. Rather than level the playing field, we try to tilt it to our advantage.

We’re advised to leverage technology to be more competitive in the market place. And we’ve been told that new systems need to be flexible.

Sadly, the technology world we live in is not as flexible as we might like. In some situations it can be down right rigid. For example: IT departments, and consultants, will demand a structured environments, architected solutions, enterprise business models, and business requirements before construction of a system can commence. The limits of the technologies they work with force them to be inflexible (highly structured) when building solutions. This can cause the technologist to become inwardly focused. As a result, they tend to think about their technology as the end in itself and not the means to achieve business goals. Technology has limits. In order to build systems using the technology we have to impose these limits on the business. While trying to get systems to work flexibility is often forgotten because of the rigidity of the technology.

Here in lies the dilemma: In defining the limits for the business we remove the business’s ability to be flexible in the market place. Limits are imposed by the technology we hoped would free us from limits.

How often have you heard, ”it’s a good plan but it can’t be done in the time frames and budget allotted because the systems won’t support the change.” Or, ”We’ll create a quick solution now & fix it up later when we have time.” But that later time never seems to come.

These practices and outcomes become the rigid foundation for everything that comes after. Once in place they constrain flexibility, forcing us to make comprises imposed on us by our previous decisions.

Funny how work emulates life.

How easy will it be to change?

Does this architecture scale?

Can we change it later as needed?

We need to define flexibility into our architectures.

But how?

Applications can be built to be data driven. Relational Databases can be built to support these applications. Object oriented technology is supposed to help us with this but many IT shops out there still code in 2GLs, or scripting languages, or interpretive markup utilities like XML for batch processing. And hard coded values imbedded in 3000+ line programs still exist in great number.

We can drag ourselves out of the old world if we can anticipate how people will perceive the introduction of new features and functions.

We need to remember that no matter how much we plan and try to anticipate the market’s needs the market will always outsmart us with a new idea we never thought of.

The business model should drive the design. Future perfect is the goal but today practical is the reality.

Make sure that the data is actually as specified. Often times transformations are made more complicated by unanticipated data, data that no one knew was there. While the business rules may drive the design the available data constrains the final deliverables.

Technology aside for a moment, the most inflexible piece of any current systems is the people using it. People will always come up with new, unintended and sometimes better ways to use the systems & data they have. People become content with the quirks of exiting systems and resist changes that may impact their work habits and routines. ”If it ain’t broke, don’t fix it.”

Building the Flexible DW

The main role of the data warehouse is to provide value to the business through a repository of data.

The business then has to find ways to get at that data so that better decisions can be made based on improved information in the database.

On the technology side we demand clear requirements so we can lock down the design and get to development. I think the challenge for IT is to accept the fact that changes to the database will occur. No matter how hard we try to get the final design locked down. Perhaps we need to think of the DW as an ongoing, evolving system, rather than a system that can be completed.

This implies we need to keep the structure of this DW as simple as possible for both maintenance and development & change reasons.

The most flexible database structure to support is 3rd Normal Form. Right about now, the star schema bigots just stopped reading. But sometimes the truth hurts. The challenge with 3rd Normal Form of course is the performance of multi-table joins is notoriously bad in most databases. So we build materialized views and star schemas to get around our performance problems and in so doing give up some flexibility. We build data marts for specific reporting purposes and thereby limit our flexibility.

Perhaps, we should start thinking about the performance issues of our databases and how they can be improved rather than building kludges around them.

Flexibility derives from simplicity of data structures that can be joined and queried dynamically at will with little or no technical intervention required.

Leave a Reply

captcha *