Why Business Intelligence is So Complex?

Often the conceptual formulation of a business intelligence project is very simple.  "Wouldn't it be great if we could track out net return by product or customer?"   As the project develops the definitions and business rules often get mired down in complexity.  The business sponsor is really needs to drive the project rather than allow it to expand into a complex formulation which is accurate to the penny.

This simple formulation can lead to a very complex project unless the vision of what is needed is driven by a business sponsor, in partnership with IT.  I believe the complexity arises in bridging the communications gap between the business and IT.

I recall a project many years ago that the VP Sales wanted to track the "Net-Net" of every deal the salesmen proposed.  The idea was to compare the proposal with the current price of the commodity on the London Metals Exchange.  This calculation was very complex because of different currencies and many locations around the world.  The key to success of the project was the VP drove the project and eliminated the complexity devised by the lower level people who wanted it accurate to three decimal points.  He knew it would be an estimate but would be a powerful decision making tool.  The project was saved from drowning in complexity by a determined sponsor.

I think a sponsor who can hold onto the original vision and creating momentum.  The sponsor needs support from key project members who share the vision.  I think often the problem is that the concept is new to the organization and people resist the change.  One way of resisting the change is to add complexity.

I recall I wanted to add a second bathroom at my cottage.  I really wanted just a toilet and a sink.  My simple vision quickly got expanded into adding a whole new wing to the cottage.  I failed to hold on to my vision and the project became huge and stalled.  On reflection, I allowed my vision got overwhelmed by complexity.  If I had stuck to my vision, I might already have a second bathroom at my cottage.  You may say that was a scope issue but it sure felt like the project was becoming more complex.

How many projects have we been involved in that got driven into complexity and the original vision was lost in the details?

  1. Jim Reply

    Another friend submitted this great comment:
    I kind of agree and disagree with the scope vs. complexity argument, here’s my thinking: I agree that there are two different issues at play here, scope/definition of scope and complexity of each scope item. “Scope creep” has been an issue on almost every project I’ve been on. However, I think there are always two ways that scope can “creep”: 1. A purely new requirement that was not originally in scope; 2. A modification/increase in complexity to an original scope item. Number 1 is easy to deal with but the difficulty may be in both parties (business sponsor and IT) agreeing the requirement is indeed new. Number 2 is a little tougher. Usually an increase in complexity arises from improper understanding of the complexity of the requirement to begin with. Your bathroom example is a good one to illustrate this because a lot of the problems that could arise from the “I need a new toilet and sink” requirement are hidden until the work starts. When work actually begins we could find that the underlying plumbing is rusted and needs to be replaced, or the subfloor is rotting, etc. This happens a lot in IT. We take a set of requirements from the business sponsor, do some preliminary analysis and give an estimate. When the actual work starts there are usually several “oh crap” moments where we realize, for example, the supporting EDW data will not give us the results the client wants. This starts a long back-and-forth discussion on how to best deal with this with turning the project into something bigger than it should be. This is where the business sponsor must inevitably agree on a “good enough” solution rather than an “accurate to the penny” solution. This was the case on a couple of EDW projects I’ve worked on where we tried to balance EOP of one month with BOP of the next

  2. Jim Reply

    A friend of mine submitted this comment to me privately and gave me permission to put it on the blog.
    “I think you’re talking about two issues here, Scope and Complexity. The first is Scope, in your bathroom analogy the scope of the project expanded beyond the bathroom into a whole wing – that’s scope creep. The second issue is complexity, you might want an outhouse, but that doesn’t conform to code. Or a simple toilet and sink with quick and dirty – do it yourself – piping might “work” for the cottage but would not pass building inspection or meet the local building code.
    Recall the cartoon in the front of Riding the Tiger – the swing problem – what the customer wants is the tire on a rope. What gets delivered is a tree supported between two ladders with the trunk cut out so the swing can pass between the tree’s crown and its stump.
    When other parties get involved, or technologies and standards are introduced or imposed, and when conformance to those standards becomes mandatory, then things start to get too complex. In order to control the complexity we introduce levels of management to make sure things don’t get out of hand.
    I was in Venice, watching children walk from the Grand Canal to their school. I saw them play soccer in a square that was a short distance from the smaller canals. If it were in Canada every kid would have been forced to wear a lifejacket in case they fell into the canal.
    As humans we easily conceptualize solutions in our imagination – vision. Execution to achieve that vision may require the introduction of complexity because the technology demands it (imposed limitations) or the technologists demand it (personal agendas). Keeping the first to make the solution work, while avoiding the second, is the only way I can think of to achieve the vision with minimal complexity.”

  3. Jim Reply

    Thank you for your comment. I think your analysis is excellent. I think your point about being well meaning is really true in my experience. Underlying the issue is often trust or confidence.
    We get caught in our silos and our need for perfection and our perceptions can hurt our ability to make it simple. “Not my job.”
    I suspect another factor is at play which I find difficult to describe. Adding complexity may be a mechanism to avoid making decisions. “It is more complex than that.”
    Good luck on your job search.

  4. Mike Reply

    In searching for a job, I found your interesting blog… I share your concern over why seemingly simple BI wishes (gee, if only we could…) turn into monster projects. In my opinion, the business thinks of BI as magic and IT doesn’t really do much to dispel that notion. Well meaning, both embark upon projects that begin death spirals once the complexity or turn-around timing is shared… often times each party has a deaf ear for the others’ concerns/needs and the lobbying for their own goals start to supersede reality. Business clients who take the time to truly understand the key elements of BI production, and, IT projects that really dig into finding out the essential business output(s)are the positional factors that make for mutual respect and trust. Once each side can appreciate the others efforts, problems of complexity and the like are substantially diminished in a collaborative and co-operative atmosphere… now, trying to find and establish those types of relationships – may be the real complexity…
    You really touched upon a subject that literally could take up a day long conversation and yet, like you intimate, seems to be pretty straightforward at the outset.

Leave a Reply to Mike Cancel reply

captcha *