A couple of months ago I was fortunate enough to be invited to a small group session run by Stephen Brobst, CTO of Teradata. It was an interesting group of senior business and technology folks interested in joining in a dialogue around the concept of extreme data warehousing. Of course I fell in love with the idea as it was a great word and it had a strong X in the beginning. So this is the start of a group of posts in this area based on my observations and thoughts from this session.
So let’s start off as to what is meant by the term – extreme data warehousing. To me this is about the business and application issues and opportunties that can be addressed through the future vision of the possible. This is not to mean that what we are talking about is not possible. In every example the components of the solution are available today. It is about how the leverage of some of the trends in data warehousing can make these a reality from a cost and business point of view. The trends that are driving this are things like
- more data, moving faster
- new uses for data (new generation of information management)
- Strategic, Tactical and Operational Intelligence
- Storage evolution > 1Tb storage on one drive is available, BUT, Storage Increase does not equal increase in performance capacity
- Massively Parallel Processing is increasing with the evolution of blade computing (grid)
- Virtual not vertical organizations – Nike
- Shift towards event based analytics
- Real-time Enterprise (driving process change – Walmart / Dell)
- Global Data Warehousing
- Enterprise technology both maturing and morphing into new immature but possibly valuable components (SOA)
- Customers expect more. More of everything, better pricing, better selection, better service. Both in and outside the organization.
So what are we talking about. What is extreme data warehousing? Well it is or can be…
- extreme performance – response in milliseconds
- extreme scalability
- meeting service oriented architecture demands,
- millions of users
- > petabytes of data – that’s a lot of info
- extreme availability – up all the time – eliminate planned and unplanned downtime
- extreme data freshness – as we move to active, service levels guarantee the freshness
- extreme comprehensiveness – if it’s not there then you don’t need it
- extreme integration – exploit all business relationships in the data
- extreme granularity
- atomic data becomes sub-atomic
- Call detail
records are no longer the detail, network packets may be once you move
to VoIP or IPtv.
- extreme business – speed to answering questions accelerates
- extreme agility – adding, integrating and delivering data is at the speed of business, not IT cycle time
So why do it?
If done, and integrated into your business processes this offers many
rewards both in revenue and cost control. We will use the examples to
follow to try and illustrate this.
What’s Stopping us?
- Business Justification – this is always hard no matter what you are
doing. With the number of project failures, this shift in mentality
becomes difficult. We can certainly help with that. - Technical feasability – the technology exists, but can it be done cost effectively for the return on investment.
- Organizational Politics
- who owns
- controls it (governance)
- gets to use it
- changing the way we approach problems and processes
- Political issues – Privacy, consumer perception
Funnily enough technical feasability can be overcome, but the real issues are
- economics
- politics
- organizational change and mobility
- the creation and acceptance of the burning platform
