What is the value of data in an Enterprise Datawarehouse (EDW)? It’s the Business Intelligence an organization gain leverage from the EDW. But all this falls short if we can not get the data into the EDW fast, from source and correctly. Data latency and quality are reflected in the EDW, but these issues live in the source systems (but I digress).
Over the past we have leverage Extract Transform and Load (ETL) as it was the way we had available to get the data from source to EDW. These are a separate (middle) tier in our EDW architecture. Typically transformations are done row by row – this has been effective, though very challenging method to be able to transform our data into the EDW so that people can start to use it.
Lately, with the emergence and stability of Extract Load and Transform, a new option as presented itself. Let’s get the data into the EDW and do the Transform in the EDW and have the existing RDBMS engines in the EDW that are usually pretty hard wired for data do the transformation. This leverages the native code in the EDW to do the manipulations in bulk mode.
There is generally a 20-30% savings on the ELT platform versus ETL, but if you already have an ETL platform how does this fit in.
Over the next couple of weeks I will identify some cases that we have piloted that show how this may be a more productive method of getting the data into the EDW and in the form needed – thus saving time and money in development and support.
Lastly this also follows the trend from Oracle and IBM where they are integrating their datamarts into the EDW to bring all the data closer, with less lifting and placing, lifting and placing.
