ETL

We don't trust this data

We Don’t Trust This Data

“Learning to trust is one of life’s most difficult tasks.” –  Isaac Watts As data professionals, there are times when our jobs are relatively easy. Back up the databases. Create the dashboard report. Move the data from flat files to the database. Create documentation. There are lots of cogs in those machines, but an experienced technologist will have little trouble…


How to get fired from an ETL developer job

How To Get Fired from an ETL Developer Job

Through the course of my 8-someodd years of building and fixing ETL processes, I’ve had the opportunity to see a lot of ETL code. Some of that code was really good, well-thought-out and carefully executed. Other load processes were – well, let’s just say that they provide plenty of consulting opportunities (and I include much of my early code in…


Fibers

The SSIS Object Variable and Multiple Result Sets

In my most recent post in this series, I talked about how to use the SSIS object variable as an ADO recordset as a source in a data flow. By loading the result set of a query into this variable, the contents of the variable can be read by an SSIS script component and sent out through the SSIS pipeline….


Internet cable concept

Using the SSIS Object Variable as a Data Flow Source

Object variables in SSIS are incredibly versatile, allowing the storage of almost any type of data (even .NET objects). In my last post on this topic, I demonstrated how an SSIS object variable containing a .NET DataSet object could be used by the for each loop container as an iterator. In this post, I’ll continue the discussion by showing how…


twoplustwo

Null, empty string, or zero?

The answer: It Depends. One of the more common problems I encounter when managing data quality, especially in an ETL process, is the handling of null values, blanks, and zeroes. When I put on my preaching shoes to talk about bad data, this is one the areas I have to spend a lot of time covering because it is so…


fire.jpg

How to burn down your house while frying a turkey

It’s an odd query, yes, but in preparation to write this post I actually typed the above phrase into my browser.  No, I’m certainly not looking to burn down my house.  In fact, wait here while I clear my search history, just in case. For the sake of argument, let’s say you’re planning to fry a turkey over the upcoming…


No Picture

Parent-Child SSIS Architecture

This is the first in a series of technical posts on using parent-child architectures in SQL Server Integration Services.  The index page for all posts can be found here. In this post, I will provide an overview of the architecture and describe the benefits of implementing a parent-child design pattern in SSIS structures. Definition The simplest definition of SSIS parent-child…