Tim Mitchell
Follow Tim Mitchell on Twitter  Like Tim Mitchell on Facebook  Subscribe on YouTube  Connect on LinkedIn  Subscribe to the Data Geek Newsletter

SSIS

MULTIFLATFILE

Using the SSIS Multiple Flat Files Connection Manager

When building an ETL pipeline to import data from a text file, it’s very common to have the incoming data spread across multiple files. For example, if you are ingesting files generated on a periodic basis (per day, per hour, etc.), you could have dozens or hundreds of files with identical structure. This is an ideal setup for building a…


FileNameColumnName

Extract the File Name in SSIS Data Flows using the FileNameColumnName Property

When extracting data from a flat file, it is handy to have the name of the file from which the data was retrieved. Whether you capture this information for auditing purposes, or you include it directly in the output table, the odds are good that you’ll want to have that filename for use later in the process. You could do…


Temp Tables in SSIS

Temp Tables in SSIS

Temp tables are very handy when you have the need to store and manipulate an interim result set during ETL or other data processing operations. However, if you use SQL Server Integration Services as your ETL tool, you may find some challenges when trying to work with temp tables in SSIS packages, especially in the SSIS data flow. In this…


Training Day in London: Building Better SSIS Packages

I am delighted to announce that I’ll be delivering my full-day course, Building Better SSIS Packages, at the SQLBits conference in London on Thursday, February 22nd. This course is aimed at data professionals with experience in SQL Server and a general understanding of SQL Server Integration Services concepts. This will be my second time to present at SQLBits, and my…


PASS Summit 2017

PASS Summit 2017 Day 2

It’s day 2 of the PASS Summit. Yesterday was a whirlwind of a day, after my early-morning post-Halloween trip to Seattle. I spent some time with my Microsoft friends in the SQL Clinic and visited several vendor receptions, followed by a quick visit to the karaoke party. Today is the Cosmos DB keynote by Rimma Nehme. I’ve heard Dr. Nehme…


Script Component with Multiple Outputs

Using the SSIS Script Component With Multiple Outputs

One of the more common questions I find in SQL Server SSIS forums is how to split apart a single text file into multiple outputs. Files extracted from foreign sources often arrive in nonstandard formats, and are too often unchangeable at the source and must be parsed during the import process. For unconventional custom transformations such as this, using the…


Upcoming Full-Day SSIS Class in Dallas

This year I’ve delivered my full-day Building Better SSIS Packages course a half dozen times across the country, but I hadn’t scheduled this class in the Dallas area this year. As of today, that has changed! I’m happy to announce that I’ll be delivering this class at the Microsoft office in Las Colinas next month, on Friday, August 25th. Registration…


Staging Load with Biml

Create a Staging Load with Biml

When designing the ETL architecture for new or changing systems, one of the more common needs is to create a process that copies all of the data from a set of tables (perhaps even all tables) from a source system to a destination database. This is especially prevalent when building an operation data store (ODS), or building a set of…


SSIS in an Azure VM

Running SSIS in an Azure VM

In the previous post in this series, I addressed how to use an on-premises instance of SSIS to move data to and from Azure databases. If you’re running in a pure Azure environment without on-prem SQL Server, that load architecture would present some challenges. However, by running SSIS in an Azure VM, you can communicate move data into or out…


Creating the SSIS Catalog

Creating the SSIS Catalog

If you are building SQL Server Integration Services (SSIS) packages, using the SSIS catalog as a deployment target is usually the easiest and most efficient solution. However, the SSIS catalog is not created by default, even when you select the SSIS components during the SQL Server installation process. The good news is that creating the SSIS catalog is a quick…