Earlier this week, I blogged about the automatic cleanup process that that will clean up the SSIS catalog logging tables. This nightly process removes data for operations that are older than 365 days. While this is useful, many SSIS admins have complained that this process is very slow and contentious on large or busy SSISDB databases. In this post, I’ll…
SQL Server Integration Services is one of the most popular extract-transform-load (ETL) tools on the market. Learning how to use SSIS to move and transform data is an essential skill for every data professional, whether you’re a DBA, SQL Server developer, data analyst, or architect. Classroom Training: Introduction to SSIS This 3-day hands-on course will teach you what you need…
In the previous post in this series, I addressed how to use an on-premises instance of SSIS to move data to and from Azure databases. If you’re running in a pure Azure environment without on-prem SQL Server, that load architecture would present some challenges. However, by running SSIS in an Azure VM, you can communicate move data into or out…
Among the new features of SSIS 2016, one of my favorite is incremental package deployment. This new functionality allows the deployment of a single package to an SSIS catalog without having to deploy the entire project. Since the release of this version, I’ve been asked the following question a few times: “How does one go about deleting a package from…
Making the most of the SSIS catalog requires an understanding of how to access the information stored in the logging tables. Although there are built-in reports to show this information, there are limitations in their use. Fortunately, the logging tables in the SSIS catalog database are (mostly) straightforward and easy to understand once you’ve worked with them a bit. In…
The FTP protocol is one of the oldest methods for sharing and moving files. Although frequently considered to be an “old-school” way to transfer data, FTP is still relied upon in most every data movement architecture. Sadly, the functionality around FTP is very limited in SQL Server Integration Services. A common project requirement is to retrieve a list of files from…
In my ongoing series on ETL best practices, I recently wrote about the importance of logging in extract-transform-load processes. For users of later versions (2012 and beyond) of SQL Server Integration Services, adding logging to those ETL processes is very simple. Since logging is managed by the catalog, adding logging to an ETL process is a one-step process and requires…
In my last post I shared how to build a package in SSIS to perform simple file archive (zip) operations. In this post, I’ll show you how to go the other direction and unzip files with SSIS. Unzip Files with SSIS Similar to the file zip operation, this solution uses the following tools: 7Zip: This free utility manages the creation…
Of the very many things SQL Server Integration Services does well, one of its shortcomings is that it does not have a built-in way to create compressed archive (ZIP) files. Generating such files is a common need in ETL processes, so it’s surprising that the fifth generation of SSIS still does not have a native task to manage this operation….
Built into the SQL Server Integration Services catalog is the ability to run a validation without actually executing the package. Running a package validation in the SSIS catalog performs a high-level check against the underlying metadata to check for common points of failure (especially those related to data flows). SSIS package validation is not designed to capture every metadata issue,…