Digital Privacy is the Wild West

Who has the legal right to access your personal and private digital assets? The answer can be complex, and will depend on where you live, where you are traveling from and to, and whether or not you’ve been suspected of a crime. The rules governing personal digital access are evolving rapidly, and are often subject to creative, on-the-fly interpretations. Digital…


Managing Bad Data in ETL

In the last post in my ongoing series about ETL best practices, I discussed the importance of error handling in ETL processes, reviewing best practices for application flow to prevent or gracefully recover from a systematic error or data anomaly. In this post, I’ll dig a bit further into that topic to explore the design patterns for managing bad data in ETL…


Retrieve A List Of Files From FTP Using SSIS

Retrieve a List of Files from FTP using SSIS

The FTP protocol is one of the oldest methods for sharing and moving files. Although frequently considered to be an “old-school” way to transfer data, FTP is still relied upon in most every data movement architecture. Sadly, the functionality around FTP is very limited in SQL Server Integration Services. A common project requirement is to retrieve a list of files from…


SQL Saturday Pensacola

Upcoming Class: Building Better SSIS Packages in Pensacola

I’ll be teaching my popular day-long course, Building Better SSIS Packages, on Thursday, June 1 in Pensacola, Florida. This class is being delivered in conjunction with SQL Saturday Pensacola. Registration is open to the public, and early-bird pricing is in effect until early May. Let me know if you have any questions about this class. I look forward to seeing…


Why Data Warehouse Projects Fail

Why Data Warehouse Projects Fail

Data warehouse projects are among the most visible and expensive initiatives an organization can undertake. Sadly, they are also among the most likely to fail. At one time, Gartner reported that more than 50% of data warehouses would fail to make it to user acceptance. Because of the size of investment (both time and money) required, the success of such…


What is ETL

What is ETL?

In my ongoing series on ETL Best Practices, I am illustrating a collection of extract-transform-load design patterns that have proven to be highly effective. In the interest of comprehensive coverage on the topic, I am adding to the list an introductory prequel to address the fundamental question: What is ETL? What Is ETL? ETL is shorthand for the extraction, transformation,…


Side Project

The Side Project

When I first started working as a technologist some 17 years ago, I was doing fairly low-level stuff: moving and reimaging computers, network cabling, and basic desktop support were the extent of my responsibilities. I decided early on that I wanted to move into a more advanced role, but faced a common quandary: I couldn’t get the job I wanted…


ETL Error Handling

ETL Error Handling

In designing a proper ETL architecture, there are two key questions that must be answered. The first is, “What should this process do?” Defining the data start and end points, transformations, filtering, and other steps must be done before any other work can proceed. The second question that must be answered is “What should happen when the process fails?” Too…


It's a Lonely Road

Survey Results: Skipping the Holidays

Last week I posted a brief survey about working the holiday. To those who normally are not required to work the holidays, I asked if they would be willing to do so if the price was right, and if so, what that price would be (relative to what they normally make). There were 46 responses to this survey, and I’ve…


It's a Lonely Road

Survey: Skipping the Holidays?

For a little pre-holiday fun, I’ve got a simple scenario to present to you, and I am interested in your response. Your holiday plans are made. All of your family and friends expect to see you at the annual holiday gathering. Your kids are expecting you there on Christmas morning as they open presents. You unexpectedly get a call from…