Speaking at PASS Summit 2018

PASS Summit 2018I am honored to have been selected to be a presenter at this year’s PASS Summit coming up this November. I will be presenting a half-day talk entitled Build a Metadata-Driven ETL Repository with Biml and SSIS:

Your boss tells you that a new field will be added to one of your vendor’s data files, and asks how long it will take to update the ETL to support this change. You start thinking through all of the pieces: the source packages, staging tables, stored procedures, and business rules. It could take weeks just to add a single column!

Managing metadata is one of the most important and time-consuming challenges an ETL developer will face. Simple changes can translate into many hours of discovery, code changes, and testing. However, there are ways to more easily manage this using tools you already have access to, which will ease the burden of SSIS package creation and maintenance. In this demo-packed session, we will discuss ETL metadata management using SSIS and Biml. We will review the challenges posed by “metadata guessing”, and will show a more predictable and robust way by building an end-to-end metadata repository to manage data types, sources and destinations, and transformations. 

Using Biml to automate the process of ETL development and management is one of my favorite topics to teach, so I am very much looking forward to this session. Since we get a double time slot (2.5 hours) for this topic, we’ll get to dive deeply into metadata management patterns in Biml. I hope to see you there!

About the Author

Tim Mitchell
Tim Mitchell is a data architect and consultant who specializes in getting rid of data pain points. Need help with data warehousing, ETL, reporting, or training? If so, contact Tim for a no-obligation 30-minute chat.

4 Comments on "Speaking at PASS Summit 2018"

  1. I agree that adding a column is not a trivial task. But I can’t imagine weeks being the answer.

    • Ron, thanks for the feedback. Making a metadata change isn’t always a weeks-long process, but it can be. Simple, single-source, single-destination ETL processes wouldn’t require that kind of time investment. But for complex, multi-step ETL processes with dozens of dependencies, the time required to inventory the scope of change, build a change script, and implement the change will be significant. Using automation in tools such as Biml can significantly shorten that cycle.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.