Recently I’ve been talking a lot with clients and others about the involvement of cloud architecture in a data warehouse design. In fact, this topic was the focus of my most recent Data Geek newsletter publication.
A few weeks ago, I had the pleasure of being interviewed by my friend Carlos Chacon for the SQL Data Partners podcast, during which we talked about the future of the on-premises data warehouse. Take a half hour to listen, and let me know in the comments whether you agree or disagree with my assessment.
Tim – Agree with your thoughts 100%. We are in process of upgrading our on premise SQL 2014 Fast Track Data Warehouse (FTDW) to SQL Server 2016 FTDW. Until Azure can give us the same TPC-H performance on premise its a no go for us, with the caveat that we are enterprise grade and have the budge to purchase elite hardware.
Matt, thanks for the comment. I’d love to hear a little more about your setup if you’ve found that the scale-up capabilities available from cloud providers can’t keep up with your needs.
Tim, would be great if you could at some point blog about challenges of DW in the cloud, either Azure or AWS. My only experience with cloud are sources such as Dynamics CRM that we pull into our DW and that’s incredibly painful as you 1) can’t query the source database directly so you can’t reverse engineer physical model/relationships, 2) data loading and extraction is like watching paint dry except slower, 3) I need special adapters to make SSIS talk to them, 4) pricing is uncertain as its based on use through complex formulas that we cannot predict or budget for accurately. Cloud solutions throttle access to point where an Access database seems really fast AND we have to pay for use which applies pressure to not use them for things like testing. Bottom line is, I get that cloud makes tons of sense for small businesses with little data and lacking data centers and expert staff. But I must be missing something because cloud solutions for DW and in fact data integration in general look like they’re only adding costs, not taking them away. Our on-prem VM environment can spin up a new server for me in a day with a SqlServer image at no added cost and its rocking fast. The minute I tough my cloud Azure I have to check my credit limit. What’s wrong with this picture?
Hi Gary – that’s a great suggestion. I’m already working on some cloud material now for publication later this year.
Hi Gary, we use webservice to push data to data-warehouse. From there we run ETL to load, it’s quick.