The Hard Problem in Database Migrations
by Brandon Carver, on Oct 15, 2020 12:16:45 PM
I don't think anyone would question that 2020 has been a year of problems. I mean, 2019 had its problems, but this is definitely a level up. More of the "hard problems" than we're collectively used to. When I was 1, I imagine my problems were centered around worrying that my steps were too wobbly to make it across the room, and what was going to be for dinner. When I was 16, speed humps on dark roads were the bane of my existence, along with what was going to be for dinner. Even when I became an adult, my main concerns were centered around how do I get this data warehouse migrated to the cloud... and what was going to be for dinner.
One of the hard problems 2020 has caused is how to take your entire organization's workforce remote. (We've done it here, so now I'm less concerned with traffic every morning, and more concerned about the interruptions by household animals and children that are ever present on any video call I happen to be on. Though I've discovered deciding what's for dinner has remained a primary concern. I guess some things are universal, even in 2020.) If you're taking your workforce remote, Mobilize.Net has a lot of experience modernizing applications and taking them to the web. Getting a complete transformation is going to be far more beneficial in the long run than trying to lift and shift, and the same is true for any data warehouse. Taking your data warehouse from on-premise servers to the cloud is practically essential in a business world where everyone is working remotely.
As a result, the demand to get to the cloud is at an all time high, and Snowflake is uniquely positioned as the leader in this space. (I mean... did you see their IPO?) Our tools have been hard at work fast tracking data warehouses to Snowflake. Just in the past 6 months, we've seen over 240 million lines of code processed by Mobilize.Net's database migration products. So we've decided to partner up with Snowflake and share a few of the lessons learned in migrating workloads from Teradata to Snowflake. We co-hosted a webinar with Snowflake called "Fast Track Your Migration from Teradata to Snowflake" on Wednesday, October 21, 2020, at 10:00 am Pacific Time. (Fear not, if you missed it the recording will be up soon.) In this webinar, we shared how we implement solutions to the "hard problem" in a migration from Teradata to Snowflake.
So... what is the "hard problem"? If you're going to Fast Track your migration to Snowflake, you'll need to know where the speed humps are. Before we get there, we've already done a webinar that gives a high level view of the migration process, in case you missed it. Every SI has their own version of this, but it's all essentially the same. Here's an image from the webinar that gives an overview of the migration process:
There are a lot of steps along the way to a migration, but we like to think of any successful migration effectively managing the transition for the people who are stakeholders, the data itself, and the code that builds the warehouse and acts as a way for the people to interact with the data. There is obviously more than one pain point in a database migration, but not all of them will cause a migration to fail. There are a lot of solutions out there that can duplicate the data and have a functioning data warehouse in both Teradata and Snowflake. (Mobilize.Net recommends CONNX DataSync.) Getting all of the stakeholders on board is essential to a successful migration, but there are a lot of resources out there provided by Snowflake to educate a team on using Snowflake.
The Hard Problem
The hard problem is the code conversion. This is the big one. The number one reason a migration fails is when a migrator underestimates the complexity in migrating the code. The codebase for a data warehouse not only builds the warehouse, but it's how your people interact with the warehouse. A Teradata instance may have millions of lines of DDL, BTEQ, sProcs, and thousands of objects. Understanding how to rebuild that code and make it functionally equivalent on the other side comes with experience. And with the volume of code that is often present in a migration, a quality automation tool to move that process along is absolutely essential. That's why migrations that feature Mobilize.Net's SnowConvert do not fail. With a high automation rate and the ability to recreate functionally equivalent code for tables, views, stored procedures, macros, and bteq files, the hard problem of code conversion can be solved by Mobilize.Net's tools for migrating from Teradata to Snowflake.
You may be ready to move to Snowflake, but there's a good chance your Teradata data warehouse isn't broken. It still works, and when you make the migration, you want it to work in Snowflake as well. But Teradata SQL and Snowflake SQL are not the same, and there's certainly no BTEQ equivalent for Snowflake. Functional equivalence can be created by the tools that Mobilize.Net has developed in partnership with Snowflake.
We gave out many of the details on how it all works at our joint webinar with Snowflake (the recording of which will be available shortly). If you were able to make it, hopefully, it was worth your time. We be made free copies of the SnowConvert Assessment Tool Beta available to anyone who attended the webinar. We will be making that available as a wider release in the very near future. This will give you the chance to get a first look at how much code you have, and how much of it can be migrated automatically.
The hard problem of code conversion doesn't need to go unsolved. As the workforce adjusts to the new normal of working remotely, cloud-based data platforms are also settling in to be the new normal. Don't let the hard problem of code conversion keep you from getting your infrastructure to the cloud. This problem has a solution.
Now if only there was a solution to 2020...