What is data migration, and why would a company consider such an undertaking?
As the world becomes more dependent on digital infrastructure, where and how companies store their data becomes even more important. But sometimes you need to make a change after the fact. Making that change safely and confidently requires planning and a process.
It requires data migration.
Let’s look at why data migration comes important for some companies as well as the typical steps involved in getting it done.
When does data migration become necessary?
Data gathering and integration becomes essential for most business practices at one point or another. But that also means any attempt to move data to another location – whether it’s a cloud provider, a data lake or data warehouse, or another repository – needs to be utterly seamless.
Data migration can mean different things in a different set of circumstances or for a different company. But it refers to the general framework for finding, auditing, preparing, converting (if necessary), and finally loading that data into its new home.
8 steps for successful data migration
Auditing and data mapping
Choose your timeline, method, and architecture
Test your migration process
Load the data
Perform another data audit
This isn’t as straightforward a process as you might think. But the need for data migration may come about due to any number of circumstances.
The company made an acquisition, or was acquired itself, and data from the newly acquired entity has to be ordered and consolidated into existing databases.
The company has chosen to upgrade or overhaul their digital infrastructure or move to a different provider. In some cases, data must be transformed (converted) to achieve compatibility with a different (or newer) application or operating system.
Sometimes a company may choose to establish a new data warehouse due to geographical or market considerations, to move processing closer to the point at which the data was gathered, or because they want to comply with (or avoid) data stewardship regulations like the EU’s General Data Protection Regulation.
Companies may also choose to partition their digital infrastructure if the company is forming separate divisions or otherwise altering its structure, and discrete file paths are necessary for each division.
No matter their reasons for embarking on this process, businesses need to know they can resume operations successfully afterward without:
Experiencing more downtime or inconvenience than is necessary
Losing any mission-critical data
Sustaining data duplication and redundancies
If you’re bringing older data into a new system, this is a prime opportunity to make a fresh and confident start. Having a plan in place for data migration is a chance to clean up and reorder your data without exporting existing issues into a new system.
Data migration only goes as smoothly as your prep work, and that means having a strategy is vital. The following steps should provide a general outline for companies and organizations contemplating a full-scale data migration project.
Before any data gets migrated into the new system, companies are advised to perform a complete data audit. Don’t confuse this with an optional step – it sets the foundation for everything that follows.
A data audit will help shine a light on questions like these:
Are there any potential data liability issues with the new system, location, or service provider, including holding personally identifying data on clients, partners or customers?
Is this data needed anymore? During a data audit, you can consider the usefulness of all of your digital clutter and jettison anything that hasn’t been useful in a while.
After the audit is complete and your decision-makers are satisfied with the results, it’s on to the next step.
2. Clean up
With the audit behind you, cleanup can begin. In some cases, cleanup might mean asking for outside help with getting your data compliant with existing, new, or potential upcoming data protection regulations.
In other cases, cleanup can be taken a bit more literally. According to one report, only around 3% of companies hold data that meets even the most basic standards for quality.
The scale of your cleanup job might require the aid of third-party data preparation tools. Reconciling older records with newer ones, such as making sure out-of-date phone numbers and addresses are discarded, and then transposing the result into the new database, is a task best left to the light touch of automation through workflow recipes.
3. Choose your timeline, method, and architecture
There are two main methods for embarking on a data migration project. We’ll discuss these in a moment. This step of the process is also where your organization should define the expected timeline for project completion and map the architecture of the new system.
One thing you might overlook without a professional (or professional-level data migration tool) on your side is the challenge of keeping your data protected while it’s being migrated. As you’re laying out your timeline and processes, it’s important to keep data integrity a top-of-mind concern.
“Integrity” here refers both to cybersecurity concerns as well as making sure there are workflows and redundancies in place to ensure data is duplicated safely before it’s moved, and that the redundancies are deleted, for cybersecurity’s sake, only after receiving confirmation of a successful transfer.
It’s fairly likely your company “runs on” data gathered from a variety of sources – including ecommerce websites, POS modules, social media and campaign reports, direct mail and email outreach, customer relationship management dashboards, and enterprise planning software.
That might mean your structured and unstructured data is in a format that isn’t immediately compatible with the architecture of your new system. A successful migration might require that your data undergoes formatting changes or other transformations before they can be used.
5. Test your migration process
Remember the saying, “measure twice, cut once”? During a data migration project, the maxim is “test twice, migrate once.”
This is an important step in the project. It’s your chance to make sure all of the required code is doing the job it’s supposed to be doing. Use a variety of data types as “dummies” to verify the completeness and fidelity of the data migration process. Look for fringe cases and outliers that might cause slowdowns, data corruption, or other problems once the system goes live.
As we said, there are two schools of thought on the methods of data migration. Each one has different ramifications for the business in question. These are: all-at-once data migrations and trickle data migrations.
All-at-once data migrations
This is where the entire data migration project is completed within a known and planned-for interval. Any live systems in use at the time will require temporary downtime as data is extracted, transformed, and loaded into the new system.
The advantage of going this route is that, if you’ve prepared well, the entire project happens within a known timeframe. It’s a one-time thing. But that’s also the primary downside: the “crunch” for the parties involved can be tough to work through, especially if the company also has to continue functioning in a reduced capacity.
Trickle data migrations
In some cases where companies need to avoid downtime, a trickle approach can be a better way to go. This is where the old data system and the new one continue operating in parallel. Your staff attached to the project can migrate your data piece by piece, as it’s processed, and without rendering any of your company’s digital assets unusable or your services unreachable.
These can be more complex migration projects, but in the right hands they can actually reduce rather than compound the run-of-the-mill risks that any data migration is susceptible to.
7. Load the data
If you’ve followed the above steps, you’re ready to flip the switch and implement your data migration protocols.
8. Perform another data audit
So we began, so the process ends. The data audit at the beginning of your migration project confirmed that you didn’t bring any issues with you from your previous data infrastructure. Capping the experience with a second data audit confirms you didn’t create any new ones in the process.
You're ready for your own data migration
Data is the present and future of business. And like it or not, that means we’ll probably always have a need to migrate data between storage or processing locations.
Now that you know what to expect, you’re better prepared than ever for the next time you have to migrate your all-important data to a new home.
Kayla Matthews is a technology journalist and professional development writer from Pittsburgh, PA.
Her work has been featured on WIRED, The Muse, Digital Trends, and Information Age. To reach more from Kayla, please visit her blog, Productivity Bytes.