In today's business environment, every critical function an organization undertakes is increasingly reliant on information that is accurate, timely, complete, and integrated.

The move by larger companies (and increasingly by SMEs) toward fully integrated systems, generically referred to as ERP software has raised the bar even higher for maintaining data integrity. With the implementation of an ERP system, a robust data migration process is central and critical to the project’s successful implementation. Every system and subsystem relies on quality data, and historical data is often critical to business continuity.

Most common data migration challenges and how to combat them

A successful data migration improves business operations substantially, from enhancing data accessibility to improving data integrity. Before we can explain how to combat data migration challenges, it's important to understand what they are and how they can affect your business. 

Most common challenges of data migration

Data migration involves the transferring of data between computer storage types or formats, making it a key consideration for any system implementation, upgrade, or consolidation. During the migration process, data needs to be extracted from the source systems, transformed and loaded into the target system. This can be (and usually is) a complex and challenging process for many reasons; the following are perhaps the most common:

Lack of involvement from management

Migrations are performed infrequently and companies are usually unprepared for the challenges that lie ahead. One of the most important and common problems with data migration is the  lack of involvement and participation by company management, thus degrading the importance of data migration projects. This is a task that requires collaboration and participation from all employees across the organization in order to ensure a successful outcome.  

An incomplete migration strategy

An incomplete or non-existent migration strategy; employees involved in data transfer should be very familiar with all the steps in the data migration process from start to finish. 

Data quality issues

Data quality issues that have accumulated over time, such as, dissimilar data structures for the same customer, incomplete or missing data, lack of legacy data standards. These problems are multiplied by the volume of data stored and used in organizations. 

Underestimating the cost

Underestimating the scale and cost of the data migration project; underestimating the scope and budget required for the project at the outset will almost certainly result in project overruns in both time and money (if not failure).

Strategy to address data migration challenges

The following four key areas should be addressed in order to satisfy the requirements of a fully integrated migration process:

Planning

Data migration is not the sexy part of the project. Consequently, data migration planning is often given less priority and seen as an administrative ‘burden’ instead of a necessary and essential component of the project.  A well thought-out strategy must be in place before attempting the data migration, with input from all the business stakeholders.

Given a broad enough scope, the data migration strategy can address issues of scope, timeline, and  resources, as well as itemizing the administrative steps to be covered in the migration.

Conceptualizing the data

All corporate data is a business asset. It belongs to the organization and is one of the most important ‘tools’ of the business. If data is conceptualized in this manner, the idea of engaging and achieving a robust commitment from business management committed to the success of this project should not be difficult. Participation from management will ensure a process that is empowered to make the necessary decisions that will drive the required actions.

Allocating proper time and resources

Proper allocation of resources and a timeline are critical aspects of the data migration project and contribute to the overall success or failure of the project. Refining and clearly defining the scope also makes it easier to determine the size of the budget and the eventual agreement from management.

Conducting an analysis of the source and target systems, in consultation with the business users (directly impacted by the data migration), will ensure a transfer process that is both fully functional and that has minimized the amount of data to be migrated. 

Addressing data integrity

Data integrity problems (as previously noted) begin as a failure to treat data as a strategic business resource. Data integrity requires the prioritizing of its maintenance and upkeep in a business strategy, with an appropriate budget allocation. This perspective should also be supported when dealing with data migration. A breakdown of data to-do’s is as follows:

• It is important to examine data and rationalize it prior to migration. This will determine the level of source information to be included in the migration. Historical data can be very costly to transfer and not always necessary. There are legal requirements which dictate the length of time some data must be retained (e.g. accounting data), but there is also obsolete data that can be eliminated.  
• Data validation, prior to migration, is a crucial part of data integrity. Transferring inaccurate data to the new system would compromise its efficiency and negate the value of the investment in money and time. Plans to correct and prevent ‘dirty’ data must be taken.
• Finally, the testing or verification stage of the migrated data is equally critical. A simple way to ensure the data migration was successful is to create a test sample on a test database - not the live system. A strategy of testing at various points of the installation can prevent storing up issues and dealing with them late in the cycle, making them more expensive and difficult to correct.

Final thoughts

A data migration is an integral part of any successful ERP implementation. As such, it must be given the priority that it deserves (at the planning stage) in order to fully and successfully participate in the overall successful ERP implementation.