Data migration

Moving the rows is the part you can script. What takes time is knowing that they all arrived, that they mean the same thing on the other side, and what you do with the records that could not be interpreted.

The exceptions set the timeline

Records that follow the data model are moved by the script. The rest are duplicates, fields used for something other than intended, the same person three times with different spellings, and leftovers from an earlier system change. How many there are, and in which categories, nobody knows until the trial run shows it. Each category needs a decision from someone in the business: merge, discard, correct or keep. Once the decision is made, the correction can be scripted. The decision cannot.

How a migration works

1. Mapping fields and meanings before anything moves. What is this field called, and what is it actually used for?
2. A trial run against a copy of live data, with the same access rules as the original. It produces the list of deviations.
3. Working through the deviations with the business, category by category.
4. Reconciliation on counts, totals and identities, backed by spot checks on content. The same number of rows and the same total are not enough: a missing customer and a duplicate cancel each other out. We reconcile record by record on the key, check the business rules, such as open orders per customer adding up, and check by hand that payment terms mean the same thing after the move.
5. A rollback plan agreed in advance, in case something turns out wrong after go-live. What can be rolled back, and for how long, depends on whether the new system has already taken in new data.

The trial run is the step that is tempting to cut when time runs short. Do not cut it. It produces the list of deviations, and without it you see them for the first time on go-live day, when there is no time left to decide anything.

If the migration is part of a system change or a legacy modernisation, we plan it together with that work. If you are moving into a new data model, it is worth getting it right while you are at it.