8 – Migrating Translated User Entities slowing migration down significantly

I have a very large site I’m attempting to migrate from D7 to D8 but am experiencing significant performance issues. We have around 35,000 users, so I don’t expect this to be fast, but if I don’t make any changes the migration will take several weeks to complete which is obviously not an option considering we can’t shut down our site for multiple weeks… Here’s my experience so far

I ran a test migration of the site via the UI which ended up stalling out when processing the d7_entity_reference_translation:user__user migration. I ended up having to use drush to finish out the migration via drush migrate-import.

After cleaning everything up post-migration (took about a week), we decided that it would be better to just keep this instance up and running and only re-migrate users and their data before going live considering that’s the only data that will have changed since the initial migration. Specifically, we plan on running these user-data related migrations using drush: d7_user, d7_entity_reference_translation:user__user, d7_google_analytics_user_settings, and d7_legal_accepted.

  • The d7_user migration took about 14 hours, not great, but it was consistent and worked.
  • The d7_entity_reference_translation:user__user translation is where we ran into performance issues. During the first hour, it processed about 3000 users (on par with the rate d7_user migrated. However, it’s slowly become slower and slower. Currently, I am sitting at the 24 hour mark and it’s only processed 25K users and in the last 3.5 hours it only processed 2000 users. Therefore, it started at a rate of about 3000/hr, and after 24 hours it’s down to only about 500/hr.

I’ve attempted the following and seen some improvements, but nothing significant:

  • Increased the max_execution_time in php.ini settings to around 600 seconds instead of 60 (little difference)
  • Increased the memory_limit up from 128M to a few GBs (note, this instance is isolated to just this task at the time, so it’s okay for us to use more memory). This seemed to work a little, but obviously not enough.

What other adjustments can be made or tips anyone might have to mitigate this performance issue?