Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate:import --delete also rolls back skipped rows #5064

Open
chrisolof opened this issue Feb 4, 2022 · 0 comments
Open

migrate:import --delete also rolls back skipped rows #5064

chrisolof opened this issue Feb 4, 2022 · 0 comments

Comments

@chrisolof
Copy link

Describe the bug
When migrate:import's --delete flag initiates a rollback, all skipped records in the ID Map are deleted as well. Example:

 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) in 1.2 seconds (0/min) - done with 'foo'
 [notice] 2 items are missing from source and will be rolled back
 5827/5827 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%
 [notice] Rolled back 2721 items - done with 'foo'

Note how it says it will roll back 2 items, yet actually rolls back 2721.

That leaves the migration in an incomplete state with unprocessed items:

 ------------------ -------- ------- -------------- ------------- --------------- 
  Migration ID       Status   Total   Imported       Unprocessed   Last Imported  
 ------------------ -------- ------- -------------- ------------- --------------- 
  foo                Idle     5827    3108 (53.3%)   2719                         
 ------------------ -------- ------- -------------- ------------- ---------------

If I run this migration a second time, the skipped rows are added back into the ID map and I'm once again at 100% processed:

5827/5827 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%
 [notice] Processed 2719 items (0 created, 0 updated, 0 failed, 2719 ignored) in 23.6 seconds (6924.9/min) - done with 'foo'
------------------ -------- ------- -------------- ------------- --------------------- 
  Migration ID       Status   Total   Imported       Unprocessed   Last Imported        
 ------------------ -------- ------- -------------- ------------- --------------------- 
  foo                Idle     5827    3108 (53.3%)   0             2022-02-03 17:07:31  
 ------------------ -------- ------- -------------- ------------- ---------------------

Ideally only the records (and ID Map rows) now missing from the source would be deleted, leaving the unprocessed column at 0.

To Reproduce
Set up a migration that will skip some source rows. Import all source rows with it - note how many rows were skipped. Remove one or two rows from the source. Re-run this migration with the --delete flag. See that both the removed rows, as well as the previously skipped rows are rolled back. See that the migration now has unprocessed rows (via migrate:status).

Expected behavior
When migrate:import's --delete flag initiates a rollback, only records missing from the source are deleted. Previously-skipped, processed rows should be left alone unless they are actually missing from the source.

Actual behavior
Records missing from the source and any skipped, previously-processed, rows are rolled back.

Workaround
Run the migration twice. First run will leave it in an incomplete state with unprocessed rows. The second run will re-process the skipped rows, skipping them again but bringing them back into the map table. The migration will then have zero unprocessed rows and allow dependent migrations to run.

System Configuration

Q A
Drush version? 11.0.4
Drupal version? 9.3.3
PHP version 7.4
OS? Linux

Additional information
To get the delete flag working in 11.x I had to first apply the change in #5063.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants