Orphaned quantities from create_quantity migrate plugin when validation fails

[mstenta]

I discovered a bug while working on a custom CSV importer (following the guide at https://farmos.org/development/module/csv/) using the create_quantity plugin provided by the quantity module in farmOS: https://github.com/farmOS/farmOS/blob/17b792aca77fe757e7f7b18a95f8e5aedae1e192/modules/core/quantity/src/Plugin/migrate/process/CreateQuantity.php

When validate: true is used in the migration's destination config, and an imported entity's validation fails, the quantity entity is still created, resulting in orphaned quantity entities.

I don't think this is really create_quantity's fault. The same would be true of entity's created by the migrate_plus module's entity_generate plugin I believe. It's the nature of how those plugins work. It's also possible to end up with orphaned taxonomy terms, for example, if you use entity_generate to create a term and validation fails. But at least with terms you can go in manually and clean them up in the UI.

There is no way to clean up orphaned quantities in farmOS.

[mstenta]

There is no way to clean up orphaned quantities in farmOS.

Maybe this is the real issue that should be solved.

Our current View of quantities joins in log information, and provides a link to the log so you can get to the entity. If a quantity is not attached to a log, it still appears in the list of quantities, but there is no link to any log that you can use to delete them.

It's also possible to create orphaned entities via the API. In fact, it's necessary to create quantity entities before you reference them from a log entity. If anything goes wrong during that process, and the log isn't created, you're left with orphans.

One option might be to add the ability to delete quantities directly from Records > Quantities in the main menu. But that's a bit complicated because we also need to clean up the reference on the log (for quantities that are not orphaned).

And that doesn't really deal with the fact that someone who imports a CSV file will have no way of knowing that these orphaned quantities created. Nor should they. They shouldn't have to clean them up.

[mstenta]

Two ideas:

1. Maybe we can hide orphaned quantities from the quantity Views. That would at least prevent them from showing up. It won't hide them from the API though.
2. Maybe we should add a hook_cron() that periodically checks for orphaned entities (that are older than a certain date perhaps) and deletes them.

Option 2 would work, and would cover ALL cases of orphaned entities. It feels a little risky to be deleting anything via cron, though... but if we are careful with the logic it would probably be OK.

[mstenta]

Maybe we should add a hook_cron() that periodically checks for orphaned entities (that are older than a certain date perhaps) and deletes them.

Also need to consider how this would work with quantity revisions. A log can have multiple revisions to it's quantity reference field, all stored in the log_revision__quantity table. Each record in that table references a quantity by id and revision_id.

So we would need to make sure we don't delete a quantity that's referenced by a previous log revision.