drush config:import seems to not support batch operations in a module's hook_install()

[sonfd]

Describe the bug
I have two modules, module_alpha and module_beta.

Both modules install a database table with hook_schema() which they use to track information about published nodes. Data is logged in the table with service methods executed from hook_node_insert() and hook_node_update() hooks. This all works.

But, I need to process all existing nodes when the module is installed to backfill the custom tables with information about the already existing published nodes. Each module implements hook_install() and uses a batch process to process all existing published nodes. However, I am seeing strange inconsistencies. Sometimes the batch process executes, sometimes it does not. A scenario may work for one module, but not the other. All of this even though the code is nearly identical. Here are the scenarios I've tested:

Module install method	Module Alpha	Module Beta
drush en MYMODULE	Batch executes	Batch executes ~~
drush cim	Batch does not execute	Batch does not execute
UI, enable module from /admin/modules	Batch executes	Batch executes
UI, sync config from /admin/config/development/configuration	Batch executes	Batch executes

Batch executes indicates that the batch executes completely as expected, with progress messages, and my table is filled with data. Batch does not execute indicates that the batch is not executed at all. No error messages shown or in logs.

~~ Update: I've tested this further and the batch process does execute in this scenario, but for some reason an entity query is returning no results, only during drush operations and only when querying for a specific bundle.

What could be causing the batch process to work in some cases but not in others? How can I consistently execute this batch process across all 4 of these methods, whether the module is enabled via UI or drush? To be honest, the only method I'll use is drush cim

Code:

In MYMODULE.install:

function MYMODULE_install() {
  $batch = new \Drupal\Core\Batch\BatchBuilder();
  $batch
    ->setTitle(t('Bulk processing existing items.'))
    // This operation references a different function per module.
    // In both cases, the function lives in MYMODULE.module and the code is nearly identical.
    ->addOperation('_MYMODULE_initialize_items', []);

  batch_set($batch->toArray());
}

In MYMODULE.module:

function _MYMODULE_initialize_items(&$context) {
  $batch_size = 25;
  $entity_type = 'node';
  // I target different bundles in module_alpha and module_beta.
  $bundle = 'MY_TARGET_BUNDLE';

  if (!isset($context['sandbox']['processed'])) {
    $context['sandbox']['processed'] = 0;
  }

  if (empty($context['sandbox']['entity_ids'])) {
    $context['sandbox']['entity_ids'] = \Drupal::entityTypeManager()
      ->getStorage($entity_type)
      ->getQuery()
      ->condition('type', $bundle)
      ->condition('status', 1)
      ->execute();

    if (is_array($context['sandbox']['entity_ids'])) {
      $context['sandbox']['total'] = count($context['sandbox']['entity_ids']);
    }
  }

  if (!empty($context['sandbox']['entity_ids'])) {
    $current_batch_ids = array_slice($context['sandbox']['entity_ids'], $context['sandbox']['processed'], $batch_size);
    $current_batch_entities = \Drupal::entityTypeManager()
      ->getStorage($entity_type)
      ->loadMultiple($current_batch_ids);

    foreach ($current_batch_entities as $entity) {
      // I use a different services and methods in module_alpha and module_beta.
      \Drupal::service('MYMODULE.my_service')
        ->processEntity($entity);

      $context['sandbox']['processed']++;
    }
  }

  if (!empty($context['sandbox']['total'])) {
    $context['finished'] = $context['sandbox']['processed'] / $context['sandbox']['total'];
    $context['message'] = t('Processed @processed of @total items.', [
      '@processed' => $context['sandbox']['processed'],
      '@total' => $context['sandbox']['total'],
    ]);
  }
  else {
    $context['finished'] = 1;
  }
}

Example service method called in the batch process:

public function processEntity(EntityInterface $entity) {
  // $this->db is injected `@database` service, i.e. \Drupal::service('database');
  return $this->db
    ->insert('MYMODULE_mytable')
    ->fields([
      'entity_type' => $entity->getEntityTypeId(),
      'entity_bundle' => $entity->bundle(),
      'entity_uuid' => $entity->uuid(),
      // etc...
    ])
    ->execute();
}

System Configuration

Q	A
Drush version?	10.6.2
Drupal version?	9.3.6
PHP version	7.x (7.4)
OS?	Mac, using ddev / linux

Additional information
I'm wondering if batch operations are just not supported here. I see that in Drush\Drupal\Commands\config::doImport(), the code states that it copies \Drupal\Core\Config\ConfigImporter::import, but import() uses a batch, and doImport() circumvents the batch.

[sonfd]

Ok, when using drush en mymodule the batch in hook_install seems to process. I noticed the code from that command has:

if (batch_get()) {
  drush_backend_batch_process();
}

If I toss that into doImport() my batches execute.

[sonfd]

Created PR #5106 with this change.

[weitzman]

Lets discuss in PR #5106