Divergence of the number of shards in the metastore vs as seen from the control plane

[fulmicoton]

As observed on airmail

select count (*) AS shards from shards;
 shards 
--------
 763471
(1 row)

While the control plane sees a much lower shard count (~3000s)

[fulmicoton]

This is probably not the cause, but this causes a divergence between metastore and CP

        if let Err(control_plane_error) = self
            .auto_create_indexes(&request.subrequests, ctx.progress())
            .await
        {
            return Ok(Err(control_plane_error));
        }

[fulmicoton]

I suspect the problem is that we

- open shard on metastore
- init shard on ingest
- only update the model for successfully initialized shards

So if an ingester happens to be unreachable etc., we end up with some dangling shard in the metastore.

[fulmicoton]

scale_up shards seems buggy too.

   let open_shards_response = match progress
            .protect_future(self.metastore.open_shards(open_shards_request))
            .await
        {
            Ok(open_shards_response) => open_shards_response,
            Err(error) => {
                warn!("failed to scale up number of shards: {error}");
                model.release_scaling_permits(&source_uid, ScalingMode::Up, NUM_PERMITS);
                return;
            }
        };

[fulmicoton]

Split into #5013 #5014 #5020