Prevent parts of VCF being reparsed

[benjeffery]

Fix sgkit-dev/sgkit-publication#35 by telling dask that it can release a future for part of a VCF parse when it is complete. This prevents re-parsing when a worker is restarted.

[benjeffery]

@phofl Thanks for the chat about this issue yesterday. I tried using a simple client.gather but this still resulted in the futures being redone on worker failure. However, after poking around the docs I realised cancelling the future after success would prevent its re-execution. Let me know what you think of this approach. Thanks again!

[benjeffery]

There has been some discussion at dask/dask#10654 although no suggestions that meet the requirement of not redoing the work AND having a sensible progress bar. I suggest for now we use the fix with the confusing progress bar, as although it doesn't fill up, it does at least show the amount to-do dropping.

[benjeffery]

I've looked at the other format parsing methods and they both (bgen and plink) load the data into cluster memory, instead of writing parts to disk as the VCF parser does. Therefore this issue doesn't apply - you workers have to last until you save the dataset.

[benjeffery]

Looks like failing tests as sometimes we don't have a dask client object - will work around.

[jeromekelleher]

I've looked at the other format parsing methods and they both (bgen and plink) load the data into cluster memory, instead of writing parts to disk as the VCF parser does. Therefore this issue doesn't apply - you workers have to last until you save the dataset.

Can we close #1152 then?

[benjeffery]

I think so - will comment there.

[benjeffery]

I'm getting failures here due to missing coverage of one line 1250. But I can't see that I've changed anything that impacted that line?

[benjeffery]

@tomwhite The line that isn't covered here is the last line in:

        if region is None:
            variants = vcf
        else:
            variants = vcf(region)

I don't think I've changed anything to uncover this line - do you know why this might not be covered?

[tomwhite]

@benjeffery I can't see what could have caused this. To debug you could print a stacktrace for the unaltered code to see which code path causes that line to run (when region is not None), which may give a clue to what's happening.