Numerical stability testing within series patch releases

[mgxd]

What would you like to see added in fMRIPrep?

From the NiPreps release documentation:

If a subject may be successfully run on 20.0.n, then the imaging derivatives should be identical if rerun with 20.0.(n+1), modulo rounding errors and the effects of nondeterministic algorithms. The changes between successful runs of 20.0.n and 20.0.(n+1) should not be larger than the changes between two successful runs of 20.0.n.

However, there is no definitive testing on this yet - one idea would be to incorporate the work done in https://arxiv.org/pdf/2307.01373 (cc @yohanchatelain) into our CI pipeline, as a workflow that is only triggered on release branches/tags.

Do you have any interest in helping implement the feature?

Yes, but I would need guidance

Additional information / screenshots

No response

[yohanchatelain]

Hi @mgxd,

I think it would be a good idea to integrate the work that have been done in fMRIPrep.
The main idea is to sample the numerical variability of a representative dataset on a X.0.0 LTS version and then do the stability test (compare output from the new version to the reference's variability) for each new minor release.

To do that, the main steps would be:

• Sample the numerical variability. Use the build_fuzzy_libmath_dockerfile.sh script to build the fuzzy version of fMRIPrep.
• Store the T1 image. Store the T1 images produced by fuzzy somewhere, to be able to download them during the CI step.
• Do the stability test. Use the stability test python package to perform the test. This would produce the fail/pass test results for all the configurations (smoothing kernel, confidence level). I have to add documentation for that module because it's empty right now but the usage is pretty simple and example can be found in the repo's paper.
• Produce a report. Extract the scripts from the paper's notebook to visualize the fail/pass stability test results.