v1.2.1 test_scalar_minimizers (test_nose.py) fails on i386

[drew-parsons]

Debian tests of lmfit-py 1.2.1 fail in test_scalar_minimizers (tests/test_nose.py) on i386. Test log here. Other tests pass, and all tests pass on other arches, full set of test logs here.

The error message is

 =================================== FAILURES ===================================
 __________________ CommonMinimizerTest.test_scalar_minimizers __________________
 
 self = <tests.test_nose.CommonMinimizerTest testMethod=test_scalar_minimizers>
 
     def test_scalar_minimizers(self):
         # test all the scalar minimizers
         for method in SCALAR_METHODS:
             if method in ['newton', 'dogleg', 'trust-ncg', 'cg', 'trust-exact',
                           'trust-krylov', 'trust-constr']:
                 continue
             self.minimizer = SCALAR_METHODS[method]
             if method == 'Nelder-Mead':
                 sig = 0.2
             else:
                 sig = 0.15
 >           self.scalar_minimizer(sig=sig)
 
 tests/test_nose.py:412: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 tests/test_nose.py:420: in scalar_minimizer
     check_wo_stderr(para, true_para.value, sig=sig)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 para = <Parameter 'amp', value=5.000000000006848, bounds=[5:20]>
 real_val = 14.0, sig = 0.15
 
     def check_wo_stderr(para, real_val, sig=0.1):
         err = abs(para.value - real_val)
 >       assert err < sig
 E       assert 8.999999999993152 < 0.15
 
 tests/test_nose.py:31: AssertionError

It's not simply a 32 bit issue. The test evidently passed on armel, armhf.

Has anyone got this test passing on i386? Perhaps it's an Issue that should be reported.

[newville]

@drew-parsons This sort of thing ("lmfit tests fail when run on system and/or architecture") has been brought up at various times in the past (#502, #686, #692, #802, and more recently #861, #879, and then the series of #880). My response is generally the same "shrug".

The thing is, we have a test suite and a CI system in place (https://dev.azure.com/lmfit/lmfit-py/) so that every PR and commit gets run on the systems, architectures, and software stacks that we choose to test. These tests tell us that the code is "working" and point out what changes cause problems.

The tests are designed to be run with our CI system. We literally do not test on other systems and architectures. You might say that we do not care about these.

The lmfit code is pure Python. It will not change between architectures or OSes: for a particular version on Python, all code using lmfit will generate identical Python bytecode. Lmfit does depend on compiled extension code from numpy and scipy that very well may vary between architectures and OSes. We cannot do anything about such differences. We test of a range of OSes to make sure that subtle numerical differences are not too problematic.

If you are only getting one failure on i386, I would say, "Great, it mostly works". Is this test failure a problem for you? Like, why are you running the lmfit CI test suite on a bunch of different architectures? What do you think that is going to reveal?

[drew-parsons]

It's CI testing for the Debian Linux Distribution, it's important to confirm the packages continue working successfully together.

I agree, it is odd that i386 alone is failing. Sometimes the CI tests reveal 32 bit issues, but then that affects all the 32 bit arches, not just i386. Perhaps it's related to pythran+xsimd, which is used by scipy.

[newville]

@drew-parsons Well, why are you using the lmfit test suite as part of your CI process for including lmfit in ubuntu?
Our tests are not testing whether packages continue to work successfully together. They test if lmfit works as we expect.

You can run our tests anywhere you want. We know they pass on the systems and environments we test with.

If you get a failure, that is telling you that there is a failure on your system/platform/architecture/choice of environment.
Why would you think that the failure you see is interesting to us?

Honestly, one failure out of 600+ tests of a 32-bit architecture seems pretty good to me. We do not support, endorse, or even suggest that a 32-bit architecture is at all useful or interesting for numerical analysis.