BUG: Segfault in scipy/sparse/linalg/_eigen/arpack

[AgilentGCMS]

Describe your issue.

I am installing scipy from source on an HPC on top of python 3.11.6. The code builds without errors, but scipy.test() segfaults at scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py. We have intel MKL on the HPC, so I built scipy with

CC=icx CXX=icpx FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp

Reproducing Code Example

python dev.py test -s sparse

Error message

ninja: Entering directory `/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build'
[4/4] Generating scipy/generate-version with a custom command
Build OK
💻  meson install -C build --only-changed
Installing, see meson-install.log...
Installation OK
SciPy from development installed path at: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages
Running tests for scipy version:1.13.0, installed at:/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0
configfile: pytest.ini
plugins: hypothesis-6.102.4
collected 11638 items / 61 deselected / 11577 selected

scipy/sparse/csgraph/tests/test_connected_components.py ........                                                                                                             [  0%]
scipy/sparse/csgraph/tests/test_conversions.py ...                                                                                                                           [  0%]
scipy/sparse/csgraph/tests/test_flow.py ...........................................                                                                                          [  0%]
scipy/sparse/csgraph/tests/test_graph_laplacian.py ......................................................................................................................... [  1%]
............................................................................................................................................................................ [  2%]
............................................................................................................................................................................ [  4%]
............................................................................................................................................................................ [  5%]
............................................................................................................................................................................ [  7%]
............................................................................................................................................................................ [  8%]
.................................................................................................................................................                            [ 10%]
scipy/sparse/csgraph/tests/test_matching.py ................................                                                                                                 [ 10%]
scipy/sparse/csgraph/tests/test_pydata_sparse.py ssssssssssssssssssssssssssssssssss                                                                                          [ 10%]
scipy/sparse/csgraph/tests/test_reordering.py ...                                                                                                                            [ 10%]
scipy/sparse/csgraph/tests/test_shortest_path.py ...................................                                                                                         [ 11%]
scipy/sparse/csgraph/tests/test_spanning_tree.py .                                                                                                                           [ 11%]
scipy/sparse/csgraph/tests/test_traversal.py ........                                                                                                                        [ 11%]
scipy/sparse/linalg/_dsolve/tests/test_linsolve.py .s.s.s.s.ss..........s.....................                                                                               [ 11%]
scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py .Fatal Python error: Segmentation fault

Current thread 0x0000146b7118b740 (most recent call first):
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 883 in extract
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1357 in eigs
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1577 in eigsh
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 248 in eval_evec
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 425 in test_hermitian_modes
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/_lib/_testutils.py", line 106 in __call__
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 745 in scipy_tests
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 761 in run
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/action.py", line 461 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/task.py", line 473 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 176 in execute_task
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 220 in run_tasks
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 254 in run_all
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_run.py", line 265 in _execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 570 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 150 in parse_execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/api.py", line 50 in run_tasks
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 251 in run_doit_task
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pydevtool/cli.py", line 267 in callback
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 783 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1434 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1688 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/rich_click/rich_command.py", line 126 in main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1157 in __call__
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 1486 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy._lib._fpumode, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2 (total: 50)
Segmentation fault (core dumped)

SciPy/NumPy/Python version and system information

Python 3.11.6 (main, May 15 2024, 11:36:54) [GCC Intel(R) C++ gcc 11.3.1 mode] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.13.0 1.26.4 sys.version_info(major=3, minor=11, micro=6, releaselevel='final', serial=0)
/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
  warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "11.3.1",
      "commands": "icx"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.10",
      "commands": "cython"
    },
    "c++": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "11.3.1",
      "commands": "icpx"
    },
    "fortran": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "2023.1.0",
      "commands": "ifx"
    },
    "pythran": {
      "version": "0.14.0",
      "include directory": "../../../python/3.11.6/lib/python3.11/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "cross-compiled": true
  },
  "Build Dependencies": {
    "blas": {
      "name": "mkl-dynamic-lp64-iomp",
      "found": true,
      "version": "2023.1",
      "detection method": "pkgconfig",
      "include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
      "lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
      "openblas configuration": "unknown",
      "pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
    },
    "lapack": {
      "name": "mkl-dynamic-lp64-iomp",
      "found": true,
      "version": "2023.1",
      "detection method": "pkgconfig",
      "include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
      "lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
      "openblas configuration": "unknown",
      "pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.12.0",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/work2/noaa/co2/sbasu/packages/python/3.11.6/bin/python3.11",
    "version": "3.11"
  }
}

[rgommers]

Thanks for the report @AgilentGCMS. That ARPACK code has had recurring issues with some BLAS functions (cdotc & co.) that are sensitive to Fortran compiler ABI conventions. MKL is built with the opposite convention as the default one (gfortran).

Maybe this particular crash was always an issue with MKL from OneAPI, or maybe it changed with #20232 (that was included in 1.13.0, which you're testing here). Do you know if this is a regression from 1.12.x or not for your setup?

[cgohlke]

I can reproduce this on Windows with scipy-1.13.1 and OneAPI MKL 2024.1.