Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: stats.skewnorm.ppf returns wrong values with moderately high skew parameters #20124

Closed
maresb opened this issue Feb 21, 2024 · 2 comments · Fixed by #20643
Closed

BUG: stats.skewnorm.ppf returns wrong values with moderately high skew parameters #20124

maresb opened this issue Feb 21, 2024 · 2 comments · Fixed by #20643
Labels
defect A clear bug or issue that prevents SciPy from being installed or used as expected scipy.stats upstream bug Items related to bugs in upstream projects
Milestone
1.14.0

Comments

@maresb
Copy link
Contributor

maresb commented Feb 21, 2024

Describe your issue.

I'm trying to compute the inverse CDF (i.e. PPF) of several skewnorm distributions. I am getting nonsense answers when the skew parameter is moderately large.

Reproducing Code Example

from scipy.stats import halfnorm, skewnorm


#  This is the correct ppf for a standard skewnorm with skew 500
q = 0.012533469508013

#  As confirmation of the above result, the skewnorm with skew 500 is
#  extremely close to a halfnorm distribution.
#  In fact, the ppf of the halfnorm agrees with q to 13 decimal places.
halfnorm.ppf(0.01, 0, 1)
#  0.012533469508069276

#  This should theoretically be 0.01, and it is very close.
skewnorm.cdf(q, 500, 0, 1)
#  0.009999999999999898

#  This should theoretically be q, but it's not at all.
#  This is the problem.
skewnorm.ppf(0.01, 500, 0, 1)
#  4.833181656374989e+142

Error message

/opt/conda/lib/python3.11/site-packages/scipy/stats/_continuous_distns.py:9141: RuntimeWarning: overflow encountered in _skewnorm_ppf
  return _boost._skewnorm_ppf(x, 0, 1, a)

SciPy/NumPy/Python version and system information

1.12.0 1.26.3 sys.version_info(major=3, minor=11, micro=7, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /opt/conda/include
    lib directory: /opt/conda/lib
    name: blas
    openblas configuration: unknown
    pc file directory: /opt/conda/lib/pkgconfig
    version: 3.9.0
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /opt/conda/include
    lib directory: /opt/conda/lib
    name: lapack
    openblas configuration: unknown
    pc file directory: /opt/conda/lib/pkgconfig
    version: 3.9.0
  pybind11:
    detection method: pkgconfig
    include directory: /opt/conda/include
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    args: -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -march=nocona, -mtune=haswell,
      -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections,
      -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -DNDEBUG, -D_FORTIFY_SOURCE=2,
      -O2, -isystem, /opt/conda/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem,
      /opt/conda/include
    commands: /home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/_build_env/bin/x86_64-conda-linux-gnu-cc
    linker: ld.bfd
    linker args: -Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now,
      -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,/opt/conda/lib,
      -Wl,-rpath-link,/opt/conda/lib, -L/opt/conda/lib, -Wl,-O2, -Wl,--sort-common,
      -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now, -Wl,--disable-new-dtags, -Wl,--gc-sections,
      -Wl,--allow-shlib-undefined, -Wl,-rpath,/opt/conda/lib, -Wl,-rpath-link,/opt/conda/lib,
      -L/opt/conda/lib, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -march=nocona, -mtune=haswell,
      -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections,
      -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -DNDEBUG, -D_FORTIFY_SOURCE=2,
      -O2, -isystem, /opt/conda/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem,
      /opt/conda/include
    name: gcc
    version: 12.3.0
  c++:
    args: -fvisibility-inlines-hidden, -fmessage-length=0, -march=nocona, -mtune=haswell,
      -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections,
      -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -fvisibility-inlines-hidden,
      -fmessage-length=0, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC,
      -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem,
      /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -DNDEBUG, -D_FORTIFY_SOURCE=2,
      -O2, -isystem, /opt/conda/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem,
      /opt/conda/include
    commands: /home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/_build_env/bin/x86_64-conda-linux-gnu-c++
    linker: ld.bfd
    linker args: -Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now,
      -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,/opt/conda/lib,
      -Wl,-rpath-link,/opt/conda/lib, -L/opt/conda/lib, -Wl,-O2, -Wl,--sort-common,
      -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now, -Wl,--disable-new-dtags, -Wl,--gc-sections,
      -Wl,--allow-shlib-undefined, -Wl,-rpath,/opt/conda/lib, -Wl,-rpath-link,/opt/conda/lib,
      -L/opt/conda/lib, -fvisibility-inlines-hidden, -fmessage-length=0, -march=nocona,
      -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt,
      -O2, -ffunction-sections, -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -fvisibility-inlines-hidden,
      -fmessage-length=0, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC,
      -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem,
      /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix, -DNDEBUG, -D_FORTIFY_SOURCE=2,
      -O2, -isystem, /opt/conda/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem,
      /opt/conda/include
    name: gcc
    version: 12.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.7
  fortran:
    args: -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix
    commands: /home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/_build_env/bin/x86_64-conda-linux-gnu-gfortran
    linker: ld.bfd
    linker args: -Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now,
      -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,/opt/conda/lib,
      -Wl,-rpath-link,/opt/conda/lib, -L/opt/conda/lib, -Wl,-O2, -Wl,--sort-common,
      -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now, -Wl,--disable-new-dtags, -Wl,--gc-sections,
      -Wl,--allow-shlib-undefined, -Wl,-rpath,/opt/conda/lib, -Wl,-rpath-link,/opt/conda/lib,
      -L/opt/conda/lib, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /opt/conda/include, -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/scipy-split_1705882420314/work=/usr/local/src/conda/scipy-split-1.12.0,
      -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix
    name: gcc
    version: 12.3.0
  pythran:
    include directory: ../../_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.11/site-packages/pythran
    version: 0.15.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /opt/conda/bin/python
  version: '3.11'
@maresb maresb added the defect A clear bug or issue that prevents SciPy from being installed or used as expected label Feb 21, 2024
@maresb
Copy link
Contributor Author

maresb commented Feb 21, 2024

Additional information:

I have isolated the error to Boost. I found a reported regression which has been fixed in boostorg/math#1080. Thus I suspect that this can be solved by updating Boost.

from scipy.stats._boost.skewnorm_ufunc import _skewnorm_ppf

_skewnorm_ppf(0.01, 0, 1, 500)
#  4.833181656374989e+142

maresb added a commit to maresb/scipy that referenced this issue Feb 21, 2024
@maresb
TST: Add CDF-PPF roundtrip check for skewnorm
b905486 
This is a regression test for scipygh-20124.
@maresb maresb mentioned this issue Feb 21, 2024
Closed
maresb added a commit to maresb/scipy that referenced this issue Feb 21, 2024
@maresb
MAINT: Update boost_math submodule
9436f0e 
This grabs the PR boostorg/math#1080 that should fix scipygh-20124.
@maresb maresb mentioned this issue Feb 21, 2024
Closed
maresb added a commit to maresb/scipy that referenced this issue Feb 21, 2024
@maresb
MAINT: Update boost_math submodule
46dc1a7 
This grabs the PR boostorg/math#1080 that should fix scipygh-20124.
@lucascolley
Copy link
Member

x-ref gh-19348 - looks like we want to try updating to Boost 1.85 once it's out

@dschmitz89 dschmitz89 mentioned this issue Apr 17, 2024
Closed
@lucascolley lucascolley added the upstream bug Items related to bugs in upstream projects label Apr 28, 2024
@dschmitz89 dschmitz89 mentioned this issue May 4, 2024
Merged
@dschmitz89 dschmitz89 added this to the 1.14.0 milestone May 7, 2024
@tylerjereddy tylerjereddy modified the milestones: 1.14.0, 1.13.1 May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
defect A clear bug or issue that prevents SciPy from being installed or used as expected scipy.stats upstream bug Items related to bugs in upstream projects
Projects
None yet
Milestone
1.14.0
6 participants
@tylerjereddy @maresb @dschmitz89 @lucascolley @j-bowhay and others