-
-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: interpolate: fix high memory usage for 2 classes #20404
BUG: interpolate: fix high memory usage for 2 classes #20404
Conversation
In MacOS, psutil seems to be returning wildly inconsistent values for the memory used in the test I made: -210 Mb and 141 Mb. For context, in my Linux machine, I always get between 40 and 60 MB. I have decided to increase the tolerance to 512 MB which should still showcase the benefits of this bug fix - before the fix, in my machine, this test would try to allocate an array of size 90 GB. |
As a quick note, I believe it's better to add an ASV memory benchmark (https://github.com/scipy/scipy/tree/main/benchmarks) than a unit test. |
Done. Did not realize that ASV also had memory benchmarks |
207f4bb
to
2988926
Compare
…rresponding unit test
d1b6ad9
to
ec461f1
Compare
I am getting an error in the Linux - 32 bit check in |
Yes, that is showing on other PRs too |
@ev-br, when you have the chance, can you take a look at my pull request? |
LinearNDInterpolator
and CloughTocher2DInterpolator
(#20357)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry about the delay @Matryoskas .
This PR looks reasonable. I checked that the main interface with qhull in interpnd.pyx
is basically equivalent to what it was before #18376 which caused the memory regression.
I think we should merge this for 1.14 when the CI comes back green.
... and CI is green, and so is this PR. Thanks @Matryoskas for the patch and @Kai-Striega for the review. |
Reference issue
Closes #20357
What does this implement/fix?
When using the LinearNDInterpolator and CloughTocher2DInterpolator classes, instead of keeping the data about the location and simplicies of all the input points in the memory at once, only keep the data about a single simplex at once, greatly reducing the memory usage with larger inputs