-
Notifications
You must be signed in to change notification settings - Fork 429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP][FIX] option to disable multithreading while multiprocessing #2547
base: master
Are you sure you want to change the base?
Conversation
Hello @skoudoro, Thank you for updating ! Cheers ! There are no PEP8 issues in this Pull Request. 🍻 Comment last updated at 2022-03-04 04:02:02 UTC |
Codecov Report
@@ Coverage Diff @@
## master #2547 +/- ##
==========================================
- Coverage 85.02% 85.02% -0.01%
==========================================
Files 127 127
Lines 17426 17445 +19
Branches 2963 2967 +4
==========================================
+ Hits 14817 14832 +15
Misses 1936 1936
- Partials 673 677 +4
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder whether it wouldn't be more elegant to implement this as an object that stores current_openblas
and current_mkl
as state, rather than storing those in environment variables.
current_openblas = os.environ.get('OPENBLAS_NUM_THREADS', '') | ||
current_mkl = os.environ.get('MKL_NUM_THREADS', '') | ||
|
||
# import ipdb; ipdb.set_trace() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove debugging statement
|
||
|
||
def disable_np_threads(): | ||
"""Reduce OPENBLAS and MKL thread numbers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you want to make the number of threads an argument to this function? Right now, this always reduces to 1, but maybe sometimes we want to reduce down to number of cpus?
Not a bad idea. I will look into it. Also, I will turn this into a work in progress. It needs testing and exploration. Thank you for the feedback! |
1419292
to
ca6268a
Compare
5935e1e
to
765963e
Compare
This PR fixes #2519. It permits to avoid thread oversubscription. In the close future, we need to improve the design and insert it in the parallel module. Small Notes: the
loky
backend fromjoblib
solve this issue. This fix might be useful for the otherjoblib
backends