Allow modules to be (soft) purged in batches #8147
Conversation
CT Test Results0 tests 0 ✅ 0s ⏱️ Results for commit 50df1f7. ♻️ This comment has been updated with latest results. To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass. See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally. Artifacts// Erlang/OTP Github Action Bot |
Erlang/OTP 27 increased 4x the maximum number of processes.
As a consequence, erlang:processes/0 got four times slower.
When running code:purge/1 and code:soft_purge/1, it has to
traverses all processes. On an Erlang VM with a low number of
running processes, the cost of erlang:processes/0 outweighs
the costs of traversal, and consequently purging/soft purging
got 3-4x more expensive. This can be seen in this simple program:
application:ensure_all_started(compiler),
{ok, Mods} = application:get_key(compiler, modules),
lists:map(fun code:ensure_loaded/1, Mods),
[code:delete(M) || M <- Mods],
lists:sum([element(1, timer:tc(code, purge, [M])) || M <- Mods]).
Which went from 89ms with +P 250000 to 285ms with +P 1000000.
This pull request augments code:purge/1 and code:soft_purge/1
to allow a list of modules to be given, computing erlang:processes/0
only once per batch. Purging the compiler application is now done
within 76ms, faster than Erlang/OTP 26.
josevalim
force-pushed
the
jv-batch-purge
branch
from
4c72f74
to
50df1f7
Compare
|
Note |
|
Can you clarify if the slowdown is just due to increased process limit, or when that limit is taken advantage of - the increased number of processes is actually spawned before this purging operation. |
| case erts_internal:purge_module(Mod, prepare) of | ||
| false -> | ||
| {{false, false}, Reqs}; | ||
| do_purge(Mods, Processes, Reqs, WasOld, DidKill); |
There was a problem hiding this comment.
This is unsafe: erlang:processes/0 must be called after purge_module(_, prepare | prepare_on_load) for fun purging to work properly. Otherwise, a new process not part of the initial erlang:processes/0 might have landed in the to-be-purged code by calling a local fun.
To make this work, we need to change the emulator side of this to handle purging a list of modules. marking their funs as part of a purge so that calls to them can be caught regardless of whether they were part of our process list or not (and in either case, erlang:processes/0 must be called after those funs are marked).
There was a problem hiding this comment.
Oh, I see. Therefore the work necessary is much larger than I expected. I don't think I have the skill to take it all the way home, I am sorry. Feel free to close this.
The only remaining question is: should you revert the default +P until the purge changes are in place?
|
Thanks for the PR!
Yes, it looks like this PR will balloon a bit in size though because of a few lambda-related wrinkles. It's doubtful whether we can get it into 27, it may have to wait until 28.
Just due to the increased process limit. |
Erlang/OTP 27 increased 4x the maximum number of processes. As a consequence, erlang:processes/0 got four times slower.
When running code:purge/1 and code:soft_purge/1, it has to traverses all processes. On an Erlang VM with a low number of running processes, the cost of erlang:processes/0 outweighs the costs of traversal, and consequently purging/soft purging got 3-4x more expensive. This can be seen in this simple program:
Which went from 89ms with +P 250000 to 285ms with +P 1000000.
This pull request augments code:purge/1 and code:soft_purge/1 to allow a list of modules to be given, computing erlang:processes/0 only once per batch. Purging the compiler application is now done within 76ms, faster than
Erlang/OTP 26.