Multiprocessing behavior differs across Ubuntu/MacOS

[anilbey]

Context

Overview of the issue

Neuron global parameters are passed to sub processes on Ubuntu.
Neuron global parameters are not passed to sub processes on MacOS.

Expected result/behavior

The behavior should not differ between operating systems.

NEURON setup

NEURON 8.2.4

Minimal working example - MWE

MWE that can be used for reproducing the issue and testing. A couple of examples:

• python script:

import neuron
from multiprocessing.pool import Pool
import os

def f(_):
    print(f" inside the process: {neuron.h.celsius}")
    print(f" process id: {os.getpid()}")


if __name__ == '__main__':
    print("the main process")
    print(neuron.h.celsius)
    neuron.h.celsius = 37.0
    print(neuron.h.celsius)

    with Pool(5) as p:
        p.map(f, range(5))

Prints the following on Ubuntu

❯ python reproducer.py
the main process
6.3
37.0
celsius is: 37.0
 process id: 22858
celsius is: 37.0
 process id: 22859
celsius is: 37.0
 process id: 22860
celsius is: 37.0
 process id: 22861
celsius is: 37.0
 process id: 22862

Prints this on MacOS

> python reproducer.py
the main process
6.3
37.0
celsius is: 6.3
 process id: 4712
celsius is: 6.3
 process id: 4712
celsius is: 6.3
 process id: 4712
celsius is: 6.3
 process id: 4712
celsius is: 6.3
 process id: 4712

The issue applies to all global parameters.

[anilbey]

cc @ilkilic @darshanmandge @stefanoantonel

[pramodk]

@anilbey : could you check this #2094 (comment) ? Just from the title, without looking into details, I wonder if it's multiprocessing module itself and not neuron.

[anilbey]

Yes it is the same issue. Thanks for pointing to it. It is indeed the difference of the multiprocessing module between the platforms.

However, this subtle but critical implementation difference in the behavior is very difficult for the users of NEURON to spot. When not considered, the simulation results become corrupted. Can we not make a decision between the fork mode / spawn mode at the NEURON level and hide this detail (and responsibility) from the Python users?

[pramodk]

thanks for confirming, @anilbey!

As neuron is loaded into Python as a C/C++ library and has a global state, I don't have clarity into how such a thing could be ("easily") achieved on the NEURON side.

Tagging @ferdonline and @1uc if they have any thoughts/ideas.

[1uc]

I don't immediately see a quick fix. In a library, e.g. bluepyopt, you can use get_context:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
(half way down the section).

Likely something close to:

import multiprocessing

# ...

    ctx = multiprocessing.get_context("fork")
    with ctx.Pool(5) as p:
        p.map(f, range(5))

[anilbey]

Thanks for the information.

And also the maxtasksperchild=1, argument has to be set in the pool running NEURON to make sure one run does not cause side-effect on the others.
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool