Multiprocessing archipelago.push_back issue

[AlexH726]

Hi all,
I am encountering an issue with the pygmo archipelago pushback command. I have gotten it to run successfully with this code but more often than not when the code hits the pushback lines I get to following error with the multiprocessing library. Any help on fixing this or workarounds would be greatly appreciated. The applicable code is included below as well as the traceback. it is running in a folder called 1runANN_f but when this error pops up it creates another folder by the same name in the original and copies most of the contents of the old folder into the new one.
Best,
Alex

Code:
algos=[pg.algorithm(pg.nsga2(gen=5, cr=0.95, eta_c=10, m=0.01, eta_m=10)), pg.algorithm(pg.de(gen=5))]
prob=pg.problem(seqopt(deb, in_prop[20], seed))
archi = pg.archipelago(topology=pg.fully_connected())
archi.push_back(pg.island(algo=algos[0],prob=prob,size=in_prop[20]*2))
archi.push_back(pg.island(algo=algos[1],prob=prob,size=in_prop[20]*2))

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\users\ahickey7\documents\dissertation\runa_n.py", line 13, in
os.mkdir(path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: '1runANN_f/.'
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\users\ahickey7\documents\dissertation\runa_n.py", line 13, in
os.mkdir(path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: '1runANN_f/.'
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\users\ahickey7\documents\dissertation\runa_n.py", line 13, in
os.mkdir(path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: '1runANN_f/.'
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\users\ahickey7\documents\dissertation\runa_n.py", line 13, in
os.mkdir(path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: '1runANN_f/.'
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\users\ahickey7\documents\dissertation\runa_n.py", line 13, in
os.mkdir(path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: '1runANN_f/.'

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\ahickey7\Anaconda3\envs\DLoc\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\users\ahickey7\documents\dissertation\runa_n.py", line 13, in
os.mkdir(path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: '1runANN_f/.'

[AlexH726]

I should also mention the computer is running Windows 10 and python 3.10

[bluescarni]

Hi @AlexH726 , it looks like the error is not arising from within pygmo code but rather in runa_n.py. From the look of it, you are doing some multiprocessing the in the implementation of your optimisation problem and Python seems to be complaining that multiple processes are attempting to create a file/directory with the same name at the same time.

I would suggest to try using the tempfile module if you need to read/write data from/to disk in your optimisation problem:

https://docs.python.org/3/library/tempfile.html

[AlexH726]

Hi @bluescarni all I am doing is creating a new file, copying in all the necessary files to run, and a target list, then changing the directory and running from that new file. What is interesting is that I have been able to run a number of native algorithms like de, sga, and aco in addition to some user-defined algorithms without problem. When I try to do this with the asynchronous island model, it seems to set off some sort of perpetual loop periodically when it hits __original_island_init(self, *args) (line 388 in init.py) line in pygmo. It does not seem to matter if I use a temporary directory or a permanent one. If I do let it all run in the file with the original function and just copy in the target list instead of copying everything into a new file I still get an error associated with the multiprocessing library:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "c:\users\alexa\documents\github\dissertation\func3\runa_n.py", line 32, in
wrapper(i)
File "C:\Users\alexa\Documents\GitHub\Dissertation\func3\Wrapper.py", line 125, in wrapper
archi.push_back(pg.island(algo=algos[0],prob=prob,size=in_prop[20]*2))
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\site-packages\pygmo_init.py", line 388, in _island_init
__original_island_init(self, *args)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\site-packages\pygmo_py_islands.py", line 119, in init
self._init(use_pool)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\site-packages\pygmo_py_islands.py", line 130, in _init
mp_island.init_pool()
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\site-packages\pygmo_py_islands.py", line 338, in init_pool
mp_island._init_pool_impl(processes)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\site-packages\pygmo_py_islands.py", line 315, in _init_pool_impl
mp_island._pool, mp_island._pool_size = _make_pool(processes)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\site-packages\pygmo_mp_utils.py", line 62, in _make_pool
pool = mp_ctx.Pool(processes=processes)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\pool.py", line 212, in init
self._repopulate_pool()
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static
w.start()
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\alexa\anaconda3\envs\DLoc\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

[bluescarni]

@AlexH726 does importing your code result in the creation of a multiprocessing pool?

[bluescarni]

@AlexH726 to expand a bit, a multiprocessing pool should never be created as a result of importing a submodule, as that leads to an endless recursion as explained by the error message. See also here:

https://stackoverflow.com/questions/18204782/runtimeerror-on-windows-trying-python-multiprocessing

Note that creating a pygmo mp island indirectly triggers the creation of a multiprocessing pool, so you should never have island-creating code executed as part of an import statement.