Support for creating python sub-interpreters

[glopesdev]

One of the most exciting features of Python 3.12 is the possibility of creating sub-interpreters programmatically, each with an independent per-interpreter lock. This feature relies on a new C API function Py_NewInterpreterFromConfig.

Needless to say, this feels like one of the most powerful combinations to expose to the managed world via Python.NET, but I was wondering how the current design with static classes would be able to accommodate this. Specifically, I was concerned with the following API comment from the Bugs and caveats section of per-interpreter GILs:

Also note that combining this functionality with PyGILState_* APIs is delicate, because these APIs assume a bijection between Python thread states and OS-level threads, an assumption broken by the presence of sub-interpreters. It is highly recommended that you don’t switch sub-interpreters between a pair of matching PyGILState_Ensure() and PyGILState_Release() calls. Furthermore, extensions (such as ctypes) using these APIs to allow calling of Python code from non-Python created threads will probably be broken when using sub-interpreters.

It feels that perhaps initial support for PyGILState_Ensure (and therefore Py.GIL()) will not work with sub-interpreters. I couldn't find a section in the manual specifically addressing this, but this SO question provides some interesting insights on what a solution may look like.

If the expectation is to avoid the use of Py.GIL altogether and simply restrict operations to individual threads, perhaps the static class design might work by simply exposing a new method to create a sub-interpreter and let users manage their own threads and use libraries compatible with the new lock system.

Are there any thoughts on supporting the Py_NewInterpreter* APIs in the near future? I don't feel like support for Python 3.12 will really be complete without them.