qudi does not work in up-to-date python 3.8 conda environment

[tobiasgehring]

What is affected by this bug?

Windows 10, current qudi version from github, anaconda with up-to-date python 3.8 environment

When does this occur?

Starting the manager gui module

Where on the platform does it happen?

Manager gui, ipython console

How do we replicate the issue?

Start qudi

Expected behavior (i.e. solution)

The console should be started

Other Comments

Python 3.8.2 (default, Mar 25 2020, 08:56:29) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.13.0
Python 3.8.2 (default, Mar 25 2020, 08:56:29) [MSC v.1916 64 bit (AMD64)] on win32
runfile('C:/Users/qpitlab/Desktop/software/qudi/start.py', wdir='C:/Users/qpitlab/Desktop/software/qudi')
Used Qt API: PyQt5
Used Qt API: PyQt5
Loading Qudi...
C:\Users\qpitlab\Desktop\software\qudi\core\__main__.py:134: DeprecationWarning: zmq.eventloop.ioloop is deprecated in pyzmq 17. pyzmq now works with default tornado and asyncio eventloops.
  ioloop.install()
============= Starting Manager configuration from C:/Users/qpitlab/Desktop/software/CVQKD/qudi-modules/config/qkd_fmc120_rx/qkd_fmc120_rx.cfg =================
============= Manager configuration complete =================
Remote connection not secured! Use a certificate!
Error during activation
Traceback (most recent call last):
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\traitlets\traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'iopub_socket'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\traitlets\traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'iopub_thread'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\qpitlab\Desktop\software\qudi\core\module.py", line 96, in wrap_event
    base_event(*args, **kwargs)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 311, in fn
    self.transition()
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 306, in _tran
    self._after_event(e)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 338, in _after_event
    return getattr(self, fnname)(e)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 95, in _callback
    return func(obj, *args, **kwargs)
  File "C:\Users\qpitlab\Desktop\software\qudi\core\module.py", line 216, in __load_status_vars_activate
    self.on_activate()
  File "C:\Users\qpitlab\Desktop\software\qudi\gui\manager\managergui.py", line 162, in on_activate
    self.startIPython()
  File "C:\Users\qpitlab\Desktop\software\qudi\gui\manager\managergui.py", line 282, in startIPython
    self.kernel_manager.start_kernel()
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\inprocess\manager.py", line 46, in start_kernel
    self.kernel = InProcessKernel(parent=self, session=self.session)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\inprocess\ipkernel.py", line 72, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\ipkernel.py", line 71, in __init__
    self.shell.displayhook.pub_socket = self.iopub_socket
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\traitlets\traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\traitlets\traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\inprocess\ipkernel.py", line 67, in _default_iopub_socket
    return self.iopub_thread.background_socket
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\traitlets\traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\traitlets\traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\inprocess\ipkernel.py", line 59, in _default_iopub_thread
    thread = IOPubThread(self._underlying_iopub_socket)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\iostream.py", line 71, in __init__
    self._setup_event_pipe()
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\ipykernel\iostream.py", line 90, in _setup_event_pipe
    self._event_puller = ZMQStream(pipe_in, self.io_loop)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\zmq\eventloop\zmqstream.py", line 127, in __init__
    self._init_io_state()
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\zmq\eventloop\zmqstream.py", line 546, in _init_io_state
    self.io_loop.add_handler(self.socket, self._handle_events, self.io_loop.READ)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\tornado\platform\asyncio.py", line 100, in add_handler
    self.asyncio_loop.add_reader(fd, self._handle_events, fd, IOLoop.READ)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\asyncio\events.py", line 501, in add_reader
    raise NotImplementedError
NotImplementedError
Error during deactivation
Traceback (most recent call last):
  File "C:\Users\qpitlab\Desktop\software\qudi\core\module.py", line 96, in wrap_event
    base_event(*args, **kwargs)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 311, in fn
    self.transition()
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 306, in _tran
    self._after_event(e)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 338, in _after_event
    return getattr(self, fnname)(e)
  File "C:\Users\qpitlab\AppData\Local\conda\conda\envs\qudi-3.8\lib\site-packages\fysom\__init__.py", line 95, in _callback
    return func(obj, *args, **kwargs)
  File "C:\Users\qpitlab\Desktop\software\qudi\core\module.py", line 226, in __save_status_vars_deactivate
    raise e
  File "C:\Users\qpitlab\Desktop\software\qudi\core\module.py", line 224, in __save_status_vars_deactivate
    self.on_deactivate()
  File "C:\Users\qpitlab\Desktop\software\qudi\gui\manager\managergui.py", line 192, in on_deactivate
    self.stopIPythonWidget()
  File "C:\Users\qpitlab\Desktop\software\qudi\gui\manager\managergui.py", line 345, in stopIPythonWidget
    self._mw.consolewidget.kernel_client.stop_channels()
AttributeError: 'NoneType' object has no attribute 'stop_channels'

[kay-jahnke]

Hi Tobias,
I can confirm this problem. The most current python branch which is working form qudi would be 3.7.x (in my case 3.7.7 works).

We have also encountered this problem when we tried extracting a qudi-core. As python 3.8 is doing some more radical changes and some of the packages have to be adopted we will (at least for the core) first get everything clean for python 3.7 and then tackle any change to python 3.8 or higher.

At least that is how I see it. @drogenlied @Neverhorst what do you think?

[Neverhorst]

Yes, I agree. Some packages do not work the same way as before with Python 3.8.
You also seem to have installed the latest available version of qudi dependencies which is usually not working due to some breaking changes that need to be addressed and thoroughly tested on all supported platforms before elevating the package dependencies for qudi. This is the reason qudi specifies its dependencies with a specific package version (== instead of >=).
Qudi only tries to guarantee intended working conditions if you stick to the supported Python and package versions. Everything else would be not properly maintainable with reasonable effort.
Every deviation from the supported Python environment is at the users responsibility. However I would be happy if someone would step forward and figure out new dependencies including thorough testing (!) on multiple platforms.

Since this is explicitly not supported by qudi at the moment and does not cripple qudi functionality it should not be considered a bug.
I would change the tag to "help wanted" in case someone feels like tackling this task to elevate all package dependencies and Python to the latest versions.
But maybe that should wait until qudi core has been released as package.

[tobiasgehring]

On linux the installation of ipykernel version 5.2.1 solves the issue. Windows is to be seen.

[tobiasgehring]

adding
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
to __main__.py just before ioloop.install() fixes the issue.
I'm not sure what on what conditions it has to be called and what effect it actually has. This would need to be investigated.

Since ioloop.install() is deprecated we may want to fix this as well.