Websocket retry for disconnects

[shibbas]

We have seen the nteract kernel websocket disconnect randomly pretty often. I saw that there used to be a websocket retry implemented in the past and jupyter does support kernel websocket message buffering for random disconnects. It looks like this might have been reverted because the websocket retry never terminated in some cases.

I tried the following and it seems to have worked a bit better for terminating the retry. Wanted to get your opinion and feedback before contributing:

• Move the logic for websocket retry to launchKernelWebsocketEpic instead of rx-jupyter. This should provide better control over when the retry should be terminated. I.e. letting retryWhen continue until KILL_KERNEL_SUCCESSFUL or tracking the retry attempt count in the redux store to be more deterministic.
• If the websocket disconnects as soon as the kernel is launched, it looks like the first message is lost. What if LAUNCH_KERNEL_SUCCESSFUL is only raised after a successful websocket message is sent back from the kernel? Ideally, this would be done via the heartbeat message and not kernel_info_request because those will be queued up on a busy kernel.

Thoughts?

[shibbas]

Thanks for the meeting @willingc, @CodeSeal. Would you be able to share the info on heartbeat messages? I couldn't find any good reference on how this was sent over the websocket specifically.

[MSeal]

Hmm so looking into it the heartbeat channel is available to the server and Python clients, but it's not exposed on the websocket channel afaik now after looking through server code bases. @rgbkrk might know if there's a different route to check for kernel existence when the kernel is in busy state?

[shibbas]

Yeah, I couldn't find it in the Jupyter source and I also tried a few tests on the kernel channel but didn't see any good example. An alternate approach in the lack of heartbeat messages I was thinking would be to just hold off sending any messages to the kernel until the websocket responds. LAUNCH_KERNEL_SUCCESSFUL can still happen when the API returns, but after that the execution messages can just be buffered on the client side until any message comes back from the kernel, which would probably be a status message or kernel_info_reply after the kernel becomes idle again.

Heartbeat messages would be better though if those can be used.

[rgbkrk]

I think that we never implemented the heartbeat messages because at the time we thought they were going to be phased out (of the protocol).