POC for new log streaming API using a zero-mq broker

[alexAubin]

The problem

• webadmin/moulinette: We should really have a more robust way of streaming what's going on the server issues#2055
• the Yunohost webadmin should be robust against scenario where the current websocket gets disconnected OR the user reloads the tab for some reason
• it should be easy to know what's going on on the server (in terms of YunoHost command/action) even when the action was started from the CLI
• now that there can be multiple YunoHost admins, it would be nice if any action/log was broadcasted to all opened webadmin tabs in real-time

Solution

• Completely rework the old websocket-based mechanism in Moulinette + corresponding clumsy job queuing (which created a necessity for a session ID etc)
• Rely on Server-Sent Event (SSE) instead : https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• The most complex part is being able to stream logs from CLI-triggered stuff to the web clients connected to the SSE endpoint. For this, we need a message broker. With the help of @Axolotle , we ended up finding ZeroMQ which is a pretty simple, flexible and powerful lib, and it allows to create various kind of messaging topologies.
• In our case, we have :
  • multiple "publishers" of logs (even though usually only 1 is running at a single time), which are the actual python-code API-triggered actions or CLI-triggered actions
  • multiple "subscribers" of logs, which are the N webadmin tabs currently opened and connected to the SSE endpoint
• This can be implemented solved using the XSUB/XPUB pattern with a central broker described in https://netmq.readthedocs.io/fr/latest/xpub-xsub/ with a zero-mq python implementation shown here
• In our case, I tweaked the OperationLogger mechanism to add a new logging Handler such that, during operations (= the actual "modifying" operations, not just any yunohost foobar list), a new PUB is created and streams logs to the broker, which forwards them to the SSE SUBscribers :

PR Status

Still mostly a POC

How to test

• You may need to pip3 install pyzmq (if on bullseye) or apt install python3-zeromq (if on bookworm)
• In one terminal : stop the yunohost-api server and manually start yunohost-api --debug
• In another terminal, run curl http://127.0.0.1:6787/sse
• In another terminal, run a modifying command, such as yunohost user create <username> or yunohost user delete <username>
• In the SSE terminal, you should observe a stream of base64-encoded jsons such as :

data: eyJ0eXBlIjogInN0YXJ0IiwgInRpbWVzdGFtcCI6IDE2ODQ1MTU1NjAuMzc0NDYyLCAib3BlcmF0aW9uX2lkIjogIjIwMjMwNTE5LTE2NTkyMC11c2VyX2NyZWF0ZS1jYW1pbGxlIn0=

data: eyJ0eXBlIjogIm1zZyIsICJ0aW1lc3RhbXAiOiAxNjg0NTE1NTY0LjIzMjE5NjMsICJsZXZlbCI6ICJTVUNDRVNTIiwgIm1zZyI6ICJVc2VyIGNyZWF0ZWQiLCAib3BlcmF0aW9uX2lkIjogIjIwMjMwNTE5LTE2NTkyMC11c2VyX2NyZWF0ZS1jYW1pbGxlIn0=

data: eyJ0eXBlIjogImVuZCIsICJzdWNjZXNzIjogdHJ1ZSwgImVycm9ybXNnIjogbnVsbCwgInRpbWVzdGFtcCI6IDE2ODQ1MTU1NjQuMjMzMzQ3LCAib3BlcmF0aW9uX2lkIjogIjIwMjMwNTE5LTE2NTkyMC11c2VyX2NyZWF0ZS1jYW1pbGxlIn0=

which translates to :

{"type": "start", "timestamp": 1684515560.374462, "operation_id": "20230519-165920-user_create-camille"}
{"type": "msg", "timestamp": 1684515564.2321963, "level": "SUCCESS", "msg": "User created", "operation_id": "20230519-165920-user_create-camille"}
{"type": "end", "success": true, "errormsg": null, "timestamp": 1684515564.233347, "operation_id": "20230519-165920-user_create-camille"}