Support bridge and manifests on different machines

[martinpitt]

Goal

We want to serve pages from the webserver (cockpit/ws container, bastion host,
flatpak), which is often not the target machine. The remote machine only gets
the bridge sent through beipack, but not the pages. This makes it difficult to
evaluate manifest conditions in the bridge, as manifests and bridge are not on
the same machine.

The main difficulty is to do this in a backwards-compatible manner: Connecting
from a RHEL 7 or Debian stable old Shell to latest bridge in e.g. Fedora 39
needs to work. Same for the opposite direction (new Shell to old bridge).

Status quo

The main interface between Shell and bridge is /manifests.json. It is currently
indexed by the resolved "name" (either directory basename or name: explicitly
given in the manifest).

For e.g. c-ostree it looks like this:

    "updates": {
        "name": "updates",
        "requires": {
            "cockpit": "125"
        },
        "conditions": [
            {
                "path-exists": "/usr/bin/rpm-ostree"
            },
            {
                "path-exists": "/sysroot/ostree"
            }
        ],
        "tools": {
            "index": {
                "label": "Software updates"
            }
        },
        "preload": [
            "index"
        ]
    }

The bridge currently resolves priorities, conditions, and conflicts, and the
resulting manifests.json just contains the one which wins. The shell displays
everything in the sent map, and does not have any checks to
conditionalize/ignore entries.

The dict key determines the URL of the navigation entry. Changing it to e.g.
updates-ostree or simply prepending an x to everything for testing, breaks
the menu entries. manifests cannot override this by adding "name": or
"path":.

-- src/cockpit/packages.py
+++ src/cockpit/packages.py
@@ -410,7 +410,7 @@ class Packages(bus.Object, interface='cockpit.Packages'):
         else:
             self.checksum = None

-        self.manifests = json.dumps({name: package.manifest for name, package in self.packages.items()})
+        self.manifests = json.dumps({'x' + name: package.manifest for name, package in self.packages.items()})

     def serve_manifests_js(self, channel):
         channel.http_ok('text/javascript')

Option 1: Send everything in manifests.json, resolve in the Shell

Export all available manifests in unresolved form to manifests.json, and let
the shell resolve priorities, renames, conflicts, and conditions. Disambiguate
conflicts by changing the manifests.json key names to the directory names.
The bridge explicitly adds name: to all manifest entries, by and large
ignores the keys, and computes the URLs using name:.

This would be straightforward for lockstep bridge + shell, but completely
breaks backwards compatibility in both directions. It would also change the
page URLs (bookmarks, documentation), but that may be less of a concern.

For maintaining backwards capability, there are some options below. With all of
those we have to keep the bridge code for at least resolving name conflicts and
priorities for several years (which seems fine, it's not very expensive). We
can take out the Python bridge's condition evaluation (as that hasn't been
released yet).

Capabilities

The shell could announce that it is capable of interpreting this "new"
manifests format. This would be very clean and elegant:

--- pkg/shell/base_index.js
+++ pkg/shell/base_index.js
@@ -446,6 +446,7 @@ function Router(index) {
                 if (source) {
                     const reply = {
                         ...cockpit.transport.options,
+                        capabilities: [...cockpit.transport.options.capabilities, "manifest-resolve"],
                         command: "init",
                         host: source.default_host,
                         "channel-seed": source.channel_seed,
--- src/cockpit/bridge.py
+++ src/cockpit/bridge.py
@@ -104,6 +104,7 @@ class Bridge(Router, PackagesListener):
             return dict(token.split('=', 1) for token in lexer)

     def do_init(self, message: Dict[str, object]) -> None:
+        logger.warning("XXX got init message: %s", message)
         superuser = message.get('superuser')
         if isinstance(superuser, dict):
             self.superuser_rule.init(superuser)

However, this just gets sent to ws and not relayed, because ws starts the
bridge first, and loads the shell from it:

got init message: {'command': 'init', 'version': 1, 'host': 'localhost', 'superuser': False}

In other words, there is no way to communicate capabilities from the shell
to the bridge.

Same for cockpit.transport.options.system.version -- that's the version of
the bridge for the UI's benefit, it doesn't work the other way round due to the
way the session is started.

New manifests.json API

The bridge could start to offer a /manifests-all.json with that new
structure. If it receives it, it uses that and resolves it, otherwise it falls
back to /manifests.json.

This is inelegant as it would commit us to the -all name forever. It's not a
big deal, but a bit "meh".

Alternatively, the GET request for manifests.json in the new Shell could add
a header like X-Cockpit-Manifest: no-resolve. Then in a few years this could
just become the default. But it's a lot more magic and error prone.

Option 2: Send manifests to the bridge for resolution

Introduce a new channel type or internal D-Bus service in the bridge which
receives the unresolved manifest (with the new structure from above), let the
bridge resolve everything, and return with a manifest dict of the current
format.

Then the shell itself could gather locally available pages (which it would have
to do anyway somehow for the flatpak/ws use case) and send them to the bridge.
If that API does not exist, then the remote system has an old bridge installed,
and the Shell can fall back to using /manifests.json as usual.

Option 3: Include local manifests into beipacks

Send them along with the Python code, and add the path to $XDG_DATA_DIRS so that the bridge finds them. The main benefit is that it's (probably) the easiest implemenation and has no backwards compatiblity concern. This might be the best "quick solution" to get this functionality working before working on option 1.

Option 4: Parallel implementation for ws-hosted pages

If the shell gets an empty result for /manifests.json, query /manifests-ws.json instead (or in parallel). cockpit-ws delivers the locally bundled pages then, in the above new format. Then the shell can easily do the resolution itself.

[martinpitt]

@allisonkarlitskaya , @mvollmer : This is my initial brain dump and investigation. I'd appreciate some thoughts and feedback from you! This may also be a good sprint topic, but I'd like to have a place to put down notes.

[allisonkarlitskaya]

So one quick note: my initial idea for dealing with the bridge configurations on the beiboot bridge was to send the bridge stanzas (in the format received from Packages.get_bridge_configs() aka cockpit-bridge --bridges) over to the remote as a special transport control message, possibly before init, but I abandoned this idea because the way we invoke sudo or pkexec is different for the beipacked scenario (sudo python -ic '# cockpit-bridge') vs normal scenario (sudo cockpit-bridge --privilged). We need to represent that difference somewhere, and I figured the best way to do that would be to simply hardcode it into a special sort of 'beipack mode' which doesn't attempt to load local packages at all and only cares about the (modified, internal) bridge configs so that it can become root.

On the side of how to resolve conditions, I imagined one of two possibilities:

• we get beiboot to do the work for us by introducing a new bootloader command for querying and reporting the existence of a particular file via the ferny command protocol. The first stage bootloader gets sent in one go, so we wouldn't have to worry about roundtrips. We'd get a new custom cockpit.file_exists('/usr/bin/rpm-ostree', True) or so for each file we request.
• we add a new commandline argument to the bridge (which would also enable the modified bridge configs) to specify a list of files we'd like to check the existence of. Something like --remote-package-checks=/usr/bin/rpm-ostree:etc:etc. Then we'd report out the result of the scan as part of the init message and pick it up in the injector.

Both of those would necessarily require some kind of a change to the Packages API in the injector in order to be fed the information about the conditions from "outside" instead of calling os.path.exists() directly.

[martinpitt]

So are you saying that simply sending all manifests along with the beipack won't work, because it would also include these bridge configurations? I.e. we would have to at least filter out all privileged: bridge configs (or just the shell's).

Note that the condition resolving is not the primary difficulty here. It's that API corner of manifests.json and renaming/priority which is hard to change in a backwards compat way, not the conditions (which don't have much legacy commitment attached to them)

[allisonkarlitskaya]

There's nothing that prevents us from doing priority/renaming resolution on the -ws system, inside of the injector, unless you're interested in mixing-and-matching packages from the client and host, which for me (at least at the current time) is a no-go.

[allisonkarlitskaya]

There's nothing that prevents us from doing priority/renaming resolution on the -ws system, inside of the injector, unless you're interested in mixing-and-matching packages from the client and host, which for me (at least at the current time) is a no-go.

Nothing except checking conditions, I mean, of course.

[martinpitt]

I'm not interested in mixing-and-matching. Yesterday a fourth option came to my mind, I added it to the description. I think that should be relatively easy to implement, and is in the direction of what you suggested in the comment above.