Cockpit behind NGINX reverse proxy closes connection after several seconds

[shindouj]

Hey there,

I run cockpit 299-1~bpo12+1 on Debian Bookworm (12) behind an NGINX proxy. I can log in just fine and browse the panel, but after a random amount of time (usually under a minute) the session dies out:

with transport closed: disconnected in the JS console. Cockpit logs are empty.

My cockpit.conf:

[WebService]
Origins = https://server.domain.tld wss://server.domain.tld
ProtocolHeader = X-Forwarded-Proto
# ForwardedForHeader = X-Forwarded-For
LoginTo = false
AllowUnencrypted = true

My nginx configuration:

server {
    listen 443 ssl;

    ssl_certificate /root/.acme.sh/*.domain.tld_ecc/*.domain.tld.cer;
    ssl_certificate_key /root/.acme.sh/*.domain.tld_ecc/*.domain.tld.key;
    ssl_trusted_certificate /root/.acme.sh/*.domain.tld_ecc/fullchain.cer;

    server_name server.domain.tld; # public server host name
 
    location / { # public context (should be the same as internal context)
        proxy_pass          http://127.0.0.1:9090/; # full internal address
        proxy_http_version  1.1;
        proxy_buffering     off;
	
        proxy_set_header    Upgrade $http_upgrade;
        proxy_set_header    Connection "upgrade";
        proxy_set_header    Host $host;
        proxy_set_header    X-Forwarded-Proto $scheme;

        gzip off;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name server.domain.tld;
    return 301 https://$host$request_uri;
}

which is basically identical with the documentation example. I also tried running it on https to no avail.
I have found at least one similar issue with the same problem that was basically unresolved - back from 2018. I have proxied multiple applications across multiple servers with similar configs and never encountered any similar issues.

At this point, I'm out of ideas; hoping the community has some.

[martinpitt]

This could also mean that the webserver (cockpit-ws) died, it may not be a proxy problem. Can you run journalctl -f -t cockpit-ws when you see the login screen, then login, and wait a bit until it logs out? Do you see any messages there?

[shindouj]

Funny! I checked all the logs with journalctl -f and guess what happened:

Sep 05 23:39:16 hostname monit[1634]: 'nginx' failed protocol test [HTTP] at [127.0.0.1]:81 [TCP/IP] -- HTTP error: Server returned status 403
Sep 05 23:39:16 hostname monit[1634]: 'nginx' trying to restart
Sep 05 23:39:16 hostname monit[1634]: 'nginx' stop: '/bin/systemctl stop nginx'
Sep 05 23:39:16 hostname systemd[1]: Stopping nginx.service - A high performance web server and a reverse proxy server...
Sep 05 23:39:21 hostname nginx[850790]: 2023/09/05 23:39:21 [alert] 850790#850790: *25 open socket #27 left in connection 15
Sep 05 23:39:21 hostname nginx[850790]: 2023/09/05 23:39:21 [alert] 850790#850790: *24 open socket #22 left in connection 16
Sep 05 23:39:21 hostname nginx[850790]: 2023/09/05 23:39:21 [alert] 850790#850790: aborting
Sep 05 23:39:21 hostname cockpit-ws[850708]: connection unexpectedly closed by peer
Sep 05 23:39:21 hostname systemd[1]: nginx.service: Deactivated successfully.
Sep 05 23:39:21 hostname systemd[1]: Stopped nginx.service - A high performance web server and a reverse proxy server.
Sep 05 23:39:21 hostname systemd[1]: nginx.service: Consumed 1.059s CPU time.
Sep 05 23:39:22 hostname monit[1634]: 'nginx' start: '/bin/systemctl start nginx'
Sep 05 23:39:22 hostname systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Sep 05 23:39:22 hostname systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.

Turns out my NGINX keeps getting restarted. Cockpit ain't faulty here. If I could have one suggestion though - by default Cockpit tells you to search through journalctl -u cockpit - feels like journalctl -u cockpit-ws might be more adequate.

Thanks for your response anyway!