Streaming query continues probing database even with all users disconnected

[TiagoJacobs]

Version Information

Server Version:

hasura-graphql-engine version
Hasura GraphQL Engine: v2.37.0

Environment

OSS

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"

What is the current behaviour?

Hasura continues sending streaming queries checks for postgresql even when all users are disconnected (I've ensured it using netstat).

What is the expected behaviour?

After all users are disconnected, I expect that hasura stops probing postgresql.

How to reproduce the issue?

1. Configure postgresql to log all the queries (log_statement = 'all')
2. Launch 400 users in a BigBlueButton 3.0 server (which uses hasura)
3. Wait for all users to leave.
4. Stop bbb-graphql-middleware and nginx (to ensure nothing else is connected to hasura).

Screenshots or Screencast

In the screencast, I monitor the number of connections with port 8085 (which hasura is listening too).
When almost all users disconnected, I stop the nginx and the bbb-graphql-middleware (they route connections to hasura).
After stopping, I ensure there is no ESTABLISHED connection in port 8085.
Even after that, postgres continues receiving the streaming queries from probers.
PS: this doesn't happen with the subscription queries, only with streaming.
I kept the server running for 1 hour, and the queries were still being emmited, so it doesn't look like a pending timeout.

Link to the video (too long for github):

https://www.dropbox.com/scl/fi/35enek9b2fjpzntc1rgey/hasura-bug-with-streaming-queries-2.mp4?rlkey=lrdgxcd7pgx1p8fgu4crhbwn6&dl=0

Keywords

always increasing system load, memory leak, cpu leak, streaming queries