gobetween hangs

[AzizDZH]

Hi team
I've gb installed on windows server, and it works as balancer for mobile apps.
sometimes it hangs and nothing helps except restart.

please help
prof.zip

[mw-cyrano]

+1 here, I just kicked gobetween out of our production environment. It has enough resources in terms of RAM, CPU and even storage. System responds normally, but https requests just hang and nothing happens. There is nothing in the logs (level warn). Scary. The worst part is, we had two gobetweens with two seperate A records in DNS and both went down approximately at the same time, happened multiple times. Load was around 15-20 Mbit/s HTTPS traffic. Three services configured (80, 443, mysql).

[nickdoikov]

have you checked stuff like slow connections etc?
you can try to play with idle timeouts here

gobetween/config/gobetween.toml

Line 53 in 3372328

[defaults]

most of all you have to check the amount of connection from GB to your backend app.

i'm using gb for years with u to 5 gbps load and do not experiencing anything like that under Linux.

[mw-cyrano]

The backend / webservers are available and up running normally. If I bypass GB by e. g. entries in the client's host file and access the webserver directly, everything runs fine. For all webservers, tested one after another.

The scary part is that two GB instances for the same domain went down at almost the same time, the second instance just 1-2 minutes after the first. Two times within an hour.

So I'm not sure timeouts would help here. To my understanding, that would just be a symptom mitigation.

Host system is Ubuntu 20.04 + gobetween via snap.

[nickdoikov]

if you have a connection leak between GB and backend you can connect to the backend from the third place easily but from the server where gb is sitting, you won't be able to connect if the amount of the connection will exceed the upper ports range (but really not sure how it works under windows).

you can check how many connections you have from the server with GB to the server with your backend servers.
co maximum connections you can have between two servers is the number of upper ports configured (32K is the default)

and in case if you are serving web traffic ie 80/443 port with http/https so it is more than possible that you could have a connections limit exceed because of semi-closed or close_wait connections.

so i suggest you to do this:

client_idle_timeout = "5s"        
backend_idle_timeout = "5s"

and add such params to your sysctl:

net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6
net.ipv4.ip_local_port_range = 10000 65535

also please check the connection amount on the gb server using netstat and grep by the IP:port of your backend and count the connections plus check the connections statuses from both sides.

[MPW1412]

Hi,

thanks for your detailed answer! We do not use Windows. Both gobetween and the webserver run on Linux / Debian / Ubuntu.

We definitely did not have 32k connections. Your guess was correct, it is http/https but also mysql in front of a galera cluster. Accesses around ~700-1000k simultaneously. In Mbit/s I've seen between 2 and 20 Mbit/s. All rather low.

But let's say your guess was correct. Wouldn't I see a log entry for that? Logging was set to ”warn“ and there was no output when it stopped working.

Best,
Matthias

[TechIsCool]

1yr mark reply... from a user that hasn't used gobetween in 3 years.

Curious if you have set the CPU Process Priority higher than Normal we continually had weird contention issues until we upgraded the priority on gobetween on windows. After that it worked great and ran stably.