Harden Jamulus server for unwanted connections

[ann0see]

Even though #2935 is already old, I think we should still consider hardening the "connection" process.
@dtinth proposed a "waiting room" approach, which we kind of already have (we already have something like this:

jamulus/src/channel.cpp

Line 39 in 2505a31

bIsIdentified ( false ),

).

I have another suggestion for a fix.

Problems with checking for valid audio packet is that we need to know 2 things in order to successfully decode a packet: (1) which decoder to use and (2) whether sequence numbers are used. These information is sent out-of-band and not in Opus audio packet, so we need to know their NETW_TRANSPORT_PROPS before we can succesfully decode an audio packet. We can also brute-force all combinations but efficiency is in question.

So here’s a new proposal:

Upon server receiving a packet from an unknown source, put them in a waiting room (do not assign a channel ID yet, dropping all audio packets in the meantime) and send it PROTMESSID_REQ_NETW_TRANSPORT_PROPS. It turns out that a Jamulus client can respond to that message before it has a channel ID assigned. Once an acknowledgement (or a reply, or a well-formed reply) is received, move them from waiting room to jamming room (where all messages are processed as usual).

Originally posted by @dtinth in #2935 (comment)

[mcfnord]

Is this code change intended to thwart the connection floods we've seen? I believe those floods aren't possible when a server's firewall blocks UDP packets from random sources. Would this code change benefit servers that block random UDP packets?

[ann0see]

Is this code change intended to thwart the connection floods we've seen?

There's no code yet, but yes, it should make the software a bit stricter.

I believe those floods aren't possible when a server's firewall blocks UDP packets from random sources.

The problem is: how would you conclude that some source is "random"? Having a firewall surely helps.
My server even uses a blacklist - but that's a protection on a lower level and mainly protects other services.

[ann0see]

It would be good to see if we can even reproduce the observed behaviour on modern servers. Maybe bIsIdentified is enough

[mcfnord]

A source is random when it doesn't rely on the directory server to open a connection prior to use. I believe this prevents random connections outside the directory protocol from reaching a server. So my servers should be immune to this particular attack. I could open UDP and make a server susceptible to that type of attack. Of course, private servers (more accurately called unlisted servers) must open UDP to anyone.

[ann0see]

Ok. But as you said that doesn't hold true for unlisted servers or direct connections. So that's only a small "solution" if you even want to call it that.

[mcfnord]

Thanks for watching out for users in all scenarios. I do limit my scope to developing the directory-based public network.