IPv6 for Directory Servers

[softins]

Now that #1938 has been merged to provide support for direct connections over IPv6, I wanted to start a discussion around the issues involved with adding IPv6 support for Directory Servers. This is a potentially complex thing to get right, so it is worth talking through the technicalities before starting to write code!

Server registration

• All servers must still support IPv4 connections.

• If a server also supports IPv6, this should be available from the same server entry. i.e. there should not be two entries in the Directory for the one server.

• Servers must register with a Directory Server using IPv4, never IPv6. This is because the Directory registers the server using the IPv4 address and port number from which the registration request originates.

• The existing server registration message contains the local IPv4 address of the registering server's public interface. If the server is behind NAT, this will be the LAN address, not the WAN address (the WAN address is the address from which the registration packet arrives, as mentioned in the above point). The Directory stores both the WAN and LAN addresses for the registering server.

• Servers will initially register for IPv4 using the existing protocol, for compatibility with Directory Servers that do not support IPv6.

• A Server that supports IPv6 will follow up with an enhanced registration message that additionally contains the IPv6 address of its public interface, as well as all the information currently in the existing registration message. This message will be ignored by a Directory Server that does not support IPv6. If the Directory Server does support IPv6, it will respond with a registration successful message. This might also need to be a new message for unambiguity.

• The Directory Server receiving the enhanced registration message (over IPv4) will create or update the existing Server entry to include the supplied IPv6 address too.

• A server that receives the enhanced registration successful message can note that the Directory Server support IPv6, and could elect to send only the enhanced registration message for registration renewals.

• A Directory Server that supports IPv6 and receives a legacy registration message for a Server it knows about and has an IPv6 address for must not clear the IPv6 address it holds.

Client operation

• It might be worthwhile to put a "Use IPv6" checkbox in the Advanced Settings and store it in the ini file.

• A client that does not support IPv6 will request the server list from the Directory as currently. It will receive the existing format list that does not contain IPv6 addresses.

• A client that supports IPv6 must still send the request for the server list over IPv4. This is because most listed servers will still need to send an empty message to the client's IPv4 address.

• A client that supports IPv6 may send an enhanced server list request. A Directory Server that does not understand the message, or does not support IPv6, will ignore this request.

• A server that understands the enhanced request and supports IPv6 will respond with a server list that includes both IPv4 and (if available) IPv6 addresses for the servers.

• A client that receives the enhanced server list can continue just to send the enhanced list request to that Directory Server. Each time the client user changes list, the client will start by sending both the legacy and enhanced list request.

• !!! Currently, when the Directory sends the list to a client, it also tells all the servers in that list to send an empty message to the client's IPv4 address and port. This IPv4 address and port are taken from the source address/port of the list request. What should we do about servers that are registered with both IPv4 and IPv6 addresses, when sending a client the enhanced server list? Possibilities:

  • The client sends the legacy list request from IPv4 and the enhanced request from IPv6?

  • The client sends both from IPv4, but includes its IPv6 address in the enhanced request? (probably this option)

  • Should we support a Directory Server that can only communicate on IPv4, but that does support the storing of servers' supplied IPv6 addresses? We probably should. If so, then the client needs to send its IPv6 address in the enhanced request.

• If the directory server is serving an enhanced list, it should tell all listed servers to send the normal empty message to the client's IPv4 address, and those servers registered with an IPv6 address also to send an empty message to the client's IPv6 address.

• !!! What about the pings performed by the Connect dialog? If a client supports IPv6, and a server has both IPv4 and IPv6 addresses, we should probably list it twice in the list, and send a ping to both addresses, so the user can observe any difference and choose which to connect to.

[softins]

There may well be other issues in using IPv6 with Jamulus, specifically to do with large UDP datagrams and fragmentation. Apparently, IPv6 does not allow intermediate routers to fragment at the IP layer, and places the onus on the sending host to determine the path MTU size and perform its own fragmentation. Whether this requires knowledge at the application layer, or if the UDP layer can handle it, I don't yet know. But Jamulus currently relies on large UDP datagrams for server lists from Directory Servers, and for client lists from servers.

The following articles provide some background:

• Evaluating IPv4 and IPv6 packet fragmentation
• Fragmenting IPv6
• Broken packets: IP fragmentation is flawed
• https://en.wikipedia.org/wiki/IPv6_packet

[ghost]

For IPv4 iptables rules, only fragmented tcp packets can be dropped
because tcp should be using:
-A INPUT ! -i lo -p icmp --icmp-type 3/4 -j ACCEPT

# Force fragmented tcp packet check (Correct)
-A INPUT -p tcp -f -j DROP

# Force fragmented packet check (STUPID! - DON'T DO THIS!)
#-A INPUT -f -j DROP

Or maybe you can disallow all fragmentation with the exception of udp.

[ann0see]

What happens with directories on an IPv6 only connection?

You say that an IPv4 connection is needed because of the registration process.

This is because the Directory registers the server using the IPv4 address and port number from which the registration request originates.

Why can’t we change that?

[ann0see]

It might be worthwhile to put a "Use IPv6" checkbox in the Advanced Settings and store it in the ini file.

Could already be implemented now for usability reasons.

What about the pings performed by the Connect dialog? If a client supports IPv6, and a server has both IPv4 and IPv6 addresses, we should probably list it twice in the list, and send a ping to both addresses, so the user can observe any difference and choose which to connect to.

Yes. Preferably the client chooses the best connection automatically.

[gene96817]

How should things work if a client or server has only IPv6 service and the IPv4-IPv6 gateway is operated by the ISP?

[rdica]

Or NAT64 is in use by the server operator?

[gene96817]

Sorry, I was saying, there are now some ISPs that deliver only IPv6 to the consumer and they operate a network-level IPv6-to-IPv4 gateway. We should include this as a use case when we implement IPv6 support. This offers the server operator great simplicity. No local vs public address issues. No port mapping issues. (Although IPv6-to-IPv4 issues are now out of the server operator's hands.)

[rdica]

Yes, I understood what you meant, my comment was yet another scenario. Servers aren't just run on residential ISP networks. Increasingly there are servers being run on cloud platforms, and some have IPv6-only. Admins may configure NAT64 in order for those servers to reach legacy (IPv4) network resources. I'll admit its more of an edge case at this time but I could see NAT64 use becoming more prevalent as time progresses and IPv6 adoption increases.
😃

[pljones]

OK, bumping because I think it's about time we revivified this. In terms of project time lines, this would be "Jamulus v4.0.0". There shouldn't be breaking changes but it does twist the protocol a little...

I've a fairly simple plan for this:

The existing "server internal address" is transfered as a UTF-8 string... At first, that might seem odd. But I had plans when I did it.

bc04ff2f 2020-04-10 Use server internal address in server list [Peter L Jones]

Previously, it was:
2 bytes number n | n bytes UTF-8 string topic
but the topic wasn't used for anything anywhere.

So I made it:
2 bytes number n | n bytes UTF-8 string server internal address
rather than saying it was definitely only the size of an IPv4 address.

So long as the total length of the field fits in the "2 bytes number n" for the length and the field encoding supports appropriate parsing, there's the possibility of putting more into it than just the server internal address.

The problem is that the Directory currently has no way of knowing the Client version - it's not sent when requesting the server list. Well, not in the current version.

So what I'm thinking is

• PROTMESSID_CLM_REQ_SERVER_LIST: pass the client version -- compatibility: if the field isn't present, it's "old" and doesn't support the feature. @softins is also suggesting passing IPv6 address here: it only needs to be the "other" address as the source address can be used for one. But generally yes, sending the extra address as well as the client version might help. It just means the client needs to know it's public address in the other protocol, a complexity most users won't want to get involved with.
• PROTMESSID_CLM_REGISTER_SERVER: change server internal address to server address details, containing four fields:
  • server IPv4 local address:port
  • server IPv4 public address:port
  • server IPv6 local address:port
  • server IPv6 public address:port
    all encoded into a base64 and prefixed with a version number and delimiter. EvaluateCLRegisterServerMes should be able to parse the new format. CServerListEntry would change to support this.
• PROTMESSID_CLM_SERVER_LIST: depending on the version received (i.e. "or not") in PROTMESSID_CLM_REQ_SERVER_LIST, send either an old format PROTMESSID_CLM_REGISTER_SERVER or a new format for each entry.

If we assume the Client will always use its chosen protocol "server public address", then the server list processing needed to ensure, based on the same magic it uses for IPv4, that the switching of local and public addresses happens for both protocols.

The Directory would need some backward compatibility around the server list persistence file.

Servers must register with a Directory Server using IPv4, never IPv6. This is because the Directory registers the server using the IPv4 address and port number from which the registration request originates.

CHostAddress already supports IPv6, so parsing the incoming IPv6 request and passing through the source address as IPv6 should just work. That is, the Directory can see the requesting IP address is v6 and store it correctly in CServerListEntry. It's up to the requesting server to ensure the other three fields are correct (in the same way as --serverpublicip is currently used, we'd need --serverpublicip6, --serverlocal and --serverlocal6 to fill in fields the Directory would otherwise get wrong).

All servers must still support IPv4 connections.

Not necessarily. The Directory would be able to see from the Client version (or lack of) that it could support the new format entries. If the Server didn't have an IPv4 address and the Client didn't have the right version, then that Server could be dropped from the list (or set to an invalid IPv4 address so it would still show in --showallservers).

Servers will initially register for IPv4 using the existing protocol, for compatibility with Directory Servers that do not support IPv6.

I don't think this is needed. We can have --directory-old be for backwards compatibility. Otherwise, just get on and use the new format. Or maybe send both if --directory-old and --directory are used, to allow for a "changeover period" with a Directory.

Client ...

Yeah, there are probably bits this doesn't quite cover. But I'm hoping it's getting us going again.

[ghost]

Note that ipv6 is udp fragmentation ignorant, thus Jamulus will need to chop up the udp packets so that they are not too large. This may be as simple as chopping the directory listing into smaller udp packets or may require trivial fiddling with the packet header at the server, and may not even affect old clients (udp is connection-less). I have not investigated this.

[pljones]

Initially, I'd suggest not changing anything on that front as part of adding IPv6 support.

Handling fragmentation of the server list is a separate issue and we need to deal with it as such. (That, I'd suggest, would be by deprecating the old protocol and introducing a new "paged" protocol for exchange.)

[softins]

OK, bumping because I think it's about time we revivified this. In terms of project time lines, this would be "Jamulus v4.0.0". There shouldn't be breaking changes but it does twist the protocol a little...

I've just picked this up again and have been giving it some more thought. I plan to produce a more complete proposal in the next week or two, but I wanted to comment on one specific part of @pljones suggestion above.

• PROTMESSID_CLM_REGISTER_SERVER: change server internal address to server address details, containing four fields:

  • server IPv4 local address:port
  • server IPv4 public address:port
  • server IPv6 local address:port
  • server IPv6 public address:port
    all encoded into a base64 and prefixed with a version number and delimiter. EvaluateCLRegisterServerMes should be able to parse the new format. CServerListEntry would change to support this.

It's not necessary to have all four of those fields above, for a few reasons:

1. If the server is behind NAT (e.g. either on a home network, or on an AWS instance), it does not know its public IPv4, so cannot include it in the message. The directory can determine the server's public IP and port from the source of the registering datagram.
2. The server's local IPv4 and port are already included in the register message protocol.
3. IPv6 by design does not have NAT. The IPv6 address detected by the server's Jamulus instance is the same public address that the world will see it by. So there is no need for both "local" and "public" IPv6s.
4. There is no need to include :port either. The local port number included by the server in the register message will be good for both IPv6 and same-subnet IPv4. If IPv4 NAT changes the port number, the server can't tell that (as in 1 above), but the Directory will receive the request from the public port number.

Also, I don't like the idea of Base64, and don't think it's needed here.

My suggestion is for the server to put into this field its local IPv4 address (or provided server public address) exactly as at present, and for an IPv6-enabled server to append to that a single space followed by its detected public IPv6 address.

For example 192.168.0.55 2001:db8::1234:5678:1111:2222.

The IPv6-enabled server will initially use that format whatever Directory it tries to register to. Once the public Directories are updated to understand this format, this registration will always succeed, so this will be the "normal" case. The Directory will just split this field on the space and use the two parts as LHostAddr and HostAddr6 (the new class member) respectively.

If trying to register with an older Directory that doesn't understand this format, the request will fail validation, so the Directory will ignore the request and not respond (I've just tried it). This is because the Local IP string will be longer than 15 characters, which is the current value of MAX_LEN_IP_ADDRESS. The server will detect this situation, and retry the registration without the appended IPv6 address, appearing to the Directory as a normal IPv4-only server. This register request will succeed as at present. The server will continue to use the same format when refreshing its registration.

If in the future we make it possible for a server instance to have only an IPv6 address and no IPv4 address, it can just send its registration request from its IPv6 address, with an empty string in the Local Address field. An IPv6-enabled Directory can then add the server entry with an IPv6 address but no IPv4 address.

[softins]

the directory code in serverlist.cpp calls CServerListManager::IndexOf() to do a linear search of the list looking for a HostAddr that matches the request origin.

Isn't CHostAddr the IP:port (i.e. not QHostAddr)? (We need to check either that or IP and server name. IP alone isn't enough.)

Yes it is. IndexOf() compares the CHostAddr, which checks both InetAddr and iPort using an operator== method:

bool operator== ( const CHostAddress& CompAddr ) const {
   return ( ( CompAddr.InetAddr == InetAddr ) && ( CompAddr.iPort == iPort ) );
}

So it is able to identify the server uniquely.

[softins]

So the suggested "Extended response" message would need to include both the address and the port, which is no problem.

[ann0see]

I think the unique ID is only a problem if we have IPv4 and 6.
so it would just be a number internal to the directory.
IPv4 address gets mapped to an ID (array index) internally. We'd have a hash map from IPv4 address to id for the "old" approach.

Then on registering we send the id with the success message if the server registers - if that works easily.

Locally if the client sees two different IDs (one from IPv4 and one from IPv6) in the respective success message, it requests a new ID from the server but now has both IPs in the request message. The server then responds with one.

But that's very close to your suggestion, I think?

[pljones]

OK, so to have the "same type" of unique key in the Directory's server list, we extend the server list entry (used in all Servers to hold their own data and in a Directory to hold each Server's data -- same structure) to have an additional field and, for IPv6 Servers we return both new fields (key and IPv6 address) and for IPv4 Servers, including legacy ones, we populate the new key but leave the IPv6 address blank. For IPv4 servers, we also return a new message with the new key in, so when they send a new IPv6 registration request, they include this key (null on first attempt if they've not received it).

The IPv4 registration request will never contain the new unique ID - it will always continue to rely on IPv4 IP and port (as legacy code will continue to know nothing about the new field - both Servers and Directories). That will be used to find the new key in Public Directories (and any updated custom Directories). So if an IPv6 server first sends the IPv6 request (or e.g. it arrives first, due to routing issues) and gets registered, it will know its new key - but not be able to use if for anything. It will then need to send an IPv4 request and wait for the new extended response with the (different) IPv4 key, and then send another IPv6 registration entry, with the (different) IPv4 key - which would leave the "other" IPv6 entry (with the earlier key) to expire or be deleted (using the source IPv6 details).

Legacy Clients, of course, must continue receiving the IPv4 entries as today.

New Clients will need the IPv4 IP and Port in the IPv6 entry to "merge" entries.

Also, if a server registration expires due to intermittent network issues (e.g. dead router), the Server cannot use the ID any more - it could have been replaced in the list. A Client wouldn't know this, either.

I'm not sure that's much better.

[softins]

I'm not sure that's much better.

No, I agree completely. I don't see any need to invent a new opaque key. A directory receiving a request has to identify the server from the source IP and source port anyway, using IndexOf() (just need to enhance IndexOf() to be intelligent about the CHostAddr that gets passed to it and compare the correct field). Since server registration is an occasional occurrence, compared to other traffic, there is very little to be gained by trying to optimise this particular lookup.

[danryu]

Seems like the strategy here has been thoroughly hashed out! @softins what's the status on this? I'd be happy to guinea-pig any code/branch you need testing ...

[pljones]

None of my servers have, or will have, JSON RPC compiled in.

[danryu]

@pljones Why's that?
Just wondering if there's a technical blocker to greater adoption of JSON RPC, or otherwise what your reasoning is - it sounds very absolute.

[softins]

• List of servers from a directory (CLM_REQ_SERVER_LIST).
• List of connected clients from a server (CLM_REQ_CONN_CLIENTS_LIST).
• Small messages such as requests, ping, version and OS, register and unregister server. These are small enough never to need > > fragmentation.

For the first two items in the list (the lists of servers and connected clients), is this not an ideal use case for the JSON RPC interface?

No, I don't think so, for a number of reasons:

• The idea is to keep the existing Jamulus protocol messages, but just allow them to travel over TCP on the same port number.
• The JSON-RPC interface is optional, and primarily designed for local use within a LAN or even localhost.
• It would involve creating a client-side implementation of JSON-RPC. At the moment, it is just a server.

Would this simplify the roadmap to IPv6 directory support?

I feel it would in fact complicate it. But no problem in posing the question!

[pljones]

@pljones Why's that? Just wondering if there's a technical blocker to greater adoption of JSON RPC, or otherwise what your reasoning is - it sounds very absolute.

Because they're publicly exposed, serve the community and I'm not competant to check the code for vulnerabilities -- as Tony says, it was originally intended solely for local host usage, so was accepted on that basis. It's still "experimental", more or less - just it's been fairly stable for a while. So I don't include it in my builds to reduce potential attack vectors. I don't know if Volker includes it in his directories, either. So it may be that it's just not there in the main public directories currently.

[ann0see]

I don't think he includes it based on some emails we sent.

[danryu]

* The idea is to keep the existing Jamulus protocol messages, but just allow them to travel over TCP on the same port number.

Is this for backward compatibility? It just seems there's a lot of work involved to do things in a "legacy" way which are much more easily done with purpose-built things such as JSON endpoints.

* The JSON-RPC interface is optional, and primarily designed for local use within a LAN or even localhost.

I admit I haven't played with it much, but if it's well suited to certain admin-related tasks (such as directory server metadata distribution), then what's the problem with bringing it more into play?

* It would involve creating a client-side implementation of JSON-RPC. At the moment, it is just a server.

Aren't we just talking about client requests to the JSON endpoints in question? Which is a cinch compared to your very involved proposals that I definitely haven't finished digesting :)

PS Wondering also about the requirement on IPv4 re server registrations - I don't think I understood those assumptions. IPv4 addresses are becoming more expensive to use with cloud providers. And you already implemented IPv6 direct connections (kudos btw) which I still have to test ...

[softins]

Well backward compatibility is definitely one good reason. Jamulus has always taken care to remain backward-compatible with earlier clients and servers. But also, the code to construct, decode and act on the existing Jamulus protocol messages is already there. So the TCP implementation is just providing a different socket wrapper around the existing functions, rather than reimplementing them in an additional and completely different way. I'm happy with that approach, having considered alternatives.

Some ISPs are still not supporting IPv6 at all for subscribers, so we need to continue supporting IPv4 for the foreseeable future.

[danryu]

It's definitely great that you can remain backward-compatible with this strategy and maintain an even UX.
I just fear (perhaps overly so) that this comes at the expense of increased complexity on top of what was already a confusing design to newcomers (eg lists of servers/clients sent over connectionless UDP).

[danryu]

Because they're publicly exposed, serve the community and I'm not competant to check the code for vulnerabilities

Fair enough! These are more operational concerns, as I see it, rather than pure design choices (web endpoint interactions are well understood and probably easier to maintain long-term security-wise than custom TCP implementations), but I get your point.
EDIT: I should perhaps say "web-ish" - looking at the RPC server code it's evidently fairly "custom" anyway, though clearly serving its function.