WIP Spec: Draft for top-level p2p spec

[josef-widder]

This is the beginning of a high-level p2p spec addressing #9573 for discussions. I started top down trying to understand what p2p is supposed to do rather than how p2p should do things. This collects findings from several discussion.

---

PR checklist

• Tests written/updated, or no tests needed
• CHANGELOG_PENDING.md updated, or no changelog entry needed
• Updated relevant documentation (docs/) and code comments, or no
  documentation updates needed

[cason]

Some comments on the introduction of the document. Possibly need of better definitions for used terms.

[sergio-mena]

Would it be worth touching on "connectedness" in this section? Maybe some definitions that give an idea of how the number connections that a node keeps to peers, together with the "amount of bad peers" (or probability of a peer to be bad) has an impact on the global properties we want to achieve (e.g. a message reaches all full nodes eventually)

[josef-widder]

Yes, you are right. I think there are basic implicit assumptions/requirements in the current solution, e.g., between any two correct validators there is a path consisting only of correct full nodes. (Most likely even more paths). This might be a requirement that the p2p needs to provide to the reactors, and it can only do so, if the underlying network has even stronger properties.

[cason]

Moreover, I hardly can see this property being ensured, it is probabilistically achieved.

[lasarojc]

You are right @sergio-mena, in that these properties cannot be guaranteed, so maybe we shouldn't try. I mean, the P2P should only guarantee a best effort guarantee of conectedness, and then we call out that if by some external forces other guarantees are made, such as the existence of a GST and a maximum number of byzantine nodes, then and stronger properties can be derived, like eventual delivery of messages from some correct processes to some correct processes. This way we modularize the guarantees.

[sergio-mena]

That's a good point. Although I'd simplify to say "if you configure a node to prevent it to normally connect to peers " (validator<-->sentry) then only the normally connected node (the sentry) can expect the properties the overlay network provides

[josef-widder]

Yes, I think we should use something like "normally" in the definitions of the specification. At this points I think it makes sense to talk about validators to motivate the definition.

[cason]

I like the idea of defining a public network, the overlay, fully using p2p and PEX, and a private network, where each operator manually configures nodes. In fact, validators also have their IDs set as "private", so that no one learns its actual address.

[josef-widder]

@sergio-mena says: let's drop the term "validators" from this spec.

Talk about the public network of full nodes. Explain the validators need to make sure to be connected to the public network

[josef-widder]

@cason says:

1. We assume validators are somehow connected to at least a correct public node

From this point we reason about correct public nodes

[sergio-mena]

Something about self-healing? (not sure, just thinking out loud)
Something like, "the network overlay can recover from important crippling network events (e.g. temporary partitions)"

[josef-widder]

You are right. I wrote the properties 1.-3. having somewhat "good periods" in mind. The question as I see it now when I look at your comment is whether there is a relation between self healing and openness. But perhaps the connection is rather in the solution side. We need to think more ;-)

[sergio-mena]

Totally agree

[lasarojc]

found the following quote while doing a quick research "I cannot define gossip, but I can recognize it when I see it”

[sergio-mena]

@josef-widder, thanks for putting together this draft top-level spec.
I added a few comments (mainly with some ideas).
I like the overall structure. IMO it's an excellent starting point: this document structure can be helpful on execution (i.e., who focuses on what and when). I think we could even structure our tracking issue (#9573) according to this document structure.

[lasarojc]

Should the reactors be exposed or have a say in the overlay? Could we have multiple ones in parallel? For example, could we use a chord like overlay for 1-to-1 communication but a fat tree rooted at the proposer for proposal broadcast? The p2p could ask the reactors for hints on how to propagate messages.

[lasarojc]

• no duplication
• no corruption

the following are "low level" and not required as long as messages are delivered.

• ability to prioritize messages.
• ability to make messages stale, for example, right after sending MSG1, still in the outbound buffers, send a MSG2, which makes MSG1 useless and dropped.
• at most once delivery
• channel isolation (or some best effort)
• to respect back pressure

[josef-widder]

For now we will stay with "one overlay for all". I will make this explicit in the write-up.

p2p properties (single hop, by channel)

• no corruption
• prioritization

seems that currently message handling by p2p is best effort

[lasarojc]

I don't know where exactly to put it, but it would be a good property not to leak information to other nodes, for example, the internal topology of a validator network.

[lasarojc]

found the following quote while doing a quick research "I cannot define gossip, but I can recognize it when I see it”

[lasarojc]

• do not forward messages already known to potential receivers

[josef-widder]

I thought this is currently implemented at reactor level, no?

[lasarojc]

I think the reactors try to filter what to send, based on the deduced state of the other nodes, but not based one what messages they have received. The first is stronger, if correctly implemented, but the latter should be easier to implement.

[lasarojc]

one interesting property is that eventually all messages sent by correct nodes to correct nodes are delivered or they become stale (e.g., a decision is reached and becomes known)

[lasarojc]

is this really required? What if there are better connections?
I thought about "In a good period, connections eventually stabilize", but even in good periods there may be changes in the network that the p2p layer might want to take into account. And even if not, if we need to ensure openness and a bounded number of connections, good connections may need to be dropped.

[lasarojc]

Wondering if there is any way we could get anonymized aggregated data from the network? For example average number of connections, average memory usage, anything that we could use to study the network while not revealing too much to malicious entities.

[lasarojc]

• to provide reasonable parameters to drive the connections of peers.

  These parameters can be low level (like what we have now, i.e., number of outgoing connections, number of incoming connections, permanent peers) or higher level (e.g., form a tree, for a ring with long n fingers...), which require a more elaborate PEX.