Aim for minimal use of TCP in the Derecho code base

[sagarjha]

Since libfabric can emulate RDMA over TCP, we should just use RDMA operations throughout the code so that we are TCP free on a cluster that has RDMA. What happens when we add external clients will not be affected by this.
We have already in the past migrated our P2P layer to use RDMA. A major (and may be the only) subsystem that uses TCP is the view change system that uses it to send states, logs, views etc. We should create some temporary RDMA P2P connections to transfer these at high speeds.

[sagarjha]

Of course, RDMA connections need to be setup and they need TCP (or some out-of-band mechanism) to exchange data such as memory addresses, queue pair information etc. When I say TCP-free, I do not mean that we get rid of that. That is also only required at connection setup time and involves only a few bytes of data transfer. It, therefore, does not factor into performance at all.

[etremel]

From my understanding of ViewManager, there are three remaining tasks that use TCP:

1. Sending View information to new members when they join, which is done via the leader_connection that only exists during ViewManager's constructor. (On the leader, this corresponds to a socket in pending_join_sockets).
2. Sending new Views and lists of prior-view shard leaders to existing members during a view change. This is done via the tcp_connections object in ViewManager, which maintains a TCP socket for every member in the View (connected to, oddly enough, the "RPC port").
3. Sending Replicated Object state from shard leaders to new members of the shard during a view change. This is also done via ViewManager's tcp_connections, although the receiving end of the transfer is actually implemented in Group::receive_objects().

I created a separate issue, #157, for the task of converting state transfer to use RDMA, because this seems like it has the greatest potential for performance improvement and is easiest to separate from the other tasks.

[KenBirman]

Edward's work seems to have resolved this issue as of v2.0