Allow applications to lease multiple buffers simultaneously from Derecho

[sagarjha]

This is from Ken.
Derecho manages application memory used for multicasts. This is so that the memory can be allocated, pinned in memory and registered with the NIC for RDMA operations. The application obtains a buffer from Derecho, sends it and then asks for another buffer. Our rigid model of giving out one buffer at a time is very restrictive and forces applications to do memory copy or otherwise, write convoluted code. The other option left for applications is to set the message size very high and use RDMC for data transfer if they expect to receive large messages from the outside world (sensors etc.).
We should implement some form of leasing multiple buffers simultaneously.
Note that we may probably side-step this issue if it becomes possible in the future to directly RDMA from unregistered-unpinned memory. In that case, Derecho will not need to manage any memory. And the application can allocate space as it pleases.

[mpmilano]

From Weijia:

The basic idea is that A Derecho user should always manage the objects to be send around using the slab allocator (Matt suggest Group::get_allocator(), which will return an allocator and capable of "create" and "delete" objects of type ObjectType). And the cooked ordered_send or p2p_send calls can check if the argument objects are managed by the allocator or not (by checking if their memory range are in a pre-allocated/pinned memory region for the allocator) and decide to send with copy or non-copy code path.

One key problem here we believe is how to manage the reserved memory region. We use this for slab allocator, SMC, and RDMC. Of course We want to avoid the copy between slab allocator and SMC/RDMC. The management of SMC and RDMC buffer is pretty ridged so far - we allocate the buffer(with pre-configured size) beforehand, fill it, and send it out. We may need flexibility to send data of variable length in the reserved memory region (possibly by using buffer pointers for SMC and RDMC?). What do you think?