New API to enable changing endpoint traffic class

[iziemba]

HPE has the use-case of being able to change the traffic class associated with an endpoint. At this point, we are looking at supporting 2+ traffic classes per endpoint. Worst-case behavior could be the libfabric user changes the traffic class per message operation.

Two requirements for this feature:

• Must not significantly increase HW resources. Allocating a new endpoint with new TX and RX queues is not acceptable.
• Mechanism to change traffic class should be lightweight.

The following is the prototype API we have been working with.

#include <rdma/fabric.h>
#include <rdma/fi_endpoint.h>
#include <rdma/fi_cxi_ext.h>

struct fid_ep *ep;
. . .

struct fid_ep *alias_ep = NULL;
uint32_t tclass = FI_TC_LOW_LATENCY;
uint64_t op_flags = FI_TRANSMIT | desired data operation flags;

ret = fi_ep_alias(ep, &alias_ep, op_flags);
if (ret)
    error;

ret = fi_set_val(&alias_ep->fid, FI_OPT_CXI_SET_TCLASS, (void *)&tlcass);
if (ret)
    error;

Note: Message ordering is undefined between traffic classes.

Thoughts on defining a proper libfabric API for this?

[shefty]

I'm in favor of extending the ep alias definition to cover this case. There may be other situations where an alias could be used to select other aspects of a transfer. Making things up: maybe using a different auth_key or transfers which could be marked as having a more relaxed ordering? Those might also be controlled using an alias.

I don't know how to easily define this or convey support.

[iziemba]

Ok. I'll have someone from HPE do a more formal API proposal and take into consideration auth_key/RO extensions listed above.

[shefty]

I'd limit any definition to only what we need. I can just imagine that we might want to extend this in the future.