libfabric 2.0 release

[shefty]

Discussion on whether we should move toward a 2.0 release, which allows breaking API/ABI compatibility, with a proposed target date in the second half of 2023, after a v1.19 release.

General proposed goal is: 2.0 should be a drop in replacement for most existing binaries that were compiled and work against version 1.X (with X to be decided).

Purpose of 2.0 release is to simplify the API by removing little used features and re-examine how narrow use features are exposed. The first objective is to list potential features to remove, identify if there are users, and update the list accordingly. This is a first pass at features to examine for removal or re-work, grouped by header files. Listing as a task list with expectation that marking as 'done' means the item has been discussed, not that it's been selected for removal. (If there's a different markdown option here

fi_log.h

• remove enum fi_log_subsys
• remove FI_LOG_TRACE and FI_LOG_MAX

fi_prov.h

• Change fi_param_get_bool() to use bool instead of int

fi_trigger.h

• Re-examine if we want to keep triggered ops

fi_tagged.h

• Keep as is

fi_rma.h

• Likely keep as is
• Possible for removal is rma_iov_count

fi_ext.h

• Keep as is
• Expand peer provider APIs for greater composability

fi_errno.h

• Keep as is

fi_eq.h

• Remove wait sets from API
• remove FI_WAIT_MUTEX_COND and FI_WAIT_POLLFD
• Remove poll sets from API
• remove fi_eq_attr::signaling_vector
• remove async memory registration (FI_MR_COMPLETE)
• remove async AV insertion (FI_AV_COMPLETE)
• remove fi_cq_wait_cond
• Can we simplify, rework, or remove counters
• Do we need separate send/recv CQs?

fi_endpoint.h

• remove FI_OPT_ENDPOINT options:
  BUFFERED_MIN, BUFFERED_LIMIT, SEND_BUF_SIZE, RECV_BUF_SIZE,
  TX_SIZE, RX_SIZE
• remove fi_ops_ep::rx_size_left / tx_size_left

fi_domain.h

• remove FI_SYMMETRIC
• remove fi_ops_av::insertsym / insertsvc
• remove fi_mr_modify
• remove FI_DATATYPE_LAST, FI_ATOMIC_OP_LAST

fi_collective.h

• Keep as is

fi_cm.h

• Keep as is

fi_atomic.h

• remove fi_ops_atomic::write_valid / readwritevalid / compwritevalid

fabric.h

• remove FI_PATH_MAX / FI_NAME_MAX / FI_VERSION_MAX
• remove FI_PROV_SPECIFIC (document instead)
• remove FI_MORE
• remove FI_PRIORITY (make internal)
• remove FI_AFFINITY
• remove FI_MATCH_COMPLETE
• remove FI_AV_USER_ID is not implemented (keep or remove?)
• remove FI_VARIABLE_MSG
• remove FI_SOURCE_ERR
• remove FI_NUMERICHOST
• Remove address formats for removed providers
• remove FI_MR_UNSPEC / FI_MR_BASIC / FI_MR_SCALABLE
• remove FI_THREAD_FID / FI_THREAD_COMPLETION / FI_THREAD_ENDPOINT
• remove fi_resource_mgmt (set to always enabled)
• remove FI_ORDER_NONE / FI_ORDER_STRICT
• remove FI_EP_SOCK_STREAM / FI_EP_SOCK_DGRAM
• Remove protocols for removed providers
• remove FI_MSG_PREFIX
• remove FI_ASYNC_IOV
• remove FI_LOCAL_MR
• remove FI_NOTIFY_FLAGS_ONLY
• remove FI_RESTRICTED_COMP
• remove FI_BUFFERED_RECV
• remove fi_tx_attr::comp_order (no order guarantees)
• remove fi_rx_attr::comp_order (no order guarantees)
• remove fi_rx_attr::total_buffered_recv
• remove fi_ep_attr::msg_prefix_size
• fi_ep_attr::max_order_raw / war / waw _size (look at alternatives)
• fi_ep_attr:mem_tag_format - replace with max_tag_bits
• remove fi_domain_attr::resource_mgmt (always enabled)
• fi_domain_attr - move/copy some cnt fields to nic attribute
  field removal has potential to impact most apps (want compatibility)
• remove fi_domain_attr::mr_iov_limit (define to 1)
• FI_SELECTIVE_COMPLETION
• fi_info - add provider specific context
• fi_info - always require that fi_info come from libfabric
  E.g. app must use fi_allocinfo or fi_dupinfo
• remove fi_alias / FI_ALIAS
• remove FI_GETFIDFLAG / FI_SETFIDFLAG
• Examine fi_deferred_work
• remove FI_REFRESH

Providers

• bgq
• gni
• netdir (now part of verbs)
• psm
• rstream
• sockets
• usnic
• require fully qualified names to pick layered providers (and peers)

[shefty]

AWS is interested in keeping triggered ops. HPE does use it. Believe it is useful and worth keeping.

Ian will check on GNI use.

EFA is interested in FI_MORE and FI_SOURCE_ERR. CXI will support both. FI_SOURCE_ERR has use case with Mercury.

No known use of fi_log_subsys - could drop. OR... add a getinfo subsys to filter/select getinfo output.

FI_LOG_TRACE - currently only used by gni provider. OPX does use FI_LOG_TRACE via FI_DBG_TRACE. Hooking providers use FI_TRACE. Will keep.

fi_param_get_bool() - would like to change, but check C standard. Supported _Bool in C99, must include stdbool.h for bool.

EFA can support rma_iov_count > 1. OPX references the field, but need to check if it's used > 1.

Seems okay to not hold fi_ext.h definitions to backwards compatible ABI/API. ?

EFA isn't using wait sets. Wait sets are used internally. No known uses of Wait Sets by attendees. Can we check?
Also check if poll sets are used by applications. There are better alternatives for wait/poll sets, such as using a single CQ.

[shefty]

Another option is to provide more significant changes. E.g. can we combine the fabric, domain, eq, av, cq into a single object? No plan to merge EPs, support for multiple EPs is required.

Can create a layering API that gives this abstraction for ease of development. This would not provide benefits to the provider implementation, unless the provider can assume apps are using the 2.0 API.

Apps that use separate CQs would need separate domains, which would require registering memory multiple times. Do we have apps that use multiple CQs? Amir believes there's at least 1 app (DDN?) that allocates 1 CQ per EP (connected EP). Check with Sylvain. Sandia openSHMEM may have an app with multiple EPs and CQs - 1 CQ per EP.

Is a single CQ per EP sufficient?

[shefty]

No opinions on wait or poll sets. Is there any use outside of internal use? Mercury obtains the fd and waits on that directly but need to determine where they get the fd from. CQ? wait set? May be okay to remove.

Async AV isn't used and has limited benefit. Usually, communication isn't established until the app tries to send to the peer, so AV insert is just adding data into an array.

There are possible benefits to async MR, and the overhead of keeping the feature seems minimal. To use, we need changes through the underlying SW stack into the kernel.

Can remove separate send/recv CQs if apps aren't using that feature. Results in more difficult progress in the provider.

MPIs do create counters for MPI RMA one-sided calls. For one-sided, MPI uses selective completion and only counters for RMA. Today an EP can be associated with up to 4 counters. Do we need 4? Are there any limits for how counters and CQs are combined? MPI may only be using 1 counter for domain for local completion operations (e.g. RMA write/read completed locally). There are inefficiencies in supported remote write/read counter updates in a provider without native HW support. Can require additional wire protocol to notify peer that a remote operation has occurred and may need to be there just in case the peer using counters. The initiator does not know if remote counters are in us.

[soumagne]

We get the fd from the cq when the provider supports it, we then add the fd to our own epoll set. We also use only one ep / cq / av per domain unless the provider supports scalable endpoints in which case we create different tx/rx contexts and cqs. Do you have any particular plans for scalable endpoints in general btw ? I don't see it mentioned in your list and I'm wondering if that's something we should pursue or not, we currently fallback to separate domains / ep / cq / av when scalable endpoints are not supported so that we keep a single cq / ep per core / thread.

[shefty]

AFAIK, only the sockets provider supports scalable endpoints and it's emulated. Most providers do not. So that would be a feature I would consider removing. Separate from any 2.0 proposal, I'm in favor of isolating threads at the domain level. Most or all providers should be able to support that, and it's something apps could target knowing that they would gain the benefit. Other threading levels end up being difficult to support, and I don't like that apps have to guess which model to develop to.

[shefty]

Updates from Nov 1 ofiwg:

No strong use case for having separate FI_SEND / FI_RECV CQs per EP. This impacts standard EPs only. Scalable EPs should be unaffected.

Should be able to cleanup the AV calls. HPE will check if FI_SYMMETRIC is used. They may have an application use of it.

No comments on removing atomic valid calls. Can check to move implementation into header files, with inline calling the query functions.

Keep FI_AFFINITY

FI_AV_USER_ID - added to shm provider. Is useful for peer providers. Want to keep and implement.

Thread model - keeping FI_THREAD_DOMAIN. Cornelis (thread safe only?) and HPE will check on their models. In many cases, the EP and CQ must be used together to provide lockless operation. Goal is guide apps on model that they can implement toward for lockless operation. Assumption is apps may code for one model, but not multiple. Proposal assumes thread_domain gives the broadest implementation option.

[bsbernd]

Would it be possible to add the libfabric event type and backend (like verbs) event types in struct fi_eq_err_entry? At a minimum it would help log/debug message.

[shefty]

From OFIWG on Feb 21: Most notes captured in slide set. There was extended discussion on whether the mr_mode could be simpler for apps. That was the original intent of BASIC and SCALABLE options, which did not work and had to be replaced. Although some providers may be able to handle a mr_mode bit being set or not (e.g. PROV_KEY), not all will be able to, and they may not be able to support having a mr_mode re-enabled. PROV_KEY is an exception, in that a provider could always just increment a value. Of course, that forces the app to exchange the key. Other bits, such as VIRT_ADDR, become part of the wire protocol and cannot be supported by some providers. Apps will need to continue handling mr_mode bits being set/cleared if they want to support multiple providers. Otherwise, they can fail at fi_getinfo.

[shefty]

libfabric2.pptx

Attaching latest copy of slides presented at ofiwg.

[j-xiong]

2024 Roadmap including 2.0 schedule (from 2/6/2024 OFIWG meeting):

[j-xiong]

Cisco asked to keep usnic.