Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prov/cxi performance regression in fi_pingpong #9802

Open
philippfriese opened this issue Feb 9, 2024 · 5 comments
Open

prov/cxi performance regression in fi_pingpong #9802

philippfriese opened this issue Feb 9, 2024 · 5 comments

Comments

@philippfriese
Copy link

Describe the bug
Using the upstreamed CXI provider (as of commit fc869ae main branch) yields reduced throughput in fi_pingpong (14GB/s for ofiwg/libfabric compared to 20GB/s for HPE-internal libfabric).

To Reproduce
Steps to reproduce the behavior:

  • Launch fi_pingpong -p cxi -e rdm on two Slingshot-connected nodes.
  • Observe performance deviation between ofiwg/libfabric and HPE-internal libfabric

Expected behavior
Equivalent performance between both libfabric-variants (~20GB/s).

Output
Deviating performance:

  • ~14GB/s for ofiwg/libfabric
  • ~20GB/s for hpe/libfabric

It is worth noting that the observed throughput of ofiwg/libfabric can be increased by setting the number of iterations from the default 10 to 100 via -I 100.
Additionally, using osu_bw and osu_latency from the OSU Microbenchmark Suite, no performance differences are observed between the two libfabric variants.

I've attached raw output of the fi_pingpong runs and osu_bw/osu_latency runs.

Environment:

  • ofiwg/libfabric at commit fc869ae main branch
  • ofiwg/libfabric configure setup: ./configure LDFLAGS=-Wl,--build-id --enable-cxi=yes --enable-only --enable-restricted-dl --enable-tcp --enable-udp --enable-rxm --enable-rxd --enable-hook_debug --enable-hook_hmem --enable-dmabuf_peer_mem --enable-verbs --enable-gdrcopy-dlopen --enable-profile=dl
  • hpe/libfabric version 1.15
  • OpenMPI 4.1.6 with --with-ofi=yes
  • OSU Microbenchmark: 7.3
  • openSUSE Leap 15.5, kernel 5.14.21
  • aarch64

Additional context
Due to a currently unresolved issue with the local Slingshot deployment on the used ARM platform, it is required to set FI_CXI_LLRING_MODE=never for both fi_pingpong and osu_bw.

@SSSSeb
Copy link

SSSSeb commented Feb 9, 2024

ping @mindstorm38

@mindstorm38
Copy link

mindstorm38 commented Feb 9, 2024

I can't reproduce the regression, here's my environment:

  • HPE/libfabric v1.20.1
  • OFI/libfabric: v1.21.0a1 (your commit)
  • Iterations: 100
  • Aarch64 (also works with x86_64)
  • With FI_CXI_LLRING_MODE=never
  • Both are manually built from sources

Not yet tested with MPI

@lflis
Copy link

lflis commented Feb 9, 2024

@mindstorm38
Which slingshot libraries version are you using?

@mindstorm38
Copy link

I'm using the latest internal sources, I don't know the version number to be honest. I configure cxi, cassini headers and UAPI headers to directly point to the sources. Please tell me if you have any command to check a version that would be interesting for you, but note that my installation is not standard compared to official SlingShot packages, I'm working in parallel on a packages-based installation but it's on x86_64 so this will not be helpful in this case I guess (I'll try anyway).

@vanderwb
Copy link

FWIW - I've replicated this result on an x86_64 platform, with pretty much the exact same pingpong bandwidth numbers and the same result if I increase the iterations to 100. We are still running Slingshot 2.1, so maybe things work better with the newly released Slingshot 2.2. But in any case, the HPE 1.15 is performing better than the built-from-source 1.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants