mpifort 5.0.1 fails with undefined refs to memcpy@GLIBC_2.14 and clock_gettime@GLIBC_2.17 #143
Comments
|
Thanks for the report Charles! 🙏 This is likely because this feedstock is now building with a CentOS 7 image on That said, the packages should carry this constraint to ensure that they are only installed on systems with a new enough GLIBC. However the GLIBC constraint wasn't being included before. PR ( #145 ) should fix that An open question is whether older GLIBC are still of interest to maintain support for here. Will defer to the feedstock maintainers on that question |
|
Actually sorry I misspoke, looking at the info provided above can see It's possible adding |
|
Thanks Jack. Yes, this was observed on an up-to-date Gentoo Linux system with current glibc 2.38. I have not previously had any issues with conda-forge packages on this system. Adding sysroot to the build deps sounds like the right solution (probably should be used by default for all packages, IMO) |
|
The Packages are building and should be uploaded soon Please look for a |
|
I'm afraid that the problem persists in both OpenMPI 5.0.1 and 5.0.2 as packaged by conda-forge. The last usable version is 5.0.0 The same failure is observed with both a very recent system (Gentoo linux with glibc 2.39) and and older system (CentOS7 with glibc 2.17) Both complain about 'memcpy@GLIBC_2.14' and 'clock_gettime@GLIBC_2.17'. (These versioned symbols sure are a pain). |
|
That last comment was from me, I was logged in under a different GitHub account without realizing. One question I have is - what changed between OpenMPI 5.0.0 and 5.0.1? Can we just go back to the way 5.0.0 was getting built? That version does not suffer from the portability problems. |
|
The most likely problem is that the the 5.0.1 image was built in a different, newer docker image. You should probably as the core conda-forge team. This is ultimately not an Open MPI issue or the fault of this feedstock (although I could be partially wrong). |
|
Thanks dalcinl. How do I bring this to the attention of the core conda-forge team? |
|
I usually contact the team via Gitter https://conda-forge.org/community/getting-in-touch/#gitter-and-element |
|
@jakirkham Do you think we are somehow messing things up in this feedstock? |
|
The issue they are seeing is they are on newer systems (GLIBC 2.28+) and are having trouble resolving symbols that should be available on their systems as we built on (GLIBC 2.17+) IOW the symbols should be available in their cases, but for some reason they are not The bug may very well be in our build, but am a little fuzzy on how it is occurring |
|
Could one of you seeing this error please trying installing |
|
Tried adding the GLIBC constraint to those packages directly ( #147 ). Maybe that helps? |
|
@jakirkham |
|
It also works with |
|
Thanks Charles! 🙏 Yeah that's what I was wondering about Then I think we should try PR: #147 |
|
New packages are building. Will probably be a bit before they upload and mirror to CDN Please test out tomorrow and let us know how it goes |
|
It's still broken, or else I'm not seeing new packages. Testing this should be very easy. Just do bash$ mamba create -n test; mamba activate test
(test)$ mamba install openmpi gfortran
Package Version Build Channel Size
──────────────────────────────────────────────────────────────────────────────────────
Install:
──────────────────────────────────────────────────────────────────────────────────────
+ mpi 1.0 openmpi conda-forge Cached
+ _libgcc_mutex 0.1 conda_forge conda-forge Cached
+ libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge Cached
+ ld_impl_linux-64 2.40 h41732ed_0 conda-forge Cached
+ ca-certificates 2024.2.2 hbcca054_0 conda-forge Cached
+ libgomp 13.2.0 h807b86a_5 conda-forge Cached
+ _openmp_mutex 4.5 2_gnu conda-forge Cached
+ libgcc-ng 13.2.0 h807b86a_5 conda-forge Cached
+ libiconv 1.17 hd590300_2 conda-forge Cached
+ libsanitizer 13.2.0 h7e041cc_5 conda-forge Cached
+ openssl 3.2.1 hd590300_1 conda-forge Cached
+ icu 73.2 h59595ed_0 conda-forge Cached
+ xz 5.2.6 h166bdaf_0 conda-forge Cached
+ libzlib 1.2.13 hd590300_5 conda-forge Cached
+ libgfortran5 13.2.0 ha4646dd_5 conda-forge Cached
+ libnl 3.9.0 hd590300_0 conda-forge Cached
+ libevent 2.1.12 hf998b51_1 conda-forge Cached
+ libxml2 2.12.6 h232c23b_1 conda-forge Cached
+ libgfortran-ng 13.2.0 h69a702a_5 conda-forge Cached
+ libhwloc 2.9.3 default_h554bfaf_1009 conda-forge Cached
+ openmpi 5.0.3 h7fc1de5_100 conda-forge 15MB
+ libgcc-devel_linux-64 13.2.0 ha9c7c90_105 conda-forge Cached
+ kernel-headers_linux-64 2.6.32 he073ed8_17 conda-forge Cached
+ sysroot_linux-64 2.12 he073ed8_17 conda-forge Cached
+ binutils_impl_linux-64 2.40 hf600244_0 conda-forge Cached
+ gcc_impl_linux-64 13.2.0 h338b0a0_5 conda-forge Cached
+ gfortran_impl_linux-64 13.2.0 h76e1118_5 conda-forge Cached
+ gcc 13.2.0 hd6cf55c_3 conda-forge Cached
+ gfortran 13.2.0 h98b45c4_3 conda-forge Cached
....
(test)$ mpifort /tmp/test_mpi.f90
/home/cgw/miniforge3/envs/test/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/cgw/miniforge3/envs/test/lib/libmpi_mpifh.so: undefined reference to `memcpy@GLIBC_2.14'
/home/cgw/miniforge3/envs/test/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/cgw/miniforge3/envs/test/lib/./libpmix.so.2: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status |
|
|
Could you show the full output? I don't see the |
|
That is the full package list, the package is called |
|
|
Sorry I meant |
|
What? bash$ which mpifort
which: no mpifort in (/home/cgw/Applications/.bin:/home/cgw/miniforge3/condabin:/home/cgw/bin:/home/cgw/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/18/bin:/usr/lib/llvm/17/bin)
bash$ mamba activate test
(test)$ which mpifort
/home/cgw/miniforge3/envs/test/bin/mpifortI have been using |
|
@leofang What is the purpose of the |
The compiler wrapper packages It appears that when using CUDA 11 to build (which is what this feedstock does) we're pinned at gfortran 11: What would be the reason that you want to ignore the ABI compatibility issue and use gfortran 13? |
|
Thanks for the reply. It's not so much that I want to ignore ABI compatibility, but that I've been maintaining a Conda package that uses OpenMPI and gfortran for several years, and we have used the |
|
Just an update that this is also an issue with clean installs of Linux Mint 21.3 (based on Ubuntu 24.04), using the most recent versions of openmpi and gcc/gfortran (version 13), as noted above for other versions. Generally it would be nice to be able to use this package with up-to-date compilers, so that packages and libraries we distribute which use openmpi can be used on systems with the latest compilers (especially seeing as how openmpi applications often find themselves on HPC clusters, where users often don't necessarily have a ton of control over compiler versions and sysadmins may be reluctant to have many installed versions of compilers). I would like to second the above question; is the current advice that this package should not be used, and instead only ever openmpi-mpifort? At least until this versioned symbols issue is fixed? |
|
(This reply is a lot shorter if you ignore the code blocks and read the text in between, I'm just trying to create a clear reproducer. Or scroll to the bottom for the conclusion.) ReproducingI'm running into this same issue (albeit in C, but it's identical): installing This installs an and it installs a Since we don't have a Conda-installed glibc, the system one is used: and since that is new enough, the dependencies can be resolved and everything works (note that this uses Installing and this breaks things: I can confirm that installing and now we have older symbols that work with If I specifically ask for the latest Indeed if we try to compile: Why it's still broken (?)Despite #142 and #145, it seems that no released package ever had a dependency on Potential solution?It seems to me that depending on That suggests that this issue should be resolved elsewhere, but I don't know enough about how conda-forge works to know where to take it. Can someone give a hint? |
|
Maybe this is actually some sort of build issue? If the package had been built with the proper sysroot installed in the build environment, then the binaries would not end up using newer symbols. Are we somehow missing something in our recipe to constrain the build-time sysroot? Or is really adding a |
|
Interesting question. Does conda-forge have a policy on which minimum glibc it supports? Then packages should adhere to that, and then the Docker image should provide that exact glibc to link against, either natively or via a sysroot package. Relying on each and every package maintainer to read that policy and adjust their package definition accordingly doesn't sound like a working strategy... Glibc 2.12 was released in 2010, and 2.17 in 2012, so even enterprise Linux distributions should have at least 2.17 by now and requiring it doesn't seem all that controversial to me. So perhaps another question is why |
|
Ah, maybe the answer to that second question is that this causes packages to be built against glibc 2.12, thus making them compatible with everything and avoiding the problem we're seeing? But the |
|
And looking at the build configuration, this package explicitly builds against glibc 2.17, first by explicitly having sysroot 2.17 as a build dependency, and after this commit in Searching for |
Solution to issue cannot be found in the documentation.
Issue
I am using cmake, gfortran, and openmpi from conda-forge to compile a Fortran package. With cmake 3.28.3, gfortran 13.2.0 and openmpi 5.0.0 everything works. When openmpi upgraded to the latest 5.0.1 version I started getting this error:
Installed packages
Environment info
The text was updated successfully, but these errors were encountered: