Missing libcholmod when building apps with filter_stdlibs = false after updating PackageCompiler

[cihga39871]

Hi,

I found some libs are missing when building apps, and it is because create_app() does not copy the libs to the new app's lib/julia folder. I did not have the same problem on Nov 10 2023, but after updating the dependencies yesterday, the following error message was shown:

┌ Error: Error during initialization of module CHOLMOD
│   exception =
│    could not load library "libcholmod"
│    libcholmod.so: cannot open shared object file: No such file or directory
│    Stacktrace:
│     [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
│       @ Base.Libc.Libdl ./libdl.jl:117
│     [2] dlopen (repeats 2 times)
│       @ ./libdl.jl:116 [inlined]
│     [3] __init__()
│       @ SuiteSparse.CHOLMOD ~/projects/app/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:161
└ @ SuiteSparse.CHOLMOD /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:245

Environment

Julia version v1.8.5 and v1.10.0 were tested, and all have the same problem.

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 32 × 13th Gen Intel(R) Core(TM) i9-13900K
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, goldmont)
  Threads: 1 on 32 virtual cores

I manually fix the issue by copying libs from julia to my app after building app:

# check if atria is successfully installed
# eg: fix libs that is not copyed to lib path
new_app_exe = joinpath(app_path, "bin", "atria")
this_bin_dir = unsafe_string(Base.JLOptions().julia_bindir)
this_lib_dir = abspath(this_bin_dir, "..", "lib", "julia")
dest_lib_dir = abspath(app_path, "lib", "julia")

function copy_missing_lib()
    if !isdir(this_lib_dir)
        return nothing # skip checking
    end

    buffer = IOBuffer()
    run(pipeline(`$new_app_exe --version`, stderr=buffer, stdout=buffer))
    res = String(take!(buffer))

    if occursin("Error during initialization of module", res)
        m = match(r"([^ \n]*\.(so|dylib|dll)([\.0-9]*)?): cannot open shared object file", res)
        if isnothing(m)
            return nothing
        end
        lib = joinpath(this_lib_dir, m.captures[1])
        dest_lib = joinpath(dest_lib_dir, m.captures[1])
        if isfile(lib) && !isfile(dest_lib)
            @info "Copying $(m.captures[1])"
            cp(lib, dest_lib, follow_symlinks=true)
            copy_missing_lib()
        end
    end
end
copy_missing_lib()

Working example

The working example is https://github.com/cihga39871/Atria

git clone https://github.com/cihga39871/Atria.git

cd Atria
julia ./build_atria.jl

[KristofferC]

Strange, according to

PackageCompiler.jl/src/library_selection.jl

Line 11 in 8cd96a1

"SuiteSparse_jll" => ["libamd", "libbtf", "libcamd", "libccolamd", "libcholmod", "libcolamd", "libklu", "libldl", "librbio", "libspqr", "libsuitesparseconfig", "libumfpack"],

it should get copied if SuiteSparse_jll is in the environment (which it is for you).

[cihga39871]

Actually, my project does not need SuiteSparse, and SuiteSparse is not in Project.toml or Manifest.toml.

[KristofferC]

Okay, but in that case filter_stdlibs should have filtered it out from the sysimage so then why does

│     [3] __init__()
│       @ SuiteSparse.CHOLMOD ~/projects/app/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:161

run...

[KristofferC]

Oh, it is with filter_stdlibs=false...

[sjkelly]

I was seeing this in the CI for #929 on <1.10 as well, but not on 1.10.

Though it appears fixed in: 15b3656
But there it now seems to be an un-caught error:
https://github.com/JuliaLang/PackageCompiler.jl/actions/runs/8329025921/job/22790369630?pr=929#step:5:434