Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy downloading artifacts with custom sysimage #639

Open
tomyun opened this issue Nov 20, 2021 · 8 comments · May be fixed by #673
Open

Lazy downloading artifacts with custom sysimage #639

tomyun opened this issue Nov 20, 2021 · 8 comments · May be fixed by #673

Comments

@tomyun
Copy link

tomyun commented Nov 20, 2021

Some images created by create_sysimage() tries downloading artifacts after it's loaded first time.

$ julia
julia> using PackageCompiler

julia> create_sysimage([:Gadfly], sysimage_path="gadfly.dylib")
✔ [01m:48s] PackageCompiler: compiling incremental system image

$ julia -Jgadfly.dylib
 Downloading artifact: MKL
    Downloading [================>                        ]  39.2 %

I believe this issue started happening since recent versions of PackageCompiler, presumably from 1.6 or 1.7. I'm not entirely sure if there was any relevant change in the said package, i.e. MKL.jl, triggering this issue.

I saw include_lazy_artifacts option was recently added to create_app() and create_library(), but there seems to be no equivalent option for create_sysimage().

julia> versioninfo()
Julia Version 1.6.4
Commit 35f0c911f4 (2021-11-19 03:54 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

PackageCompiler.jl used in the example above was v2.0.2.

@KristofferC
Copy link
Sponsor Member

Hmm, but create_sysimage never bundled any artifacts?

@tomyun
Copy link
Author

tomyun commented Nov 20, 2021

Then maybe it's happening after JuliaLinearAlgebra/MKL.jl#78 I guess. Now I have to explicitly add MKL.jl as a dependency to avoid lazy downloading and wonder if there is a better way to handle this, i.e. include_lazy_artifacts for create_sysimage().

@simonbyrne
Copy link
Collaborator

simonbyrne commented Dec 1, 2021

Ah, I just hit this: basically the problem is FFTW.jl. It has MKL_jll (for which the artifacts are lazy) in its Project.toml, but doesn't load it by default (lazy artifact downloads are triggered by __init__()). It is only loaded if you change the preferences.

However if you build FFTW.jl into your system image, it will also add MKL_jll. Since all system image __init__() statements are triggered at julia_init, starting Julia is sufficient to trigger the artifact download (even though it will never actually be used).

@simonbyrne
Copy link
Collaborator

The solution appears to be to use include_transitive_dependencies=false, which will avoid pulling in MKL_jll if it isn't actually loaded.

@simonbyrne
Copy link
Collaborator

Perhaps it is worth changing this to be the default?

@zhubonan
Copy link
Contributor

I think include_transitive_dependencies=false might have some side effects. It can be useful to allow passing an array of packages that should be excluded perhaps.

@TheCedarPrince
Copy link
Member

I had this issue as well.
My only issue is I do not know if I can ignore installing the MKL dependency.
Do we know how to determine if this can be ignored or not?

@zhubonan
Copy link
Contributor

zhubonan commented Apr 4, 2022

I had this issue as well.
My only issue is I do not know if I can ignore installing the MKL dependency.
Do we know how to determine if this can be ignored or not?

It depends on what you want to achieve and what the MKL_jll is used for. If you want a system image then it is harmless as wether including MKL or not in the system image should have minimum impact on the reduction of compilation delay (since MKL_jll wraps compiled MKL libraries).

If you are making an App then have the MKL_jll missing may cause problems, as a standalone App should include all dependencies in the system image. Somehow I could not get lazy aretefact download to work for App (it errors after the download, may be another issue to flag?), even it does work, it is a massive pain becuase MKL_jll aretefacts are huge.

However, if the reason of depending on MKL_jll is becuase of FFTW.jl, this is not a problem because FFTW.jl actually defaults to use FFTW_jll, and for an App the user should not switch backend anyway. In this case, my solution was to make a fork of FFTW.jl without MKL_jll as dependency, and include this forked version in the App instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants