Not bundle common libraries (e.g. for LLVM and C++) by default #862
Comments
|
The problem with not bundling libstdc++ by default has different flavors, and I am surely not an expert on all of them. What I can say, though, is this: Before #853, PackageCompiler.jl always included the a) the library links against system-local 3rd party library (i.e., not from a JLL artifact) In this case, it could happen that running the PC-compiled library became impossible, since it automatically loads the bundled, older Thus, I think just excluding the C++ library from the bundle by default is not really helpful in many cases, and I'd rather have users deal with a slightly larger bundle than getting weird, hard to understand errors. It could be considered to be added as (yet another) kwarg option to exclude certain libraries from the bundle, at the cost of increasing the complexity of PC. The question to me would be, how much impact is actually achieved by excluding just libstdc++? In my case, for example, the |
|
This could shave 234 MB off the download file of your app (code not common with others), assuming more stripped. More conservative 155+ MB of what can be skipped and I find likely to already be installed on computers. One way all the stripped files could be found, even if not already installed as part of your distro, is if you previously downloaded another Julia app (you would then additionally just need to check cryptographic checksums to see if you could for sure use or copy them). I might need to change the title, it only refers to phase 1, an easy part (that could be opt-in; and useful for proof-of-concept), but phase 2 makes this an even more promising approach, which could be default, and if download is implemented for any one then trivial to extend to other libs. You have Downloads.jl available precompiled, so I assume it doesn't need LLVM nor the C++ standard library. We would need to figure out a place to download from (Julia's infrastructure? or) from Ubuntu (mirrors) or Debian or which I believe works across distros. Stuff would added to your (or julia's) local folder, i.e. sudo not needed unless we want to support a a global option to save space. That could be implemented later, and opt-in or just skipped. libLLVM is largest (and likely will get larger, what would happen if you use (slightly) older version, if such is installed, would it still work, make less efficient native code, besides most code already precompiled, so for a tiny part of your app) and libjulia-codegen.so.1.10.0 is next largest (common to all apps? or not used for), except for the sysimage, sys.so, which is of course largest and no longer the default one, so will not be skipped. libgfortran.so.5.0.0 is maybe not to be relied on installed (nor is used by most), but it and all down to at least libgomp.so.1.0.0 would potentially be installed, and could be downloaded if not, since they are the same anyway, on first use. So this adds up. Potentially the same would also apply for OpenBLAS which is larger (and often not used, when used should also work and would be with some of the following?): I have on my Linux Mint though not quite latest: /snap/julia/83/lib/julia/libopenblas64_.0.3.21.so [not sure what those mean:] -rwxr-xr-x 1 pharaldsson pharaldsson 241767920 ágú 17 06:40 sys.so
I'm not seeing a problem, or possible errors assuming using the download way (or if you worry about no internet, then very non-modern concern, you could always opt into full file, or if you insist have it the default). I think even without internet (or download option) the error doesn't have to be so cryptic, something like fix your internet (or so sudo apt-get install ...). I also think the C++ standard lib part only applies to Linux(?) so that part a limited worry to often more advanced users, that could fix, or are used to... |
PallHaraldsson
changed the title
I'm assuming in most cases at least some version of libstdc++ is installed as a system library (and that you can trust it), and recent enough. Then there's no need to bundle any, just probe for it (in a few places, maybe also at e.g. /snap/core22/864/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30 if not done already).
It's a question what should be done of if no C++ library is found (or too old). This could be a proof-of-concept for other libraries.
Phase 1.
You could simply abort and state what is missing, and maybe instructions that might fix, apt-get rpm etc.
Phase 2:
You could offer or transparently download. apt-get needs sudo to install, but if you don't find it, then you don't have to download and install globally, just for julia or your app, so no priviledge is needed. Without internet, fall back to above error described.
Phase 3:
The same concept works for a lot of other code, OpenBLAS, that many may not need anyway. I know people can strip if they know it's not needed, but then they must be sure or test well. It might reduce anxiety to know it just works. Stripping it could be the default (or still opt-in? but both would work).
How large is the version we actually bundle? Checking locally it seemed to be 18 MB up from 13, so I was surprised by:
@sloede
and:
@staticfloat
If only 2.5 MB we could get away with then not as needed, and while I'm curious why "the compiler libraries are special", with downloading not as critical to get the "same treatment".
man strip
I may try, is there a reason some or most users couldn't do that, and it would work for the .so you get (from PJ.jl)? Also what happens if you simply drop the .so (and LLVM, where that applies)?
The text was updated successfully, but these errors were encountered: