GNU Guix without root access on HPC

Introduction

GNU Guix is designed to be installed in gnu/store + hash. The ‘hard-coded’ path has a reason - it means we can store paths inside binaries and create reproducible build. If apache depends on an ssl library, the link is fully referenced in the apache binary. There is no way to circumvent the pointer to the library. Also the hard paths mean they can be shared between systems, which makes trusted binary deployment possible.

For those on systems that have no root access, i.e., who can not create /gnu/store this is a real bummer. If your administrators are enlightened they may create /gnu/store for you and allow you to copy files across built on a system where you have root access. This is a solution on several HPC systems out there already and does not require a running Guix daemon.

But still, there are those who won’t even allow you to do that. For those environments the only solution is to build Guix on a relative path. It can be done, but it has downsides: these binary deployments are not the same as the main distribution and may behave differently. The Guix community will find it hard to help you with your specific setup.

One intermediate solution is to rewrite binary paths, as I (pjotr) did successfully with ldc. It has the advantage of using actual Guix built (reproducible) binaries but requires some hacking. Maybe I’ll get around to writing something generic one day.

Proot is a user land tool that runs on most Linux kernels. It intercepts path names and convert them on the fly. So you can tell it to convert everything in $HOME/opt to / (root directory). I.e., create a virtual root. The downside of using proot is that it comes at a performance price.

This document aims to find a reproducible way to bootstrap Guix in proot and get a recent Guix daemon running. The goal is to install software in a special directory that runs without proot. That way we can deploy Guix packages without ever having administrator privileges. Something that is wished for in HPC environments.

The main tricks are:

Install proot (there are many ways and it should not be hard)
Download and run the Guix binary tarball inside proot
Run the guix-daemon in proot without root privileges
Set the ACL to be able to download pre-built binaries
Install the hello package (test)
Install the build packages required for building guix
Build the source tree inside proot
Start the updated guix-daemon and tell it to use a different store relative to $HOME
Start building packages relative to $HOME
Run software packages without proot(!)

All without root access to your HPC. Note that much of this can be avoided if you actually have a build host (somewhere) with root privileges - even a laptop - as ‘guix pack’ allows you to copy files across to a different host. As long as the store path is the same.

Install proot

Proot can rewrite paths transparently so your home directory looks like a root directory (where we create /gnu/store). The first thing to do is install proot on your build system. This can be done with compilation from source. If you happen to have Guix running somewhere you can move proot across:

scp $(guix build proot-static)/bin/proot remote:

where remote is the non-root machine name.

Create a directory to install Guix, e.g.

mkdir ~/guix-no-root

Install the Guix binary installation

Fetch the binary tarball and install in your directory, e.g.,

cd ~/guix-no-root
wget https://alpha.gnu.org/gnu/guix/guix-binary-0.13.0.x86_64-linux.tar.xz
tar --warning=no-timestamp -xf  guix-binary-0.13.0.x86_64-linux.tar.xz

now you can remove the tar ball

rm guix-binary-0.13.0.x86_64-linux.tar.xz

Run proot

At this point you should be able to locate bash in the store and, for example, do

proot -r . gnu/store/02426nwiy32cscm4h83729vn5ws1gs2i-bash-static-4.4.12/bin/bash

the ‘-r .’ fakes the new root ‘/’ directory. type ‘exit’ to leave again.

Note the 02426nwiy32cscm4h83729vn5ws1gs2i hash path for bin/bash may differ on your system - this will happen over time with Guix distributions getting updated. Same for all hashes listed below. Just look inside your store with a command like

ls gnu/store/*/bin/bash

to fish out the paths.

Run guix daemon in proot

Open a new terminal window and start the guix daemon

proot -r '.' -b /proc -b /dev -b /etc gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix-daemon

and run guix in the another terminal

proot -r '.' -b /proc/ gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix package -A hello

should list packages and an install command

proot -r '.' -b /proc/ gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix package -i hello

should show something like this in the daemon terminal:

accepted connection from pid 25742, user 502

So, far so good, but the build fails with

guix package: error: symlink: No such file or directory: "/home/user/.guix-profile"

this is because in the new proot there is no such directory. So, create it. The build system also need /tmp

mkdir -p /home/user /tmp

The build daemon will still complain:

 ERROR: In procedure open-file: Permission denied: "/etc/guix/acl"

but that does not prevent guix from downloading source packages. We would like binary downloads, to import an existing acl file with something like

mkdir etc_guix
cp gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/share/guix/hydra.gnu.org.pub etc_guix/acl

and modify it to look like:

(acl
 (entry
  (public-key
   (rsa
    (n #00DB1634E3D9DFAC97AE4734DAE968CCB15EE4815C82BDC254883DBB49FE1EF32268E82D4BBE0E35298C481C9DA1551642FAFF05AEC1A60712F1BB4BE7D25D7EFF7A4F89704A5A9AC232870CB9F2476C3B538A0E990A8825DEB73081D317001FB8A188600F2FEF5F5F570E857F3EE4355077A3C3918ED72723A56BA55C466D400658974D7DAD1F6B7B63C192B9C2704D98BBFF1C3BD5B8EF11A8ADC83ACB8FD8E9F1E792FDAD262415D13F2DEE55F330908CFDA9C3C8C32B64F7DD088457D34F445E2E2C83C6D680549DC9B6E6573B89496567204ED285E67A279F2F667080BA941D80D015CE87B0FB6A91A99CECC7D91D2D210B00E4B6E611DA51DB008F1DFE3FCAC6B27393FA781D45F9A15FC7B8785A3E86BA6592B2916CA22CF1E40FC85F85CACA590461154F58F3580B16398908EF32076F411299C28727C94D88B6A618F84DD73AEBED8270BCB6690928CB1BF250C35E1F6BF3B1B30D05BA246ECE8F69D9065DE26F4B3E0D814D70A9C27CB5B7B050C9090590D3A9EF83374F2643E5446FBD39DDB124DBF6DFDAA6D18E2560AD0CBFA11C959C9B7316BF19963A191967054E9FD97DC14D71082B30B1C90A46E8996682474C3BCB51BA0882958897B6DD35E41B5174D0A6BCDE97B89043E95BD1B70DE61DA666893B417196A180005466BC3A742FDF04E89B04460E3E6BC72E7F1B5FEA5B3092FEE551A3C447C12E104E65#)
    (e #010001#)
    )
  )
 (tag
  (guix import)
  )
 )

restart the daemon with

proot -0 -b /proc -b /dev -b /etc -r . -b etc_guix/acl:/etc/guix/acl gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix-daemon --disable-chroot

(note the extra -0 and chroot switches) and you should see on a guix package install

substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%

which makes for faster installs! Btw ignore the –build-users-group warning switch here it has no real use in proot.

Now we have the guix-daemon running we can install a package

Install and run the hello package

proot -b /proc/ -r . gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix package -i hello

After installing the hello package we can run it with proot. First locate the file:

find gnu/store/ -name hello
gnu/store/prvjrj6i8cz7rvkgm643h952bv2daj1h-hello-2.10/bin/hello

and run it

proot gnu/store/prvjrj6i8cz7rvkgm643h952bv2daj1h-hello-2.10/bin/hello
Hello, world!

this has also been added to our profile in home/user/.guix-profile, so we can run it as

proot home/user/.guix-profile/bin/hello

isn’t that the coolest thing!

Compile Guix from source

So far, we bootstrapped Guix from its binary installation without root privileges. Next step is to create a reproducible build system using proot which will allow us to build packages that can be run without proot. That is necessary because proot introduces system overheads - especially when reading files. We care about that when we want to run software in HPC environments.

The following is pretty similar to what is desribed in INSTALL.org in ‘Building GNU Guix from source (using Guix)’. With proot we install all build tools in our working directory:

proot -r '.' -b /proc/ gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix package -i guix help2man git strace pkg-config less vim binutils coreutils grep gawk ld-wrapper --no-grafts

make a note of the provided shell paths. In my case

  export GIT_EXEC_PATH="/home/user/.guix-profile/libexec/git-core"
  export GUILE_LOAD_PATH="/home/user/.guix-profile/share/guile/site/2.2${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
  export GUILE_LOAD_COMPILED_PATH="/home/user/.guix-profile/lib/guile/2.2/site-ccache:/home/user/.guix-profile/share/guile/site/2.2${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_LOAD_COMPILED_PATH"

we create a full blown guix environment for building with

proot -b /proc/ -r . gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix environment guix --ad-hoc help2man git strace pkg-config less vim binutils coreutils grep --no-grafts --bash

it will end with a

guix environment: error: execlp: No such file or directory

(it would be nice we got a shell, but no matter) after installing all packages we take a clean bash

env -i proot -b /proc -b /dev -r '.' gnu/store/k7029k5va68lkapbzcycdzj7m5bjb4b8-bash-4.4.12/bin/bash --login --noprofile --norc
bash-4.4$

paste above shell paths or pick up the profile environment

. home/user/.guix-profile/etc/profile

if needed, set the path to the profile

export PATH=/home/user/.guix-profile/bin

and you should be able to do

git --version
guile --version
guix --version
gcc --version

if files are missing, simply install them with guix (we no longer depend on the underlying Linux distribution). For some reason running guix inside proot got me warn-about-old-distro, so I had to do that outside the proot shell. The full thing is

proot -r '.' -b /proc/ gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix package -i \
  autoconf automake bzip2 gcc-toolchain gettext \
  guile libgcrypt pkg-config sqlite m4 make \
  gnutls guile-json gawk ld-wrapper

when that works we clone the guix source tree (outside proot) to get the latest and greatest

git clone http://git.sv.gnu.org/r/guix.git

and run the proot bash again, setting the profile and

proot -b /proc -b /dev -b /etc -b /bin -b /lib -b /lib64 -r . gnu/store/vir3lrwqy50pr8fkaf3m091dgbrja2n6-guix-0.13.0/bin/guix environment guix --ad-hoc help2man git strace pkg-config less vim binutils coreutils grep gawk --no-grafts -- bash

once bootstrapped it actually may be enough to run

env -i proot -b /proc -b /dev -b /etc -b /bin -b /lib -b /lib64 -r '.' gnu/store/k7029k5va68lkapbzcycdzj7m5bjb4b8-bash-4.4.12/bin/bash --noprofile --norc --login

. /home/user/.guix-profile/etc/profile

export PATH=$PATH:/usr/bin:/bin
cd guix
./bootstrap
./configure --localstatedir=/var
time make -j 8 # for 8 cores though guile manages itself
  real    11m26.352s
  user    37m52.192s
  sys     0m50.012s

and you should have a fresh Guix (the build is slow though, much slower than without proot).

Now you should be able to run a very recent guix

./pre-inst-env guix --version

to update

./pre-inst-env guix package -i guix

got a conflict, so I create a new profile

mkdir ~/opt
./pre-inst-env guix package -p ~/opt/guix-latest -i guix --no-grafts

Now we should have the latest guix-daemon set up! In the second terminal again rerun it with

proot -0 -b /proc -b /dev -b /etc -r . -b etc_guix/acl:/etc/guix/acl home/user/opt/guix-latest//bin/guix-daemon --disable-chroot

Building packages to run without proot

But wait! Now we can start building new stuff ready for non-root! All we need to do is tell the daemon what store to target. To do that rebuild GNU Guix from source, but this time tell configure –with-store-dir=PATH and –localstatedir=DIR. The first defaults to /gnu/store and the second to /var/guix. So, if we target $HOME/opt we can do

rm -rf autom4te.cache/ # to be sure
echo "Installing to $HOME/opt"
./configure --localstatedir=$HOME/opt/var --with-store-dir=$HOME/opt/store --prefix=$HOME/opt/local
make clean-go
make clean
time make -j 8

Note that there exist limitations on the size of the store path for shell scripts - so, don’t go very deep. Note also that $HOME should point to a path that is valid both on your system and in proot (check it!)

We also need to build the daemon and install guix in

make guix-daemon
make install

Note that guix has been installed in $PREFIX/bin this time (also inside proot). Now we need to run the new guix and daemon.

In both terminals running Guix daemon and client you need to set both

NIX_STORE_DIR=$HOME/opt/store
NIX_STATE_DIR=$HOME/opt/var

as Roel documented. Restart the new daemon with settings:

env NIX_STORE_DIR=$HOME/opt/store NIX_STATE_DIR=$HOME/opt/var proot  -b /proc -b /dev -b /et-b /lib -r . -b etc_guix/acl:/etc/guix/acl guix/guix-daemon --disable-chroot

and inside the proot shell you can do

env NIX_STORE_DIR=/home/user/opt/store NIX_STATE_DIR=/home/user/opt/var  ./pre-inst-env guix package -i hello

or run it directly with proot:

env NIX_STORE_DIR=$HOME/opt/store NIX_STATE_DIR=$HOME/opt/var proot -r '.' -b /proc opt/local/bin/guix package -i hello --no-grafts --no-substitutes

and install at leisure. Note that, at this point, the download script needs a fix. The first hash bang needs to point to a bash in the store (see below).

Note also you can build in parallel and continue with guix package switches -c 8 -M 8 -k. Use them and use your cores!

env NIX_STORE_DIR=/home/user/opt/store NIX_STATE_DIR=/home/user/opt/var  proot -r '.' -b /proc opt/local/bin/guix package -i hello -c 8 -M 8 -k --no-grafts --no-substitutes

With those packages build in $HOME/opt/store we no longer require proot to run them. For example I built gcc

ldd store/75d52cyi8qd3cphz5am1hwp5anjihmpp-gcc-5.4.0/bin/gcc
        linux-vdso.so.1 (0x00007ffce6b1c000)
        libm.so.6 => /home/user/opt/store/k3q0r9lp2gfbkw04iy4n25vxsrfchz1f-glibc-2.25/lib/libm.so.6 (0x00007fe097e34000)
        libc.so.6 => /home/user/opt/store/k3q0r9lp2gfbkw04iy4n25vxsrfchz1f-glibc-2.25/lib/libc.so.6 (0x00007fe097a95000)
        /home/user/opt/store/k3q0r9lp2gfbkw04iy4n25vxsrfchz1f-glibc-2.25/lib/ld-linux-x86-64.so.2 (0x00007fe098146000)

and it runs without proot inside my $HOME (you need to symlink the store to a working path)

ln -s ~/guix-no-root/gnu/store ~/opt/store

./store/75d52cyi8qd3cphz5am1hwp5anjihmpp-gcc-5.4.0/bin/gcc --version
gcc (GCC) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.

I’ll show a more recent version when I get there.

Final commands

For completeness. In the final setup, after compiling and installing guix, I run the guix daemon and guix as a normal user with proot:

env NIX_STORE_DIR=$HOME/opt/store NIX_STATE_DIR=$HOME/opt/var \
  proot  -b /proc -b /dev -b /etc -b /lib -r . \
  -b etc_guix/acl:/etc/guix/acl opt/local/bin/guix-daemon --disable-chroot

env NIX_STORE_DIR=$HOME/opt/store NIX_STATE_DIR=$HOME/opt/var \
   proot -r '.' -b /proc opt/local/bin/guix package -i hello \
   -c 8 -M 8 --no-grafts --no-substitutes

the software that gets compiled runs without proot. In this case, for example

 ~/opt/store/vl87lyk8vpk7h1ifawfa39g0kzwcmb69-gzip-1.8/bin/gzip --version
gzip 1.8
Copyright (C) 2016 Free Software Foundation, Inc.

Disabling tests

GNU Guix builds with tests switched on by default. When building a system from source this can be very annoying - some packages have extremely long runnnnnnnning tests. You can patch Guix to skip testing (note this is heresy and you’ll never find it endorsed or an option added, but if someone has to come clean it might as well be me!). The following disables testing in a number of build systems, you’ll get the idea.

diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
index 1786e2e3c..2aff344df 100644
--- a/guix/build/gnu-build-system.scm
+++ b/guix/build/gnu-build-system.scm
@@ -286,7 +286,7 @@ makefiles."
 (define* (check #:key target (make-flags '()) (tests? (not target))
                 (test-target "check") (parallel-tests? #t)
                 #:allow-other-keys)
-  (if tests?
+  (if #f
       (zero? (apply system* "make" test-target
                     `(,@(if parallel-tests?
                             `("-j" ,(number->string (parallel-job-count)))
diff --git a/guix/build/perl-build-system.scm b/guix/build/perl-build-system.scm
index b2024e440..8008a7173 100644
--- a/guix/build/perl-build-system.scm
+++ b/guix/build/perl-build-system.scm
@@ -63,7 +63,7 @@
 (define-w/gnu-fallback* (check #:key target
                                (tests? (not target)) (test-flags '())
                                #:allow-other-keys)
-  (if tests?
+  (if #f
       (zero? (apply system* "./Build" "test" test-flags))
       (begin
         (format #t "test suite not run~%")
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index dd07986b9..dacf58110 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -131,7 +131,7 @@

 (define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)
   "Run the test suite of a given Python package."
-  (if tests?
+  (if #f
       ;; Running `setup.py test` creates an additional .egg-info directory in
       ;; build/lib in some cases, e.g. if the source is in a sub-directory
       ;; (given with `package_dir`). This will by copied to the output, too,
diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-system.scm
index c2d276627..2f12a4362 100644
--- a/guix/build/ruby-build-system.scm
+++ b/guix/build/ruby-build-system.scm
@@ -116,7 +116,7 @@ generate the files list."
 (define* (check #:key tests? test-target #:allow-other-keys)
   "Run the gem's test suite rake task TEST-TARGET.  Skip the tests if TESTS?
 is #f."
-  (if tests?
+  (if #f
       (zero? (system* "rake" test-target))
       #t))

Warning

WARNING: it is really easy to mix up running daemons on one machine. Always make sure the right daemon is running before you want to install software and use the matching guix client with or without NIX_STORE_DIR and NIX_STATE_DIR! A warning from experience. Testing in a VM may be an idea.

Trouble shooting

error: cannot bind to socket

turned out to be a proot switch -r

warn-about-old-distro

Guix versions won’t run if they are old. Use a more recent version of the daemon. It may also help to remove ~/.config/guix if it exists.

Also see this inside proot - probably because $HOME/.config is not mounted. Happens when the client is run inside proot bash.

refusing to run with elevated privileges (UID 0)

Use proot -0 switch on the daemon.

guix-daemon crashes X

When setting the –build-group of the guix-daemon in proot X would crash on my machine. You can avoid the switch, as mentioned above.

guix package: error: unsupported manifest format

You have probably run a newer guix-daemon and the format has changed and now you are trying to run an older one. A ‘guix pull’ saved me once because it built a new daemon after

guix pull
guix package -i guix -p ~/opt/guix

error: failed to run download program ‘/home/user/opt/local/libexec/guix/download’: No such file or directory

This is caused by the hash bang #!/bin/bash not being picked up in the script. Modify it to point to a bash in a visible store (no bash without hash), e.g. replace it with

#!/store/gnsc9ssv2zbmlw6l51fvgjy8s9lzha64-bash-4.4.12/bin/bash

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GUIX-NO-ROOT.org

GUIX-NO-ROOT.org

GNU Guix without root access on HPC

Table of Contents

Introduction

Install proot

Install the Guix binary installation

Run proot

Run guix daemon in proot

Install and run the hello package

Compile Guix from source

Building packages to run without proot

Final commands

Disabling tests

Warning

Trouble shooting

error: cannot bind to socket

warn-about-old-distro

refusing to run with elevated privileges (UID 0)

guix-daemon crashes X

guix package: error: unsupported manifest format

error: failed to run download program ‘/home/user/opt/local/libexec/guix/download’: No such file or directory

Files

GUIX-NO-ROOT.org

Latest commit

History

GUIX-NO-ROOT.org

File metadata and controls

GNU Guix without root access on HPC

Table of Contents

Introduction

Install proot

Install the Guix binary installation

Run proot

Run guix daemon in proot

Install and run the hello package

Compile Guix from source

Building packages to run without proot

Final commands

Disabling tests

Warning

Trouble shooting

error: cannot bind to socket

warn-about-old-distro

refusing to run with elevated privileges (UID 0)

guix-daemon crashes X

guix package: error: unsupported manifest format

error: failed to run download program ‘/home/user/opt/local/libexec/guix/download’: No such file or directory