devnix

This is a set of nix functions that extend the regular nix specifications to provide enhanced support for a development environment.

It’s relatively easy to work on a single package under nix, and possibly even collect some dependencies, but when the number of dependencies grows, when some of those must either be changed or newer versions of them are needed, or when applying Hydra CI to the project, it becomes fairly tedious to manage the nix environment.

The devnix tooling is designed to make the latter situations easier. Simply create a configs.nix that provides the high-level description and then invoke either the mkRelease or the mkJobsetsDrv and mkJobset functions with those descriptive inputs. These will generate the appropriate definitions to use with nix-build or nix-shell to do local development work, and with the inclusion of a minimal JSON declarative file, these will also generate the jobsets needed for Hydra variants.

Status

The devnix tooling is currently functional. Beta users are invited. Attempts will be made to keep the existing interface stable while adding new functionality, although they may change.

Pending updates

• Better/more documentation
• Currently support is focused on Haskell packages; support for other languages (e.g. Rust) should be possible in the near future.

Usage

Maintaining the devnix specification

The devnix configs.nix (and associated) files can be maintained in the project repo itself or in a separate location (e.g. a separate repository).

The advantage to maintaining the configuration in a separate repository is that it keeps the main repository clean and the configuration can be managed independently from the project (since the configuration depends on elements which change outside of the project itself, like recent dependency releases, etc.).

With nix-shell

Assumes the devnix configs.nix is in $HOME/my-devnix, and the project you want to work on is checked out in $HOME/projects/myproj.

$ cd $HOME/projects/myproj
$ nix-shell $HOME/my-devnix/configs.nix -A myproj.env
[nix-shell] $ BUILD_COMMAND

With nix-build

Assumes the same devnix location as the nix-shell example.

To build myproj from the declared source (e.g. github):

$ nix-build $HOME/my-devnix/configs.nix -A myproj

This will create a results link in the current directory that points to the myproj just installed and built in the nix store.

If the above should build a locally checked-out version of myproj instead, then edit the configs.nix file and where the myproj github declaration is, add a local parameter. For example:

...
myproj = githubsrc "myself" "myproj";
...

would become:

...
myproj = githubsrc "myself" "myproj" // { local = /home/me/projects/myproj; };
...

With hydra

Pending…

• Note that the devnix configurations can support the “GitTree” input type. This type is not available from standard Hydra, but is available from https://github.com/kquick/hydra under the “galois” branch. This is a single file that can be dropped into src/lib/Hydra/Plugins/; it will recursively follow git submodule specifications and generate a JSON file of the repositories and the corresponding git submodule revision references for those repositories.

Locally recreating a hydra build

Useful for debugging or locally working

Directly installing

$ nix copy --from ssh://hydra-system  /nix/store/path-from-details-tab.drv
$ nix-build DRV

where DRV is the derivation store path or output store path from the “Details” tab of the build.

Creating a shell and building as hydra

$ nix-shell DRV

Within this environment, there are shell variables that manage the build. They are run as phases, although if phases isn’t set, a default set of phases is used. To run the full build:

[nix-shell] genericBuild

To setup the ghc-pkgs independently:

[nix-shell] eval "$setupCompilerEnvironmentPhase"

The above uses things like pkgsHostHost and pkgsHostTarget to configure ghc-pkgs, although the proper configuration flag needs to be passed to GHC to utilize that configuration:

[nix-shell] echo ${pkgsHostTarget[*]}
[nix-shell] ghc-pkg --package-db="$packageConfDir" list

The default set of phases is:

phases="${prePhases:-} unpackPhase patchPhase ${preConfigurePhases:-}
        configurePhase ${preBuildPhases:-} buildPhase checkPhase
        ${preInstallPhases:-} installPhase ${preFixupPhases:-} fixupPhase installCheckPhase
        ${preDistPhases:-} distPhase ${postPhases:-}";

Be aware that there are also shell functions with the same name as phases, so if (for example) the configurePhase variable is defined, then the following do two different things:

[nix-shell] configurePhase
[nix-shell] eval "$configurePhase"

Running individual phases:

[nix-shell] eval "${!prePhases:-}"  # use ! because it is a list of vars
# If already in a checked-out copy of the source, skip the unpackPhase
[nix-shell] eval "$patchPhase"
[nix-shell] eval "${!preConfigurePhases:-}"
[nix-shell] eval "$configurePhase"
...

Haskell Considerations

Building

When using the nix-shell configuration described above, the BUILD_COMMAND is usually a “cabal build” or “cabal test” directive.

Note that either cabal v1-build or v2-build can be used: the nix-shell should have brought all the dependencies into scope so everything should be available for v1 and v2.

A dependency is “not found in archive”

$ nix-build -A myproj $HOME/mydevnix/configs.nix
...
building '/nix/store/HASH-all-cabal-hashes-component-th-abstraction-0.3.1.0.drv'...
tar: */th-abstraction/0.3.1.0/th-abstraction.json: Not found in archive
tar: */th-abstraction/0.3.1.0/th-abstraction.cabal: Not found in archive
tar: Exiting with failure status due to previous errors
builder for '/nix/store/HASH-all-cabal-hashes-component-th-abstraction-0.3.1.0.drv' failed with exit code 2
cannot build derivation '/nix/store/HASH-all-cabal-hashes-component-th-abstraction-0.3.1.0.drv': 1 dependencies couldn't be built
$

This happens because this is a newer version of the package (th-abstraction in this example) that is not present in the standard nix haskell packages definitions and so a cabal2nix-based specification is being auto-constructed, but the sha256 hash for the package could not be found in the hashes repository.

First, ensure that you are using a recent nixpkgs version and that your channel is updated. An 18.09 channel is known to be too old, and the unstable channel should be recent enough. You can use this by either updating your channel with $ nix-channel --update or by passing the updated version to configs.nix:

$ nix-build -A myproj $HOME/mydevnix/configs.nix --arg nixpkgs https://nixos.org/channels/nixos-unstable

If you still encounter the error, then it means that the package definition hasn’t been picked up in nix, but it has probably been updated upstream already. The upstream location is https://api.github.com/repos/commercialhaskell/all-cabal-hashes/tarball/hackage (as of Aug 2019).

The devnixlib process can automatically fetch a recent version of this for you:

$ nix-build -A myproj $HOME/mydevnix/configs.nix --arg freshHaskellHashes true

[Note that if your channel is too old, the mechanism supporting freshHaskellHashes is not yet available and you will still get the error above; use a recent channel, and 18.09 is known to be too old.

The downside to simply specifying freshHaskellHashes as true is that nix will only cache this file for a little while (usually 3600 seconds, see $ nix show-config | grep tarball), so when this cache period expires, it will get a (likely new) version of the hashes file, which will have a new sha256 hash, and therefore cause rebuilds of most of the downstream elements.

To fix this, add a recent version of the haskellHashes file to your store and then use that store location instead of true for the freshHaskellHashes argument:

$ nix build $(nix eval --raw '(builtins.fetchurl { url = \
  https://api.github.com/repos/commercialhaskell/all-cabal-hashes/tarball/hackage; \
  })') -o freshHashes

creates the freshHashes link in the local directory to allow:

$ nix-build -A myproj $HOME/mydevnix/configs.nix --arg freshHaskellHashes $(readlink -f freshHashes)

Updating dependencies along with the current project

If the myproj project has a dependency on the mydep project, and a local development process requires changes to both, it can be inconvenient to continually exit and re-enter the nix-shell enviroinment for myproj when changes have been made to a local copy of mydep. Additionally this can be slower because mydep will need to be fully rebuilt and the new version installed into the nix store before myproj’s nix-shell environment is ready.

The best way to handle this is to use cabal’s v2-build functionality with a cabal.project file:

1. Checkout myproj:

   $ git clone https://github.com/myself/myproj $HOME/projects/myproj
       

2. Checkout mydep:

   $ git clone https://github.com/myself/mydep $HOME/projects/mydep
       

3. Create a cabal.project file that references mydep:

   $ cat << EOF > cabal.project
   packages: .
             $HOME/projects/mydep
   EOF
   $
       

4. Enter the nix-shell environment for myproj

   $ nix-shell $HOME/my-devnix/configs.nix -A myproj.env
   [nix-shell] $
       

5. Normal dev and build:

   [nix-shell] $ edit myproj-files-or-mydep-files
   [nix-shell] $ cabal v2-build
   [nix-shell] $ cabal v2-test
       

   During the dev and build process, cabal’s v2-build will build version of mydep cached in the local dist-newstyle/ subdirectory, updating it as necessary as changes are made to mydep but building only the necessary elements.

FAQ

Why configs.nix and not default.nix?

There’s no critical reason, but since the default.nix is customarily used for the description of a single package, a different name for the “project configuration” was chosen as the recommendation.

Why not just use overlays?

Overlays are fine for global needs. They are less useful in cases like:

• Working on several different projects that have the same dependencies but different versions of those dependencies.
• Needing different dependency versions for different compilers.
• Configuring Hydra.

The devnix configuration and utilities are designed to be used on a per-project basis.