In the previous article we walked through how to set up our very own Nix binary cache.

It’s great being able to run attic push system /run/current-system on whichever machine we are currently using, but the the chances are that if you use Nix to manage your system configurations, you have a system configuration monorepo, and depending on how many machines and architectures you are targeting, it can quickly become tiresome to manually push to the cache from each of them.

My configuration repo presently has 4 targets:

  • NixOS VM that runs on WSL (x86_64)
  • Bare metal Hetzner server (x86_64)
  • Raspberry Pi running in my home network (aarch64)
  • Production instance of https://notado.app (x86_64)

Building NixOS system configurations⌗

NixOS system configurations are essentially build artifacts. Since they are build artifacts, it makes sense to build, cache and push them in CI pipelines, right? Right!

Although GitHub Actions is awful, it can be made siginificantly less awful by installing Nix immediately after we run actions/checkout@v4. Once we have Nix installed, we can side-step the poorly maintained npm-esque “actions” ecosystem almost entirely.

If you have subscribed to my YouTube channel you may have seen a video where I built the system configurations for my NixOS starter templates on GitHub Actions.

Here is an example build.yml from the nixos-hetzner-robot-starter template.

name: "build"

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: DeterminateSystems/nix-installer-action@main
      - uses: DeterminateSystems/magic-nix-cache-action@main
      - run: nix build -L .#deploy.nodes.robot.profiles.system.path

Painless, right? We just checkout the repo, install Nix, optionally install magic-nix-cache, and then run a nix build command!

Building more complex NixOS system configurations⌗

Starter templates are pretty easy to build, but once we start adding things like secrets and packages from private repositories, different architecture targets etc. to our system configurations, naturally we’ll need to adapt our build.yml

Handling private repositories⌗

The nix-installer-action has a handy input where you can put a generated GitHub token with access to the private repositories you need. This will be used whenever the build needs to pull something like a flake input from one of your private repos. When you add this as a GitHub Actions secret, note that you won’t be able to use the GITHUB_ prefix for the name as that is reserved.

Additionally, you can also configure git to use this token, which is useful if you need to check out any private submodules in your configuration repo before you can run nix build.

- uses: DeterminateSystems/nix-installer-action@main
  with:
    github-token: ${{ secrets.GH_TOKEN }}
- run: |
    git config --global url."https://${{ secrets.GH_TOKEN }}@github.com".insteadOf https://github.com

Handling secrets encrypted with git-crypt⌗

While secrets encrypted with sops-nix are skipped over gracefully during a system build, git-crypted files that are used in a system configuration will result in a build error if they are not decrypted.

We can handle this scenario by exporting a symmetric key, encoding it in base64, adding it as a secret variable in GitHub Actions and then decoding it to decrypt the relevant files in our build job.

git-crypt export-key ./exported-key
cat exported-key | base64 --encode > encoded-key

# Store the value of `encoded-key` as a secret variable in GitHub Actions

With the encoded key in place as a GitHub Actions secret, we can add a step to decode it and decrypt any git-crypted files in our repo.

- run: |
    echo "${{ secrets.GIT_CRYPT_KEY }}" | base64 --decode > key
    nix profile install nixpkgs#git-crypt
    git-crypt unlock key

Handling different architectures⌗

Typically if you want to build packages for an aarch64-linux machine on an x86_64-linux machine, you just need to specify that architecture as an emulated system.

{
    boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
}

I came across an article by David Wagner which showed how to replicate this in GitHub Actions with a mixture of docker-setup-qemu and specifying aarch64-linux as an extra-platform in nix.conf.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3
      - uses: DeterminateSystems/nix-installer-action@main
        with:
          extra-conf: |
            extra-platforms = aarch64-linux

This is great, but we don’t really need to run docker-setup-qemu when we are building x86_64-linux systems, so let’s tweak this a little.

We can introduce a matrix which contains each of our machines and their respective platforms, and use that information to only run the docker-setup-qemu step if we are building an aarch64 system.

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        machine:
          - host: nixsl
            platform: x86-64-linux
          - host: hetzner
            platform: x86-64-linux
          - host: notado
            platform: x86-64-linux
          - host: pi3
            platform: aarch64-linux
    steps:
      - uses: actions/checkout@v4
      - if: matrix.machine.platform == 'aarch64-linux'
        uses: docker/setup-qemu-action@v3

Configuring access to our Attic binary cache⌗

If you recall from the previous article, in order to interact with our Attic binary cache, we had to do three things.

  • Log in with our token to be able to push
- run: |
    nix run github:zhaofengli/attic#default login fly https://<your fly app name>.fly.dev ${{ secrets.ATTIC_TOKEN }}
  • Add our token to a netrc file to be authorized to query
- run: |
    sudo mkdir -p /etc/nix
    echo "machine <your fly app name>.fly.dev password ${{ secrets.ATTIC_TOKEN }}" | sudo tee /etc/nix/netrc > /dev/null
  • Update our substituters and trusted-public-keys to be able to query during builds
- uses: DeterminateSystems/nix-installer-action@main
  with:
    extra-conf: |
      substituters = https://<your fly app name>.fly.dev/system?priority=43 https://nix-community.cachix.org?priority=41 https://numtide.cachix.org?priority=42 https://cache.nixos.org/
      trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= system:5M8uBPjS68HTadSbeCs0Jiu0Z1tJBNdahtKBCXhl+Z0= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= numtide.cachix.org-1:2ps1kLBUWjxIneOy1Ik6cQjb41X0iXVXeHigGmycPPE=

Putting it all together⌗ 

name: "build"

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      # Here we specify the matrix of our hosts and their target platform architectures
      matrix:
        machine:
          - host: nixsl
            platform: x86-64-linux
          - host: hetzner
            platform: x86-64-linux
          - host: notado
            platform: x86-64-linux
          - host: pi3
            platform: aarch64-linux

    steps:
      - uses: actions/checkout@v4

      # We only run this if we are building an aarch64-linux system
      - if: matrix.machine.platform == 'aarch64-linux'
        uses: docker/setup-qemu-action@v3

      # We make our netrc file that is used to make authorized requests to Attic
      # We also make sure that we use our custom GitHub token if we need to clone submodules or anything like that
      - run: |
          sudo mkdir -p /etc/nix
          echo "machine <your fly app name>.fly.dev password ${{ secrets.ATTIC_TOKEN }}" | sudo tee /etc/nix/netrc > /dev/null
          git config --global url."https://${{ secrets.GH_TOKEN }}@github.com".insteadOf https://github.com          

      - uses: DeterminateSystems/nix-installer-action@main
        with:
          # We set our custom GitHub token for any private flake inputs we might have
          github-token: ${{ secrets.GH_TOKEN }}
          # We add all the config for extra platforms, other binary caches and to raise the number of connections that can be made
          extra-conf: |
            fallback = true
            http-connections = 128
            max-substitution-jobs = 128
            extra-platforms = aarch64-linux
            substituters = https://<your fly app name>.fly.dev/system?priority=43 https://nix-community.cachix.org?priority=41 https://numtide.cachix.org?priority=42 https://cache.nixos.org/
            trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= system:5M8uBPjS68HTadSbeCs0Jiu0Z1tJBNdahtKBCXhl+Z0= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= numtide.cachix.org-1:2ps1kLBUWjxIneOy1Ik6cQjb41X0iXVXeHigGmycPPE=            
      - uses: DeterminateSystems/magic-nix-cache-action@main
      # We make sure that any git-crypted files are decrypted before we begin to build
      - run: |
          echo "${{ secrets.GIT_CRYPT_KEY }}" | base64 --decode > key
          nix profile install nixpkgs#git-crypt
          git-crypt unlock key          

      # We build each system in a separate job, targeting the configuration using matrix.machine.host
      # Once built, we login to Attic and push the built system to our `system` cache!
      - name: Build and push system
        run: |
          nix build .#nixosConfigurations.${{ matrix.machine.host }}.config.system.build.toplevel
          nix run github:zhaofengli/attic#default login fly https://<your fly app name>.fly.dev ${{ secrets.ATTIC_TOKEN }}
          nix run github:zhaofengli/attic#default push system result -j 2

If you have any questions or comments you can reach out to me on Twitter and Mastodon.

If you’re interested in what I read to come up with solutions like this one, you can subscribe to my Software Development RSS feed.

If you’d like to watch me writing code while explaining what I’m doing, you can also subscribe to my YouTube channel.

If you found this content valuable, or if you are a happy user of the komorebi tiling window manager or my NixOS starter templates, please consider sponsoring me on GitHub or tipping me on Ko-fi.