There are a number of different approaches available for NixOS users to handle secrets. The most popular tend to be git-crypt, agenix and sops-nix. But which one should you use?

To hopefully help you in answering this question for yourself, here is an overview of a few common use cases and what I think is most appropriate for each.

Managing Your Own Physical Machines⌗

Maybe you have a desktop, a Macbook and a Raspberry Pi which you are managing from a single NixOS flake repo. Maybe you even have a NixOS dedicated server somewhere running in a datacenter which functions as your media server running Plex or Jellyfin.

If you are primarily using NixOS on your own physical machines, used exclusively by you, and you want to be able to publish your flake repo publicly, I think you can get pretty far with git-crypt.

I have been a happy user of git-crypt for a long time, even before I started using NixOS, and naturally it was my first instinct to use when setting up my own configuration flake repo.

Using git-crypt will ensure that your secret files are encrypted when you push your repo to a remote like GitHub, however, using this approach means that your secrets will end up in /nix/store unencrypted, which is readable to all users on a machine. If you’re exclusively managing your own physical machines, this isn’t really an issue for you to worry about.

Here is how I’d suggest getting started with git-crypt for a personal flake repo:

Initialize the repo with git-crypt init, make a directory dedicated to secrets, and use a .gitattributes file to ensure that every file that you create in that secrets subdirectory will always be encrypted.

secrets/** filter=git-crypt diff=git-crypt

Since the only user of this personal flake repository with access to decrypt secrets will be you, it’s more convenient to export a symmetric secret key and base64 encode it so that you can throw it in 1Password or something similar.

git-crypt export-key ./secret-key
cat ./secret-key | base64 --encode > ./secret-key-base64

You can store the value of ./secret-key-base64 in your password manager, and if you ever need to decrypt the files in this repo on another machine, you can just decode the key before using it.

pbpaste | base64 --decode > ./secret-key
git-crypt unlock ./secret-key

Up until this point, this is all pretty basic stuff. Next let’s take a look at how we can ergonomically handle secrets within a flake.

I like to have a single secrets.json file that is structured with top-level keys describing what the secret(s) relate to:

{
  "github": {
    "oauth_token": "ghp_..."
  },
  "gitlab": {
    "oauth_token": "glpat-..."
  },
  "tailscale": {
    "authkey": "tskey-auth-..."
  }
}

Then, at the top-most level of my flake I declare a variable called “secrets” which reads the values from this file and deserializes them into a Nix object.

This secrets variable can then be passed as a member of specialArgs to make it available to the various NixOS system configurations.

{
  description = "My NixOS configurations";

  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
  inputs.nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs = inputs:
    with inputs: let
      secrets = builtins.fromJSON (builtins.readFile "${self}/secrets/secrets.json");
    in {
      # make sure to "inherit secrets;" in the nixpkgs.lib.nixosSystem.specialArgs object for each of your machines
    }
}

In particular, I find this very useful and easy to use inside of home-manager to do things like set URL overrides in my gitconfig so HTTPs clones of private repositories automatically use an oauth token.

{
  secrets,
  ...
}: {
  programs.git = {
    enable = true;
    extraConfig = {
      url = {
        "https://oauth2:${secrets.github.oauth_token}@github.com" = {
          insteadOf = "https://github.com";
        };
        "https://oauth2:${secrets.gitlab.oauth_token}@gitlab.com" = {
          insteadOf = "https://gitlab.com";
        };
      };
    };
  };
}

You can also reference other encrypted files in the secrets dir and use home-manager to move them into place for you. For example, with an .npmrc file:

{
  secrets,
  ...
}: {
  home.username.LGUG2Z.file.".npmrc".source = ./secrets/.npmrc;
}

Providing Runtime Secrets to Remote Machines and VMs⌗

So git-crypt works well for personal secrets like GitHub tokens, secrets that live in configuration files under your $HOME folder, but what about when you need to provide a runtime secret to a service running on a remote server? Usually these services run on their own user accounts and groups, and home-manager is not a good fit for provisioning secrets to these kinds of non-user system accounts.

In this case, there are two options available: agenix and sops-nix.

Agenix⌗

agenix is, in my opinion, the simpler of the two to set up, and that’s probably why it has a bigger mindshare in the NixOS community right now.

You get started by creating a dedicated subdirectory and creating a secrets.nix file inside of it.

let
  personal_key = "ssh-rsa AAAA....";
  remote_server_key = "ssh-rsa AAAA....";
  keys = [personal_key remote_server_key];
in {
  "guest_accounts.json.age".publicKeys = keys;
}

In this file you declare variables for the public keys of the machines that will need access to the secrets (you can grab these by running ssh-keyscan user@remote-ip), and then create references to files that you will later create, linking each of them to one or more public keys.

At this point, we have a file, which says “we’re gonna make these encrypted files” and “the private keys for the linked public keys will be able to decrypt them”.

The next step is to create the encrypted file by running agenix -e service_account.json.age in the same subdirectory. This opens up your $EDITOR for you to type/paste your secret into. When you save and close the text editor, the file will be encrypted, and this file can be added to git.

If you ever need to add more public keys for other machines that you want to be able to decrypt these secrets, you can just run agenix -r inside the subdirectory and commit the changes.

Next, in order to get these encrypted secrets into your flake, follow the Install via Flakes steps on the project README (expand the content section) to ensure that you have the agenix.nixosModules.default module loaded, and then start mapping references to your encrypted files.

{
  inputs.agenix.url = "github:ryantm/agenix";
  # ... other inputs

  outputs = { self, nixpkgs, agenix }: {
    nixosConfigurations.yourhostname = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        # ... your config
        agenix.nixosModules.default
        {
          age.secrets."guest_accounts.json".file = ./secrets/guest_accounts.json.age;
        }
      ];
    };
  };
}

Unlike git-crypt, this approach means that your secrets will end up in /nix/store but they will be encrypted there, so even if another user or process can find the file, they won’t be able to make any sense of it.

What about making use of these secrets? Well, there is where you’ll need to change your approach a little if you are coming from git-crypt; you can’t really refer to the contents of these secrets in the /nix/store in your NixOS configuration (ie. to do string interpolation) because they are encrypted, so you have to instruct your services to read from the decrypted files in /run/agenix when they start.

Typically this is done by using systemd’s LoadCredential option, which will make a copy of the decrypted secret in /run/agenix available to a service in /run/credentials/your-service.service/filename, and then instructing the service via environment variables to use the file in that location.

Below is an example from my previous agenix article where we loaded a guest_accounts.json secret file to be read by nitter.service.

{
  systemd.services.nitter.serviceConfig.LoadCredential = [
    "guest_account.json:${config.age.secrets."guest_accounts.json".path}"
  ];

  systemd.services.nitter.serviceConfig.Environment = [
    "NITTER_CONF_FILE=/var/lib/private/nitter/nitter.conf"
    "NITTER_ACCOUNTS_FILE=%d/guest_account.json"
  ];
}

As you can see, this process is geared very much towards files, so even if you wanted to encrypt a GitHub OAuth token, you’d have to create a unique file with agenix -e github_token and then find a way to cat the value of that into your systemd script or service.

Here is an example from a systemd timer that I use to periodically update this website.

{
  systemd.services."update-lgug2z-com" = {
    startAt = "hourly";
    serviceConfig = {
      Type = "oneshot";
      ExecStart = ''
        ${pkgs.bash}/bin/bash -c "${pkgs.httpie}/bin/http POST \
            https://api.cloudflare.com/client/v4/accounts/''$(cat ${config.age.secrets."cloudflare_account_id".path})/pages/projects/lgug2z-com/deployments \
            -A bearer -a ''$(cat ${config.age.secrets."cloudflare_pages_api_token".path})"
      '';
    };
  };
}

Note that you have to wrap this in a bash -c call to enable the use of cat.

The final generated systemd timer definition which gets symlinked from /nix/store looks like this; no secrets exposed!

# /etc/systemd/system/update-lgug2z-com.service

[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/fzm1flvb7zmz3ij4sscn521shz2f76jh-glibc-locales-2.37-45/lib/locale/locale-archive"
Environment="PATH=/nix/store/w8vm09hri2zz7yacryzzzxvsapik4ps4-coreutils-9.1/bin:/nix/store/4cxs4cigh2zdxvma52ygm3mh2igq70iw-findutils-4.9.0/bin:/nix/store/b4in4hmq54h6l34a0v6ha40z97c0lzw2-gnugrep-3.7/bin:/nix/store/4mca20b13q88s6llkr8mc468rh9l9bmr-gnused-4.9/bin:/nix/store/12bynbp6y51j5449l27sy8ycgksd8npk-systemd-253.6/bin:/nix/store/w8vm09hri2zz7yacryzzzxvsapik4ps4-coreutils-9.1/sbin:/nix/store/4cxs4cigh2zdxvma52ygm3mh2igq70iw-findutils-4.9.0/sbin:/nix/store/b4in4hmq54h6l34a0v6ha40z97c0lzw2-gnugrep-3.7/sbin:/nix/store/4mca20b13q88s6llkr8mc468rh9l9bmr-gnused-4.9/sbin:/nix/store/12bynbp6y51j5449l27sy8ycgksd8npk-systemd-253.6/sbin"
Environment="TZDIR=/nix/store/951696yxqlphz378fx126wjnrih08mz3-tzdata-2023c/share/zoneinfo"

ExecStart=/nix/store/0rwyq0j954a7143p0wzd4rhycny8i967-bash-5.2-p15/bin/bash -c "/nix/store/yzl3gaf8yjd1y8mql9f36yhz2ppgz98g-python3.10-httpie-3.2.2/bin/http POST \
    https://api.cloudflare.com/client/v4/accounts/$(cat /run/secrets/cloudflare/account_id)/pages/projects/lgug2z-com/deployments \
    -A bearer -a $(cat /run/secrets/cloudflare/pages_api_token)"

Type=oneshot

# /etc/systemd/system/update-lgug2z-com.timer

[Unit]

[Timer]
OnCalendar=hourly

Sops-Nix⌗

So agenix lets us get provision secret files for services which remain encrypted in the /nix/store, but are available decrypted for the specific services to which we allow access. However the tradeoff here is that we lose some of the convenience of being able to have smaller secrets such as tokens all collected together in a single file in our repo.

sops-nix is a little trickier to get set up, but offers a way for you to retain your single encrypted file of secrets as a source of truth, both for individual secret values and larger secrets files, all while providing clean git diffs to make it easier to see what has changed.

Getting started requires you to create a .sops.yaml file at the root of your flake repo.

keys:
  - &remote age1...
  - &personal age1...
creation_rules:
  - path_regex: secrets/[^/]+\.(yaml|json|env|ini|sops)$
    key_groups:
      - age:
          - *remote
          - *personal

Let’s walk through this. At the top we define some keys, just like we did in secrets.nix for agenix. However, the keys here are a little different. You will need to convert your SSH public keys into age public keys, and it’s simpler to do this using ed25519 keys instead of rsa keys.

You can generate these age public keys for remote servers by running ssh-keygen user@remote-ip | ssh-to-age (you might need to nix-shell -p ssh-to-age first if you don’t have the package on your system), and for your local machine by running ssh-to-age -i ~/.ssh/id_ed25519.pub.

Next is the creation_rules section, where instead of providing explicit names for each encrypted file like we did in secrets.nix with agenix, we just define a regex of encryptable files, and the public keys corresponding to the private keys that we want to allow to decrypt them.

Before we start creating rules, it’s important to make sure that we have ssh-to-age in our system packages and create an .envrc for our flake repo, otherwise we won’t be able to decrypt any of our secrets locally!

# make this point to wherever your own es25519 ssh key is
export SOPS_AGE_KEY=$(ssh-to-age -i ~/.ssh/id_ed25519 -private-key)

With the .sops.yaml and .envrc files created, we can create an encrypted file in our dedicated subdirectory by running sops secrets/secrets.yaml and fill it with some values.

github:
  oauth_token: ghp_...
gitlab:
  oauth_token: glpat-...
tailscale:
  authkey: tskey-auth-...`
guest_accounts.json: |
  [
      {
          "id": "some-id",
          "token": "some-token",
          "grants": ["some" "grants"],
      }
  ]  
npmrc: |
  //registry.npmjs.org/:_authToken=npm_...
  //some.other.registry.org/:_authToken=npm_...

Notice that we can store both file contents and individual secrets like tokens here; this is kind of the best of both worlds from the git-crypt and agenix approaches, especially if like me you find it convenient to only have to encrypt a single file.

Again, in order to get these encrypted secrets into your flake, follow the Install sops-nix (Flakes) steps on the project README (expand the content section) to ensure that you have the sops.nixosModules.default module loaded, and then start mapping references to your encrypted secretsa.

{
  inputs.sops-nix.url = "github:Mic92/sops-nix";

  outputs = { self, nixpkgs, sops-nix }: {
    # change `yourhostname` to your actual hostname
    nixosConfigurations.yourhostname = nixpkgs.lib.nixosSystem {
      # customize to your system
      system = "x86_64-linux";
      modules = [
        # ... your config
        sops-nix.nixosModules.sops
        {
          sops = {
              defaultSopsFile = ./secrets/secrets.yaml;
              age.sshKeyPaths = ["/etc/ssh/ssh_host_ed25519_key"];
              secrets = {
                "github/oauth_token" = {};
                "gitlab/oauth_token" = {};
                "tailscale/authkey" = {};
                "guest_accounts.json" = {};
                "npmrc" = {
                    owner = "youruser";
                    path = "/home/youruser/.npmrc"
                };
              };
          };
        }
      ];
    };
  };
}

There is a little more code here, but it’s not as bad as it looks. Let’s walk through it. Since we can have multiple files encrypted by sops, sops-nix helpfully allows us to define a defaultSopsFile, which is useful if you’re like me and you like to just keep a single encrypted file.

Then we have to set the paths to the private keys on the remote machine that will be used to decrypt the encrypted defaultSopsFile that will end up in the /nix/store.

Finally, we can declare our secrets. Each secret is output into /run/secrets into a separate file, with a path that is taken from the object structure. For example, github.oauth_token’s value will be output to /run/secrets/github/oauth_token.

We can also use this to our advantage by storing stringified verisons of entire files in the way that we have for guest_accounts.json and npmrc, to ensure that these will be output to /run/secrets/guest_accounts.json and /run/secrets/.npmrc respectively.

Notice that in the case of npmrc, we can actually set the owner for this file and a location for it to be symlinked to, which is a useful way to replicate the placing of files in the $HOME directory used in the git-crypt approach (you can also do this with agenix).

To make these secrets in their decrypted forms available to systemd services, just follow the same steps outlined for agenix, but reference the path from sops instead of age.

Summary⌗

Hopefully this overview of when you might want to use these different approaches will help you to decide what is right for your use case. As a little tl;dr:

  • Just managing your own machines? git-crypt is fine
  • Managing your own machines and remote servers? You’ll need agenix or sops-nix to provide secrets to services
  • Wanna keep as much as possible in a single encrypted file? sops-nix is probably the way to go

However, keep in mind that you don’t have to pick only one of these! For example, you can use a git-crypted secrets.json to populate values on machines that you use for secret values in your configuration files and also use agenix or sops-nix to provide secrets to services on remote servers and VMs!

If you have any questions you can reach out to me on Twitter and Mastodon.

If you’re interested in what I read to come up with solutions like this one, you can subscribe to my Software Development RSS feed.

If you’d like to watch me writing code while explaining what I’m doing, you can also subscribe to my YouTube channel.

On 11/14/2023 I was impacted by large scale layoffs at my previous employer. I am currently looking for work. I am an experienced SRE with a strong passion for developer enablement. Please reach out if you are hiring for a role that you think I’d be a good fit for.

If you found this content valuable, please consider sponsoring me on GitHub or tipping me on Ko-fi to help me through this uncertain period.