Handling Secrets in NixOS: An Overview
There are a number of different approaches available for NixOS users to handle
secrets. The most popular tend to be
git-crypt
,
agenix
and
sops-nix
. But which one should you use?
To hopefully help you in answering this question for yourself, here is an overview of a few common use cases and what I think is most appropriate for each.
Managing Your Own Physical Machines⌗
Maybe you have a desktop, a Macbook and a Raspberry Pi which you are managing from a single NixOS flake repo. Maybe you even have a NixOS dedicated server somewhere running in a datacenter which functions as your media server running Plex or Jellyfin.
If you are primarily using NixOS on your own physical machines, used
exclusively by you, and you want to be able to publish your flake repo
publicly, I think you can get pretty far with git-crypt
.
I have been a happy user of git-crypt
for a long time, even before I started
using NixOS, and naturally it was my first instinct to use when setting up my
own configuration flake repo.
Using git-crypt
will ensure that your secret files are encrypted when you
push your repo to a remote like GitHub, however, using this approach means that
your secrets will end up in /nix/store
unencrypted, which is readable to all
users on a machine. If you’re exclusively managing your own physical machines,
this isn’t really an issue for you to worry about.
Here is how I’d suggest getting started with git-crypt
for a personal flake
repo:
Initialize the repo with git-crypt init
, make a directory dedicated to
secrets, and use a .gitattributes
file to ensure that every file that you
create in that secrets
subdirectory will always be encrypted.
secrets/** filter=git-crypt diff=git-crypt
Since the only user of this personal flake repository with access to decrypt secrets will be you, it’s more convenient to export a symmetric secret key and base64 encode it so that you can throw it in 1Password or something similar.
git-crypt export-key ./secret-key
cat ./secret-key | base64 --encode > ./secret-key-base64
You can store the value of ./secret-key-base64
in your password manager, and
if you ever need to decrypt the files in this repo on another machine, you can
just decode the key before using it.
pbpaste | base64 --decode > ./secret-key
git-crypt unlock ./secret-key
Up until this point, this is all pretty basic stuff. Next let’s take a look at how we can ergonomically handle secrets within a flake.
I like to have a single secrets.json
file that is structured with top-level
keys describing what the secret(s) relate to:
{
"github": {
"oauth_token": "ghp_..."
},
"gitlab": {
"oauth_token": "glpat-..."
},
"tailscale": {
"authkey": "tskey-auth-..."
}
}
Then, at the top-most level of my flake I declare a variable called “secrets” which reads the values from this file and deserializes them into a Nix object.
This secrets
variable can then be passed as a member of specialArgs
to make
it available to the various NixOS system configurations.
{
description = "My NixOS configurations";
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
inputs.nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = inputs:
with inputs: let
secrets = builtins.fromJSON (builtins.readFile "${self}/secrets/secrets.json");
in {
# make sure to "inherit secrets;" in the nixpkgs.lib.nixosSystem.specialArgs object for each of your machines
}
}
In particular, I find this very useful and easy to use inside of
home-manager
to do things
like set URL overrides in my gitconfig
so HTTPs clones of private
repositories automatically use an oauth token.
{
secrets,
...
}: {
programs.git = {
enable = true;
extraConfig = {
url = {
"https://oauth2:${secrets.github.oauth_token}@github.com" = {
insteadOf = "https://github.com";
};
"https://oauth2:${secrets.gitlab.oauth_token}@gitlab.com" = {
insteadOf = "https://gitlab.com";
};
};
};
};
}
You can also reference other encrypted files in the secrets
dir and use
home-manager
to move them into place for you. For example, with an .npmrc
file:
{
secrets,
...
}: {
home.username.LGUG2Z.file.".npmrc".source = ./secrets/.npmrc;
}
Providing Runtime Secrets to Remote Machines and VMs⌗
So git-crypt
works well for personal secrets like GitHub tokens, secrets that
live in configuration files under your $HOME
folder, but what about when you
need to provide a runtime secret to a service running on a remote server?
Usually these services run on their own user accounts and groups, and
home-manager
is not a good fit for provisioning secrets to these kinds of
non-user system accounts.
In this case, there are two options available: agenix
and sops-nix
.
Agenix⌗
agenix
is, in my opinion, the simpler of the two to set up, and that’s
probably why it has a bigger mindshare in the NixOS community right now.
You get started by creating a dedicated subdirectory and creating a
secrets.nix
file inside of it.
let
personal_key = "ssh-rsa AAAA....";
remote_server_key = "ssh-rsa AAAA....";
keys = [personal_key remote_server_key];
in {
"guest_accounts.json.age".publicKeys = keys;
}
In this file you declare variables for the public keys of the machines that
will need access to the secrets (you can grab these by running ssh-keyscan user@remote-ip
), and then create references to files that you will later
create, linking each of them to one or more public keys.
At this point, we have a file, which says “we’re gonna make these encrypted files” and “the private keys for the linked public keys will be able to decrypt them”.
The next step is to create the encrypted file by running agenix -e service_account.json.age
in the same subdirectory. This opens up your
$EDITOR
for you to type/paste your secret into. When you save and close the
text editor, the file will be encrypted, and this file can be added to git
.
If you ever need to add more public keys for other machines that you want to be
able to decrypt these secrets, you can just run agenix -r
inside the
subdirectory and commit the changes.
Next, in order to get these encrypted secrets into your flake, follow the
Install via Flakes
steps on the project README (expand the content section) to ensure that you
have the agenix.nixosModules.default
module loaded, and then start mapping
references to your encrypted files.
{
inputs.agenix.url = "github:ryantm/agenix";
# ... other inputs
outputs = { self, nixpkgs, agenix }: {
nixosConfigurations.yourhostname = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
# ... your config
agenix.nixosModules.default
{
age.secrets."guest_accounts.json".file = ./secrets/guest_accounts.json.age;
}
];
};
};
}
Unlike git-crypt
, this approach means that your secrets will end up in
/nix/store
but they will be encrypted there, so even if another user or
process can find the file, they won’t be able to make any sense of it.
What about making use of these secrets? Well, there is where you’ll need to
change your approach a little if you are coming from git-crypt
; you can’t
really refer to the contents of these secrets in the /nix/store
in your NixOS
configuration (ie. to do string interpolation) because they are encrypted, so
you have to instruct your services to read from the decrypted files in
/run/agenix
when they start.
Typically this is done by using systemd’s LoadCredential
option, which will
make a copy of the decrypted secret in /run/agenix
available to a service in
/run/credentials/your-service.service/filename
, and then instructing the
service via environment variables to use the file in that location.
Below is an example from my previous agenix
article
where we loaded a guest_accounts.json
secret file to be read by
nitter.service
.
{
systemd.services.nitter.serviceConfig.LoadCredential = [
"guest_account.json:${config.age.secrets."guest_accounts.json".path}"
];
systemd.services.nitter.serviceConfig.Environment = [
"NITTER_CONF_FILE=/var/lib/private/nitter/nitter.conf"
"NITTER_ACCOUNTS_FILE=%d/guest_account.json"
];
}
As you can see, this process is geared very much towards files, so even if you
wanted to encrypt a GitHub OAuth token, you’d have to create a unique file with
agenix -e github_token
and then find a way to cat
the value of that into
your systemd
script or service.
Here is an example from a systemd
timer that I use to periodically update
this
website.
{
systemd.services."update-lgug2z-com" = {
startAt = "hourly";
serviceConfig = {
Type = "oneshot";
ExecStart = ''
${pkgs.bash}/bin/bash -c "${pkgs.httpie}/bin/http POST \
https://api.cloudflare.com/client/v4/accounts/''$(cat ${config.age.secrets."cloudflare_account_id".path})/pages/projects/lgug2z-com/deployments \
-A bearer -a ''$(cat ${config.age.secrets."cloudflare_pages_api_token".path})"
'';
};
};
}
Note that you have to wrap this in a bash -c
call to enable the use of cat
.
The final generated systemd
timer definition which gets symlinked from
/nix/store
looks like this; no secrets exposed!
# /etc/systemd/system/update-lgug2z-com.service
[Unit]
[Service]
Environment="LOCALE_ARCHIVE=/nix/store/fzm1flvb7zmz3ij4sscn521shz2f76jh-glibc-locales-2.37-45/lib/locale/locale-archive"
Environment="PATH=/nix/store/w8vm09hri2zz7yacryzzzxvsapik4ps4-coreutils-9.1/bin:/nix/store/4cxs4cigh2zdxvma52ygm3mh2igq70iw-findutils-4.9.0/bin:/nix/store/b4in4hmq54h6l34a0v6ha40z97c0lzw2-gnugrep-3.7/bin:/nix/store/4mca20b13q88s6llkr8mc468rh9l9bmr-gnused-4.9/bin:/nix/store/12bynbp6y51j5449l27sy8ycgksd8npk-systemd-253.6/bin:/nix/store/w8vm09hri2zz7yacryzzzxvsapik4ps4-coreutils-9.1/sbin:/nix/store/4cxs4cigh2zdxvma52ygm3mh2igq70iw-findutils-4.9.0/sbin:/nix/store/b4in4hmq54h6l34a0v6ha40z97c0lzw2-gnugrep-3.7/sbin:/nix/store/4mca20b13q88s6llkr8mc468rh9l9bmr-gnused-4.9/sbin:/nix/store/12bynbp6y51j5449l27sy8ycgksd8npk-systemd-253.6/sbin"
Environment="TZDIR=/nix/store/951696yxqlphz378fx126wjnrih08mz3-tzdata-2023c/share/zoneinfo"
ExecStart=/nix/store/0rwyq0j954a7143p0wzd4rhycny8i967-bash-5.2-p15/bin/bash -c "/nix/store/yzl3gaf8yjd1y8mql9f36yhz2ppgz98g-python3.10-httpie-3.2.2/bin/http POST \
https://api.cloudflare.com/client/v4/accounts/$(cat /run/secrets/cloudflare/account_id)/pages/projects/lgug2z-com/deployments \
-A bearer -a $(cat /run/secrets/cloudflare/pages_api_token)"
Type=oneshot
# /etc/systemd/system/update-lgug2z-com.timer
[Unit]
[Timer]
OnCalendar=hourly
Sops-Nix⌗
So agenix
lets us get provision secret files for services which remain
encrypted in the /nix/store
, but are available decrypted for the specific
services to which we allow access. However the tradeoff here is that we lose
some of the convenience of being able to have smaller secrets such as tokens
all collected together in a single file in our repo.
sops-nix
is a little trickier to get set up, but offers a way for you to
retain your single encrypted file of secrets as a source of truth, both for
individual secret values and larger secrets files, all while providing clean
git diff
s to make it easier to see what has changed.
Getting started requires you to create a .sops.yaml
file at the root of your
flake repo.
keys:
- &remote age1...
- &personal age1...
creation_rules:
- path_regex: secrets/[^/]+\.(yaml|json|env|ini|sops)$
key_groups:
- age:
- *remote
- *personal
Let’s walk through this. At the top we define some keys, just like we did in
secrets.nix
for agenix
. However, the keys here are a little different. You
will need to convert your SSH public keys into age
public keys, and it’s
simpler to do this using ed25519
keys instead of rsa
keys.
You can generate these age
public keys for remote servers by running
ssh-keygen user@remote-ip | ssh-to-age
(you might need to nix-shell -p ssh-to-age
first if you don’t have the package on your system), and for your
local machine by running ssh-to-age -i ~/.ssh/id_ed25519.pub
.
Next is the creation_rules
section, where instead of providing explicit names
for each encrypted file like we did in secrets.nix
with agenix
, we just
define a regex of encryptable files, and the public keys corresponding to the
private keys that we want to allow to decrypt them.
Before we start creating rules, it’s important to make sure that we have
ssh-to-age
in our system packages and create an .envrc
for our flake repo,
otherwise we won’t be able to decrypt any of our secrets locally!
# make this point to wherever your own es25519 ssh key is
export SOPS_AGE_KEY=$(ssh-to-age -i ~/.ssh/id_ed25519 -private-key)
With the .sops.yaml
and .envrc
files created, we can create an encrypted
file in our dedicated subdirectory by running sops secrets/secrets.yaml
and
fill it with some values.
github:
oauth_token: ghp_...
gitlab:
oauth_token: glpat-...
tailscale:
authkey: tskey-auth-...`
guest_accounts.json: |
[
{
"id": "some-id",
"token": "some-token",
"grants": ["some" "grants"],
}
]
npmrc: |
//registry.npmjs.org/:_authToken=npm_...
//some.other.registry.org/:_authToken=npm_...
Notice that we can store both file contents and individual secrets like tokens
here; this is kind of the best of both worlds from the git-crypt
and agenix
approaches, especially if like me you find it convenient to only have to
encrypt a single file.
Again, in order to get these encrypted secrets into your flake, follow the
Install sops-nix (Flakes)
steps on the project README (expand the content section) to ensure that you
have the sops.nixosModules.default
module loaded, and then start mapping
references to your encrypted secretsa.
{
inputs.sops-nix.url = "github:Mic92/sops-nix";
outputs = { self, nixpkgs, sops-nix }: {
# change `yourhostname` to your actual hostname
nixosConfigurations.yourhostname = nixpkgs.lib.nixosSystem {
# customize to your system
system = "x86_64-linux";
modules = [
# ... your config
sops-nix.nixosModules.sops
{
sops = {
defaultSopsFile = ./secrets/secrets.yaml;
age.sshKeyPaths = ["/etc/ssh/ssh_host_ed25519_key"];
secrets = {
"github/oauth_token" = {};
"gitlab/oauth_token" = {};
"tailscale/authkey" = {};
"guest_accounts.json" = {};
"npmrc" = {
owner = "youruser";
path = "/home/youruser/.npmrc"
};
};
};
}
];
};
};
}
There is a little more code here, but it’s not as bad as it looks. Let’s walk
through it. Since we can have multiple files encrypted by sops
, sops-nix
helpfully allows us to define a defaultSopsFile
, which is useful if you’re
like me and you like to just keep a single encrypted file.
Then we have to set the paths to the private keys on the remote machine that
will be used to decrypt the encrypted defaultSopsFile
that will end up in the
/nix/store
.
Finally, we can declare our secrets. Each secret is output into /run/secrets
into a separate file, with a path that is taken from the object structure. For
example, github.oauth_token
’s value will be output to
/run/secrets/github/oauth_token
.
We can also use this to our advantage by storing stringified verisons of entire
files in the way that we have for guest_accounts.json
and npmrc
, to ensure
that these will be output to /run/secrets/guest_accounts.json
and
/run/secrets/.npmrc
respectively.
Notice that in the case of npmrc
, we can actually set the owner for this file
and a location for it to be symlinked to, which is a useful way to replicate
the placing of files in the $HOME
directory used in the git-crypt
approach
(you can also do this with agenix
).
To make these secrets in their decrypted forms available to systemd
services,
just follow the same steps outlined for agenix
, but reference the path from
sops
instead of age
.
Summary⌗
Hopefully this overview of when you might want to use these different approaches will help you to decide what is right for your use case. As a little tl;dr:
- Just managing your own machines?
git-crypt
is fine - Managing your own machines and remote servers? You’ll need
agenix
orsops-nix
to provide secrets to services - Wanna keep as much as possible in a single encrypted file?
sops-nix
is probably the way to go
However, keep in mind that you don’t have to pick only one of these! For
example, you can use a git-crypt
ed secrets.json
to populate values on
machines that you use for secret values in your configuration files and also
use agenix
or sops-nix
to provide secrets to services on remote servers and
VMs!
If you have any questions you can reach out to me on Twitter and Mastodon.
If you’re interested in what I read to come up with solutions like this one, you can subscribe to my Software Development RSS feed.
If you’d like to watch me writing code while explaining what I’m doing, you can also subscribe to my YouTube channel.
On 11/14/2023 I was impacted by large scale layoffs at my previous employer. I am currently looking for work. I am an experienced SRE with a strong passion for developer enablement. Please reach out if you are hiring for a role that you think I’d be a good fit for.
If you found this content valuable, please consider sponsoring me on GitHub or tipping me on Ko-fi to help me through this uncertain period.