Last October, Plex started blocking access to instances running on servers hosted by Hetzner.

I have a Hetzner Auction server that I renew every year or so to make use of newer hardware, which I use to run various workloads, from web services, to scheduled jobs and self-hosted instances of privacy-friendly alternative web frontends like Nitter.

Another one of those workloads, until recently, was Plex.

I didn’t have the time to put too much effort into getting around the Hetzner network ban when it was first implemented, so I just started running Jellyfin instead. I even made a video demonstrating how easy it was to get Jellyfin up and running on a VPS.

The Initial View from Europe⌗

Since getting laid off I finally had some time to try and think about how to get Plex working again.

My circumstances have changed since I first started hosting Plex back in the early 2010s:

  • I no longer travel for work (previously, I was an ICRC field delegate and later, for a time, a software development consultant)
  • I now have a fixed address and a desktop computer(!)
  • I no longer live/work/travel in Europe
  • I now have a lot of spare computer parts lying around the house

When I still lived in Europe, hosting Plex on Hetzner made a lot of sense because it was incredibly fast (both in terms of raw bandwidth and transcoding speed) when serving content in Europe and SWANA, and because I only had a laptop that I carried with me when I traveled for work.

The flow of network requests at that time were pretty simple:

Me (Anywhere)

🔻

Plex (Hetzner Data Center)

🔻

Google Workspace (Rclone)

🔻

Plex (Hetzner Data Center)

🔻

Me (Anywhere)

The Current View from Washington⌗

Upon moving to the US, I was shocked at the poor quality of service provided by ISPs here. After a lot of reading, I came to realize that my own ISP, Xfinity, in particular was/is known for incredibly poor peering with Hetzner data centers.

To get around the poor peering, I proxied my requests to Plex through a VPS running in the US, and this worked quite well, despite the complexity creep.

The flow of network requests at that time became:

Me (Home)

🔻

VPS (US Cloud Provider)

🔻

Plex (Hetzner Data Center)

🔻

Google Workspace (Rclone)

🔻

Plex (Hetzner Data Center)

🔻

VPS (US Cloud Provider)

🔻

Me (Home)

Storage Considerations⌗

Last summer, Google Workspace, which I was using to host the media served by my Plex instance, had a pricing restructure and effectively did away with their “unlimited” storage option.

This wasn’t too bad for me as I had less than 15TB of data in total on my Google Workspace account which was still possible to host on a Google Workspace with three active user accounts.

First Steps Forward⌗

Once I have started working again (and after I have restored my emergency fund) I would like to start buying some hard drives and setting up a local storage solution at home, making use of the old computer parts I have collected over the years.

Until then, with my media storage still all on Google Workspace, I think it makes sense to reduce the complexity of my current setup by removing the VPS that is being used to proxy my media streaming requests from the Hetzner server.

And since we are removing the VPS, why not just run the media server locally on a virtual machine for now too, with the storage mounted via rclone from Google Workspace?

The flow of network requests can now be simplified again:

Me (Home)

🔻

Plex (Home)

🔻

Google Workspace (Rclone)

🔻

Plex (Home)

🔻

Me (Home)

This works nicely because US ISPs generally don’t have peering issues with Google Workspace, so we can avoid the extra hop, while also benefiting from the traffic being served from the Plex server to the Plex client at home over the LAN.

Virtual Machine Setup⌗

My main desktop computer at home runs Windows 11 and a bunch of VMs on WSL2.

I figure that for now, until I am ready to assemble a dedicated server at home (I still need a motherboard and a case in addition to the hard drives), I can create another WSL2 VM running NixOS, which will make the eventual migration to bare metal a largely lift-and-shift operation.

This is how my Plex VM on WSL2 is configured, based on my various NixOS starter templates:

{
  config,
  pkgs,
  username,
  ...
}: let
  uid = username:
    if config.users.users.${username}.uid == null
    then "1000"
    else toString config.users.users.${username}.uid;

  gid = group:
    if config.users.groups.${group}.gid == null
    then "100"
    else toString config.users.groups.${group}.gid;

  inherit (config.users.users.${username}) home;

  media = "${home}/media";
  plexData = "${home}/plex";
  remote = "${media}/remote";
  mount = "gsuite-encrypted";
in {
  imports = [
    ../modules/nix/base.nix
    ../modules/nix/linux.nix
  ];

  system.stateVersion = "22.05";
  time.timeZone = "America/Los_Angeles";

  wsl = {
    enable = true;
    wslConf.automount.root = "/mnt";
    wslConf.interop.appendWindowsPath = false;
    wslConf.network.generateHosts = true;
    defaultUser = username;
    startMenuLaunchers = true;
    docker-desktop.enable = false;
    interop.register = true;
  };

  environment.systemPackages = [
    (import ../pkgs/win32yank.nix {inherit pkgs;})
  ];

  systemd.tmpfiles.rules = [
    "d '${media}' 0755 ${username} users - -"
    "d '${plexData}' 0755 ${username} users - -"
  ];

  services.plex = let
    plexPass = pkgs.plex.override {
      plexRaw = pkgs.plexRaw.overrideAttrs (_: rec {
        version = "1.32.8.7639-fb6452ebf";
        src = pkgs.fetchurl {
          url = "https://downloads.plex.tv/plex-media-server-new/${version}/debian/plexmediaserver_${version}_amd64.deb";
          sha256 = "sha256-jdGVAdvm7kjxTP3CQ5w6dKZbfCRwSy9TrtxRHaV0/cs=";
        };
      });
    };
  in {
    enable = true;
    dataDir = "${plexData}";
    user = username;
    group = "users";
    openFirewall = true;
    package = plexPass;
  };

  systemd.services.remote_tv = {
    wantedBy = ["multi-user.target"];
    serviceConfig = {
      ExecStartPre = "/run/wrappers/bin/sudo -u ${username} /run/current-system/sw/bin/mkdir -p ${remote}/tv";
      ExecStart = ''
        ${pkgs.rclone}/bin/rclone mount '${mount}:media/TV Shows' ${remote}/tv \
          --config=${config.sops.secrets."rclone/rclone.conf".path} \
          --read-only \
          --allow-other \
          --allow-non-empty \
          --uid ${uid username} \
          --gid ${gid "users"} \
          --log-level=INFO \
          --buffer-size=50M \
          --drive-acknowledge-abuse=true \
          --vfs-cache-mode full \
          --vfs-cache-max-size 100G \
          --vfs-read-chunk-size=32M \
          --vfs-read-chunk-size-limit=256M
      '';
      ExecStop = "/run/wrappers/bin/fusermount3 -u ${remote}/tv";
      Type = "notify";
      Restart = "always";
      RestartSec = "10s";
      EnvironmentFile = [config.sops.secrets."rclone/environment".path];
      Environment = ["PATH=${pkgs.fuse}/bin:/run/wrappers/bin:$PATH"];
    };
  };

  systemd.services.remote_movies = {
    wantedBy = ["multi-user.target"];
    serviceConfig = {
      ExecStartPre = "/run/wrappers/bin/sudo -u ${username} /run/current-system/sw/bin/mkdir -p ${remote}/movies";
      ExecStart = ''
        ${pkgs.rclone}/bin/rclone mount '${mount}:media/Movies' ${remote}/movies \
          --config=${config.sops.secrets."rclone/rclone.conf".path} \
          --read-only \
          --allow-other \
          --allow-non-empty \
          --uid ${uid username} \
          --gid ${gid "users"} \
          --log-level=INFO \
          --buffer-size=50M \
          --drive-acknowledge-abuse=true \
          --vfs-cache-mode full \
          --vfs-cache-max-size 100G \
          --vfs-read-chunk-size=32M \
          --vfs-read-chunk-size-limit=256M
      '';
      ExecStop = "/run/wrappers/bin/fusermount3 -u ${remote}/movies";
      Type = "notify";
      Restart = "always";
      RestartSec = "10s";
      EnvironmentFile = [config.sops.secrets."rclone/environment".path];
      Environment = ["PATH=${pkgs.fuse}/bin:/run/wrappers/bin:$PATH"];
    };
  };
}

Plex Connectivity⌗

With Plex running in a WSL2 VM, it will have an IP address matching 172.*.*.*, which differs from my LAN’s 192.168.111.1/24 address range, so there are a few values that we need to add in the settings.

  • Settings -> Remote Access -> Enable Remote Access since 192.168.111.1/24 addresses appear as “remote” to Plex
  • For Settings -> Network -> LAN Networks I set 192.168.111.1/24
  • For Settings -> Network -> Custom server access URLs I set http://192.168.111.221:32400, where 192.168.111.221 is the static address of the machine the Plex VM is running on
  • I grab the address of the VM from /etc/resolv.conf and set a netsh rule so that requests to 192.168.111.221:32400 are forwarded on to the VM
    • sudo netsh interface portproxy add v4tov4 listenport=32400 listenaddress=0.0.0.0 connectport=32400 connectaddress=172.28.96.1

One thing to note however, is that the connectaddress changes when the host is restarted. This is very rare for me so I don’t mind updating this by hand every now and then.

Keeping the Data Fresh⌗

Due to the poor quality of internet service in my apartment and across the US in general, it makes sense to keep network-heavy workloads running on the Hetzner server.

For a long time, I have used mergerfs to create a consolidated view on the server of remote files on Google Workspace and local files that “appear” on the server from time to time.

directory description
media/remote/tv Google Workspace files mounted via rclone
media/local/tv Local hard drive files
media/merged/tv All the files! 🎉

The media files under merged are what Plex and Jellyfin running on the server look at, providing a consolidated view of newer files on the hard drive and older files mounted from Google Workspace, and every day a systemd timer runs to move any files older than 5 days to Google Workspace, freeing up space on the local hard drive.

{
  systemd.services.move_tv = {
    enable = enableCron;
    startAt = "daily";
    serviceConfig = {
      Type = "oneshot";
      User = username;
      Group = "users";
      EnvironmentFile = [config.sops.secrets."rclone/environment".path];
      ExecStart = ''
        ${pkgs.rclone}/bin/rclone \
            --config=${config.sops.secrets."rclone/rclone.conf".path} \
            --drive-upload-cutoff 1000T \
            --tpslimit 5 \
            move \
            "${local}/tv" \
            "${mount}:media/TV Shows" \
            --log-file ${home}/logs/upload_tv.log \
            --delete-empty-src-dirs \
            --fast-list \
            --stats-one-line \
            -v \
            --min-age 5d
      '';
    };
  };
}

These days it’s exceedingly rare that new files “appear” on my server given that there are so many family subscriptions of Netflix, Hulu etc. going around in our extended family.

Nevertheless, given that Plex is now running in a VM on my home network, I have removed the --min-age 5d restriction from my systemd timers so that whatever new files may “appear” on the server on any given day are always moved to Google Workspace within 24 hours.

Thinking Ahead⌗

When I am able to buy the remaining parts to put together a dedicated home server, the migration path seems pretty clear:

  • Zip up the plex folder containing my Plex server configuration
  • Provision the new server with a NixOS template based on the existing VM configuration
  • Unzip the Plex server configuration on the dedicated server before enabling the plex service
  • Remove the VM-specific network configuration from the Plex server settings
  • Slowly sync my media from Google Workspace to the dedicated server
  • Once the sync is complete, add a new SFTP remote remote to my rclone configuration for new files to also be pushed directly to the dedicated server at home from Hetzner

If you have any questions or comments you can reach out to me on Twitter and Mastodon.

If you’re interested in what I read to come up with solutions like this one, you can subscribe to my Software Development RSS feed.

If you’d like to watch me writing code while explaining what I’m doing, you can also subscribe to my YouTube channel.

If you found this content valuable, or if you are a happy user of komorebi or my NixOS starter templates, please consider sponsoring me on GitHub or tipping me on Ko-fi.