configuration for deploying Headscale + Headplane + Pocket ID
  • Shell 45.2%
  • Go Template 44.4%
  • Jinja 10.4%
Find a file
Matthew Ford f025d60e8b Headscale policy: allow GRE between tag:gre-router nodes
A rule with no proto matches only TCP/UDP/ICMP, so the admin *:* rule does not carry GRE. Routers are grouped by tag because Headscale does not resolve bare node names as policy aliases.
2026-10-23 12:35:26 -04:00
ansible Proxy inbound mail with HAProxy to preserve the client IP 2026-10-16 12:01:52 -04:00
certs Headscale + Headplane + Pocket ID behind Envoy on rootless podman 2026-10-15 14:04:46 -04:00
config Headscale policy: allow GRE between tag:gre-router nodes 2026-10-23 12:35:26 -04:00
scripts Expose OIDC allowed domains/users/groups in .env 2026-10-16 12:05:12 -04:00
secrets Headscale + Headplane + Pocket ID behind Envoy on rootless podman 2026-10-15 14:04:46 -04:00
systemd Fixes To Get It To Work 2026-10-16 14:28:43 +00:00
.env.example Expose OIDC allowed domains/users/groups in .env 2026-10-16 12:05:12 -04:00
.gitignore Backups: take the schedule from .env, rendered into the timer 2026-10-15 14:29:13 -04:00
AGENTS.md Fixes To Get It To Work 2026-10-16 14:28:43 +00:00
compose.yaml Security hardening from audit 2026-10-16 11:53:32 -04:00
Dockerfile.backup Headscale + Headplane + Pocket ID behind Envoy on rootless podman 2026-10-15 14:04:46 -04:00
README.md Expose OIDC allowed domains/users/groups in .env 2026-10-16 12:05:12 -04:00

Headscale + Headplane + Pocket ID behind Envoy

A self-hosted Tailscale control plane on rootless podman:

Service URL Role
Headscale https://vpn.example.com/ Tailscale control server
Headplane https://vpn.example.com/admin Admin dashboard
Pocket ID https://auth.vpn.example.com OIDC identity provider (passkeys)

MagicDNS domain: tailnet.example.com

Those hostnames are placeholders. Nothing under config/ is deployment-specific: every name comes from .env, and bootstrap.sh substitutes it into the config/**/*.tmpl templates. Substituting your own domain in the examples below is the whole setup — see Deployment-specific values.

TLS certificates are obtained from Let's Encrypt by a lego sidecar and handed to Envoy through a watched directory, so renewals never require a restart.

All three databases are SQLite files in podman volumes, and they are backed up to S3 by scripts/backup.sh on a systemd timer whose schedule lives in .env (BACKUP_SCHEDULE). See Backups.


Architecture

                 tcp/80                                   tcp/443
Internet ──────►  Envoy ──► /.well-known/acme-challenge/   Envoy (SDS certs)
                  ├─► 301 to https                           ├─► /admin ──► headplane:3000
                  │                                         ├─► /      ──► headscale:8080
                  │                                         ├─► auth.   ──► pocket-id:1411
                  └─► any other Host ──► FALLBACK_UPSTREAM:80 └─► any other SNI ──┐
                                                                                  │
                                          FALLBACK_UPSTREAM:443 ◄── raw TLS ─────────┘
                 udp/3478
Internet ──────►  headscale (DERP STUN)

                 ┌──────────┐  writes      ┌────────┐
                 │   lego   ├─────────────►│ /certs │◄── envoy watches
                 │ renew 12h│              └────────┘
                 └──────────┘

Three headscale-specific details that are easy to get wrong:

  1. Envoy needs upgrade_configs: { upgrade_type: tailscale-control-protocol }. Tailscale clients upgrade to the control protocol using a POST request with a non-standard Upgrade header value. Envoy only recognises websocket by default. Without this rule clients hang on tailscale up and never connect.

  2. Headscale ignores X-Forwarded-For unless trusted_proxies matches Envoy's address. Without it every node appears to originate from the Envoy container in logs and rate limiting.

  3. tls_inspector on :443 is load-bearing. Once the listener selects a filter chain on SNI, deleting the inspector means the SNI is never read, both chains become unreachable, and every connection — including vpn.example.com — is silently forwarded to the fallback.


Prerequisites

On the Linux host:

  • podman and podman-compose
  • DNS for vpn.example.com and auth.vpn.example.com → this host's IP
  • Inbound tcp/80, tcp/443, udp/3478
  • Rootless ports unprivileged — see systemd/sysctl-instructions.md

udp/3478 is not optional. Without it, NAT traversal fails for peers behind strict NAT and the embedded DERP relay cannot accept STUN.

Provisioning the host

Everything above is host configuration, and on a cloud image it is not a matter of the ports alone — Oracle's Ubuntu image in particular ships an iptables chain that accepts SSH and rejects everything else, and reapplies it at boot. Envoy binds its ports successfully, so nothing errors and nothing arrives.

ansible/provision.yml does this part. It prepares the host and stops there: .env still holds every deployment-specific value and bootstrap.sh still renders the configs.

cd ansible && ansible-playbook provision.yml

It also sets up an inbound port forwarder for the fallback server — see Port forwards. That forwarder has no representation in compose.yaml or in any systemd unit; its documentation is ansible/README.md.


Deploy

git clone <this-repo> ~/vpn-ops && cd ~/vpn-ops

# 1. Host prerequisites
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0
echo 'net.ipv4.ip_unprivileged_port_start=0' | sudo tee /etc/sysctl.d/99-rootless-ports.conf

# 2. Configure and generate secrets
cp .env.example .env
$EDITOR .env          # VPN_DOMAIN, AUTH_DOMAIN, TAILNET_DOMAIN, ADMIN_USERS,
                     # ACME_EMAIL and VPN_SUBNET are all required - see below.
                     # Leave ACME_STAGING=true for the first run.
                     # FALLBACK_UPSTREAM is optional — see "Fallback upstream"
./scripts/bootstrap.sh

# 3. Start
podman-compose up -d

~/vpn-ops is not an arbitrary path. The units in systemd/ hard-code WorkingDirectory=%h/vpn-ops — vpn-stack.service, telemetry.service and backup.service — and scripts/deploy.sh defaults to the same place. Clone somewhere else and you have to edit WorkingDirectory in every unit and pass the path to deploy.sh, or they will quietly act on a directory that is not there. This is also what podman-compose uses as the compose project name, so it shows up in every generated volume and container name — see Troubleshooting.

backup.timer is the one exception: it holds no paths at all, and bootstrap.sh renders it from .env and installs it for you.

Deployment-specific values

Everything deployment-specific lives in .env, and nothing else. There are six values with no default, because bootstrap.sh fails rather than fall back to a hostname that is not yours:

Value Example What it becomes
VPN_DOMAIN vpn.example.com Headscale server_url, Headplane base_url and public_url, Envoy's vhost and SNI chain, the ACME SAN
AUTH_DOMAIN auth.vpn.example.com Both OIDC issuers, Pocket ID APP_URL, the second ACME SAN
TAILNET_DOMAIN tailnet.example.com MagicDNS base_domain, so nodes resolve as <hostname>.<this>
ADMIN_USERS alice@example.com,bob@example.com group:admin in the ACL policy; the first entry is also the policy configtest's source
ACME_EMAIL ops@example.com Let's Encrypt contact address
VPN_SUBNET 10.89.0.0/24 The podman bridge subnet — and therefore Headscale's trusted_proxies

TAILNET_DOMAIN must differ from VPN_DOMAIN, and ADMIN_USERS must not contain spaces — bootstrap.sh sources .env with ., so an unquoted space is parsed as a command.

VPN_SUBNET is required because a wrong one is silently wrong: it decides the bridge subnet in compose.yaml and the list of addresses Headscale trusts for X-Forwarded-For, and neither complains. .env.example ships a working default, and .env is the only place it is ever written.

Everything else is derived: FALLBACK_UPSTREAM, PUBLIC_IPV4 and PUBLIC_IPV6 are optional, and LEGO_PROFILE / ACME_STAGING select the certificate.

Editing .env and re-running bootstrap.sh re-renders all six config files from the templates, so a hostname change is a one-line edit rather than a find-and-replace across six files. The run prints which services need recreating, as a podman rm -f --depend … && podman-compose up -d line (see The rendered configs are generated for why it is not --force-recreate).

Second pass

Headplane needs a Headscale API key, which can only be minted against a live database. bootstrap.sh is idempotent — re-run it once the stack is up:

./scripts/bootstrap.sh

Upgrading an existing deployment: run bootstrap.sh before you start the stack, not after. The Pocket ID encryption key has moved out of the pocket-id-data volume into secrets/, because a key stored in the volume it protects cannot survive that volume being rebuilt. bootstrap.sh adopts the key actually in use; skip it and Pocket ID comes up with a different key and cannot read its own database. scripts/deploy.sh already does this in the right order.

Autostart

mkdir -p ~/.config/systemd/user
cp systemd/vpn-stack.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now vpn-stack.service
sudo loginctl enable-linger $USER   # start at boot without login

The unit hard-codes WorkingDirectory=%h/vpn-ops and podman-compose -f %h/vpn-ops/compose.yaml, so it only works if the repo really is at ~/vpn-ops. Anywhere else, change those three lines before installing it — the same applies to telemetry.service and backup.service, which point at the same directory.


Updating images

Every service in compose.yaml is pinned as tag@sha256:digest:

image: docker.io/goacme/lego:v5.5.2@sha256:1944e8c36055beec47c7de6f15202b41128be75eea0ffa257f0c14d93c5155fd

podman resolves the image by digest. The tag is there to be read by humans; if the registry ever serves a different build under that tag, the pull fails instead of quietly changing what is deployed. Each digest is an OCI image index, so this pins the content without pinning a CPU architecture.

Every tag is also an exact version — no latest, no bare major:

Image Pinned version
docker.io/envoyproxy/envoy v1.34.14
docker.io/goacme/lego v5.5.2
docker.io/headscale/headscale v0.29.5
ghcr.io/tale/headplane 0.8.0
docker.io/pocketid/pocket-id v2.18.0
otel/opentelemetry-collector-contrib 0.162.0
docker.io/alpine (Dockerfile.backup) 3.20.10

The first six are compose image: entries — the collector behind the telemetry profile. alpine:3.20.10 is a FROM in Dockerfile.backup — pinned the same way, but in the Dockerfile. The backup service's own image (localhost/bootstrapserver-backup:1) is built locally and never pulled, so there is nothing to pin for it.

Why it matters here

These tags used to float. envoy:v1.34-latest, lego:v5, pocket-id:v2 and alpine:3.20 all moved as their maintainers published patches, so before the pins a podman-compose up -d months after the original deploy could have pulled a different Envoy and a different lego than the ones this stack was tested against. That matters more than usual here: a six-day certificate means a broken lego is an outage, not a slow burn (see "Certificate lifetime and renewal").

A digest on its own would have stopped that, but it leaves the tag free to drift away from it. v5 says nothing about which 5.x is running, and the moment Alpine pushed a patch, 3.20 would no longer be the tag that answers to the digest pinned beside it — which is a reference some tools reject outright. Both halves are now exact, so each reference is a self-consistent statement of what is deployed: this version, these bytes.

The same reasoning applies in the other direction. The lego CLI changed breakingly between majors — v5 removed the renew subcommand and renamed --http.port to --http.address — and a patch or minor can do the same. With the image pinned, that cannot reach you without a commit.

Bumping

The tags in compose.yaml are exact, so nothing will ever tell you a newer patch exists. Pull the floating tag to find out what the current build is — the float is the discovery mechanism, not what you deploy:

podman pull docker.io/goacme/lego:v5
podman image inspect docker.io/goacme/lego:v5 --format '{{.Digest}}'

Or for everything at once, the Dockerfile base included:

for i in \
  docker.io/envoyproxy/envoy:v1.34-latest \
  docker.io/goacme/lego:v5 \
  docker.io/headscale/headscale:v0.29.5 \
  ghcr.io/tale/headplane:0.8.0 \
  docker.io/pocketid/pocket-id:v2 \
  otel/opentelemetry-collector-contrib:0.162.0 \
  docker.io/alpine:3.20 ; do
  podman pull "$i" >/dev/null \
    && printf '%-45s %s\n' "$i" "$(podman image inspect "$i" --format '{{.Digest}}')"
done

A digest that already matches compose.yaml means there is nothing to do. For one that does not, you need the exact patch tag — take it from the upstream release notes or the registry's tag list (https://hub.docker.com/r/goacme/lego/tags), not from the digest — then check that the tag and the digest agree before writing them down. If they disagree, podman pull on the pair fails with manifest unknown:

podman pull docker.io/goacme/lego:v5.5.3@sha256:<digest-from-above>

That doubles as the check that the reference you are about to commit is one the registry will serve.

Then edit compose.yaml: change both the tag and the sha256:…. Under an exact tag that is the entire bump — a new patch is a new tag, and writing only the digest would leave a reference that claims one version and serves another. Read the upstream release notes first; nothing about this process tests anything on your behalf.

The alpine tag and digest go in Dockerfile.backup instead, and need a rebuild to take effect:

podman-compose build backup

Apply the compose changes with:

podman-compose up -d --remove-orphans

podman-compose recreates any service whose image reference changed — and since the tag is part of that reference, that is every service you bumped, even when only a patch moved. Each one is a brief restart, so do it deliberately rather than in a hurry.

Checking what is running

podman ps --format '{{.Names}}\t{{.Image}}'   # shows the digest each is on

Pocket ID OIDC client

This is the one step that must be done by hand, in the Pocket ID UI.

  1. Browse to https://auth.vpn.example.com and create your first administrator account (register a passkey).

  2. Go to OIDC Clients → Add OIDC Client.

  3. Configure:

    Field Value
    Name headscale
    Callback URLs https://vpn.example.com/api/oidc/callback
    https://vpn.example.com/admin/oidc/callback
    Public client off
    PKCE S256
    Scopes openid, profile, email
  4. Copy the Client ID and Client Secret into .env:

    OIDC_CLIENT_ID=<the client id>
    OIDC_CLIENT_SECRET=<the client secret>
    
  5. Re-run bootstrap and restart:

    ./scripts/bootstrap.sh
    podman-compose up -d
    

Why one client for both services

Headplane links an OIDC login to an existing Headscale user by comparing the sub claim, falling back to email. A single client guarantees an identical sub in both, so users link on first sign-in with no manual selection.


Headplane agent

Headplane runs a small agent that registers itself as a node in your own tailnet. It gives the dashboard per-node OS and client version, and it is what remote web SSH from the browser rides on. It is enabled here.

The agent is not a sixth service. compose.yaml has no entry for it, and adding one would be wrong: Headplane spawns /usr/libexec/headplane/agent from inside its own container as a child process, sharing the container's network namespace. The image ships it, and that is the whole of the deployment story.

headplane container
├── node /app/build/server/index.js ──spawns──▶ hp_agent
│                                            tsnet node, tag:headplane-agent,
│                                            control URL http://headscale:8080
└── /var/lib/headplane/agent/tailscaled.state  its identity, kept across restarts

What it needs

Requirement Why Status here
Headscale 0.28+ It mints a tag-only pre-auth key; older versions have no such thing pinned at v0.29.5
headscale.api_key Creates that key on every agent start api_key_path, written by bootstrap.sh
tag:headplane-agent in the policy A node no ACL rule can reach is a node the agent cannot use added to policy.hujson
/var/lib/headplane writable Holds tailscaled.state, so the agent keeps its identity across restarts already the headplane-data volume

Applying it

podman rm -f --depend headplane && podman-compose up -d

Then watch it register. The node appears as soon as Headplane starts; the host info it collects for the other nodes follows on the first sync, three minutes later by default:

podman-compose logs -f headplane | grep -i agent
podman exec headscale headscale nodes list | grep headplane-agent

Status, last sync time and any error are in the UI at https://vpn.example.com/admin/settings/agent.

Why the policy change is not optional

Headplane creates the key with aclTags: ["tag:headplane-agent"], so the agent is a tagged node: it belongs to no user and appears in the tailnet as tagged-devices. Two consequences that are easy to miss:

  1. A tag that is not declared in tagOwners resolves to nothing, so the ACL rules naming it silently match no traffic and the agent is registered but inert.
  2. Host-info sync needs no ACL at all — the agent reads hostinfo tailscaled already holds. Only web SSH needs packet-level reach, which is why the rule is scoped to port 22 rather than *:*, and why the ssh block needs its own entry: Tailscale SSH sessions are authorised by ssh, not by acls.

Also worth knowing: the agent node is a real node. It will show up in the dashboard's node list, in headscale nodes list, and in MagicDNS as headplane-agent.tailnet.example.com.

Two things that are deliberately not done here

  • No tests entry for the agent tag. A test's src must resolve to at least one node's IPs; until the agent has registered once, nothing carries that tag, so headscale records source resolved to no IP addresses and refuses to start. Add the test after the agent is up.
  • No agent-specific compose volume or capability. tsnet runs in userspace, so no /dev/net/tun, and tailscale_netns is left at its default because this container keeps CAP_NET_ADMIN and so can set SO_MARK.

The agent talks to Headscale over the podman network on http://headscale:8080, not through Envoy, so ACME_STAGING has no effect on it either way.


Connect a device

# Create a pre-auth key (headplane admin console also has one)
podman exec headscale headscale preauthkeys create --user <you> --reusable

# On the client
tailscale up --login-server https://vpn.example.com

Or skip the key entirely and use passkey auth through Pocket ID:

tailscale up --login-server https://vpn.example.com --auth-key ""

Headplane access

Browse to https://vpn.example.com/admin — / redirects there.

The first user to sign in via OIDC becomes Owner. Everyone after that starts as Member with no access and must be promoted from the Users page.


Switching to production certificates

The stack starts with ACME_STAGING=true. Staging certificates are not trusted by browsers, so the OIDC flow will fail with certificate errors — this is expected and not a bug.

Once the full flow works end to end:

sed -i 's/^ACME_STAGING=.*/ACME_STAGING=false/' .env

# Remove the existing certificate from inside the certs VOLUME.
#
# NOT `rm certs/*.crt`: certs is a named podman volume, not a host directory.
# That command deletes nothing, lego then sees a still-valid staging
# certificate, declines to renew, and the untrusted-certificate problem
# persists for up to another three days.
#
# certificates/, because that is where lego keeps them - /certs/*.crt matches
# nothing and fails the same silent way.
#
# Leave /certs/sds.json ALONE. Envoy will not start without it, and lego
# `--requires` envoy, so deleting it means the next Envoy restart takes both
# down and nothing ever writes it back. Envoy keeps serving the staging
# certificate from it until lego replaces it with the new one.
podman exec lego sh -c 'rm -f /certs/certificates/*.crt /certs/certificates/*.key /certs/certificates/*.json'

# Recreate lego, don't just `up -d`: the renewal loop sleeps for RENEW_INTERVAL
# (12h) between checks, so without a restart you wait up to half a day for it
# to notice the files are gone. It also has to re-read ACME_STAGING.
podman rm -f lego && podman-compose up -d

Confirm with echo | openssl s_client -connect vpn.example.com:443 \ -servername vpn.example.com 2>/dev/null | openssl x509 -noout -issuer.


Certificate lifetime and renewal

Certificates use Let's Encrypt's shortlived profile — 160 hours, a little over six days — set via LEGO_PROFILE in .env. Change it to tlsserver (45d) or classic (90d) if that window is too tight.

lego issues and renews from a single lego run call every 12 hours. It obtains on the first pass, then renews only when due. lego decides when — no threshold is configured here, because a fixed one would be wrong: a six-day certificate never has 30 days left, so any --renew-days threshold would mark it due on every single check. Instead lego asks the CA's renewalInfo endpoint (ARI) when one exists, and otherwise works from the certificate's own lifetime, renewing at half-life for anything under ten days.

Why this needs watching

A 90-day certificate that fails to renew is a problem you find out about later. A six-day one is an outage you find out about in about three days: vpn.example.com stops completing TLS, and Tailscale nodes stop enrolling. Let's Encrypt's own guidance for this profile is "only if you fully trust your automation to renew on time."

The cheapest liveness signal is already in the logs — every check that decides not to renew prints the expiry and how long until renewal is possible:

podman-compose logs -f lego | grep "Skip renewal"

Watch for that line appearing regularly. If it stops appearing, the loop has stalled. That is the thing to alert on; a proper check on remaining lifetime belongs in monitoring.

Verifying renewal end to end

Renewal normally happens around day 3, which is a long time to wait for confidence. Force one instead:

podman-compose exec lego /lego run --renew-force \
  --email "$ACME_EMAIL" --domains "$VPN_DOMAIN,$AUTH_DOMAIN" \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --profile shortlived --http --http.address :5002 --path /certs --accept-tos

Envoy picks up the new certs/sds.json on its own — watched_directory means no restart. Confirm both halves landed:

podman-compose logs lego | tail -5                       # wrote /certs/sds.json
podman exec envoy curl -s http://127.0.0.1:9901/certs | head   # secret present
echo | openssl s_client -connect localhost:443 -servername vpn.example.com \
  2>/dev/null | openssl x509 -noout -dates                # new notAfter

Fallback upstream

Envoy answers on every Host header and SNI it receives. For a hostname that is neither VPN_DOMAIN nor AUTH_DOMAIN, it forwards to the address in FALLBACK_UPSTREAM in .env rather than returning 404:

Inbound Destination TLS
:80, unknown Host FALLBACK_UPSTREAM:80 cleartext
:443, unknown SNI FALLBACK_UPSTREAM:443 passthrough

The upstream therefore needs a web server of its own, and for :443 it must present a valid certificate for whatever name is being requested — Envoy never decrypts that traffic. Leave the value blank and unmatched hostnames get a 503.

curl -sSo /dev/null -w '%{http_code}\n' -H 'Host: unknown.example.com' http://localhost/
openssl s_client -connect localhost:443 -servername unknown.example.com 2>/dev/null \
  | openssl x509 -noout -subject    # the upstream's cert, not the VPN one

FALLBACK_UPSTREAM is usually a LAN address, which is outside the compose bridge subnet. Egress from a rootless podman user-defined bridge is NAT'd out through slirp4netns/pasta, so check the host can actually route there — and check it from the host running the stack, not from the machine you edit this repo on: podman run --rm --network vpn curlimages/curl -v http://<address>/

An unreachable address is not a startup failure. Envoy starts normally and unmatched hostnames simply time out after connect_timeout (5s) before returning 503. Blanking the value fails fast against loopback instead.

The rendered config/envoy/envoy.yaml is a template output — see The rendered configs are generated. Validate the rendered file, which is the one the container actually reads.

Why it is built this way

Envoy's filter_chain_match.server_names does not support * — only exact names and *.suffix wildcards — and a chain with no server_names is the least specific match. So "catch every SNI except these two" can only be expressed by naming the two domains explicitly and letting the unnamed chain be the fallback. That is the reason tls_inspector sits on :443.

The cost is that a client sending no SNI (e.g. openssl s_client without -servername) matches neither and lands on the passthrough chain.

The rendered configs are generated

All six config files are templates, rendered on every bootstrap.sh run. The backup timer is a seventh, rendered only when S3 backups are configured:

Template Rendered to Placeholders
config/envoy/envoy.yaml.tmpl config/envoy/envoy.yaml VPN_DOMAIN, AUTH_DOMAIN, escaped domain for CORS, FALLBACK_UPSTREAM
config/headscale/config.yaml.tmpl config/headscale/config.yaml VPN_DOMAIN, AUTH_DOMAIN, TAILNET_DOMAIN, VPN_SUBNET, PUBLIC_IPV4/6, OIDC_CLIENT_ID (and, with OIDC_CLIENT_SECRET, whether the issuer is set), OVERRIDE_LOCAL_DNS, DNS_SERVERS
config/headplane/config.yaml.tmpl config/headplane/config.yaml VPN_DOMAIN, AUTH_DOMAIN, OIDC_CLIENT_ID (and, with OIDC_CLIENT_SECRET, whether OIDC is enabled)
config/headscale/policy.hujson.tmpl config/headscale/policy.hujson ADMIN_USERS as a JSON array, first admin for the configtest, the DNS test target
config/otelcol/config.yaml.tmpl config/otelcol/config.yaml VPN_DOMAIN, both Grafana Cloud endpoints, instance id, Loki dataset
config/pocket-id/.env.example config/pocket-id/.env AUTH_DOMAIN, your uid/gid
systemd/backup.timer.tmpl systemd/backup.timer the OnCalendar expression assembled from BACKUP_SCHEDULE + BACKUP_SCHEDULE_TZ, plus BACKUP_RANDOM_DELAY and BACKUP_ACCURACY

The templates are the source of truth. The rendered files are gitignored; editing one is pointless, because the next bootstrap overwrites it. Each template's header documents the placeholders it takes.

systemd/backup.timer goes one step further than the rest: it is also copied into ~/.config/systemd/user/ and enabled, because a rendered file sitting in the repo would not fire anything. See Schedule.

After a bootstrap that changed something:

podman rm -f --depend envoy headscale headplane pocket-id && podman-compose up -d

(bootstrap.sh prints the exact list. Only the ones that changed. If the collector is among them, the profile has to come before the subcommand, so it is printed separately as podman rm -f otel-collector && podman-compose --profile telemetry up -d otel-collector.)

This is rm then up, not podman-compose up -d --force-recreate, because podman-compose 1.0.6 accepts that flag and ignores it. For each service it runs podman run, gets the container name … is already in use (exit 125, which looks like an error and is the normal noise), and falls back to podman start on the old container. A changed mount, env_file or image then silently never applies, and a container created against a stale mount can refuse to start at all (mount … Not a directory). --depend removes the containers that --requires the target too, which podman would otherwise refuse to do, and up -d recreates all of them. Named volumes and bind-mounted state survive.

Validate before restarting a live stack. The image reference mirrors compose.yaml on purpose — validating against a different Envoy than the one serving traffic proves nothing:

# --tmpfs and the certs volume because validate opens the access-log files and
# checks /certs/sds.json exists; without them it fails on any config.
podman run --rm --tmpfs /var/log/podman -v vpn-ops_certs:/certs:ro \
  -v "$PWD/config/envoy/envoy.yaml:/e.yaml:ro" \
  docker.io/envoyproxy/envoy:v1.34.14@sha256:cfc0678bc03cca19cbb031688acb31d510bff501ff97e163026a375fe0515d69 \
  --mode validate -c /e.yaml

Rate limiting

Nothing here throttles requests. Everything is on public hostnames, including two OIDC callbacks and three login surfaces, so the one piece of abuse mitigation is Envoy's local_ratelimit filter on exactly those callbacks:

Route Bucket Cluster
/admin/oidc/callback 5 tokens, refilled 5/60s headplane
/api/oidc/callback 5 tokens, refilled 5/60s headscale

Five attempts a minute is generous for a human finishing a login and tight enough to make credential stuffing unprofitable. Over the limit returns 429 with an x-local-rate-limit: true header.

Three deliberate choices:

  • Per route, not per listener. The filter is in http_filters with no global bucket of its own; it enforces only the typed_per_filter_config on those two routes. The Headplane SPA fires many requests per page load, so a bucket on prefix: /admin would break the dashboard. The Tailscale control protocol and Pocket ID's own login paths are untouched.
  • Per downstream address. That is what makes it a speed bump rather than a denial-of-service someone else triggers for you.
  • Exact path match, ahead of the prefix: /admin and prefix: / catch-alls. Route order matters: a prefix match placed first would swallow the callback and the limit would never apply. Exact match means a trailing-slash variant is not throttled — the redirect URI registered with Pocket ID has no trailing slash, so the form an attacker can replay is the covered one.

What this is not: a WAF. It is per-Envoy-instance (fine — there is one), it does nothing against a distributed attacker, and it does not cover Pocket ID's passkey endpoints, which are on a different hostname and were left alone rather than guessed at. Check the paths against a live login before adding rules for them.

Changing it means editing config/envoy/envoy.yaml.tmpl and re-running bootstrap.sh, then podman rm -f --depend envoy && podman-compose up -d. Validate with the command in The rendered configs are generated.


Backups

All three databases are SQLite files inside podman volumes. There is no pg_dump here — the thing that goes wrong with these is copying a live SQLite file, which is what Why not just copy the files is about.

What is captured

Path in the backup What it is Lose it and
headscale/db.sqlite Control plane: every node, key, ACL, DNS record re-enrol every device in the tailnet
headscale/noise_private.key, derp_server_private.key Server identity every client has to re-authorise
headplane/hp_persist.db Dashboard settings cosmetic
headplane/agent/ The agent's tailscaled.state the agent registers as a brand new node
pocket-id/pocket-id.db, uploads/ Users, OIDC clients, avatars everyone re-onboards their passkeys
secrets/ Headscale API key, OIDC client secrets, Pocket ID encryption key, Grafana Cloud token below

The last row is the one people leave out. secrets/pocket-id/encryption_key encrypts the token-signing keys and OIDC client secrets inside pocket-id.db — a restored database without the matching key is not a recovery, it is a file nobody can read.

Enabling it

# 1. Create a bucket, and a principal with write access to it.
#    A dedicated key scoped to the prefix below, not an account-wide one.

# 2. Configure
$EDITOR .env      # S3_ENDPOINT, S3_REGION, S3_BUCKET, AWS_ACCESS_KEY_ID,
                  # AWS_SECRET_ACCESS_KEY

# 3. Generates secrets/backup/restic_password
./scripts/bootstrap.sh

# 4. Prove it works before trusting it
podman-compose build backup
podman-compose --profile backup run --rm backup

# 5. Schedule it - set BACKUP_SCHEDULE in .env, then re-run bootstrap
./scripts/bootstrap.sh

Step 5 is just step 3 again. bootstrap.sh renders systemd/backup.timer from the BACKUP_SCHEDULE* values in .env, copies it and backup.service into ~/.config/systemd/user/, and enables the timer. There is no manual cp left to forget, and no installed copy that can drift from .env.

Requires the same lingering as the stack itself, or the timer will only fire while you are logged in:

sudo loginctl enable-linger $USER

secrets/backup/restic_password is the only way to read these backups. It is encrypted with restic's own key derivation; nobody, including AWS, can recover a repository without it. Copy it somewhere that is not this host. Losing the bucket is an inconvenience; losing this file is the end of the backups.

Schedule

Four variables in .env, and nothing else to edit:

Variable Default What it is
BACKUP_SCHEDULE "*-*-* 03:17:00" A systemd calendar expression
BACKUP_SCHEDULE_TZ UTC Timezone that expression is read in. local for the host's own
BACKUP_RANDOM_DELAY 30m Jitter, so many hosts do not all fire at the same second
BACKUP_ACCURACY 1m How precisely systemd tries to hit the time

To change when backups run, edit BACKUP_SCHEDULE and re-run ./scripts/bootstrap.sh. Then confirm what actually got installed — the jitter and the timezone both apply on top of the expression, so the next fire time is not simply what you typed:

systemctl --user list-timers backup.timer   # when it next fires
systemctl --user cat backup.timer           # the installed unit, in full

bootstrap.sh prints the next three fire times as it goes, because it runs systemd-analyze calendar on the expression before installing it. That is also the check that makes a typo safe: an expression systemd cannot parse stops the run with the parser's own message, rather than becoming a timer that is installed, enabled, and never fires.

Persistent=true is not configurable. It is the guarantee that a backup missed because the host was off still runs at next boot, and that is not a preference to be edited.

BACKUP_SCHEDULE must stay quoted. It contains a space, and bootstrap.sh sources .env with . (bootstrap.sh:38), so an unquoted value is parsed as a command and the run dies on 03:17:00: command not found. Same mechanism as ADMIN_USERS.

Running and inspecting

podman-compose --profile backup run --rm backup       # one run, now
systemctl --user start backup.service                 # same, via systemd
journalctl --user -u backup.service -f                # watch it
journalctl --user -u backup.service -n 50 --no-pager  # last run
systemctl --user list-timers backup.timer             # when it next fires

Every run logs [backup] ... and ends with the snapshot list, so the simplest liveness check is:

journalctl --user -u backup.service --since '48 hours ago' | grep '\[backup\]'

Why a timer, and not another sleep loop

lego runs a while true; do sleep; done loop because a certificate has a lifetime and the deadline moves on its own — a resident process that wakes up and asks "is it due yet?" is the natural shape. A daily backup is a scheduled event, not a continuous condition, so it gets a systemd timer instead: no container sits idle between runs, the unit gets a real exit status, and Persistent=true recovers the one property the loop had for free — if the host was powered off when the timer was due, the backup happens at next boot rather than being skipped in silence.

Why the timer is a template

The schedule is in .env with everything else, but systemd expands ${VAR} in Environment= and in no timer directive at all — OnCalendar= included. There is no unit syntax that takes a calendar expression from an EnvironmentFile=, so .env cannot simply hand it over. bootstrap.sh renders systemd/backup.timer from systemd/backup.timer.tmpl instead, the same mechanism it already uses for every file under config/.

That also dictates where the reasoning lives: render_tmpl substitutes only on non-comment lines, so the explanation of why 03:17 and not 03:00 sits next to the variable in .env, next to the choice it justifies, rather than in the generated unit where it would describe a value nobody chose there.

Why not just copy the files

Because it silently produces a backup that restores cleanly and is missing data.

Headscale runs SQLite in WAL mode (config/headscale/config.yaml). In WAL mode a commit is written to db.sqlite-wal first and folded into db.sqlite later, at a checkpoint. So cp /var/lib/headscale/db.sqlite can capture a database that is missing recent commits — verified against a WAL database held open with autocheckpoint disabled, where a plain copy of the main file did not even contain the table schema.

scripts/backup.sh therefore copies db.sqlite, db.sqlite-wal and db.sqlite-shm together into a scratch directory, and then calls SQLite's online backup API on the copy:

sqlite3 scratch/db.sqlite ".backup '/work/headscale/db.sqlite'"

That reads through the WAL and writes one self-contained file. Every volume is mounted read-only, so the backup never takes a lock against the running service and never modifies live data. The old database is left where it is — deleting it is not the backup's job.

.backup preserves the source's WAL mode in the file header, so the artifact is itself a WAL-mode database. That is harmless: a WAL-mode file with no sidecars is complete and correct, and SQLite recreates the sidecars on first open.

Encryption

restic encrypts the repository client-side before anything is uploaded. The objects sitting in the bucket are ciphertext — bucket policy and SSE-KMS are not what protects this, and on MinIO, B2 and Spaces SSE-KMS is usually unavailable, which is exactly why the client-side encryption is the thing that matters.

Retention

Daily/weekly/monthly counts are configured in .env and applied with restic forget --prune on every run. --prune repacks and re-uploads, which is the only expensive step here; the payloads are a few hundred KB. If the data ever grows enough to matter, move the prune onto its own schedule rather than dropping it.

restic check runs weekly (BACKUP_CHECK_DAY). That is the command which detects a repository that has quietly rotted — without it, a broken backup is indistinguishable from one that simply did not run, which is the failure mode nobody notices until they need the data.

BACKUP_CHECK_DAY is read by scripts/backup.sh with date +%u inside the container, which runs at TZ=UTC. BACKUP_SCHEDULE_TZ is what keeps that honest: it defaults to UTC so a run at 03:17 UTC is a run on the UTC day the check is comparing against. Set BACKUP_SCHEDULE_TZ to anything else and the two drift apart — east of UTC, a nightly backup in the small hours lands on the previous UTC day, and a check set for Sunday quietly runs on Saturday. If you move the timezone, move the check day to match.

Restoring

Restores are manual. Stop the stack first: restoring underneath a running service overwrites a SQLite file it has open, and the running process will keep serving from the old inode until it is restarted.

# 1. See what you have. Pick a snapshot id, or use 'latest'.
podman run --rm -it \
  -v "$PWD/secrets:/run/secrets:ro" \
  -e RESTIC_REPOSITORY="s3:$S3_ENDPOINT/$S3_BUCKET/$S3_PREFIX" \
  -e AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY \
  -e RESTIC_PASSWORD_FILE=/run/secrets/backup/restic_password \
  localhost/bootstrapserver-backup:1 restic snapshots

# 2. Restore it to a scratch directory on the host
podman run --rm \
  -v "$PWD/secrets:/run/secrets:ro" \
  -v /tmp/restore:/restore \
  -e RESTIC_REPOSITORY="s3:$S3_ENDPOINT/$S3_BUCKET/$S3_PREFIX" \
  -e AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY \
  -e RESTIC_PASSWORD_FILE=/run/secrets/backup/restic_password \
  localhost/bootstrapserver-backup:1 \
  restic restore latest --target /restore

ls -R /tmp/restore      # headscale/ headplane/ pocket-id/ secrets/

Then put the files back and restart:

cd ~/vpn-ops
podman-compose stop        # leave the volumes in place; we are overwriting files

# Copy /tmp/restore/secrets/. over secrets/. Do this FIRST - without the
# encryption key the restored Pocket ID database is unreadable.
cp -a /tmp/restore/secrets/. secrets/
chmod 600 secrets/pocket-id/encryption_key secrets/backup/restic_password

# Put each database back. Rootless podman volumes are owned by your user, so
# this can be done straight from the host - no throwaway container, and no
# guessing at uids inside one.
for vol in headscale-data headplane-data pocket-id-data; do
  mp=$(podman volume inspect -f '{{.Mountpoint}}' "$vol")
  cp -a "/tmp/restore/${vol%-data}/." "$mp/"
done

podman-compose up -d

Check podman-compose logs headplane headscale pocket-id before you trust it, and confirm the tailnet still lists your devices.

If you no longer have secrets/backup/restic_password, there is no restore. Every other problem here is recoverable; that one is not.


Telemetry

Sends this stack's logs and metrics to Grafana Cloud. Optional: the collector sits behind the telemetry compose profile, so podman-compose up -d does not start it, and nothing about deploying the stack depends on Grafana being reachable.

 container stdout            headscale:9090/metrics
 envoy.log ─────────────┐    envoy:9902/stats/prometheus
 headscale.log          │    pocket-id:9000/metrics
 headplane.log ─────────┼──►  otel-collector ──► prometheusremotewrite ──► metrics
 pocket-id.log          │      (2 pipelines)  └─► otlphttp/logs  ──────► Loki
 lego.log               │
 otel-collector.log ────┘
 envoy-access.log      (written by Envoy itself)
 envoy-access-http.log

Enabling it

In Grafana Cloud, on your stack, copy four values from Getting started → Send metrics and Send logs: the OTLP gateway base URL, the Prometheus remote-write base URL, the instance id, and a Cloud Access Policy Token scoped to push on that one stack. They map to GRAFANA_CLOUD_OTLP_ENDPOINT, GRAFANA_CLOUD_PROM_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID and GRAFANA_CLOUD_TOKEN; the fifth, GRAFANA_LOGS_DATASET, usually stays at default.

$EDITOR .env         # GRAFANA_CLOUD_* - see .env.example, all optional
./scripts/bootstrap.sh

mkdir -p ~/.config/systemd/user
cp systemd/telemetry.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now telemetry.service

The unit being enabled is the flag. Turn telemetry off with systemctl --user disable --now telemetry.service.

There is no TELEMETRY_ENABLED in .env, and that is deliberate. The compose profile is the one thing podman-compose actually reads; a second toggle in .env could disagree with it and leave you with a collector that starts and exports nothing. The unit's enabled state is checked instead, in deploy.sh, which restarts the collector after a deploy because the profile-less podman-compose up -d --remove-orphans it runs can remove it.

It could not have been a plain .env toggle anyway: podman-compose does not support COMPOSE_PROFILES (containers/podman-compose#1083), so there is no way to make the profile sticky from .env. This is the same shape as backup.timer — a unit you enable.

Then confirm delivery before trusting it. In Grafana Cloud → Explore, look for headscale metrics within one 30-second scrape interval — the scrape succeeds as up{...} with the headscale job, and Headscale's own counters (headscale_…) follow it:

podman-compose --profile telemetry logs -f otel-collector

There is no metric literally named headscale_up: the Prometheus receiver keeps the scraped names and their labels, so filter on job="headscale" rather than looking for a prefix that is not there.

A collector that starts and quietly exports nothing looks exactly like a collector that is working.

Why a collector and not Prometheus

Prometheus scrapes and ships metrics; it has no opinion about logs. Promtail is EOL and is being folded into Grafana Alloy. Alloy and Vector are logs/metrics-first and bolt traces on. The OpenTelemetry Collector is the one that takes logs, metrics and traces as first-class pipelines, which matters when the components of a stack each expose a different subset of the three.

What it collects

Signal Source
Headscale metrics headscale:9090/metrics
Envoy metrics envoy:9902/stats/prometheus (forwarded from the loopback-only admin interface)
Pocket ID metrics pocket-id:9000 — unverified, see below
Container logs headscale, headplane, pocket-id, lego, envoy
Envoy access logs :443 and :80, written by Envoy itself

Everything is reachable only on the vpn network. The collector publishes no host ports.

There are no traces

Nothing in this stack emits OTLP. Not Headscale, not Headplane, not Pocket ID, not tailscaled, so the collector has no otlp receiver and no traces pipeline. It used to: an unauthenticated receiver on 0.0.0.0 lets any container on the shared vpn network inject forged records, and it was carrying nothing. Add the receiver and pipeline back together, with a producer and something in front of it, if that changes. There are two ways of getting traces:

  • Derive spans from the Envoy access logs. Gives RED as traces. Useful, but a derived view of a single hop, not distributed tracing.
  • Instrument Headplane (@opentelemetry/sdk-node) and Pocket ID (Go otelhttp). That gives a real span across the OIDC handshake, which is the only genuinely multi-service flow here worth tracing.

Two Envoy changes it depends on

The admin interface is bound to 127.0.0.1, and metrics have their own listener. Binding admin to 0.0.0.0 would let the collector scrape it, but every container shares the vpn network, and admin can drain listeners, stop Envoy (/quitquitquit) and change runtime settings — a compromised pocket-id or lego could take the proxy down. So listener_metrics on :9902 forwards /stats/prometheus to admin and answers 404 to everything else. Nothing publishes 9902 either. The compose healthcheck runs inside the Envoy container against 127.0.0.1:9901, so it is unaffected.

Envoy now writes an access log. There was none. It goes to a file, not to stdout, and to a separate file per listener:

podman exec envoy tail -f /var/log/podman/envoy-access.log

The split is deliberate. The collector parses the access log as JSON, which needs it distinguishable from Envoy's own stderr prose in the same stream; and two buffered access loggers sharing one path can interleave a partial line from each. The cost is that podman-compose logs envoy no longer shows request logs, which is this repo's usual debugging idiom — hence the podman exec tail above.

The port 80 log covers the ACME HTTP-01 challenge and the HTTPS redirect. It is low volume, but an ACME challenge that never arrives is the documented cause behind "too many failed authorizations", and this is where you see it.

Query strings are stripped before anything leaves the host

/api/oidc/callback carries ?code=…&state=…, and Envoy's %REQ(:PATH)% includes the query string. Shipping that to Grafana Cloud would export live authorization codes.

Envoy can strip it at the source with %REQ_WITHOUT_QUERY(…), but that command only works against a request header and returns - when the header is absent — so whether you got a path at all would depend on whether Envoy happened to populate x-envoy-original-path on that particular request. Instead the collector strips it, which cannot have that failure mode, because by then the field is always populated.

Both the body and the path attribute are stripped. The JSON parser leaves the original line in the body, so fixing only the attribute would leave the code sitting in body, fully exported.

The unstripped lines still exist in the container-logs volume on this host, exactly as they do in podman logs envoy. Same trust boundary as before; what changes is that they no longer cross it.

Volume control

Grafana Cloud bills on ingested bytes. filter/volume drops three things:

  1. The collector's own logs — already on the host, pure duplication.
  2. Envoy's TLS scanner noise on stderr. :443 is public, so every unauthenticated connection produces a handshake-failure line whose rate is set by hostile traffic rather than by your users.
  3. Successful (< 400) static-asset fetches from the Headplane SPA. The < 400 guard is what keeps the 404s and 500s on those same paths — the interesting ones — falling through.

The filter processor does not touch metrics. For that you would need a transform with metric_statements, which is deliberately absent because there is nothing to tune until the first bill arrives.

Tune against measured numbers, not guesses: Grafana Cloud → Usage, grouped by service.name. Filtering blind is how you end up dropping the one log line that explained an outage — which here is plausibly lego's Skip renewal, the cert-expiry liveness signal this README calls the cheapest one you have.

Known gaps

  • Pocket ID metrics are unverified. The scrape target on :9000 was not confirmed against the pinned digest. A target that does not exist just fails to scrape; everything else is unaffected. Check it and delete the block from config/otelcol/config.yaml.tmpl if Pocket ID has no /metrics.
  • Envoy access log timestamps are ingest time, not request time. Envoy writes those two files, so there is no podman timestamp prefix to parse. Envoy's file logger flushes promptly, so this is within a few seconds. Every other source has its real timestamp.
  • A service that cannot write to container-logs loses its logs silently. Pocket ID in particular runs as PUID/PGID (0, which is your host user under rootless podman). See Troubleshooting.
  • The collector has no healthcheck. The image is distroless and ships no shell, so there is nothing to exec a probe with — same as headplane.

Validating the collector config

The collector's config schema has been moving: redact was renamed to redaction, exporter credentials moved out from under auth:, and filelog was deprecated in favour of file_log — all inside the 0.16x range. After any version bump, validate before deploying:

podman run --rm \
  -e GRAFANA_CLOUD_TOKEN=dummy \
  -v "$PWD/config/otelcol/config.yaml:/etc/otelcol/config.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.162.0@sha256:39923a8e… \
  validate --config=/etc/otelcol/config.yaml

Silent exit means the config is good. It fails loudly and specifically otherwise, which is the only reason the config in this repo looks the way it does.


Troubleshooting

tailscale up hangs, or the client never connects The WebSocket upgrade is not working. Confirm upgrade_configs with tailscale-control-protocol exists on both the HCM and the headscale route in config/envoy/envoy.yaml.tmpl.

If POST /ts2021 shows 404 with response flag NR in envoy-access.log and an authority of vpn.example.com:443, the client's explicit port is defeating the virtual host match — both HCMs need strip_any_host_port: true. The symptom is tailscale up never printing a login URL.

Everything, including vpn.example.com, is going to the fallback upstream tls_inspector is missing from the :443 listener, so no chain matches on SNI and all traffic reaches the default passthrough chain. Or .env holds stale domain names and bootstrap.sh has not been re-run since — the generated config/envoy/envoy.yaml is what Envoy actually reads.

Unknown hostname returns 503 instead of being forwarded FALLBACK_UPSTREAM is blank in .env. bootstrap.sh substitutes 127.0.0.1 for it, where nothing listens. Check it, re-run bootstrap, then podman rm -f --depend envoy && podman-compose up -d.

bootstrap.sh fails with "Unsubstituted PLACEHOLDER" or "Unexpanded ${VAR} reference" A placeholder in a template is misspelled, or ${VAR} was used where __PLACEHOLDER__ was meant. Either way nothing substituted it, and the app it belongs to would have started with that text in place of a hostname, a CIDR or a URL — so bootstrap.sh stops first. The message names the file; the placeholder must then appear both in that template and in the render_tmpl call for it in scripts/bootstrap.sh.

Login fails with an issuer/audience error APP_URL in config/pocket-id/.env, oidc.issuer in the Headscale config, and the issuer in the Headplane config must all read exactly https://auth.vpn.example.com. Headscale validates the issuer against the discovery document Pocket ID actually serves.

Browser reports an untrusted certificate Still on staging. See "Switching to production certificates" above.

A login returns 429, or you see x-local-rate-limit in a response The rate limiter, working. The two OIDC callbacks get 5 requests a minute each per client address. Confirm which route is responsible and raise the bucket in config/envoy/envoy.yaml.tmpl if it is a real user — but note that a bucket generous enough for shared NAT addresses stops being useful against credential stuffing. See "Rate limiting".

podman-compose refuses to start after a compose edit Most likely an older podman-compose rejecting security_opt, pids_limit or mem_limit in compose.yaml. Confirm what it parsed rather than assuming:

podman-compose config | grep -A2 -e security_opt -e pids_limit -e mem_limit

Whatever is missing from that output is the key it did not understand. Removing it costs only that specific protection; it does not affect anything else in the file.

Headplane reports "Session cookie is empty" or loops on login server.cookie_secure must be true behind TLS-terminating Envoy. It is.

ACME fails with "too many certificates already issued" Staging has a tight rate limit. Set ACME_STAGING=false and use the production endpoint, which has far more headroom.

Peers cannot connect directly, only via relay udp/3478 is not reachable. Check the firewall and any upstream NAT.

envoy restarts because /certs/sds.json is missing Should clear within seconds of lego starting: lego-renew.sh writes an empty placeholder SDS file before the first issuance, because Envoy will not start without the file and HTTP-01 cannot pass until Envoy serves :80. If it persists, lego is not running — check podman-compose logs lego. Until the real certificate lands, Envoy logs Missing SDS resources for vpn_tls and :443 resets handshakes; that is expected.

Envoy serves no certificate and TLS handshakes reset Almost always the SDS file failing to parse. Envoy logs it and keeps running, so a healthy container is not evidence the certificate loaded. Check:

podman-compose logs envoy | grep -i 'config update failure'

Anything mentioning Unable to parse JSON as proto means Envoy is reading a PEM where it expects a DiscoveryResponse. version_info, the resource "@type", and the secret name must all be present and must match tls_certificate_sds_secret_configs in envoy.yaml.tmpl. Confirm what Envoy actually parsed with podman exec envoy curl -s http://127.0.0.1:9901/certs, which returns {} when no secret is loaded.

vpn.example.com stops completing TLS, or clients report an expired certificate Almost always the renewal loop is not running. With a six-day certificate this is an outage, not a warning. Check podman-compose logs lego — you should see a Skip renewal: The certificate expires at ... line every 12 hours, and its absence means the loop is dead. Then check the current expiry:

echo | openssl s_client -connect vpn.example.com:443 \
  -servername vpn.example.com 2>/dev/null | openssl x509 -noout -dates

lego logs flag provided but not defined A lego CLI change. v5 has no renew subcommand and renamed --http.port to --http.address and --days to --renew-days; its flags also sit under run, so run has to come first. compose.yaml pins goacme/lego:v5.5.2 for exactly this reason. If you see this, the image tag was overridden, or the pin was bumped to a build with the breaking change.

lego logs lego: not found The image ships the binary at /lego and its PATH does not include /, so the script resolves it explicitly. If this appears, the image is not the expected one, or /lego is missing.

ACME fails with "too many failed authorizations" Let's Encrypt limits failed authorizations per account per hostname. Usually means :80 is not actually reaching lego — confirm curl http://<host>/.well-known/acme-challenge/<anything> is routed to lego:5002 and that ACME_STAGING matches what you intend. The issuance loop backs off on repeated failure rather than retrying on a fixed timer, so a persistent outage does not burn the limit.

Headplane cannot write DNS settings The rootless podman socket mount is the usual culprit. Confirm XDG_RUNTIME_DIR resolves on the host (echo $XDG_RUNTIME_DIR). If it is genuinely unavailable, set integration.docker.enabled: false in config/headplane/config.yaml — everything else keeps working, you just lose UI-based DNS editing.

Headplane says "Agent executable not accessible" or "requires Headscale 0.28" The binary is missing from the image, or Headscale is too old for tag-only pre-auth keys. Check both:

podman exec headplane ls -l /usr/libexec/headplane/agent
podman exec headscale headscale version

The agent node registers but the UI still shows nothing It registered, so the pre-auth key and the tag both worked. The sync reads hostinfo out of the agent's netmap, and a node is shown only the peers the policy lets it reach — so if tag:headplane-agent is missing from tagOwners, the agent sees a tailnet of one. Confirm both halves:

podman exec headscale headscale nodes list | grep headplane-agent
grep -A3 'tag:headplane-agent' config/headscale/policy.hujson

Web SSH opens a connection and every session is refused That is the ssh block, not acls. Tailscale SSH sessions are authorised separately, so tag:headplane-agent needs its own entry in the ssh section — the ACL rule only gets the packets to port 22.

Headscale refuses to start right after a policy edit A tests entry whose src resolves to nothing does this — most easily caused by adding a test for a tag that no node carries yet. See the note in policy.hujson. Check before restarting:

podman exec headscale headscale policy check -f /etc/headscale/policy.hujson

That talks to the running server over its unix socket, and the server resolves tests against its live users and nodes before answering. Run it against the live container, which is the situation this section describes.

Only if Headscale is down — which is exactly when you cannot exec into it — run the same check with --bypass-grpc-and-access-database-directly on a host that has the config and database. That flag opens the database directly, so headscale prompts DO NOT run this command if an instance of headscale is running, are you sure headscale is not running? and waits for y; add --force to answer it non-interactively.

otel-collector starts but Grafana Cloud shows nothing

Check, in order: the token (podman-compose --profile telemetry logs otel-collector shows 401s), then the endpoints in .env — they must start with https:// and must be the two different hostnames, since metrics go to Prometheus remote-write and not to the OTLP gateway. Then confirm a scrape is happening at all: the collector logs Everything is ready on startup but says nothing per scrape, so absence of errors means nothing here.

otel-collector logs failed to get config or failed to build pipelines

The config schema moved between versions. Validate against the pinned image to get the precise complaint — see Validating the collector config. The errors name the exact key ('configauth.Config' has invalid keys: basic_auth), which is what makes them fixable.

A service's logs never appear, or its logs show permission errors

The container-logs volume must be writable by the service's own uid. Pocket ID is the one to watch: it runs as PUID/PGID from config/pocket-id/.env, which bootstrap sets to 0. Under rootless podman that is your host user, not real root. Confirm what landed:

podman volume ls | grep container-logs          # <project>_container-logs
podman logs pocket-id | grep -i 'permission\|denied'

container-logs is a named volume, so it appears under a project-prefixed name — the prefix is the compose project, which podman-compose derives from the directory name, so a clone at ~/vpn-ops gives vpn-ops_container-logs.

If a permission error is the cause, give that service its own volume and mount it :U, or run it as root.

Every service is Created or Exited, and podman-compose up -d fails with dependencies of container … are not running: container state improper

That error is a symptom. podman-compose turns each depends_on into --requires, so one dependency that cannot start blocks everything above it: pocket-id ← headscale ← headplane, with envoy requiring all three and lego requiring envoy. Find the one at the bottom that is actually failing:

podman ps -a --format '{{.Names}} {{.Status}}'
podman logs --tail 50 pocket-id

open /run/secrets/encryption_key: permission denied means PUID/PGID in config/pocket-id/.env are your host uid rather than 0. Re-run ./scripts/bootstrap.sh, then podman-compose down to clear the stale dependency graph and podman-compose up -d.

Envoy will not start after the access log was added

unable to open file '/var/log/podman/envoy-access.log' means the container-logs volume is not mounted into envoy. The access log path must be writable inside the container before Envoy will start.

podman logs envoy no longer shows requests

Expected. Envoy writes request logs to files now, so that they can be parsed as JSON without colliding with Envoy's own stderr prose:

podman exec envoy tail -f /var/log/podman/envoy-access.log

See Telemetry for why.

Useful commands

podman-compose logs -f envoy headscale lego
podman-compose logs -f lego | grep "Skip renewal"   # expiry + time until renewal
podman-compose logs -f headplane | grep -i agent     # agent registration + syncs
podman-compose ps
podman exec headscale headscale nodes list
podman exec headscale headscale apikeys create --expiration 999d

# Backups
podman-compose --profile backup run --rm backup
journalctl --user -u backup.service -n 50 --no-pager
systemctl --user list-timers backup.timer
$EDITOR .env && ./scripts/bootstrap.sh        # change BACKUP_SCHEDULE, re-apply

# Telemetry
systemctl --user enable --now telemetry.service     # the enable/disable flag
podman-compose --profile telemetry logs -f otel-collector
podman exec envoy tail -f /var/log/podman/envoy-access.log

Repository layout

secrets/                      generated secrets (gitignored), one
                              directory per consumer:
                                headscale/ headplane/
                                pocket-id/ backup/
certs/                        placeholder only. The ACME certs and lego
                              account state live in a named podman VOLUME of
                              this name, not on the host filesystem - so
                              `rm certs/*.crt` from the host deletes nothing.
                              See "Switching to production certificates".
compose.yaml                  five services, plus backup and otel-collector
                              behind profiles
Dockerfile.backup             restic + sqlite for the backup service
.env                          every deployment-specific value (gitignored)
.env.example                  ^ its documented template
config/
  envoy/envoy.yaml.tmpl        listeners, routes, SDS, CORS, access log (template)
  envoy/envoy.yaml             ^ generated by bootstrap.sh         (gitignored)
  headscale/config.yaml.tmpl   control server                      (template)
  headscale/config.yaml        ^ generated by bootstrap.sh         (gitignored)
  headscale/policy.hujson.tmpl ACLs                                (template)
  headscale/policy.hujson      ^ generated by bootstrap.sh         (gitignored)
  headplane/config.yaml.tmpl   dashboard + its OIDC client         (template)
  headplane/config.yaml        ^ generated by bootstrap.sh         (gitignored)
  otelcol/config.yaml.tmpl     collectors -> Grafana Cloud         (template)
  otelcol/config.yaml          ^ generated by bootstrap.sh         (gitignored)
  pocket-id/.env.example       identity provider                   (template)
  pocket-id/.env               ^ generated by bootstrap.sh         (gitignored)
scripts/
  bootstrap.sh                secrets + config rendering (idempotent)
  deploy.sh                   rsync to a remote host and restart
  lego-renew.sh               issuance, renewal, cert -> SDS DiscoveryResponse
  backup.sh                   consistent SQLite snapshots -> restic -> S3
systemd/
  vpn-stack.service           rootless autostart
  telemetry.service           ships logs+metrics to Grafana (opt-in)
  backup.service              one backup run (driven by backup.timer)
  backup.timer.tmpl           schedule template; bootstrap.sh renders
  backup.timer                the schedule, from .env (gitignored)
  sysctl-instructions.md      :80/:443 under rootless podman

Notes on how the config is wired

Secrets. OIDC and session secrets live in secrets/, bind-mounted at /run/secrets, and are referenced with the client_secret_path / cookie_secret_path variants. These files are gitignored.

secrets/ is one directory per consumer, not one flat pile:

Directory Files Mounted into
secrets/headscale/ oidc_client_secret headscale
secrets/headplane/ api_key, cookie_secret, info_secret, oidc_client_secret headplane
secrets/pocket-id/ encryption_key pocket-id
secrets/backup/ restic_password, grafana_cloud_token backup

This was one flat directory mounted into everything, and the cost was that the internet-facing Headscale container could read headplane_cookie_secret — which forges an admin session cookie — along with the Headscale API key, restic_password (the only key that opens any backup) and the Pocket ID encryption key. Headscale and Pocket ID needed exactly one file each and were holding every credential in the stack. backup is the deliberate exception: it gets the whole tree, because archiving every secret is its job.

otel-collector is mounted nothing. It used to get the whole directory and read none of it — its Grafana token arrives through compose environment, because there is no _path equivalent for an exporter.

bootstrap.sh migrates an older flat secrets/ in place, by moving files rather than regenerating them, and never overwrites a file that already exists at the new path. That direction matters: restic_password, pocket-id/encryption_key and headplane/api_key cannot be regenerated without losing the backups, the Pocket ID database or the control-plane API.

Each subdirectory is a host bind mount (./secrets/headplane:/run/secrets:ro), not a podman volume. That matters more than it looks: it was briefly declared under the top-level volumes: block, which made secrets: a named volume that bootstrap.sh never wrote to — every service was reading an empty directory. Bootstrap generates these as files on the host, and deploy.sh deliberately excludes the directory from the rsync so each host generates its own, so a bind mount is the only shape that works. backup.sh mounts the tree the same way, and fails loudly if it comes back empty.

backup.sh and the restore procedure both cp -a .../secrets/., which is recursive and structure-preserving, so neither needed changing for the extra directory level.

The Pocket ID encryption key lives outside the data volume. It encrypts the token-signing keys and OIDC client secrets stored inside pocket-id.db, so storing it in pocket-id-data means a volume rebuild takes the database and the means of decrypting any replacement with it. It is written to secrets/pocket-id/encryption_key and read via ENCRYPTION_KEY_FILE=/run/secrets/encryption_key.

Deployments that predate this have the live key at /app/data/encryption_key inside the volume, written there by Pocket ID itself on first boot, next to a different key in secrets/ that nothing ever used. bootstrap.sh adopts the one actually in use, because repointing ENCRYPTION_KEY_FILE at the unused one would make every existing row undecryptable — silently and irreversibly. That migration runs once and is idempotent afterwards.

Headplane does not get the podman socket. Its DNS/Settings UI needs the rootless podman API to restart headscale and a read-write mount of the rendered Headscale config. The socket is host-root-equivalent for your user — it can bind-mount anything, read or write every file you can, and because it can replace containers a compromise survives podman-compose down — and Headplane is a browser-facing Node app. no-new-privileges would not limit it (a unix socket connect is not a privilege transition through execve). Since bootstrap.sh reverts every edit that UI makes, the grant bought a feature that could not be used, so integration.docker.enabled is false, headscale.config_path is absent and neither mount exists.

To bring the UI back you need all four: the socket mount in compose.yaml, the read-write config mount, integration.docker.enabled: true and headscale.config_path. bootstrap.sh still resolves PODMAN_SOCKET into .env whenever docker.enabled is true, and it avoids a nested ${XDG_RUNTIME_DIR:-…} default in compose.yaml because podman-compose does not reliably expand one.

This matters because neither app expands ${VAR} throughout its config file. Headscale applies os.ExpandEnv to client_secret_path only, and Headplane to _path values only. Writing ${OIDC_CLIENT_SECRET} into any other field would be taken literally and fail in a confusing way — hence bootstrap.sh substitutes the client id into the config files and the secrets are read from disk.

Because that is true, and because .env is full of the same names, writing ${VPN_DOMAIN} into a template instead of __VPN_DOMAIN__ is the tempting mistake — and it produces a config that parses, boots, and uses the literal string ${VPN_DOMAIN} as a hostname. bootstrap.sh guards the rendered output for it, rejecting both an unsubstituted __PLACEHOLDER__ and a leftover ${VAR}.

Every config under config/ is generated from a .tmpl. bootstrap.sh renders all six on every run, substituting values from .env; the rendered files are gitignored and deploy.sh excludes them from the rsync so each host produces its own from its own .env.

Three details matter. The render uses cat tmp > out rather than mv or sed -i because every one of these files is bind-mounted into a container and all three replace the inode — the container would otherwise keep serving the old file until it was recreated, with no error anywhere. This matters most for config/headscale/config.yaml, which was mounted read-write into Headplane and would lose its writes if it were ever mounted that way again. And the CORS allow_origin pattern uses a dot-escaped form of the domain, because . is a regex metacharacter and substituting the bare domain would widen the policy.

DNS_SERVERS is rendered as a list. bootstrap.sh turns the comma-separated value in .env into a YAML block sequence for dns.nameservers.global, and OVERRIDE_LOCAL_DNS into the boolean beside it. Two details are load-bearing.

Each address is restricted to hex digits, dots and colons. That is not a tidiness check — .env is interpolated into a YAML file that Headplane can also rewrite, so a value carrying a newline and an indented key would otherwise add settings of whoever wrote it to headscale's config: no error at bootstrap, none at startup. Admitting nothing beyond the characters an IP literal can contain makes that structurally impossible rather than relying on headscale to reject the result later.

The same character-class check applies to every other value substituted into YAML, JSON or HUJSON: the three domains (hostname charset), FALLBACK_UPSTREAM and PUBLIC_IPV4/6 (IP literal), and VPN_SUBNET (CIDR). VPN_DOMAIN is checked before the dot-escaped VPN_DOMAIN_REGEX is derived from it, so the CORS allow_origin pattern inherits the guarantee.

Worth stating plainly what that is and is not: hardening against a malformed .env, not a trust boundary. bootstrap.sh sources .env with ., so anyone who can write .env already has code execution and needs none of it. It is there because the argument above is sound, and leaving these values unvalidated while DNS_SERVERS was validated would be an inconsistency rather than a decision.

One value it does catch in practice: FALLBACK_UPSTREAM must be an IP literal, because it lands in a STATIC Envoy cluster, which cannot resolve a hostname. A hostname there produces a load failure that never mentions the endpoint.

The list is emitted at six spaces of indentation, and the script owns that indentation rather than the template. YAML nesting is significant, so re-indenting global: without changing the number in bootstrap.sh attaches the list to the wrong key — or to nothing at all — and nothing reports it.

Substitution skips comment lines. Every template documents its own placeholder names in a header comment, so render_tmpl replaces per line and leaves # and // lines alone. A whole-file replacement rewrites those headers too, which was merely cosmetic while every value was a single line; __DNS_NAMESERVERS__ is a whole YAML list, and splicing it into the middle of a comment block would drop uncommented lines into a # header and leave the file unparseable. Neither guard catches that, because both strip comments before looking.

Change values in .env and the .tmpl, not in a Headplane UI. The DNS/Settings UI is off (see above); were it on, it would edit /etc/headscale/config.yaml, which is the rendered file, and bootstrap.sh would overwrite it on the next run or deploy.

Who can enroll is OIDC_ALLOWED_*, not just ADMIN_USERS. group:admin in the policy is matched on the email claim, and email_verified_required is false (Pocket ID does not set it reliably). Unless Pocket ID forbids users from editing their own email and signing themselves up, any account the OIDC client can see could claim an admin's address. OIDC_ALLOWED_DOMAINS, OIDC_ALLOWED_USERS and OIDC_ALLOWED_GROUPS (comma-separated, no spaces) render Headscale's oidc.allowed_domains, allowed_users and allowed_groups; set at least one, and in Pocket ID turn off "users can edit their own account details" and "user signups". Groups are the sturdiest choice, since a domain admits anyone holding an address there. Each renders nothing when blank, so existing deployments are unchanged until you set one. How the three combine is Headscale's rule, not this repo's — check its OIDC docs before setting more than one.

The certs volume. Envoy's file-based SDS does not read PEM. path_config_source parses the file as a protobuf-JSON DiscoveryResponse containing the Secret, with the certificate chain and private key as inline strings. Pointing it at a concatenated PEM fails to parse, and Envoy does not treat that as fatal — it logs one warning, starts anyway, and then serves no certificate, resetting every TLS handshake while the container still reports healthy.

So lego-renew.sh builds that JSON envelope from .crt + .issuer.crt + .key and replaces /certs/sds.json atomically. Envoy has watched_directory set on /certs, so a renewal is picked up without a restart. The script only rewrites the file when the contents actually change, to avoid a pointless reload. The secret name (vpn_tls) appears in both the script and envoy.yaml.tmpl and must agree.

base_url has no /admin. Headplane serves under /admin, but its server.base_url must be the bare origin or OIDC redirect URIs come out as /admin/admin/oidc/callback. Envoy redirects exact / to /admin in the browser instead.

Cloudflare note. Do not put this behind a Cloudflare proxy or tunnel. Cloudflare does not support WebSocket POSTs, which the Tailscale control protocol requires — it will not work, by design.