- Shell 45.2%
- Go Template 44.4%
- Jinja 10.4%
A rule with no proto matches only TCP/UDP/ICMP, so the admin *:* rule does not carry GRE. Routers are grouped by tag because Headscale does not resolve bare node names as policy aliases. |
||
|---|---|---|
| ansible | ||
| certs | ||
| config | ||
| scripts | ||
| secrets | ||
| systemd | ||
| .env.example | ||
| .gitignore | ||
| AGENTS.md | ||
| compose.yaml | ||
| Dockerfile.backup | ||
| README.md | ||
Headscale + Headplane + Pocket ID behind Envoy
A self-hosted Tailscale control plane on rootless podman:
| Service | URL | Role |
|---|---|---|
| Headscale | https://vpn.example.com/ |
Tailscale control server |
| Headplane | https://vpn.example.com/admin |
Admin dashboard |
| Pocket ID | https://auth.vpn.example.com |
OIDC identity provider (passkeys) |
MagicDNS domain: tailnet.example.com
Those hostnames are placeholders. Nothing under
config/is deployment-specific: every name comes from.env, andbootstrap.shsubstitutes it into theconfig/**/*.tmpltemplates. Substituting your own domain in the examples below is the whole setup — see Deployment-specific values.
TLS certificates are obtained from Let's Encrypt by a lego sidecar and
handed to Envoy through a watched directory, so renewals never require a
restart.
All three databases are SQLite files in podman volumes, and they are backed up
to S3 by scripts/backup.sh on a systemd timer whose schedule lives in .env
(BACKUP_SCHEDULE). See Backups.
Architecture
tcp/80 tcp/443
Internet ──────► Envoy ──► /.well-known/acme-challenge/ Envoy (SDS certs)
├─► 301 to https ├─► /admin ──► headplane:3000
│ ├─► / ──► headscale:8080
│ ├─► auth. ──► pocket-id:1411
└─► any other Host ──► FALLBACK_UPSTREAM:80 └─► any other SNI ──┐
│
FALLBACK_UPSTREAM:443 ◄── raw TLS ─────────┘
udp/3478
Internet ──────► headscale (DERP STUN)
┌──────────┐ writes ┌────────┐
│ lego ├─────────────►│ /certs │◄── envoy watches
│ renew 12h│ └────────┘
└──────────┘
Three headscale-specific details that are easy to get wrong:
-
Envoy needs
upgrade_configs: { upgrade_type: tailscale-control-protocol }. Tailscale clients upgrade to the control protocol using a POST request with a non-standardUpgradeheader value. Envoy only recogniseswebsocketby default. Without this rule clients hang ontailscale upand never connect. -
Headscale ignores
X-Forwarded-Forunlesstrusted_proxiesmatches Envoy's address. Without it every node appears to originate from the Envoy container in logs and rate limiting. -
tls_inspectoron:443is load-bearing. Once the listener selects a filter chain on SNI, deleting the inspector means the SNI is never read, both chains become unreachable, and every connection — includingvpn.example.com— is silently forwarded to the fallback.
Prerequisites
On the Linux host:
- podman and podman-compose
- DNS for
vpn.example.comandauth.vpn.example.com→ this host's IP - Inbound
tcp/80,tcp/443,udp/3478 - Rootless ports unprivileged — see
systemd/sysctl-instructions.md
udp/3478is not optional. Without it, NAT traversal fails for peers behind strict NAT and the embedded DERP relay cannot accept STUN.
Provisioning the host
Everything above is host configuration, and on a cloud image it is not a matter of the ports alone — Oracle's Ubuntu image in particular ships an iptables chain that accepts SSH and rejects everything else, and reapplies it at boot. Envoy binds its ports successfully, so nothing errors and nothing arrives.
ansible/provision.yml does this part. It prepares the host and
stops there: .env still holds every deployment-specific value and
bootstrap.sh still renders the configs.
cd ansible && ansible-playbook provision.yml
It also sets up an inbound port forwarder for the fallback server — see
Port forwards. That forwarder has no
representation in compose.yaml or in any systemd unit; its documentation is
ansible/README.md.
Deploy
git clone <this-repo> ~/vpn-ops && cd ~/vpn-ops
# 1. Host prerequisites
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0
echo 'net.ipv4.ip_unprivileged_port_start=0' | sudo tee /etc/sysctl.d/99-rootless-ports.conf
# 2. Configure and generate secrets
cp .env.example .env
$EDITOR .env # VPN_DOMAIN, AUTH_DOMAIN, TAILNET_DOMAIN, ADMIN_USERS,
# ACME_EMAIL and VPN_SUBNET are all required - see below.
# Leave ACME_STAGING=true for the first run.
# FALLBACK_UPSTREAM is optional — see "Fallback upstream"
./scripts/bootstrap.sh
# 3. Start
podman-compose up -d
~/vpn-opsis not an arbitrary path. The units insystemd/hard-codeWorkingDirectory=%h/vpn-ops—vpn-stack.service,telemetry.serviceandbackup.service— andscripts/deploy.shdefaults to the same place. Clone somewhere else and you have to editWorkingDirectoryin every unit and pass the path todeploy.sh, or they will quietly act on a directory that is not there. This is also what podman-compose uses as the compose project name, so it shows up in every generated volume and container name — see Troubleshooting.
backup.timeris the one exception: it holds no paths at all, andbootstrap.shrenders it from.envand installs it for you.
Deployment-specific values
Everything deployment-specific lives in .env, and nothing else. There are six
values with no default, because bootstrap.sh fails rather than fall back to a
hostname that is not yours:
| Value | Example | What it becomes |
|---|---|---|
VPN_DOMAIN |
vpn.example.com |
Headscale server_url, Headplane base_url and public_url, Envoy's vhost and SNI chain, the ACME SAN |
AUTH_DOMAIN |
auth.vpn.example.com |
Both OIDC issuers, Pocket ID APP_URL, the second ACME SAN |
TAILNET_DOMAIN |
tailnet.example.com |
MagicDNS base_domain, so nodes resolve as <hostname>.<this> |
ADMIN_USERS |
alice@example.com,bob@example.com |
group:admin in the ACL policy; the first entry is also the policy configtest's source |
ACME_EMAIL |
ops@example.com |
Let's Encrypt contact address |
VPN_SUBNET |
10.89.0.0/24 |
The podman bridge subnet — and therefore Headscale's trusted_proxies |
TAILNET_DOMAIN must differ from VPN_DOMAIN, and ADMIN_USERS must not
contain spaces — bootstrap.sh sources .env with ., so an unquoted space is
parsed as a command.
VPN_SUBNET is required because a wrong one is silently wrong: it decides the
bridge subnet in compose.yaml and the list of addresses Headscale trusts for
X-Forwarded-For, and neither complains. .env.example ships a working
default, and .env is the only place it is ever written.
Everything else is derived: FALLBACK_UPSTREAM, PUBLIC_IPV4 and PUBLIC_IPV6
are optional, and LEGO_PROFILE / ACME_STAGING select the certificate.
Editing .env and re-running bootstrap.sh re-renders all six config files
from the templates, so a hostname change is a one-line edit rather than a
find-and-replace across six files. The run prints which services need
recreating, as a podman rm -f --depend … && podman-compose up -d line (see
The rendered configs are generated for
why it is not --force-recreate).
Second pass
Headplane needs a Headscale API key, which can only be minted against a live
database. bootstrap.sh is idempotent — re-run it once the stack is up:
./scripts/bootstrap.sh
Upgrading an existing deployment: run
bootstrap.shbefore you start the stack, not after. The Pocket ID encryption key has moved out of thepocket-id-datavolume intosecrets/, because a key stored in the volume it protects cannot survive that volume being rebuilt.bootstrap.shadopts the key actually in use; skip it and Pocket ID comes up with a different key and cannot read its own database.scripts/deploy.shalready does this in the right order.
Autostart
mkdir -p ~/.config/systemd/user
cp systemd/vpn-stack.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now vpn-stack.service
sudo loginctl enable-linger $USER # start at boot without login
The unit hard-codes WorkingDirectory=%h/vpn-ops and
podman-compose -f %h/vpn-ops/compose.yaml, so it only works if the repo really is
at ~/vpn-ops. Anywhere else, change those three lines before installing it — the
same applies to telemetry.service and backup.service, which point at the
same directory.
Updating images
Every service in compose.yaml is pinned as tag@sha256:digest:
image: docker.io/goacme/lego:v5.5.2@sha256:1944e8c36055beec47c7de6f15202b41128be75eea0ffa257f0c14d93c5155fd
podman resolves the image by digest. The tag is there to be read by humans; if the registry ever serves a different build under that tag, the pull fails instead of quietly changing what is deployed. Each digest is an OCI image index, so this pins the content without pinning a CPU architecture.
Every tag is also an exact version — no latest, no bare major:
| Image | Pinned version |
|---|---|
docker.io/envoyproxy/envoy |
v1.34.14 |
docker.io/goacme/lego |
v5.5.2 |
docker.io/headscale/headscale |
v0.29.5 |
ghcr.io/tale/headplane |
0.8.0 |
docker.io/pocketid/pocket-id |
v2.18.0 |
otel/opentelemetry-collector-contrib |
0.162.0 |
docker.io/alpine (Dockerfile.backup) |
3.20.10 |
The first six are compose image: entries — the collector behind the
telemetry profile. alpine:3.20.10 is a FROM in Dockerfile.backup — pinned
the same way, but in the Dockerfile. The backup service's own image
(localhost/bootstrapserver-backup:1) is built locally and never pulled, so
there is nothing to pin for it.
Why it matters here
These tags used to float. envoy:v1.34-latest, lego:v5, pocket-id:v2 and
alpine:3.20 all moved as their maintainers published patches, so before the
pins a podman-compose up -d months after the original deploy could have
pulled a different Envoy and a different lego than the ones this stack was
tested against. That matters more than usual here: a six-day certificate means
a broken lego is an outage, not a slow burn (see "Certificate lifetime and
renewal").
A digest on its own would have stopped that, but it leaves the tag free to drift
away from it. v5 says nothing about which 5.x is running, and the moment
Alpine pushed a patch, 3.20 would no longer be the tag that answers to the
digest pinned beside it — which is a reference some tools reject outright.
Both halves are now exact, so each reference is a self-consistent statement of
what is deployed: this version, these bytes.
The same reasoning applies in the other direction. The lego CLI changed
breakingly between majors — v5 removed the renew subcommand and renamed
--http.port to --http.address — and a patch or minor can do the same. With
the image pinned, that cannot reach you without a commit.
Bumping
The tags in compose.yaml are exact, so nothing will ever tell you a newer
patch exists. Pull the floating tag to find out what the current build is — the
float is the discovery mechanism, not what you deploy:
podman pull docker.io/goacme/lego:v5
podman image inspect docker.io/goacme/lego:v5 --format '{{.Digest}}'
Or for everything at once, the Dockerfile base included:
for i in \
docker.io/envoyproxy/envoy:v1.34-latest \
docker.io/goacme/lego:v5 \
docker.io/headscale/headscale:v0.29.5 \
ghcr.io/tale/headplane:0.8.0 \
docker.io/pocketid/pocket-id:v2 \
otel/opentelemetry-collector-contrib:0.162.0 \
docker.io/alpine:3.20 ; do
podman pull "$i" >/dev/null \
&& printf '%-45s %s\n' "$i" "$(podman image inspect "$i" --format '{{.Digest}}')"
done
A digest that already matches compose.yaml means there is nothing to do. For
one that does not, you need the exact patch tag — take it from the upstream
release notes or the registry's tag list
(https://hub.docker.com/r/goacme/lego/tags), not from the digest — then check
that the tag and the digest agree before writing them down. If they disagree,
podman pull on the pair fails with manifest unknown:
podman pull docker.io/goacme/lego:v5.5.3@sha256:<digest-from-above>
That doubles as the check that the reference you are about to commit is one the registry will serve.
Then edit compose.yaml: change both the tag and the sha256:…. Under an
exact tag that is the entire bump — a new patch is a new tag, and writing only
the digest would leave a reference that claims one version and serves another.
Read the upstream release notes first; nothing about this process tests anything
on your behalf.
The alpine tag and digest go in Dockerfile.backup instead, and need a
rebuild to take effect:
podman-compose build backup
Apply the compose changes with:
podman-compose up -d --remove-orphans
podman-compose recreates any service whose image reference changed — and since the tag is part of that reference, that is every service you bumped, even when only a patch moved. Each one is a brief restart, so do it deliberately rather than in a hurry.
Checking what is running
podman ps --format '{{.Names}}\t{{.Image}}' # shows the digest each is on
Pocket ID OIDC client
This is the one step that must be done by hand, in the Pocket ID UI.
-
Browse to
https://auth.vpn.example.comand create your first administrator account (register a passkey). -
Go to OIDC Clients → Add OIDC Client.
-
Configure:
Field Value Name headscaleCallback URLs https://vpn.example.com/api/oidc/callbackhttps://vpn.example.com/admin/oidc/callbackPublic client off PKCE S256 Scopes openid,profile,email -
Copy the Client ID and Client Secret into
.env:OIDC_CLIENT_ID=<the client id> OIDC_CLIENT_SECRET=<the client secret> -
Re-run bootstrap and restart:
./scripts/bootstrap.sh podman-compose up -d
Why one client for both services
Headplane links an OIDC login to an existing Headscale user by comparing the
sub claim, falling back to email. A single client guarantees an identical
sub in both, so users link on first sign-in with no manual selection.
Headplane agent
Headplane runs a small agent that registers itself as a node in your own tailnet. It gives the dashboard per-node OS and client version, and it is what remote web SSH from the browser rides on. It is enabled here.
The agent is not a sixth service. compose.yaml has no entry for it, and
adding one would be wrong: Headplane spawns /usr/libexec/headplane/agent from
inside its own container as a child process, sharing the container's network
namespace. The image ships it, and that is the whole of the deployment story.
headplane container
├── node /app/build/server/index.js ──spawns──▶ hp_agent
│ tsnet node, tag:headplane-agent,
│ control URL http://headscale:8080
└── /var/lib/headplane/agent/tailscaled.state its identity, kept across restarts
What it needs
| Requirement | Why | Status here |
|---|---|---|
| Headscale 0.28+ | It mints a tag-only pre-auth key; older versions have no such thing | pinned at v0.29.5 |
headscale.api_key |
Creates that key on every agent start | api_key_path, written by bootstrap.sh |
tag:headplane-agent in the policy |
A node no ACL rule can reach is a node the agent cannot use | added to policy.hujson |
/var/lib/headplane writable |
Holds tailscaled.state, so the agent keeps its identity across restarts |
already the headplane-data volume |
Applying it
podman rm -f --depend headplane && podman-compose up -d
Then watch it register. The node appears as soon as Headplane starts; the host info it collects for the other nodes follows on the first sync, three minutes later by default:
podman-compose logs -f headplane | grep -i agent
podman exec headscale headscale nodes list | grep headplane-agent
Status, last sync time and any error are in the UI at
https://vpn.example.com/admin/settings/agent.
Why the policy change is not optional
Headplane creates the key with aclTags: ["tag:headplane-agent"], so the agent
is a tagged node: it belongs to no user and appears in the tailnet as
tagged-devices. Two consequences that are easy to miss:
- A tag that is not declared in
tagOwnersresolves to nothing, so the ACL rules naming it silently match no traffic and the agent is registered but inert. - Host-info sync needs no ACL at all — the agent reads hostinfo tailscaled
already holds. Only web SSH needs packet-level reach, which is why the
rule is scoped to port 22 rather than
*:*, and why thesshblock needs its own entry: Tailscale SSH sessions are authorised byssh, not byacls.
Also worth knowing: the agent node is a real node. It will show up in the
dashboard's node list, in headscale nodes list, and in MagicDNS as
headplane-agent.tailnet.example.com.
Two things that are deliberately not done here
- No
testsentry for the agent tag. A test'ssrcmust resolve to at least one node's IPs; until the agent has registered once, nothing carries that tag, so headscale recordssource resolved to no IP addressesand refuses to start. Add the test after the agent is up. - No agent-specific compose volume or capability. tsnet runs in userspace,
so no
/dev/net/tun, andtailscale_netnsis left at its default because this container keepsCAP_NET_ADMINand so can setSO_MARK.
The agent talks to Headscale over the podman network on
http://headscale:8080, not through Envoy, so ACME_STAGING has no effect on
it either way.
Connect a device
# Create a pre-auth key (headplane admin console also has one)
podman exec headscale headscale preauthkeys create --user <you> --reusable
# On the client
tailscale up --login-server https://vpn.example.com
Or skip the key entirely and use passkey auth through Pocket ID:
tailscale up --login-server https://vpn.example.com --auth-key ""
Headplane access
Browse to https://vpn.example.com/admin — / redirects there.
The first user to sign in via OIDC becomes Owner. Everyone after that starts as Member with no access and must be promoted from the Users page.
Switching to production certificates
The stack starts with ACME_STAGING=true. Staging certificates are not
trusted by browsers, so the OIDC flow will fail with certificate errors —
this is expected and not a bug.
Once the full flow works end to end:
sed -i 's/^ACME_STAGING=.*/ACME_STAGING=false/' .env
# Remove the existing certificate from inside the certs VOLUME.
#
# NOT `rm certs/*.crt`: certs is a named podman volume, not a host directory.
# That command deletes nothing, lego then sees a still-valid staging
# certificate, declines to renew, and the untrusted-certificate problem
# persists for up to another three days.
#
# certificates/, because that is where lego keeps them - /certs/*.crt matches
# nothing and fails the same silent way.
#
# Leave /certs/sds.json ALONE. Envoy will not start without it, and lego
# `--requires` envoy, so deleting it means the next Envoy restart takes both
# down and nothing ever writes it back. Envoy keeps serving the staging
# certificate from it until lego replaces it with the new one.
podman exec lego sh -c 'rm -f /certs/certificates/*.crt /certs/certificates/*.key /certs/certificates/*.json'
# Recreate lego, don't just `up -d`: the renewal loop sleeps for RENEW_INTERVAL
# (12h) between checks, so without a restart you wait up to half a day for it
# to notice the files are gone. It also has to re-read ACME_STAGING.
podman rm -f lego && podman-compose up -d
Confirm with echo | openssl s_client -connect vpn.example.com:443 \ -servername vpn.example.com 2>/dev/null | openssl x509 -noout -issuer.
Certificate lifetime and renewal
Certificates use Let's Encrypt's shortlived profile — 160 hours, a little
over six days — set via LEGO_PROFILE in .env. Change it to tlsserver
(45d) or classic (90d) if that window is too tight.
lego issues and renews from a single lego run call every 12 hours. It
obtains on the first pass, then renews only when due. lego decides when —
no threshold is configured here, because a fixed one would be wrong: a six-day
certificate never has 30 days left, so any --renew-days threshold would mark
it due on every single check. Instead lego asks the CA's renewalInfo
endpoint (ARI) when one exists, and otherwise works from the certificate's own
lifetime, renewing at half-life for anything under ten days.
Why this needs watching
A 90-day certificate that fails to renew is a problem you find out about
later. A six-day one is an outage you find out about in about three days:
vpn.example.com stops completing TLS, and Tailscale nodes stop enrolling.
Let's Encrypt's own guidance for this profile is "only if you fully trust
your automation to renew on time."
The cheapest liveness signal is already in the logs — every check that decides not to renew prints the expiry and how long until renewal is possible:
podman-compose logs -f lego | grep "Skip renewal"
Watch for that line appearing regularly. If it stops appearing, the loop has stalled. That is the thing to alert on; a proper check on remaining lifetime belongs in monitoring.
Verifying renewal end to end
Renewal normally happens around day 3, which is a long time to wait for confidence. Force one instead:
podman-compose exec lego /lego run --renew-force \
--email "$ACME_EMAIL" --domains "$VPN_DOMAIN,$AUTH_DOMAIN" \
--server https://acme-v02.api.letsencrypt.org/directory \
--profile shortlived --http --http.address :5002 --path /certs --accept-tos
Envoy picks up the new certs/sds.json on its own — watched_directory means
no restart. Confirm both halves landed:
podman-compose logs lego | tail -5 # wrote /certs/sds.json
podman exec envoy curl -s http://127.0.0.1:9901/certs | head # secret present
echo | openssl s_client -connect localhost:443 -servername vpn.example.com \
2>/dev/null | openssl x509 -noout -dates # new notAfter
Fallback upstream
Envoy answers on every Host header and SNI it receives. For a hostname
that is neither VPN_DOMAIN nor AUTH_DOMAIN, it forwards to the address in
FALLBACK_UPSTREAM in .env rather than returning 404:
| Inbound | Destination | TLS |
|---|---|---|
:80, unknown Host |
FALLBACK_UPSTREAM:80 |
cleartext |
:443, unknown SNI |
FALLBACK_UPSTREAM:443 |
passthrough |
The upstream therefore needs a web server of its own, and for :443 it must
present a valid certificate for whatever name is being requested — Envoy never
decrypts that traffic. Leave the value blank and unmatched hostnames get a 503.
curl -sSo /dev/null -w '%{http_code}\n' -H 'Host: unknown.example.com' http://localhost/
openssl s_client -connect localhost:443 -servername unknown.example.com 2>/dev/null \
| openssl x509 -noout -subject # the upstream's cert, not the VPN one
FALLBACK_UPSTREAMis usually a LAN address, which is outside the compose bridge subnet. Egress from a rootless podman user-defined bridge is NAT'd out through slirp4netns/pasta, so check the host can actually route there — and check it from the host running the stack, not from the machine you edit this repo on:podman run --rm --network vpn curlimages/curl -v http://<address>/An unreachable address is not a startup failure. Envoy starts normally and unmatched hostnames simply time out after
connect_timeout(5s) before returning 503. Blanking the value fails fast against loopback instead.
The rendered
config/envoy/envoy.yamlis a template output — see The rendered configs are generated. Validate the rendered file, which is the one the container actually reads.
Why it is built this way
Envoy's filter_chain_match.server_names does not support * — only
exact names and *.suffix wildcards — and a chain with no server_names is
the least specific match. So "catch every SNI except these two" can only be
expressed by naming the two domains explicitly and letting the unnamed chain be
the fallback. That is the reason tls_inspector sits on :443.
The cost is that a client sending no SNI (e.g. openssl s_client without
-servername) matches neither and lands on the passthrough chain.
The rendered configs are generated
All six config files are templates, rendered on every bootstrap.sh run.
The backup timer is a seventh, rendered only when S3 backups are configured:
| Template | Rendered to | Placeholders |
|---|---|---|
config/envoy/envoy.yaml.tmpl |
config/envoy/envoy.yaml |
VPN_DOMAIN, AUTH_DOMAIN, escaped domain for CORS, FALLBACK_UPSTREAM |
config/headscale/config.yaml.tmpl |
config/headscale/config.yaml |
VPN_DOMAIN, AUTH_DOMAIN, TAILNET_DOMAIN, VPN_SUBNET, PUBLIC_IPV4/6, OIDC_CLIENT_ID (and, with OIDC_CLIENT_SECRET, whether the issuer is set), OVERRIDE_LOCAL_DNS, DNS_SERVERS |
config/headplane/config.yaml.tmpl |
config/headplane/config.yaml |
VPN_DOMAIN, AUTH_DOMAIN, OIDC_CLIENT_ID (and, with OIDC_CLIENT_SECRET, whether OIDC is enabled) |
config/headscale/policy.hujson.tmpl |
config/headscale/policy.hujson |
ADMIN_USERS as a JSON array, first admin for the configtest, the DNS test target |
config/otelcol/config.yaml.tmpl |
config/otelcol/config.yaml |
VPN_DOMAIN, both Grafana Cloud endpoints, instance id, Loki dataset |
config/pocket-id/.env.example |
config/pocket-id/.env |
AUTH_DOMAIN, your uid/gid |
systemd/backup.timer.tmpl |
systemd/backup.timer |
the OnCalendar expression assembled from BACKUP_SCHEDULE + BACKUP_SCHEDULE_TZ, plus BACKUP_RANDOM_DELAY and BACKUP_ACCURACY |
The templates are the source of truth. The rendered files are gitignored; editing one is pointless, because the next bootstrap overwrites it. Each template's header documents the placeholders it takes.
systemd/backup.timer goes one step further than the rest: it is also copied
into ~/.config/systemd/user/ and enabled, because a rendered file sitting in
the repo would not fire anything. See Schedule.
After a bootstrap that changed something:
podman rm -f --depend envoy headscale headplane pocket-id && podman-compose up -d
(bootstrap.sh prints the exact list. Only the ones that changed. If the
collector is among them, the profile has to come before the subcommand, so it is
printed separately as
podman rm -f otel-collector && podman-compose --profile telemetry up -d otel-collector.)
This is rm then up, not podman-compose up -d --force-recreate, because
podman-compose 1.0.6 accepts that flag and ignores it. For each service it runs
podman run, gets the container name … is already in use (exit 125, which
looks like an error and is the normal noise), and falls back to podman start
on the old container. A changed mount, env_file or image then silently
never applies, and a container created against a stale mount can refuse to start
at all (mount … Not a directory). --depend removes the containers that
--requires the target too, which podman would otherwise refuse to do, and
up -d recreates all of them. Named volumes and bind-mounted state survive.
Validate before restarting a live stack. The image reference mirrors
compose.yaml on purpose — validating against a different Envoy than the one
serving traffic proves nothing:
# --tmpfs and the certs volume because validate opens the access-log files and
# checks /certs/sds.json exists; without them it fails on any config.
podman run --rm --tmpfs /var/log/podman -v vpn-ops_certs:/certs:ro \
-v "$PWD/config/envoy/envoy.yaml:/e.yaml:ro" \
docker.io/envoyproxy/envoy:v1.34.14@sha256:cfc0678bc03cca19cbb031688acb31d510bff501ff97e163026a375fe0515d69 \
--mode validate -c /e.yaml
Rate limiting
Nothing here throttles requests. Everything is on public hostnames, including
two OIDC callbacks and three login surfaces, so the one piece of abuse
mitigation is Envoy's local_ratelimit filter on exactly those callbacks:
| Route | Bucket | Cluster |
|---|---|---|
/admin/oidc/callback |
5 tokens, refilled 5/60s | headplane |
/api/oidc/callback |
5 tokens, refilled 5/60s | headscale |
Five attempts a minute is generous for a human finishing a login and tight
enough to make credential stuffing unprofitable. Over the limit returns 429
with an x-local-rate-limit: true header.
Three deliberate choices:
- Per route, not per listener. The filter is in
http_filterswith no global bucket of its own; it enforces only thetyped_per_filter_configon those two routes. The Headplane SPA fires many requests per page load, so a bucket onprefix: /adminwould break the dashboard. The Tailscale control protocol and Pocket ID's own login paths are untouched. - Per downstream address. That is what makes it a speed bump rather than a denial-of-service someone else triggers for you.
- Exact
pathmatch, ahead of theprefix: /adminandprefix: /catch-alls. Route order matters: aprefixmatch placed first would swallow the callback and the limit would never apply. Exact match means a trailing-slash variant is not throttled — the redirect URI registered with Pocket ID has no trailing slash, so the form an attacker can replay is the covered one.
What this is not: a WAF. It is per-Envoy-instance (fine — there is one), it does nothing against a distributed attacker, and it does not cover Pocket ID's passkey endpoints, which are on a different hostname and were left alone rather than guessed at. Check the paths against a live login before adding rules for them.
Changing it means editing config/envoy/envoy.yaml.tmpl and re-running
bootstrap.sh, then podman rm -f --depend envoy && podman-compose up -d. Validate with the command in
The rendered configs are generated.
Backups
All three databases are SQLite files inside podman volumes. There is no
pg_dump here — the thing that goes wrong with these is copying a live SQLite
file, which is what Why not just copy the files
is about.
What is captured
| Path in the backup | What it is | Lose it and |
|---|---|---|
headscale/db.sqlite |
Control plane: every node, key, ACL, DNS record | re-enrol every device in the tailnet |
headscale/noise_private.key, derp_server_private.key |
Server identity | every client has to re-authorise |
headplane/hp_persist.db |
Dashboard settings | cosmetic |
headplane/agent/ |
The agent's tailscaled.state |
the agent registers as a brand new node |
pocket-id/pocket-id.db, uploads/ |
Users, OIDC clients, avatars | everyone re-onboards their passkeys |
secrets/ |
Headscale API key, OIDC client secrets, Pocket ID encryption key, Grafana Cloud token | below |
The last row is the one people leave out. secrets/pocket-id/encryption_key
encrypts the token-signing keys and OIDC client secrets inside
pocket-id.db — a restored database without the matching key is not a
recovery, it is a file nobody can read.
Enabling it
# 1. Create a bucket, and a principal with write access to it.
# A dedicated key scoped to the prefix below, not an account-wide one.
# 2. Configure
$EDITOR .env # S3_ENDPOINT, S3_REGION, S3_BUCKET, AWS_ACCESS_KEY_ID,
# AWS_SECRET_ACCESS_KEY
# 3. Generates secrets/backup/restic_password
./scripts/bootstrap.sh
# 4. Prove it works before trusting it
podman-compose build backup
podman-compose --profile backup run --rm backup
# 5. Schedule it - set BACKUP_SCHEDULE in .env, then re-run bootstrap
./scripts/bootstrap.sh
Step 5 is just step 3 again. bootstrap.sh renders systemd/backup.timer from
the BACKUP_SCHEDULE* values in .env, copies it and backup.service into
~/.config/systemd/user/, and enables the timer. There is no manual cp left
to forget, and no installed copy that can drift from .env.
Requires the same lingering as the stack itself, or the timer will only fire while you are logged in:
sudo loginctl enable-linger $USER
secrets/backup/restic_passwordis the only way to read these backups. It is encrypted with restic's own key derivation; nobody, including AWS, can recover a repository without it. Copy it somewhere that is not this host. Losing the bucket is an inconvenience; losing this file is the end of the backups.
Schedule
Four variables in .env, and nothing else to edit:
| Variable | Default | What it is |
|---|---|---|
BACKUP_SCHEDULE |
"*-*-* 03:17:00" |
A systemd calendar expression |
BACKUP_SCHEDULE_TZ |
UTC |
Timezone that expression is read in. local for the host's own |
BACKUP_RANDOM_DELAY |
30m |
Jitter, so many hosts do not all fire at the same second |
BACKUP_ACCURACY |
1m |
How precisely systemd tries to hit the time |
To change when backups run, edit BACKUP_SCHEDULE and re-run
./scripts/bootstrap.sh. Then confirm what actually got installed — the jitter
and the timezone both apply on top of the expression, so the next fire time is
not simply what you typed:
systemctl --user list-timers backup.timer # when it next fires
systemctl --user cat backup.timer # the installed unit, in full
bootstrap.sh prints the next three fire times as it goes, because it runs
systemd-analyze calendar on the expression before installing it. That is also
the check that makes a typo safe: an expression systemd cannot parse stops the
run with the parser's own message, rather than becoming a timer that is
installed, enabled, and never fires.
Persistent=true is not configurable. It is the guarantee that a backup
missed because the host was off still runs at next boot, and that is not a
preference to be edited.
BACKUP_SCHEDULEmust stay quoted. It contains a space, andbootstrap.shsources.envwith.(bootstrap.sh:38), so an unquoted value is parsed as a command and the run dies on03:17:00: command not found. Same mechanism asADMIN_USERS.
Running and inspecting
podman-compose --profile backup run --rm backup # one run, now
systemctl --user start backup.service # same, via systemd
journalctl --user -u backup.service -f # watch it
journalctl --user -u backup.service -n 50 --no-pager # last run
systemctl --user list-timers backup.timer # when it next fires
Every run logs [backup] ... and ends with the snapshot list, so the simplest
liveness check is:
journalctl --user -u backup.service --since '48 hours ago' | grep '\[backup\]'
Why a timer, and not another sleep loop
lego runs a while true; do sleep; done loop because a certificate has a
lifetime and the deadline moves on its own — a resident process that wakes up
and asks "is it due yet?" is the natural shape. A daily backup is a scheduled
event, not a continuous condition, so it gets a systemd timer instead: no
container sits idle between runs, the unit gets a real exit status, and
Persistent=true recovers the one property the loop had for free — if the host
was powered off when the timer was due, the backup happens at next boot rather
than being skipped in silence.
Why the timer is a template
The schedule is in .env with everything else, but systemd expands ${VAR}
in Environment= and in no timer directive at all — OnCalendar= included.
There is no unit syntax that takes a calendar expression from an
EnvironmentFile=, so .env cannot simply hand it over. bootstrap.sh
renders systemd/backup.timer from systemd/backup.timer.tmpl instead, the
same mechanism it already uses for every file under config/.
That also dictates where the reasoning lives: render_tmpl substitutes only on
non-comment lines, so the explanation of why 03:17 and not 03:00 sits
next to the variable in .env, next to the choice it justifies, rather than in
the generated unit where it would describe a value nobody chose there.
Why not just copy the files
Because it silently produces a backup that restores cleanly and is missing data.
Headscale runs SQLite in WAL mode (config/headscale/config.yaml). In WAL mode
a commit is written to db.sqlite-wal first and folded into db.sqlite later,
at a checkpoint. So cp /var/lib/headscale/db.sqlite can capture a database
that is missing recent commits — verified against a WAL database held open with
autocheckpoint disabled, where a plain copy of the main file did not even
contain the table schema.
scripts/backup.sh therefore copies db.sqlite, db.sqlite-wal and
db.sqlite-shm together into a scratch directory, and then calls SQLite's
online backup API on the copy:
sqlite3 scratch/db.sqlite ".backup '/work/headscale/db.sqlite'"
That reads through the WAL and writes one self-contained file. Every volume is mounted read-only, so the backup never takes a lock against the running service and never modifies live data. The old database is left where it is — deleting it is not the backup's job.
.backup preserves the source's WAL mode in the file header, so the artifact is
itself a WAL-mode database. That is harmless: a WAL-mode file with no sidecars
is complete and correct, and SQLite recreates the sidecars on first open.
Encryption
restic encrypts the repository client-side before anything is uploaded. The objects sitting in the bucket are ciphertext — bucket policy and SSE-KMS are not what protects this, and on MinIO, B2 and Spaces SSE-KMS is usually unavailable, which is exactly why the client-side encryption is the thing that matters.
Retention
Daily/weekly/monthly counts are configured in .env and applied with
restic forget --prune on every run. --prune repacks and re-uploads, which is
the only expensive step here; the payloads are a few hundred KB. If the data
ever grows enough to matter, move the prune onto its own schedule rather than
dropping it.
restic check runs weekly (BACKUP_CHECK_DAY). That is the command which
detects a repository that has quietly rotted — without it, a broken backup is
indistinguishable from one that simply did not run, which is the failure mode
nobody notices until they need the data.
BACKUP_CHECK_DAY is read by scripts/backup.sh with date +%u inside the
container, which runs at TZ=UTC. BACKUP_SCHEDULE_TZ is what keeps that
honest: it defaults to UTC so a run at 03:17 UTC is a run on the UTC day the
check is comparing against. Set BACKUP_SCHEDULE_TZ to anything else and the
two drift apart — east of UTC, a nightly backup in the small hours lands on the
previous UTC day, and a check set for Sunday quietly runs on Saturday. If you
move the timezone, move the check day to match.
Restoring
Restores are manual. Stop the stack first: restoring underneath a running service overwrites a SQLite file it has open, and the running process will keep serving from the old inode until it is restarted.
# 1. See what you have. Pick a snapshot id, or use 'latest'.
podman run --rm -it \
-v "$PWD/secrets:/run/secrets:ro" \
-e RESTIC_REPOSITORY="s3:$S3_ENDPOINT/$S3_BUCKET/$S3_PREFIX" \
-e AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY \
-e RESTIC_PASSWORD_FILE=/run/secrets/backup/restic_password \
localhost/bootstrapserver-backup:1 restic snapshots
# 2. Restore it to a scratch directory on the host
podman run --rm \
-v "$PWD/secrets:/run/secrets:ro" \
-v /tmp/restore:/restore \
-e RESTIC_REPOSITORY="s3:$S3_ENDPOINT/$S3_BUCKET/$S3_PREFIX" \
-e AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY \
-e RESTIC_PASSWORD_FILE=/run/secrets/backup/restic_password \
localhost/bootstrapserver-backup:1 \
restic restore latest --target /restore
ls -R /tmp/restore # headscale/ headplane/ pocket-id/ secrets/
Then put the files back and restart:
cd ~/vpn-ops
podman-compose stop # leave the volumes in place; we are overwriting files
# Copy /tmp/restore/secrets/. over secrets/. Do this FIRST - without the
# encryption key the restored Pocket ID database is unreadable.
cp -a /tmp/restore/secrets/. secrets/
chmod 600 secrets/pocket-id/encryption_key secrets/backup/restic_password
# Put each database back. Rootless podman volumes are owned by your user, so
# this can be done straight from the host - no throwaway container, and no
# guessing at uids inside one.
for vol in headscale-data headplane-data pocket-id-data; do
mp=$(podman volume inspect -f '{{.Mountpoint}}' "$vol")
cp -a "/tmp/restore/${vol%-data}/." "$mp/"
done
podman-compose up -d
Check podman-compose logs headplane headscale pocket-id before you trust it,
and confirm the tailnet still lists your devices.
If you no longer have secrets/backup/restic_password, there is no restore. Every
other problem here is recoverable; that one is not.
Telemetry
Sends this stack's logs and metrics to Grafana Cloud. Optional: the
collector sits behind the telemetry compose profile, so podman-compose up -d
does not start it, and nothing about deploying the stack depends on Grafana
being reachable.
container stdout headscale:9090/metrics
envoy.log ─────────────┐ envoy:9902/stats/prometheus
headscale.log │ pocket-id:9000/metrics
headplane.log ─────────┼──► otel-collector ──► prometheusremotewrite ──► metrics
pocket-id.log │ (2 pipelines) └─► otlphttp/logs ──────► Loki
lego.log │
otel-collector.log ────┘
envoy-access.log (written by Envoy itself)
envoy-access-http.log
Enabling it
In Grafana Cloud, on your stack, copy four values from Getting started →
Send metrics and Send logs: the OTLP gateway base URL, the Prometheus
remote-write base URL, the instance id, and a Cloud Access Policy Token scoped
to push on that one stack. They map to GRAFANA_CLOUD_OTLP_ENDPOINT,
GRAFANA_CLOUD_PROM_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID and
GRAFANA_CLOUD_TOKEN; the fifth, GRAFANA_LOGS_DATASET, usually stays at
default.
$EDITOR .env # GRAFANA_CLOUD_* - see .env.example, all optional
./scripts/bootstrap.sh
mkdir -p ~/.config/systemd/user
cp systemd/telemetry.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now telemetry.service
The unit being enabled is the flag. Turn telemetry off with
systemctl --user disable --now telemetry.service.
There is no TELEMETRY_ENABLED in .env, and that is deliberate. The compose
profile is the one thing podman-compose actually reads; a second toggle in
.env could disagree with it and leave you with a collector that starts and
exports nothing. The unit's enabled state is checked instead, in deploy.sh,
which restarts the collector after a deploy because the profile-less
podman-compose up -d --remove-orphans it runs can remove it.
It could not have been a plain .env toggle anyway: podman-compose does not
support COMPOSE_PROFILES
(containers/podman-compose#1083),
so there is no way to make the profile sticky from .env. This is the same shape
as backup.timer — a unit you enable.
Then confirm delivery before trusting it. In Grafana Cloud → Explore, look
for headscale metrics within one 30-second scrape interval — the scrape
succeeds as up{...} with the headscale job, and Headscale's own counters
(headscale_…) follow it:
podman-compose --profile telemetry logs -f otel-collector
There is no metric literally named headscale_up: the Prometheus receiver keeps
the scraped names and their labels, so filter on job="headscale" rather than
looking for a prefix that is not there.
A collector that starts and quietly exports nothing looks exactly like a collector that is working.
Why a collector and not Prometheus
Prometheus scrapes and ships metrics; it has no opinion about logs. Promtail is EOL and is being folded into Grafana Alloy. Alloy and Vector are logs/metrics-first and bolt traces on. The OpenTelemetry Collector is the one that takes logs, metrics and traces as first-class pipelines, which matters when the components of a stack each expose a different subset of the three.
What it collects
| Signal | Source |
|---|---|
| Headscale metrics | headscale:9090/metrics |
| Envoy metrics | envoy:9902/stats/prometheus (forwarded from the loopback-only admin interface) |
| Pocket ID metrics | pocket-id:9000 — unverified, see below |
| Container logs | headscale, headplane, pocket-id, lego, envoy |
| Envoy access logs | :443 and :80, written by Envoy itself |
Everything is reachable only on the vpn network. The collector publishes no
host ports.
There are no traces
Nothing in this stack emits OTLP. Not Headscale, not Headplane, not Pocket
ID, not tailscaled, so the collector has no otlp receiver and no traces
pipeline. It used to: an unauthenticated receiver on 0.0.0.0 lets any
container on the shared vpn network inject forged records, and it was carrying
nothing. Add the receiver and pipeline back together, with a producer and
something in front of it, if that changes. There are two ways of getting traces:
- Derive spans from the Envoy access logs. Gives RED as traces. Useful, but a derived view of a single hop, not distributed tracing.
- Instrument Headplane (
@opentelemetry/sdk-node) and Pocket ID (Gootelhttp). That gives a real span across the OIDC handshake, which is the only genuinely multi-service flow here worth tracing.
Two Envoy changes it depends on
The admin interface is bound to 127.0.0.1, and metrics have their own
listener. Binding admin to 0.0.0.0 would let the collector scrape it, but
every container shares the vpn network, and admin can drain listeners, stop
Envoy (/quitquitquit) and change runtime settings — a compromised pocket-id
or lego could take the proxy down. So listener_metrics on :9902 forwards
/stats/prometheus to admin and answers 404 to everything else. Nothing
publishes 9902 either. The compose healthcheck runs inside the Envoy container
against 127.0.0.1:9901, so it is unaffected.
Envoy now writes an access log. There was none. It goes to a file, not to stdout, and to a separate file per listener:
podman exec envoy tail -f /var/log/podman/envoy-access.log
The split is deliberate. The collector parses the access log as JSON, which
needs it distinguishable from Envoy's own stderr prose in the same stream; and
two buffered access loggers sharing one path can interleave a partial line from
each. The cost is that podman-compose logs envoy no longer shows request logs,
which is this repo's usual debugging idiom — hence the podman exec tail above.
The port 80 log covers the ACME HTTP-01 challenge and the HTTPS redirect. It is low volume, but an ACME challenge that never arrives is the documented cause behind "too many failed authorizations", and this is where you see it.
Query strings are stripped before anything leaves the host
/api/oidc/callback carries ?code=…&state=…, and Envoy's %REQ(:PATH)%
includes the query string. Shipping that to Grafana Cloud would export live
authorization codes.
Envoy can strip it at the source with %REQ_WITHOUT_QUERY(…), but that command
only works against a request header and returns - when the header is absent —
so whether you got a path at all would depend on whether Envoy happened to
populate x-envoy-original-path on that particular request. Instead the
collector strips it, which cannot have that failure mode, because by then the
field is always populated.
Both the body and the path attribute are stripped. The JSON parser leaves
the original line in the body, so fixing only the attribute would leave the code
sitting in body, fully exported.
The unstripped lines still exist in the container-logs volume on this host,
exactly as they do in podman logs envoy. Same trust boundary as before; what
changes is that they no longer cross it.
Volume control
Grafana Cloud bills on ingested bytes. filter/volume drops three things:
- The collector's own logs — already on the host, pure duplication.
- Envoy's TLS scanner noise on stderr.
:443is public, so every unauthenticated connection produces a handshake-failure line whose rate is set by hostile traffic rather than by your users. - Successful (
< 400) static-asset fetches from the Headplane SPA. The< 400guard is what keeps the 404s and 500s on those same paths — the interesting ones — falling through.
The filter processor does not touch metrics. For that you would need a
transform with metric_statements, which is deliberately absent because
there is nothing to tune until the first bill arrives.
Tune against measured numbers, not guesses: Grafana Cloud → Usage, grouped by
service.name. Filtering blind is how you end up dropping the one log line
that explained an outage — which here is plausibly lego's Skip renewal, the
cert-expiry liveness signal this README calls the cheapest one you have.
Known gaps
- Pocket ID metrics are unverified. The scrape target on
:9000was not confirmed against the pinned digest. A target that does not exist just fails to scrape; everything else is unaffected. Check it and delete the block fromconfig/otelcol/config.yaml.tmplif Pocket ID has no/metrics. - Envoy access log timestamps are ingest time, not request time. Envoy writes those two files, so there is no podman timestamp prefix to parse. Envoy's file logger flushes promptly, so this is within a few seconds. Every other source has its real timestamp.
- A service that cannot write to
container-logsloses its logs silently. Pocket ID in particular runs asPUID/PGID(0, which is your host user under rootless podman). See Troubleshooting. - The collector has no healthcheck. The image is distroless and ships no
shell, so there is nothing to exec a probe with — same as
headplane.
Validating the collector config
The collector's config schema has been moving: redact was renamed to
redaction, exporter credentials moved out from under auth:, and filelog
was deprecated in favour of file_log — all inside the 0.16x range. After any
version bump, validate before deploying:
podman run --rm \
-e GRAFANA_CLOUD_TOKEN=dummy \
-v "$PWD/config/otelcol/config.yaml:/etc/otelcol/config.yaml:ro" \
otel/opentelemetry-collector-contrib:0.162.0@sha256:39923a8e… \
validate --config=/etc/otelcol/config.yaml
Silent exit means the config is good. It fails loudly and specifically otherwise, which is the only reason the config in this repo looks the way it does.
Troubleshooting
tailscale up hangs, or the client never connects
The WebSocket upgrade is not working. Confirm upgrade_configs with
tailscale-control-protocol exists on both the HCM and the headscale route in
config/envoy/envoy.yaml.tmpl.
If POST /ts2021 shows 404 with response flag NR in
envoy-access.log and an authority of vpn.example.com:443, the client's
explicit port is defeating the virtual host match — both HCMs need
strip_any_host_port: true. The symptom is tailscale up never printing a
login URL.
Everything, including vpn.example.com, is going to the fallback upstream
tls_inspector is missing from the :443 listener, so no chain matches on SNI
and all traffic reaches the default passthrough chain. Or .env holds stale
domain names and bootstrap.sh has not been re-run since — the generated
config/envoy/envoy.yaml is what Envoy actually reads.
Unknown hostname returns 503 instead of being forwarded
FALLBACK_UPSTREAM is blank in .env. bootstrap.sh substitutes
127.0.0.1 for it, where nothing listens. Check it, re-run bootstrap, then
podman rm -f --depend envoy && podman-compose up -d.
bootstrap.sh fails with "Unsubstituted PLACEHOLDER" or "Unexpanded
${VAR} reference"
A placeholder in a template is misspelled, or ${VAR} was used where
__PLACEHOLDER__ was meant. Either way nothing substituted it, and the app it
belongs to would have started with that text in place of a hostname, a CIDR or
a URL — so bootstrap.sh stops first. The message names the file; the
placeholder must then appear both in that template and in the render_tmpl
call for it in scripts/bootstrap.sh.
Login fails with an issuer/audience error
APP_URL in config/pocket-id/.env, oidc.issuer in the Headscale config,
and the issuer in the Headplane config must all read exactly
https://auth.vpn.example.com. Headscale validates the issuer against the
discovery document Pocket ID actually serves.
Browser reports an untrusted certificate Still on staging. See "Switching to production certificates" above.
A login returns 429, or you see x-local-rate-limit in a response
The rate limiter, working. The two OIDC callbacks get 5 requests a minute each
per client address. Confirm which route is responsible and raise the bucket in
config/envoy/envoy.yaml.tmpl if it is a real user — but note that a bucket
generous enough for shared NAT addresses stops being useful against credential
stuffing. See "Rate limiting".
podman-compose refuses to start after a compose edit
Most likely an older podman-compose rejecting security_opt, pids_limit or
mem_limit in compose.yaml. Confirm what it parsed rather than assuming:
podman-compose config | grep -A2 -e security_opt -e pids_limit -e mem_limit
Whatever is missing from that output is the key it did not understand. Removing it costs only that specific protection; it does not affect anything else in the file.
Headplane reports "Session cookie is empty" or loops on login
server.cookie_secure must be true behind TLS-terminating Envoy. It is.
ACME fails with "too many certificates already issued"
Staging has a tight rate limit. Set ACME_STAGING=false and use the
production endpoint, which has far more headroom.
Peers cannot connect directly, only via relay
udp/3478 is not reachable. Check the firewall and any upstream NAT.
envoy restarts because /certs/sds.json is missing
Should clear within seconds of lego starting: lego-renew.sh writes an empty
placeholder SDS file before the first issuance, because Envoy will not start
without the file and HTTP-01 cannot pass until Envoy serves :80. If it persists,
lego is not running — check podman-compose logs lego. Until the real
certificate lands, Envoy logs Missing SDS resources for vpn_tls and :443
resets handshakes; that is expected.
Envoy serves no certificate and TLS handshakes reset Almost always the SDS file failing to parse. Envoy logs it and keeps running, so a healthy container is not evidence the certificate loaded. Check:
podman-compose logs envoy | grep -i 'config update failure'
Anything mentioning Unable to parse JSON as proto means Envoy is reading a
PEM where it expects a DiscoveryResponse. version_info, the resource
"@type", and the secret name must all be present and must match
tls_certificate_sds_secret_configs in envoy.yaml.tmpl. Confirm what
Envoy actually parsed with
podman exec envoy curl -s http://127.0.0.1:9901/certs, which returns {}
when no secret is loaded.
vpn.example.com stops completing TLS, or clients report an expired certificate
Almost always the renewal loop is not running. With a six-day certificate this
is an outage, not a warning. Check podman-compose logs lego — you should see
a Skip renewal: The certificate expires at ... line every 12 hours, and its
absence means the loop is dead. Then check the current expiry:
echo | openssl s_client -connect vpn.example.com:443 \
-servername vpn.example.com 2>/dev/null | openssl x509 -noout -dates
lego logs flag provided but not defined
A lego CLI change. v5 has no renew subcommand and renamed --http.port to
--http.address and --days to --renew-days; its flags also sit under
run, so run has to come first. compose.yaml pins goacme/lego:v5.5.2
for exactly this reason. If you see this, the image tag was overridden, or the
pin was bumped to a build with the breaking change.
lego logs lego: not found
The image ships the binary at /lego and its PATH does not include /, so
the script resolves it explicitly. If this appears, the image is not the
expected one, or /lego is missing.
ACME fails with "too many failed authorizations"
Let's Encrypt limits failed authorizations per account per hostname. Usually
means :80 is not actually reaching lego — confirm
curl http://<host>/.well-known/acme-challenge/<anything> is routed to
lego:5002 and that ACME_STAGING matches what you intend. The issuance loop
backs off on repeated failure rather than retrying on a fixed timer, so a
persistent outage does not burn the limit.
Headplane cannot write DNS settings
The rootless podman socket mount is the usual culprit. Confirm
XDG_RUNTIME_DIR resolves on the host (echo $XDG_RUNTIME_DIR). If it is
genuinely unavailable, set integration.docker.enabled: false in
config/headplane/config.yaml — everything else keeps working, you just lose
UI-based DNS editing.
Headplane says "Agent executable not accessible" or "requires Headscale 0.28" The binary is missing from the image, or Headscale is too old for tag-only pre-auth keys. Check both:
podman exec headplane ls -l /usr/libexec/headplane/agent
podman exec headscale headscale version
The agent node registers but the UI still shows nothing
It registered, so the pre-auth key and the tag both worked. The sync reads
hostinfo out of the agent's netmap, and a node is shown only the peers the
policy lets it reach — so if tag:headplane-agent is missing from tagOwners,
the agent sees a tailnet of one. Confirm both halves:
podman exec headscale headscale nodes list | grep headplane-agent
grep -A3 'tag:headplane-agent' config/headscale/policy.hujson
Web SSH opens a connection and every session is refused
That is the ssh block, not acls. Tailscale SSH sessions are authorised
separately, so tag:headplane-agent needs its own entry in the ssh section —
the ACL rule only gets the packets to port 22.
Headscale refuses to start right after a policy edit
A tests entry whose src resolves to nothing does this — most easily caused
by adding a test for a tag that no node carries yet. See the note in
policy.hujson. Check before restarting:
podman exec headscale headscale policy check -f /etc/headscale/policy.hujson
That talks to the running server over its unix socket, and the server resolves
tests against its live users and nodes before answering. Run it against the
live container, which is the situation this section describes.
Only if Headscale is down — which is exactly when you cannot exec into it —
run the same check with --bypass-grpc-and-access-database-directly on a host
that has the config and database. That flag opens the database directly, so
headscale prompts DO NOT run this command if an instance of headscale is running, are you sure headscale is not running? and waits for y; add
--force to answer it non-interactively.
otel-collector starts but Grafana Cloud shows nothing
Check, in order: the token (podman-compose --profile telemetry logs otel-collector
shows 401s), then the endpoints in .env — they must start with https:// and
must be the two different hostnames, since metrics go to Prometheus
remote-write and not to the OTLP gateway. Then confirm a scrape is happening at
all: the collector logs Everything is ready on startup but says nothing per
scrape, so absence of errors means nothing here.
otel-collector logs failed to get config or failed to build pipelines
The config schema moved between versions. Validate against the pinned image to
get the precise complaint — see
Validating the collector config. The errors
name the exact key ('configauth.Config' has invalid keys: basic_auth), which
is what makes them fixable.
A service's logs never appear, or its logs show permission errors
The container-logs volume must be writable by the service's own uid. Pocket ID
is the one to watch: it runs as PUID/PGID from config/pocket-id/.env,
which bootstrap sets to 0. Under rootless podman that is your host user, not
real root. Confirm what landed:
podman volume ls | grep container-logs # <project>_container-logs
podman logs pocket-id | grep -i 'permission\|denied'
container-logs is a named volume, so it appears under a project-prefixed
name — the prefix is the compose project, which podman-compose derives from the
directory name, so a clone at ~/vpn-ops gives vpn-ops_container-logs.
If a permission error is the cause, give that service its own volume and mount
it :U, or run it as root.
Every service is Created or Exited, and podman-compose up -d fails with
dependencies of container … are not running: container state improper
That error is a symptom. podman-compose turns each depends_on into
--requires, so one dependency that cannot start blocks everything above it:
pocket-id ← headscale ← headplane, with envoy requiring all three and
lego requiring envoy. Find the one at the bottom that is actually failing:
podman ps -a --format '{{.Names}} {{.Status}}'
podman logs --tail 50 pocket-id
open /run/secrets/encryption_key: permission denied means PUID/PGID in
config/pocket-id/.env are your host uid rather than 0. Re-run
./scripts/bootstrap.sh, then podman-compose down to clear the stale
dependency graph and podman-compose up -d.
Envoy will not start after the access log was added
unable to open file '/var/log/podman/envoy-access.log' means the
container-logs volume is not mounted into envoy. The access log path must
be writable inside the container before Envoy will start.
podman logs envoy no longer shows requests
Expected. Envoy writes request logs to files now, so that they can be parsed as JSON without colliding with Envoy's own stderr prose:
podman exec envoy tail -f /var/log/podman/envoy-access.log
See Telemetry for why.
Useful commands
podman-compose logs -f envoy headscale lego
podman-compose logs -f lego | grep "Skip renewal" # expiry + time until renewal
podman-compose logs -f headplane | grep -i agent # agent registration + syncs
podman-compose ps
podman exec headscale headscale nodes list
podman exec headscale headscale apikeys create --expiration 999d
# Backups
podman-compose --profile backup run --rm backup
journalctl --user -u backup.service -n 50 --no-pager
systemctl --user list-timers backup.timer
$EDITOR .env && ./scripts/bootstrap.sh # change BACKUP_SCHEDULE, re-apply
# Telemetry
systemctl --user enable --now telemetry.service # the enable/disable flag
podman-compose --profile telemetry logs -f otel-collector
podman exec envoy tail -f /var/log/podman/envoy-access.log
Repository layout
secrets/ generated secrets (gitignored), one
directory per consumer:
headscale/ headplane/
pocket-id/ backup/
certs/ placeholder only. The ACME certs and lego
account state live in a named podman VOLUME of
this name, not on the host filesystem - so
`rm certs/*.crt` from the host deletes nothing.
See "Switching to production certificates".
compose.yaml five services, plus backup and otel-collector
behind profiles
Dockerfile.backup restic + sqlite for the backup service
.env every deployment-specific value (gitignored)
.env.example ^ its documented template
config/
envoy/envoy.yaml.tmpl listeners, routes, SDS, CORS, access log (template)
envoy/envoy.yaml ^ generated by bootstrap.sh (gitignored)
headscale/config.yaml.tmpl control server (template)
headscale/config.yaml ^ generated by bootstrap.sh (gitignored)
headscale/policy.hujson.tmpl ACLs (template)
headscale/policy.hujson ^ generated by bootstrap.sh (gitignored)
headplane/config.yaml.tmpl dashboard + its OIDC client (template)
headplane/config.yaml ^ generated by bootstrap.sh (gitignored)
otelcol/config.yaml.tmpl collectors -> Grafana Cloud (template)
otelcol/config.yaml ^ generated by bootstrap.sh (gitignored)
pocket-id/.env.example identity provider (template)
pocket-id/.env ^ generated by bootstrap.sh (gitignored)
scripts/
bootstrap.sh secrets + config rendering (idempotent)
deploy.sh rsync to a remote host and restart
lego-renew.sh issuance, renewal, cert -> SDS DiscoveryResponse
backup.sh consistent SQLite snapshots -> restic -> S3
systemd/
vpn-stack.service rootless autostart
telemetry.service ships logs+metrics to Grafana (opt-in)
backup.service one backup run (driven by backup.timer)
backup.timer.tmpl schedule template; bootstrap.sh renders
backup.timer the schedule, from .env (gitignored)
sysctl-instructions.md :80/:443 under rootless podman
Notes on how the config is wired
Secrets. OIDC and session secrets live in secrets/, bind-mounted at
/run/secrets, and are referenced with the client_secret_path /
cookie_secret_path variants. These files are gitignored.
secrets/ is one directory per consumer, not one flat pile:
| Directory | Files | Mounted into |
|---|---|---|
secrets/headscale/ |
oidc_client_secret |
headscale |
secrets/headplane/ |
api_key, cookie_secret, info_secret, oidc_client_secret |
headplane |
secrets/pocket-id/ |
encryption_key |
pocket-id |
secrets/backup/ |
restic_password, grafana_cloud_token |
backup |
This was one flat directory mounted into everything, and the cost was that the
internet-facing Headscale container could read headplane_cookie_secret — which
forges an admin session cookie — along with the Headscale API key,
restic_password (the only key that opens any backup) and the Pocket ID
encryption key. Headscale and Pocket ID needed exactly one file each and were
holding every credential in the stack. backup is the deliberate exception: it
gets the whole tree, because archiving every secret is its job.
otel-collector is mounted nothing. It used to get the whole directory and
read none of it — its Grafana token arrives through compose environment,
because there is no _path equivalent for an exporter.
bootstrap.sh migrates an older flat secrets/ in place, by moving files
rather than regenerating them, and never overwrites a file that already exists
at the new path. That direction matters: restic_password,
pocket-id/encryption_key and headplane/api_key cannot be regenerated without
losing the backups, the Pocket ID database or the control-plane API.
Each subdirectory is a host bind mount (./secrets/headplane:/run/secrets:ro),
not a podman volume. That matters more than it looks: it was briefly declared
under the top-level volumes: block, which made secrets: a named volume that
bootstrap.sh never wrote to — every service was reading an empty directory.
Bootstrap generates these as files on the host, and deploy.sh deliberately
excludes the directory from the rsync so each host generates its own, so a bind
mount is the only shape that works. backup.sh mounts the tree the same way,
and fails loudly if it comes back empty.
backup.sh and the restore procedure both cp -a .../secrets/., which is
recursive and structure-preserving, so neither needed changing for the extra
directory level.
The Pocket ID encryption key lives outside the data volume. It encrypts the
token-signing keys and OIDC client secrets stored inside pocket-id.db, so
storing it in pocket-id-data means a volume rebuild takes the database and the
means of decrypting any replacement with it. It is written to
secrets/pocket-id/encryption_key and read via
ENCRYPTION_KEY_FILE=/run/secrets/encryption_key.
Deployments that predate this have the live key at /app/data/encryption_key
inside the volume, written there by Pocket ID itself on first boot, next to a
different key in secrets/ that nothing ever used. bootstrap.sh adopts the
one actually in use, because repointing ENCRYPTION_KEY_FILE at the unused one
would make every existing row undecryptable — silently and irreversibly. That
migration runs once and is idempotent afterwards.
Headplane does not get the podman socket. Its DNS/Settings UI needs the
rootless podman API to restart headscale and a read-write mount of the rendered
Headscale config. The socket is host-root-equivalent for your user — it can
bind-mount anything, read or write every file you can, and because it can
replace containers a compromise survives podman-compose down — and Headplane is
a browser-facing Node app. no-new-privileges would not limit it (a unix socket
connect is not a privilege transition through execve). Since bootstrap.sh
reverts every edit that UI makes, the grant bought a feature that could not be
used, so integration.docker.enabled is false, headscale.config_path is
absent and neither mount exists.
To bring the UI back you need all four: the socket mount in compose.yaml,
the read-write config mount, integration.docker.enabled: true and
headscale.config_path. bootstrap.sh still resolves PODMAN_SOCKET into
.env whenever docker.enabled is true, and it avoids a nested
${XDG_RUNTIME_DIR:-…} default in compose.yaml because podman-compose does not
reliably expand one.
This matters because neither app expands ${VAR} throughout its config
file. Headscale applies os.ExpandEnv to client_secret_path only, and
Headplane to _path values only. Writing ${OIDC_CLIENT_SECRET} into any
other field would be taken literally and fail in a confusing way — hence
bootstrap.sh substitutes the client id into the config files and the
secrets are read from disk.
Because that is true, and because .env is full of the same names, writing
${VPN_DOMAIN} into a template instead of __VPN_DOMAIN__ is the tempting
mistake — and it produces a config that parses, boots, and uses the literal
string ${VPN_DOMAIN} as a hostname. bootstrap.sh guards the rendered
output for it, rejecting both an unsubstituted __PLACEHOLDER__ and a
leftover ${VAR}.
Every config under config/ is generated from a .tmpl. bootstrap.sh
renders all six on every run, substituting values from .env; the rendered
files are gitignored and deploy.sh excludes them from the rsync so each host
produces its own from its own .env.
Three details matter. The render uses cat tmp > out rather than mv or
sed -i because every one of these files is bind-mounted into a container and
all three replace the inode — the container would otherwise keep serving the
old file until it was recreated, with no error anywhere. This matters most for
config/headscale/config.yaml, which was mounted read-write into Headplane and
would lose its writes if it were ever mounted that way again. And the CORS
allow_origin pattern uses a dot-escaped form of the domain, because . is a
regex metacharacter and substituting the bare domain would widen the policy.
DNS_SERVERS is rendered as a list. bootstrap.sh turns the comma-separated
value in .env into a YAML block sequence for dns.nameservers.global, and
OVERRIDE_LOCAL_DNS into the boolean beside it. Two details are load-bearing.
Each address is restricted to hex digits, dots and colons. That is not a
tidiness check — .env is interpolated into a YAML file that Headplane can also
rewrite, so a value carrying a newline and an indented key would otherwise add
settings of whoever wrote it to headscale's config: no error at bootstrap, none
at startup. Admitting nothing beyond the characters an IP literal can contain
makes that structurally impossible rather than relying on headscale to reject the
result later.
The same character-class check applies to every other value substituted into
YAML, JSON or HUJSON: the three domains (hostname charset), FALLBACK_UPSTREAM
and PUBLIC_IPV4/6 (IP literal), and VPN_SUBNET (CIDR). VPN_DOMAIN is
checked before the dot-escaped VPN_DOMAIN_REGEX is derived from it, so the
CORS allow_origin pattern inherits the guarantee.
Worth stating plainly what that is and is not: hardening against a malformed
.env, not a trust boundary. bootstrap.sh sources .env with ., so
anyone who can write .env already has code execution and needs none of it. It
is there because the argument above is sound, and leaving these values
unvalidated while DNS_SERVERS was validated would be an inconsistency rather
than a decision.
One value it does catch in practice: FALLBACK_UPSTREAM must be an IP literal,
because it lands in a STATIC Envoy cluster, which cannot resolve a hostname.
A hostname there produces a load failure that never mentions the endpoint.
The list is emitted at six spaces of indentation, and the script owns that
indentation rather than the template. YAML nesting is significant, so re-indenting
global: without changing the number in bootstrap.sh attaches the list to the
wrong key — or to nothing at all — and nothing reports it.
Substitution skips comment lines. Every template documents its own
placeholder names in a header comment, so render_tmpl replaces per line and
leaves # and // lines alone. A whole-file replacement rewrites those headers
too, which was merely cosmetic while every value was a single line;
__DNS_NAMESERVERS__ is a whole YAML list, and splicing it into the middle of a
comment block would drop uncommented lines into a # header and leave the file
unparseable. Neither guard catches that, because both strip comments before
looking.
Change values in .env and the .tmpl, not in a Headplane UI. The
DNS/Settings UI is off (see above); were it on, it would edit
/etc/headscale/config.yaml, which is the rendered file, and bootstrap.sh
would overwrite it on the next run or deploy.
Who can enroll is OIDC_ALLOWED_*, not just ADMIN_USERS. group:admin in
the policy is matched on the email claim, and email_verified_required is
false (Pocket ID does not set it reliably). Unless Pocket ID forbids users from
editing their own email and signing themselves up, any account the OIDC client can
see could claim an admin's address. OIDC_ALLOWED_DOMAINS, OIDC_ALLOWED_USERS
and OIDC_ALLOWED_GROUPS (comma-separated, no spaces) render Headscale's
oidc.allowed_domains, allowed_users and allowed_groups; set at least one, and
in Pocket ID turn off "users can edit their own account details" and "user
signups". Groups are the sturdiest choice, since a domain admits anyone holding an
address there. Each renders nothing when blank, so existing deployments are
unchanged until you set one. How the three combine is Headscale's rule, not this
repo's — check its OIDC docs before setting more than one.
The certs volume. Envoy's file-based SDS does not read PEM.
path_config_source parses the file as a protobuf-JSON DiscoveryResponse
containing the Secret, with the certificate chain and private key as inline
strings. Pointing it at a concatenated PEM fails to parse, and Envoy does not
treat that as fatal — it logs one warning, starts anyway, and then serves no
certificate, resetting every TLS handshake while the container still reports
healthy.
So lego-renew.sh builds that JSON envelope from .crt + .issuer.crt +
.key and replaces /certs/sds.json atomically. Envoy has watched_directory
set on /certs, so a renewal is picked up without a restart. The script only
rewrites the file when the contents actually change, to avoid a pointless
reload. The secret name (vpn_tls) appears in both the script and
envoy.yaml.tmpl and must agree.
base_url has no /admin. Headplane serves under /admin, but its
server.base_url must be the bare origin or OIDC redirect URIs come out as
/admin/admin/oidc/callback. Envoy redirects exact / to /admin in the
browser instead.
Cloudflare note. Do not put this behind a Cloudflare proxy or tunnel. Cloudflare does not support WebSocket POSTs, which the Tailscale control protocol requires — it will not work, by design.