warm: self-heal wedged dnsmasq, run after deploy, shared state dir

dnsmasq adds IPs to an nft set only on a fresh upstream reply; a cache
hit never refreshes the set. On Archie the :5350 instance also wedged:
TCP child processes first, then the main process, logged
'nftset ... netlink: Error: cache initialization failed: Protocol error'
on every add and stopped filling bbrkn_v4/v6 until restarted.

warm-nftset.sh now counts those errors for the main PID during the run
(journalctl _PID=) and, if any, restarts the service and warms again;
exit 1 if it still fails. The state dir defaults to /var/lib/bbrkn so the
root cron job and the deploy share one lock (LOCK_WAIT lets deploy wait).

deploy-to-gateway.sh runs the warmer right after flushing the sets,
since clients keep old IPs cached and would bypass the tunnel until the
next fresh resolve. A warm failure only warns; it does not roll back.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DR7zpSsdX28T4Kn6v4jPSS

CLAUDE.md

@@ -38,7 +38,7 @@ Two bash scripts, driven by env vars (defaults in each script's header, overridd
 - **Ordering pass** emits each base domain, a `# domain — N subdomains` comment, then its not-yet-written related domains; a related domain is attributed to whichever base comes first alphabetically. A final uniqueness check `exit 5`s (config NOT written) if any domain would appear twice — treat that as a real bug in the ordering logic, not a data issue.
 - Output lines: `nftset=/<domain>/$NFTSET_SPEC` (NFTSET_CONF → `90-nftset.conf`; default `NFTSET_SPEC=4#inet#filter#bbrkn_v4,6#inet#filter#bbrkn_v6`) and `server=/<domain>/$DNS_SERVER` (RESOLVE_CONF → `92-resolve-bbrkn.conf`; `DNS_SERVER` default `127.0.0.1#5350`, the host resolver pihole delegates to).
 
-**`scripts/deploy-to-gateway.sh`** — deploys to **two targets**: `90-nftset.conf` → host `$NFTSET_TARGET_DIR` (`/etc/dnsmasq.d`, the `:5350` instance) and `92-resolve-bbrkn.conf` → pihole `$RESOLVE_TARGET_DIR` (`/opt/appdata/pihole/etc/dnsmasq.d`). Flow: timestamped-backup both → copy → **`systemctl restart $HOST_DNSMASQ_SVC` + `docker restart $DOCKER_CONTAINER`** (a full restart is required — dnsmasq **SIGHUP does not re-read `nftset=`/`server=` directives**, only hosts files) → wait for `:5350` listening + pihole `running` → `nft flush set` on `bbrkn_v4`/`bbrkn_v6` → DNS health check via pihole `:53` → end-to-end nftset-capture check (resolve a bbrkn domain via `:5350`, assert the set populated). **On any failure it calls `rollback()`**, restoring both backups and restarting both instances. Edits must preserve that rollback path — a broken deploy can break DNS for every client on the gateway.
+**`scripts/deploy-to-gateway.sh`** — deploys to **two targets**: `90-nftset.conf` → host `$NFTSET_TARGET_DIR` (`/etc/dnsmasq.d`, the `:5350` instance) and `92-resolve-bbrkn.conf` → pihole `$RESOLVE_TARGET_DIR` (`/opt/appdata/pihole/etc/dnsmasq.d`). Flow: timestamped-backup both → copy → **`systemctl restart $HOST_DNSMASQ_SVC` + `docker restart $DOCKER_CONTAINER`** (a full restart is required — dnsmasq **SIGHUP does not re-read `nftset=`/`server=` directives**, only hosts files) → wait for `:5350` listening + pihole `running` → `nft flush set` on `bbrkn_v4`/`bbrkn_v6` → DNS health check via pihole `:53` → end-to-end nftset-capture check (resolve a bbrkn domain via `:5350`, assert the set populated). Then it runs `warm-nftset.sh` (non-fatal) to refill the just-flushed sets. Backups go to `$BACKUP_DIR` (`/var/backups/bbrkn`, newest `BACKUP_KEEP`=10) — never into a dnsmasq conf dir, which would load them as live config. **On any failure it calls `rollback()`**, restoring both backups and restarting both instances. Edits must preserve that rollback path — a broken deploy can break DNS for every client on the gateway.
 
 ## Deployment / CI
 
@@ -46,6 +46,7 @@ Two bash scripts, driven by env vars (defaults in each script's header, overridd
 
 ## Gotchas
 
+- **dnsmasq only adds IPs to an nft set on a fresh upstream reply**, never on a cache hit — hence `cache-size=0` on the `:5350` instance and the hourly warmer (`/etc/cron.d/bbrkn-warm` → `/usr/local/sbin/bbrkn-warm`, a manually installed copy of `scripts/warm-nftset.sh`). The `:5350` main process can also wedge (`nftset … cache initialization failed: Protocol error` on every add until restart); the warmer detects that via `journalctl _PID=` and restarts it.
 - `domains.txt` is the primary data file; commits are typically domain additions. Run `make check` after editing it.
 - `IGNORE_PARTS` is space-separated substrings; any domain (base or related) containing one is dropped.
 - **`#` gotcha**: `DNS_SERVER=127.0.0.1#5350` — `#` starts a comment in Makefiles (escape as `127.0.0.1\#5350`) and can start one in YAML (quote it). The scripts themselves take the plain value.