Nightly demo reset: the demos were killed, not the migration

documenta-, lenbachhaus- and stadtmuseum-demo, night of 2026-11-13

What it looked like

Demo	Log ends at	podman ps	podman exec
stadtmuseum-demo	Applying 0113_link_contract	Up (unhealthy)	crun: not running
documenta-demo	Applying 0149_virtual_field_cache	Up (unhealthy)	crun: not running
lenbachhaus-demo	Applying 0159_english_index_names	Up (unhealthy)	crun: not running

Three different migrations, no error, no OOM, no restart. The processes inside the containers were gone, but podman still listed the containers as running.

What happened (from the host journal)

03:03–03:04moliri-demo-reset.service recreates the demos with podman-compose up --force-recreate. The conmon of each new container stays in the service's cgroup.
03:04:52moliri-ctl prints "Alle Demos gesund" and exits. The healthcheck only pings MariaDB, and MariaDB answers while migrations run.
03:04:52The oneshot unit ends. With the default KillMode=control-group, systemd sends SIGTERM to everything left in its cgroup.
03:04:56mkg-hamburg-demo (nothing to migrate) exits on the signal. Its restart policy starts it again in a scope of its own, so it survives.
03:06:22State 'stop-sigterm' timed out. Killing process … (conmon) with signal SIGKILL ×3. The three demos that were still migrating die. With their conmon gone, podman never notices.
03:08 →Healthcheck fails with no output, Traefik answers 404, the watchdog runs out of restarts.

The fix

Before

[Service]
Type=oneshot
ExecStart=…/moliri-ctl.sh demo-reset
TimeoutStartSec=420
# KillMode=control-group (default)

After

[Service]
Type=oneshot
ExecStart=…/moliri-ctl.sh demo-reset
TimeoutStartSec=420
KillMode=process

Only moliri-ctl is signalled, and it has exited by then. The containers keep running. In addition, demo-reset now waits until boot_state.sh reports that the demo has finished starting, instead of trusting "healthy" alone. 0113 and the other migrations need no change.

On the host, once it is on master

cd /pods/moliri_sveltekit; git pull --ff-only
systemctl daemon-reload
podman rm -f moliri-db-stadtmuseum-demo moliri-db-documenta-demo moliri-db-lenbachhaus-demo
rm -f /var/lib/moliri-watchdog/state/recover_moliri-db-*-demo
systemctl start moliri-demo-reset.service

The last command takes a few minutes. Afterwards, journalctl -u moliri-demo-reset -n 20 shows "remains running after unit stopped" for each conmon. That message is expected now.