Troubleshooting

Deployment

MongoDB will crash-loop / deploy aborts with AVX error The logs (or single) VM’s CPU doesn’t expose AVX. On the Proxmox host: qm set <vmid> --cpu host && qm stop <vmid> && qm start <vmid>, re-run the deploy. (The setup script creates VMs with --cpu host; this bites VMs created some other way.)

Guest agent never reports an IP Cloud-init may still be installing qemu-guest-agent. Check the VM console (qm terminal <vmid>). If the vendor snippet didn’t apply, verify local storage allows snippets (Datacenter → Storage → local → Content: Snippets).

Graylog stuck “waiting for opensearch” docker logs graylog-opensearch — usually heap (raise OPENSEARCH_HEAP in /opt/alethurgy/.env, re-run deploy) or vm.max_map_count (the script sets 262144; verify with sysctl vm.max_map_count).

Wazuh dashboard 503 for several minutes Normal on first boot — the indexer initializes security indices. If it persists >10 min: docker logs single-node-wazuh.indexer-1.

Dashboard downloads failed fetch-dashboards.sh needs outbound HTTPS to grafana.com. Re-run it any time: bash scripts/fetch-dashboards.sh — cached files are kept, and Grafana picks up new JSON within a minute.

Runtime

A target shows DOWN in Prometheus curl http://<target>:<port>/metrics from the metrics VM. Firewalls on the node (9100/8080) are the usual cause. Remember up == 0 alerts wait 3–5 minutes before firing.

No flows in ntopng Confirm the router exports NetFlow v5/v9/IPFIX (not sFlow-only) to <metrics-vm>:2055/udp; check docker logs netflow2ng for received packets; in ntopng select the tcp://netflow2ng:5556 interface.

Kuma metrics scrape 401 You created an API key — set KUMA_API_KEY in .env and enable the basic_auth block in the Prometheus template (see Post-setup §3).

Caddy serves the wrong cert / connection refused on 443 DNS record missing (name must resolve to the metrics VM) or the Caddyfile wasn’t re-rendered after changing DOMAIN — re-run deploy, then docker compose -f stack/caddy/compose.yml restart caddy.

No Let’s Encrypt certificate (cloudflare mode) / browser warning docker logs caddy. Typical causes: token lacks Zone:Read + DNS:Edit on the zone containing DOMAIN (curl -H "Authorization: Bearer $TOKEN" https://api.cloudflare.com/client/v4/user/tokens/verify); DOMAIN is not inside any zone on that account; DNS propagation timeouts (the Caddyfile already sets resolvers 1.1.1.1 1.0.0.1); or Let’s Encrypt rate limits after many failed attempts — wait an hour. Fix, then docker restart caddy. To fall back temporarily set TLS_MODE=internal in .env and re-run deploy.

docker compose up fails building caddy The caddy image is built locally (stack/caddy/Dockerfile, xcaddy + caddy-dns/cloudflare). Needs Internet access to proxy.golang.org and Docker ≥ 23 with BuildKit/buildx (installed by get.docker.com). Retry with docker compose -f stack/caddy/compose.yml build --no-cache.

Alloy ships nothing to Loki docker logs alloy. The journal mount requires the host to use persistent journald (/var/log/journal exists); if not: mkdir -p /var/log/journal && systemctl restart systemd-journald.

Useful commands

# status of every stack on a VM
docker ps --format 'table \t\t'

# re-render configs + re-up everything (idempotent)
sudo /opt/alethurgy/repo/scripts/deploy-stack.sh --profile <core|logs|siem|all> --non-interactive

# reload Prometheus after editing scrape targets/rules
docker compose -f stack/prometheus/compose.yml exec prometheus kill -HUP 1

# Prometheus config sanity check
docker compose -f stack/prometheus/compose.yml exec prometheus promtool check config /etc/prometheus/prometheus.yml