Configuration & customization

Where configuration lives

What	File	Applied by
Image versions	stack/versions.env	deploy-stack.sh (compose --env-file)
Deployment answers/secrets	/opt/alethurgy/.env on each VM	same
Scrape targets	stack/prometheus/prometheus.yml.tpl	re-run deploy, or edit rendered file + kill -HUP 1
Alert rules	stack/prometheus/alerts/*.yml	Prometheus reload
Alert routing	stack/alertmanager/alertmanager.yml.tpl	re-run deploy or docker compose restart alertmanager
Reverse proxy names	stack/caddy/Caddyfile.tpl (one *.<domain> site, host matchers)	re-run deploy or docker compose restart caddy
TLS mode / domain / Cloudflare token	TLS_MODE, DOMAIN, ACME_EMAIL, CLOUDFLARE_API_TOKEN, CF_CREATE_DNS in .env	re-run deploy (re-renders Caddyfile; --build rebuilds the caddy+cloudflare image if versions changed)
Loki retention	stack/loki/loki.yml (retention_period)	restart loki
Log shipping	stack/alloy/config.alloy	restart alloy

Templates (*.tpl) are rendered with your .env values at deploy time. Prefer editing the template and re-running deploy-stack.sh — it’s idempotent and keeps the rendered file reproducible.

The preloaded dashboard library

scripts/fetch-dashboards.sh downloads these community dashboards from grafana.com into stack/grafana/dashboards/downloaded/, rewires them to the provisioned datasources, and Grafana provisions them into the Preloaded Library folder (subfolders mirror the categories below):

Category	Dashboard (grafana.com ID)
nodes	Node Exporter Full (1860), Windows Exporter (14694)
containers	cAdvisor Docker Insights (19908)
availability	Blackbox Exporter (13659), Uptime Kuma Metrics (18278)
network	SNMP Interface Detail (12492), SNMP Network Overview (11368), Speedtest (13665)
logs	Loki Logs/App (13639)
alerting	Alertmanager (9578), Alerts Overview (4181)
proxmox	Proxmox VE (10347)
hardware	smartctl (22381), NUT UPS (19308)

Provisioned dashboards allow UI edits (allowUiUpdates: true), but a re-provision can overwrite them — to customize, open the dashboard → Save As into your own folder, then edit freely. To add more preloaded dashboards, append a line to the DASHBOARDS list in fetch-dashboards.sh (id||subfolder|filename|datasource-uid) and re-run it.

Alert rules

Baseline rules ship in stack/prometheus/alerts/baseline.yml: host down, probe failed, disk full/predicted-full, CPU/memory/load, TLS cert expiry, ICMP latency/loss. Tune thresholds there, or add new files to stack/prometheus/alerts/ (all *.yml are loaded). Test an expression in Grafana Explore before committing it as a rule.

Notification channels

stack/alertmanager/alertmanager.yml.tpl routes everything to ntfy and to the Node-RED webhook bus. To switch or add channels, uncomment the Telegram/email receivers in the template, add the secrets to /opt/alethurgy/.env, and re-run the deploy script. Severity handling: critical repeats every 2 h, everything else every 12 h; a HostDown alert inhibits that host’s resource alerts.

SNMP v3 for snmp_exporter

The bundled snmp.yml in the exporter image covers if_mib with SNMP v2c. For v3 credentials or vendor MIBs, generate a custom config with the official generator, mount it in stack/snmp-exporter/compose.yml (volume line is stubbed), and add scrape jobs like:

- job_name: snmp-switches
  metrics_path: /snmp
  params: { module: [if_mib], auth: [v3auth] }
  static_configs:
    - targets: ["192.168.1.3"]
      labels: { host: core-switch, role: network }
  relabel_configs:
    - { source_labels: [__address__], target_label: __param_target }
    - { source_labels: [__param_target], target_label: instance }
    - { target_label: __address__, replacement: snmp-exporter:9116 }

Optional exporters

stack/optional-exporters/compose.yml uses compose profiles:

docker compose --env-file stack/versions.env --env-file /opt/alethurgy/.env \
  -f stack/optional-exporters/compose.yml \
  --profile nut --profile smartctl --profile speedtest --profile pihole up -d

Set NUT_SERVER_IP, PIHOLE_HOST, PIHOLE_PASSWORD in .env first, then add the matching scrape jobs (ports: nut 9199, smartctl 9633, speedtest 9798, pihole 9617). Pi-hole v6 changed its API — if ekofr/pihole-exporter misbehaves, try Mosher-Labs/pihole6-exporter and dashboard 21043 instead of 10176.

Retention

• Prometheus: PROM_RETENTION in .env (default 90d). Long-term storage (Thanos/VictoriaMetrics) is deliberately deferred to v2.
• Loki: 31 d (retention_period in loki.yml) — app logs only.
• Graylog: per index set in the UI (System → Indices), size it to the logs VM disk.
• ntopng: community edition keeps limited history; export to Prometheus via its timeseries settings if you need more.

Upgrading components

1. Edit the pin in stack/versions.env (one component at a time; read the release notes — the file carries warnings for known traps like OpenSearch 3.x being unsupported by Graylog).
2. Re-run deploy-stack.sh on the affected VM, or docker compose --env-file ... -f stack/<svc>/compose.yml up -d.
3. Wazuh upgrades: change WAZUH_VERSION, remove /opt/wazuh-docker, re-run stack/wazuh/deploy-wazuh.sh (volumes persist).