Troubleshooting
Start with an observable status.
Use the smallest safe diagnostic.
Do not print environment values.
Do not print custody files.
Do not copy tokens into commands, logs, or tickets.
Use operations for normal procedures.
Safe first checks
Section titled “Safe first checks”Check the service state.
systemctl status mint.service --no-pager
Check live admission.
curl --fail http://127.0.0.1:4949/healthz
Check effective startup configuration.
mint doctor
Check policy syntax.
mint policy check <policy-file>
Inspect only sanitized audit fields.
mint audit tail --path "$AUDIT_LOG" -n 20
Use journalctl -u mint.service -n 50 --no-pager for service lifecycle messages.
Do not paste raw journal output into public issues.
Status routing
Section titled “Status routing”| Observable result | Likely boundary | Safe next action |
|---|---|---|
| Doctor policy fail | Policy syntax or schema | Run policy check and correct the declaration |
| Doctor capabilities fail | Capability schema or expiry | Check version and expiry without printing tokens |
| Doctor authenticator fail | Auth mode and capability mismatch | Use Tailnet with an empty capability file in production |
| Doctor secret-backend fail | Custody backend construction | Check backend selection and directory availability |
| Doctor broker-ready fail | Ledger damage or unresolved effect | Run unresolved and verify commands |
| Health HTTP 503 | Audit gate unhealthy, draining, or unresolved | Read the sanitized health line and audit tail |
| HTTP 400 | Malformed ingress or target | Recreate the request with a placeholder and valid target |
| HTTP 401 | Authentication failed | Check local capability scope or Tailnet peer identity |
| HTTP 403 | Policy, scheme, private-network, or header-alias denial | Check the matching policy rule and the header's alias; a body or query value that merely quotes placeholder syntax never denies |
| HTTP 429 | Call guard exhausted | Wait for the configured window or review observed capacity |
| HTTP 502 | DNS, custody, transport, response decoding or bounds, or disclosure control | Read the sanitized audit reason first; inspect never_relay_credentials only for credential_in_response_denied |
| HTTP 503 from a proxy call | Audit gate or upstream breaker | Check health and sanitized audit reason |
| Release preflight fail | Bundle shape or declaration mismatch | Re-stage the candidate and rerun preflight |
| Release readiness fail | Service command, health, restart, or audit evidence | Keep the prior release and inspect runtime evidence |
Doctor failures
Section titled “Doctor failures”Doctor does not inspect a running broker.
A doctor pass means a matching serve process can pass startup checks.
A policy failure blocks startup.
A capabilities failure blocks startup.
Tailnet mode rejects non-empty capabilities.
A call-guard path set through the environment must exist.
A directory backend requires a selected directory.
A custody self-test needs an alias argument.
A skipped self-test is not a pass.
Rotation retention requires byte or age rotation.
Rotation bounds must be positive integers.
Run doctor again after correcting one declaration.
Health stays at 503
Section titled “Health stays at 503”Check the audit gate reason in the health response.
An unhealthy audit sink blocks new effects.
A draining broker blocks new effects.
An unresolved prepared effect blocks startup admission.
Run the unresolved query.
mint audit unresolved --path "$AUDIT_LOG"
Run full ledger verification.
mint audit verify --path "$AUDIT_LOG"
Check available disk without exposing file content.
mint doctor
Restart only after resolving the reported condition.
Do not delete audit records to make health green.
Authentication failures
Section titled “Authentication failures”SharedSecretAuth accepts loopback only.
A remote caller with a valid bearer still receives a denial.
Use SharedSecretAuth only for local development.
TailnetWhoisAuth ignores identity headers.
A peer without a resolved Tailnet identity fails closed.
Check that the Tailnet daemon is active.
systemctl is-active tailscaled.service
Check the broker account's permission to query the daemon.
Do not add identity headers as a workaround.
Do not put a production bearer in a remote request.
Alias denials (403)
Section titled “Alias denials (403)”A header placeholder is an injection slot.
An unpermitted alias in a header value denies the call.
Check the audit reason for alias_not_permitted_by_rule.
A body value or a query value is not always an injection slot.
A well-formed placeholder there resolves only when its alias is permitted.
An unpermitted alias in a body or a query value stays inert literal text and forwards unresolved.
Do not expect a 403 from a body or a query value that merely quotes placeholder syntax.
Check the header first when a placeholder call returns 403 unexpectedly.
Reload appears ineffective
Section titled “Reload appears ineffective”Run doctor to compute on-disk declaration fingerprints.
Query health to read the live generation and fingerprints.
A changed generation proves a successful reload.
An unchanged generation can indicate a rejected candidate.
A rejected candidate leaves the last-known-good snapshot active.
Inspect sanitized reload events.
mint audit tail --path "$AUDIT_LOG" --actor mint-operator -n 20
Send one SIGHUP after correcting the candidate.
systemctl reload mint.service
Restart for startup-only settings.
Audit recovery
Section titled “Audit recovery”A failed audit write marks the gate unhealthy.
The fixed recovery loop retries automatically.
SIGHUP triggers an immediate recovery retry.
A successful retry does not close old unresolved effects.
Reconcile only after confirming the broker that prepared the effect is stopped.
mint audit reconcile <effect-id> --path "$AUDIT_LOG" --reason operator-reconciled-no-safe-retry --operator example-operator
Run verification after reconciliation.
mint audit verify --path "$AUDIT_LOG"
Do not retry an unknown upstream effect.
Do not record an allow result without upstream evidence.
Legacy ledger startup refusal
Section titled “Legacy ledger startup refusal”Mint refuses to append to unreviewed legacy audit content.
Read the legacy tail first.
Record an operator review outside the ledger value stream.
Migrate once with a sanitized reason.
mint audit migrate --path "$AUDIT_LOG" --reason reviewed-legacy --operator example-operator
Run verification after migration.
A second migration attempt fails.
Breaker opens
Section titled “Breaker opens”The breaker is keyed by upstream host.
Transport and 5xx failures count.
Client errors reset the consecutive count.
An open breaker returns HTTP 503 without dialing upstream.
Wait for the configured open interval.
Check the upstream health outside the broker.
Do not lower the threshold to hide a failing vendor.
Review the host's sanitized audit events.
Call guard returns 429
Section titled “Call guard returns 429”The guard tracks actor and service capacity.
The counter is in memory.
A restart resets the counter.
A guard requires a positive count and window.
A wildcard service matches every valid service authority.
Review observed load before changing the guard declaration.
Restart after changing call-guard configuration.
Release failure
Section titled “Release failure”Keep the previous current target when candidate readiness fails.
Check the candidate preflight output without publishing it.
Check the effective systemd contract.
Check service restart count and unresolved effects.
Run the release command with the expected previous target.
Do not bypass the expected-old check.
Do not replace the stable service command by editing environment paths.
Use deployment for rollback sequencing.
Source map
Section titled “Source map”- Configuration parsing defines defaults and fail-closed rotation validation.
- Authentication errors defines local bearer boundaries.
- Tailnet resolution defines bounded identity lookup.
- Proxy statuses defines safe denial statuses and audit ordering.
- Placeholder alias permission defines the header-only 403 denial; an unpermitted alias in a body value or a query value is left inert at L536-L550 and L659-L668.
- Audit gate defines health and drain reasons.
- Doctor scope defines offline checks.
- Ledger recovery defines unresolved, reconcile, migrate, and verify behavior.
- Breaker behavior defines failure classes.
- CLI declarations defines safe audit commands.
- Authentication tests cover caller boundaries.
- Audit tests cover fail-closed recovery.
- Breaker tests cover timeout and body-read failures.
- Probe tests cover sanitized receipt failures.