Operations in v0.5: audit stats, system health, jobs overview, retention, maintenance mode, API exports and `aps doctor`

Status: Accepted (2026-09-15) · Amends: ADR-0026, ADR-0027

Context#

v0.5 finishes the operations work ADR-0026 planned: GET /ops/audit/stats, GET /ops/system, a jobs overview, retention, maintenance mode, plus the Postman collection and llms.txt ADR-0027 planned, and aps doctor. Its definition of done: ops endpoints require platform roles and 2FA. ADR-0050 already delivered the upgrade half.

What exists:

Constraints: generated apps own their ops module and wiring; the library holds queries and engines (app-owns-modules); every change must reach existing apps through aps upgrade (ADR-0050), so no new migration unless needed; ops responses carry no secrets (ADR-0026).

Decision#

1. Audit stats: GET /ops/audit/stats#

• Query: from, to (default the last 7 days, at most 90 days apart), group_by one of action, outcome, actor_kind, resource_type, day; the same filters as the list (action_prefix, org_id, outcome, …).
• Response: total, and up to 50 groups ordered by count, with other for the rest.
• Library: auditpg.Store.Stats(ctx, StatsFilter), one GROUP BY bounded by occurred_at (uses the existing index), with a 5-second statement timeout.
• Permission: ops.audit.read.

2. System health: GET /ops/system#

Describes the instance that answers (other instances: GET /ops/releases/instances).

• Never includes the database URL, host names of dependencies, environment variables or settings values.
• Permission: new ops.system.read (platform admins and ops viewers).
• A slow or failing database still returns 200 with the failure in checks and database.error, so operators can see the problem.

3. Jobs overview: GET /ops/jobs/overview#

• Counts of jobs per queue and state (available, scheduled, running, retryable, discarded in the last 24 hours), paused queues, and job definitions whose latest run failed with its error summary.
• Library: jobs.Manager.Overview(ctx), one grouped query over River's job table on its state index.
• Permission: ops.jobs.read.

4. Retention#

Retention periods are runtime settings, one per kind of data, so they're changed, bounded, audited and synchronised by the existing machinery:

• GET /ops/retention is a read-only summary: each kind of data, its setting, current value, oldest row, and the enforcing job's last run. Changes go through PUT /ops/settings/{key}, so there is one write path with reasons, versions, history and audit events (amends ADR-0026's PUT /ops/retention).
• The new retention job (daily, internal/jobs/retention, editable in /ops/jobs) deletes in batches of 5,000 rows per statement until nothing is older, so it never holds long locks; library auditpg.Store.DeleteBefore(ctx, before, limit), settings.Store.DeleteHistoryBefore, jobs.Manager.DeleteHistoryBefore.
• It records one retention.purged audit event per kind with the count and cutoff, so shortened retention is visible even after the rows are gone.
• River's finished-job retention stays a code option: River applies it at client start, not live.
• Permission: ops.settings.read for the summary.

5. Maintenance mode#

• Settings maintenance.enabled (bool, default false), maintenance.message (string, at most 500 characters) and maintenance.retry_after (duration, default 5 minutes). They reach every instance within a second (LISTEN/NOTIFY) and turning it on records an audit event like any setting change.
• While on, a middleware in the app answers 503 with problem code maintenance, the message and a Retry-After header.
• Still served: /livez and /readyz (so load balancers keep instances in rotation), /docs and /openapi.json, /ops/*, and /v1/auth/* (so staff can sign in and turn it off).
• Break-glass: go run ./cmd/api maintenance on|off [--message …] writes the setting directly, for when nobody can reach /ops.
• The middleware reads the setting from memory; it adds no database query per request.

6. Postman collection and llms.txt#

• go run ./cmd/api openapi gains --dir api, writing openapi.json, postman_collection.json and llms.txt together; without --dir it prints the spec as today.
• Library modules/openapi/export: Postman(spec) builds a Postman v2.1 collection (a folder per tag, {{baseUrl}} and {{token}} variables, bearer auth, example bodies from the schema's examples); LLMs(spec) writes llms.txt Markdown (the API's title and description, how to authenticate, each tag's operations as METHOD path: summary, links to /openapi.json and /docs). Both are deterministic, so the committed files diff cleanly.
• TestOpenAPIUpToDate checks all three files; aps dev and aps upgrade regenerate all three (ADR-0050's derived paths gain the two new files).
• These describe the generated app's API. The framework's own llms.txt on docs.apistock.dev is ADR-0049's.

7. aps doctor#

Read-only; prints each check as ok, warn or fail with the fix; exit 1 on any fail; --json.

• The CLI stays free of database drivers: the database check runs the app's go run ./cmd/migrate --status --json (new flag, built on postgres.Migrations).
• Each check has a timeout; an unreachable database is a warn with the address to check, not a hang.

8. Permissions and 2FA#

• New permissions ops.system.read; existing ones cover the rest. platform_admin gets every ops.* permission, ops_viewer every read.
• Every new endpoint goes through authorize, so it requires a platform role and a session with 2FA, as ADR-0043 made true for all of /ops. A table test calls every /ops operation in the OpenAPI document without a session, without the permission, and without 2FA, and expects 401, 403 and 403 mfa_required, so a future endpoint that forgets can't pass.

9. How existing apps receive it#

• Library changes (auditpg, jobs, settings, releases, openapi/export): go get.
• App changes (ops module, retention job, maintenance middleware and command, settings, exports): template changes to both golden apps, delivered by aps upgrade, the first release to exercise it on real template changes. No new migration.
• releases.WithRetention stays as a deprecated option that sets the setting's default, removed after v1.0.

10. Order#

1. Permission table test, ops.system.read, GET /ops/system.
2. Audit stats and jobs overview.
3. Retention settings, retention job, GET /ops/retention.
4. Maintenance mode and the break-glass command.
5. API exports.
6. aps doctor.
7. Upgrade note, docs, threat model rows, and the v0.5 definition of done checked end to end.

Why#

• Settings already give retention and maintenance mode bounds, reasons, history, audit and live propagation; a second configuration store would duplicate all of it.
• One write path for retention means shortening audit retention is itself audited, and the purge leaves a summary event.
• Maintenance mode that keeps health checks, sign-in and /ops open can always be turned off, and doesn't make load balancers drop every instance.
• Exports generated by the app, from its own spec, keep generated apps independent of aps (architecture principle 8).
• aps doctor calling the app for database state keeps the CLI small and uses the app's own migration files.

Trade-offs#

• GET /ops/system shows one instance per request; a cluster view is v1.1's live observability.
• Deleting audit events at all is a policy choice; teams with longer obligations raise the setting (up to 10 years) or export events elsewhere.
• Maintenance mode leaves /v1/auth/* open to everyone, not only staff: sign-in works, but everything after it answers 503.
• aps doctor runs go run for exports and database checks, which takes seconds on a cold build cache.

Consequences#

• ADR-0026: PUT /ops/retention becomes settings writes plus a read-only summary; GET /ops/jobs/overview added; maintenance mode's exempt routes and break-glass command defined.
• ADR-0027: Postman and llms.txt export defined.
• ADR-0015: new permissions, setting keys, problem code maintenance and aps doctor output are public API.
• Threat model: rows for erasing audit trails through retention, maintenance mode lockout or abuse, and information disclosure from /ops/system and aps doctor.

Implementation notes#

Maintainer's answers (2026-09-15)#

1. Retention is changed through /ops/settings, with a read-only GET /ops/retention summary (section 4).
2. Audit events are kept 365 days by default, at least 30 days and at most 10 years.
3. Maintenance mode keeps /v1/auth/*, /ops/*, health checks and docs open, answers 503 elsewhere, and has a maintenance off command (section 5); no read-only mode.
4. aps doctor checks the database through the app's cmd/migrate --status (section 7).