Monitoring

發布於 2026年10月7日更新於 2026年10月15日

An install tells you how it is doing in four ways: every process answers a liveness and a readiness check, publira server and publira worker write a structured log, the same two export OpenTelemetry metrics once you turn them on, and every process but email-renderer can export OpenTelemetry traces. This page says what each of them reports, how to wire them into an orchestrator and an alerting system, and what is worth an alert. Which monitoring system receives them is your choice; the page names only what an install sends.

The variables are described in full in the publira server and publira worker reference, the server's tracing and metrics reference, and the web apps' tracing reference; this page links to them rather than repeating them.

Health checks

Every process answers two requests over plain HTTP:

• GET /livez answers 200 with the text ok whenever the process can answer at all. It checks nothing else, so it fails only when the process has hung or stopped.
• GET /readyz checks what the process depends on and answers 200 when every check passes, and 503 when any one fails. The body names each check.

A publira server --with-worker process answers on both ports of the first two rows, each with the checks its row names, so probe it as you would probe the two processes.

None of them checks the object store, the SMTP server, or the search engine. A failure there shows in the log and in what readers see, not in /readyz.

A healthy answer looks like this:

{
  "status": "ok",
  "checks": {
    "db.outbox": { "status": "ok" },
    "db.ticker": { "status": "ok" },
    "db.content_stats": { "status": "ok" }
  }
}

status is ok when every check passed, unavailable when one failed, and starting while the worker's job queue has not started yet. A failed check carries an error that says only which kind of failure it was — timeout, dependency_failed, or not configured — and never a host name or a connection string, so the answer is safe to show anyone. The reason itself goes to the process's log, as a readiness check failed line naming the check.

Each check is given two seconds. publira server and publira worker run their checks one after another, so when every login hangs their answer takes about six seconds; the web apps run theirs side by side and answer within about two.

Probing from an orchestrator

Point the liveness probe of every process at /livez, and its readiness probe at /readyz. Never use /readyz as a liveness probe: a database that stops answering would make the orchestrator restart every process at once, which does not bring the database back and takes away whatever the processes could still serve from cache.

A web app's /readyz fails whenever the /readyz of publira server does, so one database login that stops answering makes every web app not ready together with publira server. That is deliberate — a web app that cannot reach a working API cannot serve a page that is not cached — but it means a load balancer that follows readiness stops sending traffic to all of them at once. Decide whether that is what you want from your load balancer before you point it at /readyz.

Give a /readyz probe of publira server or publira worker a timeout longer than six seconds, so that a slow check is reported as the failure it names rather than as a probe timeout. On Kubernetes, the worker's probes look like this, and Kubernetes shows those of every process:

livenessProbe:
  httpGet:
    path: /livez
    port: 8003
readinessProbe:
  httpGet:
    path: /readyz
    port: 8003
  timeoutSeconds: 8

On ECS, every health check of a service's tasks acts as a liveness probe, since the service replaces a task that fails one; Health checks points them all at /livez.

The images declare no Docker HEALTHCHECK, and the web images carry no shell to run one from, so the Docker Compose install probes none of Publira's processes: Compose restarts a process that exits, not one that hangs. Probe it from outside instead, from a monitor that can reach the containers' network. The tenant site's /livez and /readyz are also reachable through the reverse proxy on every tenant's domain, such as https://comics.example.com/readyz, which an uptime monitor on the internet can use; the API's and the worker's are not, since the proxy forwards only /rpc, /api, and /images to publira server.

Logs

publira server, publira worker, and publiractl write their log to standard output, one record per line, as key=value pairs:

time=2026-10-07T09:41:03.402Z level=ERROR msg="outbox event dead" event_id=0199b5f0-1c2e-7c41-9a8e-3f5d2a6b7c80 event_type=contact_message_reply_email idempotency_key=… attempts=10 trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7 error="dial tcp: connection refused"

Every record has time, level (INFO, WARN, or ERROR), and msg. A record logged while handling a request or a job that is being traced also has trace_id and span_id, which lead to that trace in your tracing backend. The rest depends on what logged it: the outbox's records name the entry (event_id, event_type, attempts), a maintenance job's name the run (job_kind, job_id, attempt), and a failure carries error. The level is fixed at INFO: there is no variable to make the log quieter or more detailed.

The web apps and email-renderer write Next.js's and Node.js's own plain-text output instead. Their failures are lines prefixed with the app's name, such as [web-storefront] getTenantSiteInfo failed, and a readiness check that failed is logged as health check api failed.

What to alert on

An alert on every level=ERROR record of publira server and publira worker is a reasonable start: a request the API could not answer is logged at that level, with the error behind it. These records are worth an alert of their own, because each one means something was lost or is stuck:

How the worker's scheduled jobs record their failures, in the log and in the river_job table, is in Where a job's failures are recorded.

Metrics

publira server and publira worker export OpenTelemetry metrics once PUBLIRA_METRICS_ENABLED=true is set on them. Without it they export none. The web apps and email-renderer export no metrics.

Choose how they are exported with OTEL_METRICS_EXPORTER:

• otlp, the default, pushes them every 60 seconds to OTEL_EXPORTER_OTLP_ENDPOINT, the same collector address the traces use.
• prometheus serves them for a Prometheus scraper at /metrics on port 9464. Set OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0 as well: the default, localhost, cannot be reached from outside the container. When publira server and publira worker share a network namespace, such as one Kubernetes pod, give each its own OTEL_EXPORTER_PROMETHEUS_PORT.

On the Docker Compose install, set PUBLIRA_METRICS_ENABLED, OTEL_METRICS_EXPORTER, and OTEL_EXPORTER_PROMETHEUS_HOST in .env, which passes them to server, and run docker compose up -d again. The other variables, and the full list of what is recorded, are in the metrics reference. Each process reports under one service.name: publira-api-server for publira server and publira-worker for publira worker. A publira server --with-worker process, which the Compose file's server is, reports everything under publira-api-server.

The outbox

The worker's metrics follow the outbox, through which every mail, push notification, and cache revalidation is sent, as The outbox describes. Each carries the entry's kind as outbox.event_type.

The counters say what the worker did, not what is still waiting. A worker that has stopped sending reports nothing at all, and its /readyz does not notice either: it checks the database logins, not whether entries are moving. To see what is waiting, count the entries that are due and unsent, and the ones picked up and never finished, with psql, connected as the database's administrative login:

SELECT status, count(*), min(updated_at)
FROM outbox_events
WHERE (status = 'pending' AND available_at < now() - interval '5 minutes')
   OR (status = 'processing' AND updated_at < now() - interval '15 minutes')
GROUP BY status;

A healthy worker sends an entry within a few seconds of it becoming due, and puts an entry it was in the middle of back to wait after 15 minutes, so any row this returns means the worker is not draining.

The audit log

publira server writes the audit logs of the tenant console and the Platform Console through a queue, so a request does not wait for its entry. publira.auditlog.queue.depth is the number of entries waiting, and publira.auditlog.entries.dropped counts the entries lost, by auditlog.drop_reason. Alert on any drop, and on a queue depth that keeps growing: the database is not keeping up. The rest of the audit log's metrics are in its reference.

Tracing

Every process but email-renderer exports OpenTelemetry traces once PUBLIRA_TRACING_ENABLED=true is set on it. Turn it on for publira server, publira worker, and the web apps together: a page request is then one trace, from the web app through the API to the database queries it made.

Point them at your collector with the variables the OpenTelemetry SDK reads itself:

On the Docker Compose install, set these and PUBLIRA_TRACING_ENABLED in .env, which passes them to server, which runs the worker too, and to the web apps, and run docker compose up -d again.

Each process reports under a service.name of its own: publira-api-server, publira-admin-api-server, and publira-platform-api-server for the three APIs publira server serves, publira-image-server for its image delivery, publira-worker and a name per scheduled job for publira worker, all of them kept when publira server --with-worker runs both in one process, and publira-web-storefront, publira-web-admin, and publira-web-platform for the web apps. The full list, and what is recorded on each span, is in the server's tracing reference. The health checks of publira server are not traced, so a probe does not fill the backend with a trace every few seconds.

Sampling

With PUBLIRA_DEPLOYMENT_ENVIRONMENT unset, every process counts itself as development and records every trace. Any other value records one trace in ten. The decision is made once, where the trace starts, and every process the request goes on to reaches keeps it, so a trace is recorded either whole or not at all. Set the same value on every process.

To sample at another rate, set the SDK's own OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, such as parentbased_traceidratio and 0.25, which replace the default on every process they are set on.

Why the edge removes trace headers

Every process trusts the trace context a request arrives with: a traceparent header makes its span part of the caller's trace, and carries the caller's sampling decision. That is what joins the web app's span to the API's. A browser or the mobile app could send the same header, though, and with it attach its spans to a trace of someone else's, or mark every request as sampled and export a trace for each one past the one-in-ten rate.

So the edge in front of the install removes traceparent, tracestate, and baggage from every request it forwards, as Trace context requires of every edge, and every request from outside starts a new trace. The web apps call the API, and the API and the worker call the web apps, directly on the private network rather than through the edge, so their trace context is kept.