Operating on Kubernetes
Published Updated
These are the changes made to an install once Kubernetes has brought it into service. Each one changes the resources that page and The edge on Kubernetes created.
Adding a tenant
A tenant created with publiractl tenant create or in the Platform Console needs its two host names on the edge, as Adding a tenant describes. On a cluster that is:
- With a Gateway: two HTTPS listeners on the Gateway, one for the tenant's domain and one for its console host, each with a Secret name of its own, and each listener's name in the
parentRefsof the HTTPRoute of its web app. cert-manager issues both certificates from the Gateway's annotation. A Gateway holds at most 64 listeners, so one Gateway serves up to 31 tenants beside itshttplistener; put the next tenants on a Gateway of their own, with HTTPRoutes of their own. - With an Ingress: the domain added to the
tlsentry and the rules of theweb-storefrontIngress, the console host to those ofweb-admin, and both to those ofapi-not-found. The YAML anchors of With an Ingress (&storefront-paths,&admin-paths,¬-found-paths) let a new rule reuse the paths withpaths: *storefront-paths. cert-manager issues the certificates for the new names when thetlsentries change.
Point both names at the edge's address before you apply the change, so cert-manager's challenge reaches it.
Keeping the consoles' credentials off the tenants' domains
The edge already keeps the consoles' APIs off the internet, but the publira-server it forwards the tenants' domains to still serves them, and holds their database logins and the web service token to do so. To take all three out of the process readers reach, a layer apart from the edge rule as Only the public API goes through the edge describes, run a second Deployment with --role storefront, and route the tenants' domains and web-storefront to it. The split also lets reader traffic scale apart from the consoles, since each Deployment has replicas of its own, as Splitting the server by role describes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: publira-storefront
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: publira-storefront
template:
metadata:
labels:
app.kubernetes.io/name: publira-storefront
spec:
terminationGracePeriodSeconds: 45
containers:
- name: publira-storefront
image: ghcr.io/publira/publira:<tag>
args: ["server", "--role", "storefront"]
lifecycle:
preStop:
sleep:
seconds: 5
ports:
- name: http
containerPort: 8000
envFrom:
- secretRef:
name: publira-encryption
- secretRef:
name: publira-storefront
env:
- name: PUBLIRA_WEB_STOREFRONT_INTERNAL_URL
value: http://web-storefront:3000
- name: PUBLIRA_WEB_ADMIN_INTERNAL_URL
value: http://web-admin:4000
- name: PUBLIRA_TRUSTED_PROXIES
value: 10.244.0.0/16
livenessProbe:
httpGet:
path: /livez
port: http
readinessProbe:
httpGet:
path: /readyz
port: http
timeoutSeconds: 8
---
apiVersion: v1
kind: Service
metadata:
name: publira-storefront
spec:
selector:
app.kubernetes.io/name: publira-storefront
ports:
- name: http
port: 8000
targetPort: http
Its Secret holds the publira server values it reads, which leaves out the web service token and every connection but publira_public. Create it in the shell you created the other Secrets in, which still holds their values:
kubectl create secret generic publira-storefront \
--from-literal=PUBLIRA_AUTH_JWT_SECRET="$AUTH_JWT_SECRET" \
--from-literal=PUBLIRA_REVALIDATE_TOKEN="$REVALIDATE_TOKEN" \
--from-literal=PUBLIRA_REDIS_URL="$REDIS_URL" \
--from-literal=PUBLIRA_PUBLIC_DB_URL="postgres://publira_public:$PUBLIC_DB_PASSWORD@$DB"
Then change the backend of /images and the public API under /rpc on the tenants' domains alone, in the web-storefront HTTPRoute or Ingress, from publira-server to publira-storefront, and point web-storefront at it too, since it calls the public API alone: set PUBLIRA_GRPC_URL in its Deployment to http://publira-storefront:8000/rpc. The console hosts' routes, web-admin, and web-platform keep calling publira-server, which serves every role, until Leaving the public API off the consoles' server takes the public API off it.
The edge answers a console API with 404 on every host either way, so the check calls the two Services from inside the cluster instead:
for service in publira-storefront publira-server; do
kubectl run publira-role-check --rm -i --restart=Never --image=curlimages/curl -- \
-s -o /dev/null -w "$service %{http_code}\n" -X POST -H 'Content-Type: application/json' -d '{}' \
"http://$service:8000/rpc/publira.admin.v1.AdminAuditLogService/ListAuditLogs"
done
publira-storefront answers 404, since it does not serve that API. publira-server answers 400: the API is served there, and refuses the empty request.
Leaving the public API off the consoles' server
publira-server still serves the public API and holds publira_public, because web-admin calls the public API for the tenant's theme and tags, and the console hosts' route forwards it there. Send both to publira-storefront, and publira-server can run --role admin,platform, as step 4 of Splitting the server by role describes. Make the changes in this order, so that nothing still sends publira-server a public API request when it stops serving them.
First point web-admin's public API calls at the storefront Service. Add the variable to the env of its Deployment, beside PUBLIRA_GRPC_URL, which keeps naming publira-server, and apply it:
- name: PUBLIRA_PUBLIC_GRPC_URL
value: http://publira-storefront:8000/rpc
Its /readyz now checks that Service as well, as public-api.
Then route the console host's public API to publira-storefront, which answers it on any host, and keep its /images on publira-server, since the images a console page shows are the admin role's. In the web-admin HTTPRoute, split the first rule in two:
- matches:
- path: { type: PathPrefix, value: /images }
filters: &strip-request-headers
- type: RequestHeaderModifier
requestHeaderModifier:
remove:
- traceparent
- tracestate
- baggage
- forwarded
- x-forwarded-host
- x-forwarded-countryregion
backendRefs:
- name: publira-server
port: 8000
- matches:
- path: { type: RegularExpression, value: '/rpc/publira\.v1\..*' }
filters: *strip-request-headers
backendRefs:
- name: publira-storefront
port: 8000
In the web-admin Ingress, change the backend of the /rpc/publira.v1. path to publira-storefront instead.
Last, give the publira-server container args: ["server", "--role", "admin,platform"] in publira-server.yaml, take PUBLIRA_PUBLIC_DB_URL out of its Secret, which nothing else reads it from, and apply the Deployment:
kubectl patch secret publira-server --type=json \
-p '[{"op":"remove","path":"/data/PUBLIRA_PUBLIC_DB_URL"}]'
kubectl apply -f publira-server.yaml
kubectl rollout status deployment/publira-server
Check that publira-server no longer serves the public API. This call answers 404, where it answered 400 before:
kubectl run publira-role-check --rm -i --restart=Never --image=curlimages/curl -- \
-s -o /dev/null -w "%{http_code}\n" -X POST -H 'Content-Type: application/json' -d '{}' \
"http://publira-server:8000/rpc/publira.v1.TenantService/GetTenant"
Sign in to the tenant console: it draws the tenant's theme and lists the tags as before, now read from publira-storefront.
Metrics
publira server and publira worker serve Prometheus metrics on port 9464 once Metrics turns them on with PUBLIRA_METRICS_ENABLED=true, OTEL_METRICS_EXPORTER=prometheus, and OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0. Each runs in a Pod of its own in Start the processes, so each listens on that port without colliding. A Pod that runs the two as two containers shares one network namespace, and the second to start cannot listen on a port the first holds: give each its own OTEL_EXPORTER_PROMETHEUS_PORT, or keep them in Pods of their own.
Upgrading
Upgrading brings an install to a new release: a backup, the migrations and the roles with the new release's publiractl, and then the processes. Read it, and its release notes, first. On a cluster its steps become:
-
Stop the processes when you decided to stop them first, which Decide whether to stop the processes first weighs, by scaling every Deployment of the install to zero:
kubectl scale deployment --all --replicas=0 kubectl wait --for=delete pod -l app.kubernetes.io/name --timeout=2mThe label selects the Pods of every Deployment of the install and leaves out those of the Jobs, which have finished and are kept for their logs.
-
Back up the database, with
pg_dumpfrom a host that reaches it, or a snapshot of the managed database, as Back up the database describes. -
Apply the migrations and the roles: delete the previous release's Jobs if they are still there, since a Job cannot be changed, then change
<tag>in the two Jobs of Apply the migrations and Create the roles to the new release, and run them in that order, waiting for each to complete:kubectl delete job publira-db-migrate publira-db-roles --ignore-not-foundThe roles Job gives each role the password
publira-db-rolesholds for it, which is the one it already has, so it runs with its flags unchanged. A release that adds a role needs that role's password inpublira-db-rolesand its flag added. -
Start the new release: change
<tag>in every Deployment, add the variables the release notes name, and apply them. Applying the manifests restores the replicas you scaled to zero. -
Check the install, as Check the install describes.
When the processes keep running through the upgrade instead, apply every Deployment at once after the roles Job: a Deployment rolls one replica at a time, so the two releases serve side by side until the last replica is replaced, and the shorter that is the better, as the same page explains.