Day-2 operations
Keep Marshal current and healthy with a few repeatable steps: upgrade with one changed value, roll back in one command, renew your license, and switch on durable storage for spaces and audit events. Each step has a Terraform path and an ArgoCD path, matching the installer you used. The settings named here are in the Configuration reference.
The commands below use the default release name and namespace, marshal. Substitute yours if you
changed them.
Before you upgrade
An upgrade runs database migrations when the new API starts. Take two backups first, so you can always return to the state you had.
Back up Postgres
Take a snapshot of your managed database (an RDS snapshot or a Cloud SQL backup), or dump it:
pg_dump "<database_url>" --format=custom --file="marshal-$(date +%F).dump"Back up the app secret
The app secret holds JWT_SECRET and KMS_LOCAL_KEY. Without KMS_LOCAL_KEY, stored
credentials cannot be decrypted, so keep a copy in your own secret store:
kubectl get secret marshal-app-secrets -n marshal -o yaml > marshal-app-secrets.yamlOn ArgoCD, or with secrets.provider: existing, back up the Secret you named in
secrets.existingSecret.
Pick the target version
Use the version from your Marshal onboarding materials or your Marshal contact.
Upgrading
Marshal ships as one Helm chart, and the chart version is the product version. An upgrade is always the same move: point at the new chart version and apply.
Terraform
Each installer tag matches one chart version, so move your installer clone to the tag
for the new version first. Use the newest v<chart_version>-r<n> tag when one exists.
Your terraform.tfvars, override files and state stay in place.
git fetch --tags
git checkout v<chart_version>Set chart_version in your terraform.tfvars (or TF_VAR_chart_version) to the same
version.
Next, settle every new setting before Terraform sees the new defaults. The installer’s
capabilities.yaml lists each capability and changed default, with the installer version
that introduced it. For every entry newer than the version you run, decide enable,
disable or keep what I have, and write the answer into terraform.tfvars (the
entry’s enable and disable lines are the exact text). An explicit value means a later
default never moves under you. Entries marked disruptive: true deserve the most care:
-
Node sizes. If you never set them, a new default size replaces the spaces node group and stops every running space. Pin your current values to keep the group as it is (on EKS
node_instance_typesandnode_disk_size_gb, on GKE themachine_typeanddisk_size_gbof eachspaces_poolsentry). Move to a new size later by adding a second group. See Sizing. -
Platform pool and bundled datastores. When this upgrade turns
taint_control_plane_poolon (an install from 0.29 or older, or one that set it tofalse) and you run the bundled Redis or Postgres, setpin_bundled_datastores = falsewith it, so the datastores keep their volumes where they are while the platform pods move. Check where they run first:kubectl get pod -n marshal marshal-redis-master-0 marshal-postgresql-0 -o wide kubectl get nodes -l workload=control-planeIf they already run on a control-plane node, keep the default
true. Withfalsethey stay where they are until they are next rescheduled, which then needs a node outside the control-plane pool in their volume’s zone.
The install-marshal skill in the installer walks you through these questions one at a
time. Then download the providers the new tag needs, and read the plan before you apply:
terraform init -upgrade
terraform plan
terraform applyThe plan should show no replacement you did not choose: a node group or node pool marked
must be replaced means a size is not pinned yet. Terraform then pulls the new chart and
runs helm upgrade in place, with the installer built for that version.
Verify after the upgrade
Check the version
Terraform
helm list -n marshalThe CHART column shows marshal-<new version>.
Check the rollout
kubectl rollout status deployment/marshal-api -n marshal
kubectl get pods -n marshalWhat you should see: the rollout finishes, and every pod is Running and ready.
Use it
Open Marshal in the browser, sign in, and start a space. It reaches running.
Upgrade the docker package
From API 2.11.0, spaces run Docker rootless, which needs the docker package at 1.1.1 or
later. In each organization that uses Docker, open Marketplace, choose the Installed
tab, and upgrade docker to the newest version. Stacks with Auto-upgrade modules move to
it on their own. See Keep packages up to date and
Docker in spaces.
Rolling back
Terraform
A Terraform install goes through Helm, so roll back with Helm:
helm rollback marshal -n marshalThen set chart_version back to the previous value in your tfvars, so the next
terraform apply keeps you there.
A rollback returns the services to the previous version. It does not undo database migrations. To return the data to its state before the upgrade, restore the Postgres backup you took.
Renewing the license
Your Marshal contact issues a new installation JWT before the current one expires, over the same secure channel as the first. Then:
Terraform
Set installation_jwt in your terraform.tfvars (or the environment variable), then run:
terraform applyObject storage: space storage and audit export
Two features run on object storage, and you set each one up on its own:
- Space storage saves every space’s files, so people can stop, resume, fork and archive spaces. Without it, a space keeps its files across stop and resume only on a compute size with a persistent disk. See Spaces and storage.
- Audit export writes audit events to a bucket. See Audit log.
Setup takes two steps.
Create the bucket
On EKS and GKE the installer can create each bucket and grant Marshal access with no keys (IRSA or
Workload Identity). Set create_spaces_persistence_bucket = true,
create_audit_persistence_bucket = true, or both, and run terraform apply. On any other setup,
create the bucket yourself.
Point Marshal at it
An owner or admin opens the settings page in the web app:
- Settings > Space Storage for space storage. Choose the Provider (Amazon S3, Google
Cloud Storage, Azure Blob Storage or Filesystem / NFS), enter the Bucket name (for Azure the
Container name, for a filesystem the Mount path), and select Save & enable. For
Amazon S3, also enter the Region the bucket lives in. A bucket the EKS installer created
is in your cluster’s
region. - Settings > Auditing for audit export. In the Storage card, choose the Provider, enter the Bucket name, and select Save & enable.
For a bucket the installer created, leave the credential fields empty. Marshal uses IRSA or Workload Identity. For a bucket you created yourself, enter its credentials on the same page.
Space storage and audit export are independent. Turn on one, both or neither, with the same bucket or different ones. Both are set per organization.
Set up from a terminal
Everything the owner does in the browser after an install has an API call behind it, so you can
finish a headless install from a shell. Run these against the port-forward
(kubectl port-forward -n marshal svc/web 8080:80) or your Marshal address. They use curl and
jq.
Claim the owner account
Follow Claim from a terminal.
Sign in
Read the password without echoing it, and keep the session token in a shell variable. The
password reaches jq through its environment, not its command line, so it never shows in the
process list:
URL=http://localhost:8080
read -rs PASSWORD
TOKEN=$(PASSWORD="$PASSWORD" jq -n --arg email you@example.com \
'{email: $email, password: env.PASSWORD}' |
curl -sS -X POST "$URL/api/auth/login" -H 'Content-Type: application/json' --data @- |
jq -r .token)
unset PASSWORD
AUTH=(-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json')The CLI signs in the same way with marshal login --server "$URL" --password.
Connect space storage
Save the backend, then enable it. Saving alone leaves space storage off.
curl -sS -X PUT "$URL/api/org/space-storage" "${AUTH[@]}" \
-d '{"provider": "s3", "bucket": "<bucket>", "region": "<region>", "path_prefix": "space-volumes"}'
curl -sS -X PATCH "$URL/api/org/space-storage/enabled" "${AUTH[@]}" -d '{"enabled": true}'| Field | Value |
|---|---|
provider | s3, gcs, azure or filesystem |
bucket | The bucket name with no s3:// or gs:// prefix (for Azure the container, for a filesystem the mount path) |
region | Amazon S3 only, and required there: the bucket’s region |
endpoint | Optional, Amazon S3 only: the URL of an S3-compatible store, such as https://minio.internal:9000 |
path_prefix | The folder inside the bucket, space-volumes by default |
credentials | Leave it out for a bucket the installer created: IRSA or Workload Identity signs every request |
POST /api/org/space-storage/test with the same body runs the same connection test as the
Test connection button.
Publish a first stack
Install the default-engineering stack from the marketplace, which installs its packages and
creates a draft, then publish the draft:
VERSION=$(curl -sS "$URL/api/marketplace/browse?q=default-engineering" "${AUTH[@]}" |
jq -r '.packages[] | select(.name == "default-engineering") | .latest_compatible')
STACK=$(curl -sS -X POST "$URL/api/marketplace/install" "${AUTH[@]}" \
-d "{\"name\": \"default-engineering\", \"version\": \"$VERSION\"}" | jq -r .stack_id)
STACK_VERSION=$(curl -sS -X POST "$URL/api/stacks/$STACK/publish" "${AUTH[@]}" -d '{}' | jq -r .id)The publish response is the new stack version, and its id is what a manifest pins.
Create a manifest and a space
curl -sS -X POST "$URL/api/manifests" "${AUTH[@]}" \
-d "{\"name\": \"engineering\", \"stack_version_id\": \"$STACK_VERSION\"}"
marshal spaces new --manifest engineering --name first-space --detachWhat you should see: marshal manifests list shows engineering, and marshal spaces list
shows first-space. Work in it with marshal spaces exec first-space -- <command>, and stop and
resume it with marshal spaces stop first-space and marshal spaces resume first-space. With
space storage connected, files under /home/agent come back after the resume.