Skip to Content
Day-2 operations

Day-2 operations

Keep Marshal current and healthy with a few repeatable steps: upgrade with one changed value, roll back in one command, renew your license, and switch on durable storage for spaces and audit events. Each step has a Terraform path and an ArgoCD path, matching the installer you used. The settings named here are in the Configuration reference.

The commands below use the default release name and namespace, marshal. Substitute yours if you changed them.

Before you upgrade

An upgrade runs database migrations when the new API starts. Take two backups first, so you can always return to the state you had.

Back up Postgres

Take a snapshot of your managed database (an RDS snapshot or a Cloud SQL backup), or dump it:

pg_dump "<database_url>" --format=custom --file="marshal-$(date +%F).dump"

Back up the app secret

The app secret holds JWT_SECRET and KMS_LOCAL_KEY. Without KMS_LOCAL_KEY, stored credentials cannot be decrypted, so keep a copy in your own secret store:

kubectl get secret marshal-app-secrets -n marshal -o yaml > marshal-app-secrets.yaml

On ArgoCD, or with secrets.provider: existing, back up the Secret you named in secrets.existingSecret.

Pick the target version

Use the version from your Marshal onboarding materials or your Marshal contact.

Upgrading

Marshal ships as one Helm chart, and the chart version is the product version. An upgrade is always the same move: point at the new chart version and apply.

Each installer tag matches one chart version, so move your installer clone to the tag for the new version first. Use the newest v<chart_version>-r<n> tag when one exists. Your terraform.tfvars, override files and state stay in place.

git fetch --tags git checkout v<chart_version>

Set chart_version in your terraform.tfvars (or TF_VAR_chart_version) to the same version.

Next, settle every new setting before Terraform sees the new defaults. The installer’s capabilities.yaml lists each capability and changed default, with the installer version that introduced it. For every entry newer than the version you run, decide enable, disable or keep what I have, and write the answer into terraform.tfvars (the entry’s enable and disable lines are the exact text). An explicit value means a later default never moves under you. Entries marked disruptive: true deserve the most care:

  • Node sizes. If you never set them, a new default size replaces the spaces node group and stops every running space. Pin your current values to keep the group as it is (on EKS node_instance_types and node_disk_size_gb, on GKE the machine_type and disk_size_gb of each spaces_pools entry). Move to a new size later by adding a second group. See Sizing.

  • Platform pool and bundled datastores. When this upgrade turns taint_control_plane_pool on (an install from 0.29 or older, or one that set it to false) and you run the bundled Redis or Postgres, set pin_bundled_datastores = false with it, so the datastores keep their volumes where they are while the platform pods move. Check where they run first:

    kubectl get pod -n marshal marshal-redis-master-0 marshal-postgresql-0 -o wide kubectl get nodes -l workload=control-plane

    If they already run on a control-plane node, keep the default true. With false they stay where they are until they are next rescheduled, which then needs a node outside the control-plane pool in their volume’s zone.

The install-marshal skill in the installer walks you through these questions one at a time. Then download the providers the new tag needs, and read the plan before you apply:

terraform init -upgrade terraform plan terraform apply

The plan should show no replacement you did not choose: a node group or node pool marked must be replaced means a size is not pinned yet. Terraform then pulls the new chart and runs helm upgrade in place, with the installer built for that version.

Verify after the upgrade

Check the version

helm list -n marshal

The CHART column shows marshal-<new version>.

Check the rollout

kubectl rollout status deployment/marshal-api -n marshal kubectl get pods -n marshal

What you should see: the rollout finishes, and every pod is Running and ready.

Use it

Open Marshal in the browser, sign in, and start a space. It reaches running.

Upgrade the docker package

From API 2.11.0, spaces run Docker rootless, which needs the docker package at 1.1.1 or later. In each organization that uses Docker, open Marketplace, choose the Installed tab, and upgrade docker to the newest version. Stacks with Auto-upgrade modules move to it on their own. See Keep packages up to date and Docker in spaces.

Rolling back

A Terraform install goes through Helm, so roll back with Helm:

helm rollback marshal -n marshal

Then set chart_version back to the previous value in your tfvars, so the next terraform apply keeps you there.

A rollback returns the services to the previous version. It does not undo database migrations. To return the data to its state before the upgrade, restore the Postgres backup you took.

Renewing the license

Your Marshal contact issues a new installation JWT before the current one expires, over the same secure channel as the first. Then:

Set installation_jwt in your terraform.tfvars (or the environment variable), then run:

terraform apply

Object storage: space storage and audit export

Two features run on object storage, and you set each one up on its own:

  • Space storage saves every space’s files, so people can stop, resume, fork and archive spaces. Without it, a space keeps its files across stop and resume only on a compute size with a persistent disk. See Spaces and storage.
  • Audit export writes audit events to a bucket. See Audit log.

Setup takes two steps.

Create the bucket

On EKS and GKE the installer can create each bucket and grant Marshal access with no keys (IRSA or Workload Identity). Set create_spaces_persistence_bucket = true, create_audit_persistence_bucket = true, or both, and run terraform apply. On any other setup, create the bucket yourself.

Point Marshal at it

An owner or admin opens the settings page in the web app:

  • Settings > Space Storage for space storage. Choose the Provider (Amazon S3, Google Cloud Storage, Azure Blob Storage or Filesystem / NFS), enter the Bucket name (for Azure the Container name, for a filesystem the Mount path), and select Save & enable. For Amazon S3, also enter the Region the bucket lives in. A bucket the EKS installer created is in your cluster’s region.
  • Settings > Auditing for audit export. In the Storage card, choose the Provider, enter the Bucket name, and select Save & enable.

For a bucket the installer created, leave the credential fields empty. Marshal uses IRSA or Workload Identity. For a bucket you created yourself, enter its credentials on the same page.

Space storage and audit export are independent. Turn on one, both or neither, with the same bucket or different ones. Both are set per organization.

Set up from a terminal

Everything the owner does in the browser after an install has an API call behind it, so you can finish a headless install from a shell. Run these against the port-forward (kubectl port-forward -n marshal svc/web 8080:80) or your Marshal address. They use curl and jq.

Claim the owner account

Follow Claim from a terminal.

Sign in

Read the password without echoing it, and keep the session token in a shell variable. The password reaches jq through its environment, not its command line, so it never shows in the process list:

URL=http://localhost:8080 read -rs PASSWORD TOKEN=$(PASSWORD="$PASSWORD" jq -n --arg email you@example.com \ '{email: $email, password: env.PASSWORD}' | curl -sS -X POST "$URL/api/auth/login" -H 'Content-Type: application/json' --data @- | jq -r .token) unset PASSWORD AUTH=(-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json')

The CLI signs in the same way with marshal login --server "$URL" --password.

Connect space storage

Save the backend, then enable it. Saving alone leaves space storage off.

curl -sS -X PUT "$URL/api/org/space-storage" "${AUTH[@]}" \ -d '{"provider": "s3", "bucket": "<bucket>", "region": "<region>", "path_prefix": "space-volumes"}' curl -sS -X PATCH "$URL/api/org/space-storage/enabled" "${AUTH[@]}" -d '{"enabled": true}'
FieldValue
providers3, gcs, azure or filesystem
bucketThe bucket name with no s3:// or gs:// prefix (for Azure the container, for a filesystem the mount path)
regionAmazon S3 only, and required there: the bucket’s region
endpointOptional, Amazon S3 only: the URL of an S3-compatible store, such as https://minio.internal:9000
path_prefixThe folder inside the bucket, space-volumes by default
credentialsLeave it out for a bucket the installer created: IRSA or Workload Identity signs every request

POST /api/org/space-storage/test with the same body runs the same connection test as the Test connection button.

Publish a first stack

Install the default-engineering stack from the marketplace, which installs its packages and creates a draft, then publish the draft:

VERSION=$(curl -sS "$URL/api/marketplace/browse?q=default-engineering" "${AUTH[@]}" | jq -r '.packages[] | select(.name == "default-engineering") | .latest_compatible') STACK=$(curl -sS -X POST "$URL/api/marketplace/install" "${AUTH[@]}" \ -d "{\"name\": \"default-engineering\", \"version\": \"$VERSION\"}" | jq -r .stack_id) STACK_VERSION=$(curl -sS -X POST "$URL/api/stacks/$STACK/publish" "${AUTH[@]}" -d '{}' | jq -r .id)

The publish response is the new stack version, and its id is what a manifest pins.

Create a manifest and a space

curl -sS -X POST "$URL/api/manifests" "${AUTH[@]}" \ -d "{\"name\": \"engineering\", \"stack_version_id\": \"$STACK_VERSION\"}" marshal spaces new --manifest engineering --name first-space --detach

What you should see: marshal manifests list shows engineering, and marshal spaces list shows first-space. Work in it with marshal spaces exec first-space -- <command>, and stop and resume it with marshal spaces stop first-space and marshal spaces resume first-space. With space storage connected, files under /home/agent come back after the resume.