Install onto GKE

Marshal runs in your own Google Cloud project: on a GKE cluster, behind an internal load balancer, with images served from an Artifact Registry cache in your project. Nothing is exposed to the internet by default. One terraform apply installs into a cluster you already run, or builds a private cluster and installs onto it.

What you can do with it

• Add Marshal to the cluster you already run. Point the installer at your GKE cluster and your Cloud SQL database. It installs the chart, plus an image cache and an ingress IP in your project.
• Give Marshal its own private cluster. Let Terraform build the VPC, Cloud NAT, a private control plane and two node pools in the same apply.
• Build with Terraform, deploy with ArgoCD. Provision the cluster here and hand the release to your GitOps pipeline.

What you need

• The Prerequisites, including Terraform 1.5 or newer, Helm 3.14 or newer, gcloud and kubectl.
• Application Default Credentials: gcloud auth application-default login. Installing into an existing cluster needs container.clusters.get. Building a cluster needs container.admin and compute network admin on the project.
• A Postgres 16 database the cluster reaches over the VPC, such as Cloud SQL with a private IP. See Create Cloud SQL. For an evaluation, database_mode = "bundled" runs Postgres in the cluster instead.

Choose your path

cluster_name defaults to marshal, which is the name a new cluster gets.

Quickstart

git clone --branch v<chart_version> https://github.com/marshal-org/marshal-install.git
cd marshal-install/gcp-gke

cp ../examples/gcp-gke.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars

terraform init
terraform plan     # review
terraform apply

terraform output next_steps

What you should see

Every pod in the marshal namespace is Running, and the api log reports license state: LICENSED.

Open Marshal from your own machine through a port-forward. It needs no DNS and no route to the load balancer, and serves the console, the API and the CLI on one port. On a cluster the installer built with dns_endpoint_enabled = true, it works from anywhere:

$(terraform output -raw kubeconfig_command)
kubectl port-forward -n marshal svc/web 8080:80

Then browse http://localhost:8080, and sign the CLI in with marshal login --server http://localhost:8080 --password.

For your team, point your hostname’s DNS at the ingress IP. terraform output app_url prints the address. The load balancer is internal, so that URL answers from inside your network, 5 to 15 minutes after the apply while Google programs it. HTTPS is on with a self-signed placeholder certificate until you add your own, so the browser warns at first.

Claim your owner account

kubectl get secret marshal-app-secrets -n marshal -o jsonpath='{.data.SETUP_TOKEN}' | base64 -d

You are signed in as the owner. Next, follow Getting started as an owner to invite your team, then run your first task. Working without a browser? See Set up from a terminal.

What gets created

With deploy_cluster = true, one apply creates:

deploy_cluster       = true
cluster_name         = "marshal"
location             = "europe-west1"
dns_endpoint_enabled = true
install_marshal      = false   # first apply; then Cloud SQL, database_url and install_marshal = true

With Cloud SQL on this new network, the install is two applies. See Create Cloud SQL.

location takes a region or a zone. A regional cluster runs every node pool in each of the region’s zones, so it needs capacity in all of them at once. If the create stops with GCE_STOCKOUT, choose a zone such as europe-west1-b, which also needs a third of the quota.

On every path, the installer also creates the image cache (see Images) and, with the default internal ingress and TLS, a static internal IP for the ingress.

Reach a private cluster

The control plane endpoint is private by default (enable_private_endpoint = true), so the part of the apply that installs the chart needs a way to reach it. Three ways, best first:

The nodes stay private every way. With the DNS endpoint, fetch credentials with the --dns-endpoint flag; terraform output kubeconfig_command prints the exact command:

gcloud container clusters get-credentials <cluster_name> --location <location> \
  --project <project_id> --dns-endpoint

The DNS endpoint covers the control plane: terraform, kubectl, helm and kubectl port-forward. The app itself stays on the internal load balancer.

# 1. Network and cluster. Works from anywhere.
terraform apply -target=google_compute_network.vpc \
                -target=google_compute_subnetwork.subnet \
                -target=google_compute_subnetwork.proxy_only \
                -target=google_compute_router.router \
                -target=google_compute_router_nat.nat
terraform apply -target=google_container_cluster.this \
                -target='google_container_node_pool.spaces["spaces"]'
#    And the IAM bindings, as a project owner, so the bastion needs no IAM-admin role.
terraform apply -target=google_project_iam_member.nodes_default_node_sa \
                -target=google_project_iam_member.nodes_service_usage \
                -target=google_service_account_iam_member.storage_workload_identity \
                -target=google_service_account_iam_member.storage_workload_identity_supervisor
 
# 2. Set master_authorized_networks to the subnet the bastion will use,
#    for example the default subnet_cidr, 10.10.0.0/20.
 
# 3. A bastion with its own service account and no external IP, reached over IAP.
gcloud iam service-accounts create marshal-bastion --project=<project>
SA=marshal-bastion@<project>.iam.gserviceaccount.com
for r in compute.networkAdmin container.admin storage.admin artifactregistry.admin \
         secretmanager.admin certificatemanager.editor serviceusage.serviceUsageAdmin \
         iam.securityReviewer iam.serviceAccountViewer logging.logWriter; do
  gcloud projects add-iam-policy-binding <project> --member=serviceAccount:$SA \
    --role=roles/$r --condition=None
done
gcloud iam service-accounts add-iam-policy-binding <cluster>-nodes@<project>.iam.gserviceaccount.com \
  --member=serviceAccount:$SA --role=roles/iam.serviceAccountUser
gcloud compute instances create marshal-bastion --zone=<region>-b --subnet=<subnet> \
  --no-address --service-account=$SA --scopes=cloud-platform \
  --image-family=debian-12 --image-project=debian-cloud
gcloud compute firewall-rules create allow-iap-ssh --network=<network> --direction=INGRESS \
  --action=ALLOW --rules=tcp:22 --source-ranges=35.235.240.0/20
gcloud compute ssh marshal-bastion --zone=<region>-b --tunnel-through-iap
 
# 4. On the bastion: the same GCS state backend and the same tfvars, then terraform apply.

Ingress and TLS

The ingress is internal by default (internal_ingress = true, class gce-internal), reachable only inside your VPC. ingress_host is required on the first apply, before DNS exists: apply, then point DNS at the ingress IP.

An internal ingress needs a static internal IP to serve HTTP and HTTPS on one address. The installer reserves it for you (create_ingress_static_ip = true, named by ingress_static_ip_name). Set it to false when you manage that IP yourself, and the installer uses the existing address with that name.

For previews, a wildcard certificate covers *.preview_domain. The Google-managed certificate below includes it, and cert-manager issues one with a DNS-01 solver.

Google-managed certificate

Serve Marshal through a Gateway on the same internal load balancer, and Google issues and renews the certificate for you. Nothing in the cluster rotates it. Set this in terraform.tfvars:

ingress_mode               = "gateway"
create_managed_certificate = true

gcloud services enable certificatemanager.googleapis.com --project <project_id>

gcloud container clusters update <cluster_name> --location <location> --gateway-api=standard

terraform output managed_certificate_dns_records

gcloud certificate-manager certificates describe <cluster_name>-marshal-cert --location <region>

Already have a Certificate Manager certificate in the same project and region? Name it in gateway_certificate_names instead. The certificate covers ingress_host and the preview domain; hosts in extra_ingress_hosts keep routing on the same Gateway.

Datastores

Postgres comes from your own database (database_mode = "external", the default) through database_url. Redis runs in the cluster (redis_mode = "bundled", the default): everything Marshal keeps in Redis can be rebuilt, so a managed Memorystore adds cost without adding safety. See Datastore options.

Create Cloud SQL

The installer leaves the database to you, so it fits the way you already run Cloud SQL. A Postgres 16 instance with a private IP lives in a VPC, and that decides the order:

• Your own cluster or network. The VPC exists. Create Cloud SQL first, then run one apply with database_url set.
• A new cluster and network (deploy_cluster = true). The apply creates the VPC, so the install takes two applies:
  1. Set install_marshal = false and run terraform apply. It builds the network, the cluster and the node pools, enables the Cloud SQL APIs, and installs nothing yet.
  2. Create Cloud SQL on the new VPC, <cluster_name>-vpc, with the commands below.
  3. Set database_url and install_marshal = true, and run terraform apply again. It installs Marshal onto the cluster from step 1.

terraform plan checks that database_url is set whenever it installs Marshal with an external database.

# The two APIs Cloud SQL needs. On a new cluster and network the installer turns them on;
# run this on your own cluster or network.
gcloud services enable sqladmin.googleapis.com servicenetworking.googleapis.com --project <project>
 
# Private Services Access, once per VPC.
gcloud compute addresses create google-managed-services-<vpc> --project <project> \
  --global --purpose=VPC_PEERING --prefix-length=16 --network=<vpc>
gcloud services vpc-peerings connect --project <project> --service=servicenetworking.googleapis.com \
  --ranges=google-managed-services-<vpc> --network=<vpc>
 
# Postgres 16, private IP only.
gcloud sql instances create <name> --project <project> --database-version=POSTGRES_16 --edition=ENTERPRISE \
  --tier=db-custom-2-7680 --region=<region> \
  --network=projects/<project>/global/networks/<vpc> --no-assign-ip --availability-type=ZONAL
gcloud sql databases create marshal --instance=<name> --project <project>
gcloud sql users create marshal --instance=<name> --project <project> --prompt-for-password
gcloud sql instances describe <name> --project <project> --format='value(ipAddresses[0].ipAddress)'

--prompt-for-password keeps the password out of your shell history. database_url is then postgres://marshal:<password>@<private-ip>:5432/marshal?sslmode=require.

--edition=ENTERPRISE accepts custom sizes such as db-custom-2-7680 (2 vCPU, 7.5 GB). On ENTERPRISE_PLUS, pick one of its predefined tiers instead. See Sizing.

Node pools

Space pods run on the spaces_pools pools, and the platform runs on the control-plane pool. The control-plane pool uses e2-standard-8 because the chart’s baseline of two replicas each for the api, gateway, web and marketplace needs about 4.4 vCPU, more than a smaller machine offers.

Space pods stay off the control-plane pool by default: taint_control_plane_pool is true. It taints the pool and adds the matching toleration to the chart in the same apply. Keep control_plane_max_node_count at its default of 2: a machine-type change drains the node, and the chart’s disruption budgets need a second node to move the platform pods to.

Space pods request a lot of ephemeral storage, so check your project’s CPUS_ALL_REGIONS and regional SSD_TOTAL_GB quotas against every pool before you raise sizes. The defaults need 48 vCPU and about 1050 GB of SSD_TOTAL_GB at create (300 GB × 3 zones for spaces, plus the control-plane pool), and the default SSD_TOTAL_GB quota is 500 GB, so raise it first. See Sizing.

To change a pool’s machine type or disk without stopping spaces, add a new entry to spaces_pools, cordon the old nodes, ask your team to stop and start their spaces, and remove the old entry once it is empty. The steps are in Change node size without stopping spaces.

Faster space start on new nodes

When the autoscaler adds a node, that node has not pulled any space images yet. GKE image streaming lets containers start while their images are still downloading, so the first spaces on a new node are ready in a fraction of the time. Turn it on if your spaces mount many tool images or your usage comes in bursts that add nodes often.

image_streaming_enabled  = true
max_parallel_image_pulls = 4 # optional

On a fresh node, a space that mounts 12 tool images (about 4.9 GB) started in about 30 seconds with streaming on, down from about 180 seconds. A standard space went from about 36 seconds to about 16 seconds. The speed-up applies once each image has been streamed once in the region. The first start after you publish a new image version takes about as long as it does with streaming off, because streaming fills its cache on first use.

max_parallel_image_pulls does not shorten the start of a single space, because a node pulls one pod’s images one after another. Use it for scale-up bursts, and use image_streaming_enabled for the start time of each space.

Requirements

• GKE 1.30.1-gke.1329000 or later, with the default COS_CONTAINERD node image.
• If you set node_service_account to your own service account, grant it roles/serviceusage.serviceUsageConsumer on the project.
• If a VPC Service Controls perimeter protects the project, include containerfilesystem.googleapis.com in it.

To turn streaming off again, set image_streaming_enabled = false and run this for each pool. GKE keeps the live value on existing pools, so the command is what switches it off:

gcloud container node-pools update <pool> --cluster <cluster> \
  --location <location> --no-enable-image-streaming

Both settings apply to clusters the installer builds (deploy_cluster = true).

Images

create_registry_cache is on by default. It creates an Artifact Registry remote repository that mirrors registry.marshal.codes, stores your pull token in Secret Manager for it, and points the install at the cache with Workload Identity. Nodes pull from your project, and space starts stay fast as usage grows. See Registry options.

Storage buckets

Turn on both. Space storage saves every space’s files, so people can stop, resume, fork and archive spaces. Audit export streams the audit log to a bucket in your own project.

create_spaces_persistence_bucket = true   # space storage
create_audit_persistence_bucket  = true   # audit export

Access is keyless. The installer creates two Google service accounts and binds each through Workload Identity to one of Marshal’s service accounts. No key is created or downloaded.

The supervisor has its own account so that the code running in a space never reaches the audit bucket.

After the install, an owner or admin connects each bucket in Settings > Space Storage and Settings > Auditing. See Object storage.

Hand the release to ArgoCD

Set install_marshal = false and the apply builds the cluster, node pools, image cache, buckets and static IP, and installs nothing. Terraform still computes the GKE-specific chart values, such as the registry endpoints and the storage service account. Write them to a file and use them in your ArgoCD Application:

terraform output -raw argocd_derived_values > values-derived.yaml
terraform output next_steps      # the secrets to seed before the first sync

Then follow Install with ArgoCD. Render the values again after any apply that changes the cluster, cache, buckets or taint_control_plane_pool.

If the cluster create fails

A cluster whose create failed, for example on GCE_STOCKOUT, is replaced on the next apply. The installer guards the cluster with deletion_protection, so delete the failed one first, then apply again:

gcloud container clusters delete <cluster_name> --location <location> --project <project_id>
terraform apply

To retry after a stockout, change location to a zone or another region before the apply.

Teardown

Run these steps in this order. A default install takes about 15 to 20 minutes to remove.

deletion_protection   = false   # a cluster the installer built
force_destroy_buckets = true    # if you created the storage buckets: empties them

terraform apply

gcloud sql instances delete <name> --project <project>
gcloud services vpc-peerings delete --service=servicenetworking.googleapis.com \
  --network=<vpc> --project <project>
gcloud compute addresses delete google-managed-services-<vpc> --global --project <project>

terraform destroy

Afterwards, gcloud container clusters list, gcloud sql instances list and gcloud storage buckets list in the project show nothing of this install.

Upgrades

Check out the installer tag for the new version and bump chart_version to match. Then work through the installer’s capabilities.yaml before you plan: answer every entry newer than your version in terraform.tfvars, pinning your current node sizes so the spaces node pools are not replaced. Run terraform init -upgrade and terraform plan, check it replaces nothing you did not choose, and terraform apply. The full steps are in Day-2.