Sizing and capacity planning

Size your cluster from one number: how many spaces you want running at the same time. This page turns that number into nodes and disk, with the arithmetic shown, so you can plan a pilot or a company-wide rollout with confidence. The exact machine-type variables for each cloud are on its install page. How the pieces fit together is on Architecture.

Two pools, two jobs

Every Marshal cluster splits its compute into two node pools:

The installer keeps the pools separate by default: the control-plane pool is tainted, and the chart schedules the platform onto it. A burst of spaces then never takes CPU from the services everything else depends on. The switch is taint_control_plane_pool, on by default, and each cloud’s install page covers it.

Control plane

The chart runs at least 2 replicas each of the API, gateway, web app and marketplace: 8 pods before the first person signs in. Their requests add up as follows:

(1 + 0.5 + 0.2 + 0.5) × 2 = 4.4 vCPU. (1024 + 512 + 256 + 512) MiB × 2 = 4608 MiB = 4.5 GiB.

A 4 vCPU machine has less than 4 vCPU allocatable (about 3.9 on an e2-standard-4), so this baseline does not fit, and pods stay Pending with Insufficient cpu. Start the control plane on the 8 vCPU class. Each service scales out under load (the API and gateway up to 10 replicas, the web app and marketplace up to 6), so leave room above the baseline.

Spaces

Each space asks the scheduler for the requests of its compute class. The class is set per manifest, stack or org, and Medium is the default. See Compute sizes for every class and how a space gets its class. Admins can move every class to persistent disk, or to one storage class, in a single action: see Set storage on many sizes at once.

Spaces per node

On the default spaces node (8 vCPU with about 7.9 allocatable, 32 GB memory with about 29 GiB allocatable, and about 178 GB of allocatable disk on EKS or 146 GiB on GKE), the number of spaces that fit is the smallest of three divisions. Disk is shown for EKS first, then GKE, because the clouds leave different shares of the boot disk for pods (about 89% on EKS, about 49% on GKE):

These are upper bounds. The operating system, the image cache and system pods also use part of each node. On this node shape CPU and disk run out close together, so neither strands the other. Disk is the first limit for spaces on Persistent disk storage: each one reserves 16 GiB of node disk, which fits about 11 spaces on an EKS node and 9 on a GKE node, against 15 by CPU. The installer’s disk defaults come from this: the built-in classes reserve 20 GiB of disk per vCPU, far more than a node’s boot disk offers per vCPU, so size the disk before the machine.

To run 4x Large spaces, add a pool with a larger machine type. On GKE, add an entry to spaces_pools. On EKS, add an entry to spaces_node_groups. See Change node size without stopping spaces.

On GKE, image streaming makes the first spaces on a freshly added node start faster. See Faster space start on new nodes.

Nodes for your peak

Nodes needed = running spaces ÷ spaces per node, rounded up. The counts below use the lower of the EKS and GKE figures, so they hold on both. Disk is nodes × the boot disk (200 GB on EKS, 300 GB on GKE).

Worked example: 50 running Medium spaces. 50 ÷ 14 = 3.6, so 4 spaces nodes: 800 GB of disk on EKS (4 × 200 GB) or 1.2 TB on GKE (4 × 300 GB), plus one 8 vCPU control-plane node. Real packing is looser than the division, because spaces start and stop all day. Add about 30% for a steady load and 45% for the peak: here, 5 to 6 nodes.

A space only uses node resources while it runs. A stopped space frees its place on the node. A class with the Persistent disk storage mode puts the home folder on its own disk, and the space asks the node for only 16 GiB of local disk, for /tmp, container layers and logs. An admin tunes that with Network Disk Space Node Reservation (GiB) in Settings > Configuration, Spaces group, from 4 to 200. The limit is twice the value. Lower it to fit more spaces on a node, raise it if spaces are evicted for using too much node disk. Spaces launched after the change get the new value.

Cloud quotas

Your cloud tracks CPU quota and disk quota separately, and either one can stop the cluster from scaling. Both look the same from the cluster: new nodes do not appear and pods stay Pending.

• CPU quota (for example GCP CPUS_ALL_REGIONS, or the AWS vCPU quota for your instance family) caps the vCPUs of all node pools together. The defaults need 72 vCPU on AWS and 48 vCPU on GCP at create.
• Disk quota (for example GCP’s regional SSD_TOTAL_GB) caps the total disk of all nodes, separately from CPU. The spaces pool’s disk size × node count counts against it, and so does the control plane’s. The GKE defaults need about 1050 GB of regional SSD_TOTAL_GB.

Before you raise a pool’s maximum node count, check both quotas at the new maximum. A CPU limit needs more CPU quota or a smaller machine. A disk limit needs more disk quota, a smaller disk or fewer nodes.

Putting it together

1. Start the control plane at 8 vCPU. The chart’s own baseline sets the floor, whatever your traffic on day one.
2. Pick the class most of your spaces use, and read spaces per node from the table.
3. Divide your peak running spaces by it to get the spaces pool’s maximum node count.
4. Check CPU and disk quota at that maximum, for both pools.

For the exact machine-type variables and disk settings on your cloud, continue to Installing Marshal.

Change node size without stopping spaces

Changing the machine type or boot disk of a running pool replaces it, and a replacement stops every space on it. Move over instead, with a second pool beside the first:

1. Check quota for the old and new pools together. Both run during the move.

2. Add the new pool. On GKE, add an entry to spaces_pools. On EKS, add an entry to spaces_node_groups. Leave the existing pool untouched, and confirm terraform plan shows only additions.

3. Cordon the old nodes. Running spaces keep running, and new spaces land on the new pool:

   # GKE
   kubectl cordon -l cloud.google.com/gke-nodepool=<old-pool-name>
   # EKS
   kubectl cordon -l eks.amazonaws.com/nodegroup=<cluster_name>-spaces

   Before the control-plane pool was tainted, a space could also start on a control-plane node. List where every space runs with kubectl get pods -n marshal-spaces -o wide, and include any space on a control-plane node in the next step too.

4. Ask your team to stop and start their spaces over the next few hours. A space that starts again comes back on the new pool with its files.

5. Remove the old pool when it is empty. Check that no space pods remain on the old nodes (kubectl get pods -n marshal-spaces -o wide). On GKE, delete the old key from spaces_pools. On EKS, set create_default_spaces_node_group = false, or delete the old entry from spaces_node_groups. The plan should show only the old pool going away.

To roll back before step 5, run kubectl uncordon on the old nodes.

Related

• Compute sizes
• Spaces and storage
• Configuration reference
• Troubleshooting