Install onto EKS

Marshal runs in your own AWS account: an EKS cluster, Postgres, and an internal load balancer in front. Nothing is exposed to the internet by default. One terraform apply installs into a cluster you already run, or builds the network, the cluster and the database and installs onto them.

What you can do with it

• Give Marshal its own AWS account. Let the installer build a dedicated VPC, a private EKS cluster, two node groups and an RDS database in one apply.
• Add Marshal to the cluster you already run. Point the installer at your EKS cluster and your Postgres. It installs the chart and creates no AWS infrastructure.
• Build the cluster inside your existing network. Keep your VPC and subnets, and let the installer create everything else.

Choose your path

Two questions decide everything else.

Who owns the cluster? If you already run an EKS cluster you are happy to install into, use it. If you want Marshal isolated, or you have no cluster, let the installer build one.

Who owns the network? This matters only when the installer builds the cluster. It can create a dedicated VPC, or use a VPC you already run.

Quickstart

git clone --branch v<chart_version> https://github.com/marshal-org/marshal-install.git
cd marshal-install/aws-eks

MRPT_FILE=$(umask 077; mktemp)                                   # a new file only you can read
read -rs MRPT; printf '%s' "$MRPT" > "$MRPT_FILE"; unset MRPT     # paste the pull token
../scripts/preflight-aws.sh --region <region> --vpc-id <vpc> \
  --private-subnets <a>,<b> --ingress-host <host> \
  --deployment-id <deployment_id> --pull-token-file "$MRPT_FILE" --chart-version <chart_version>
rm -f "$MRPT_FILE"

cp ../examples/aws-eks.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars

terraform init
terraform plan     # review
terraform apply

terraform output next_steps

What you should see

Every pod in the marshal namespace is Running, and the api log reports license state: LICENSED.

Open Marshal from your own machine through a port-forward. It needs no DNS and no route to the load balancer, and serves the console, the API and the CLI on one port:

aws eks update-kubeconfig --name <cluster_name> --region <region>
kubectl port-forward -n marshal svc/web 8080:80

Then browse http://localhost:8080, and sign the CLI in with marshal login --server http://localhost:8080 --password.

For your team, point your hostname’s DNS at the load balancer. terraform output app_url prints the address. The load balancer is internal, so that URL answers from inside your network. Until you add a real certificate, it serves a self-signed placeholder and the browser warns. See TLS certificate.

Claim your owner account

kubectl get secret marshal-app-secrets -n marshal -o jsonpath='{.data.SETUP_TOKEN}' | base64 -d

You are signed in as the owner. Next, follow Getting started as an owner to invite your team, then run your first task. Working without a browser? See Set up from a terminal.

What gets created

Path 1: install into your existing cluster

deploy_cluster = false, the default. The installer adds Marshal to a cluster you already run and creates no AWS infrastructure. Four things must be in place first:

EKS does not enable the EBS CSI driver by default. Check it with kubectl get deploy -n kube-system ebs-csi-controller before you start.

Path 2: let the installer build the cluster

deploy_cluster = true. You get, in one apply:

Bring your own VPC

To build the cluster in a network you already run, name it. The cluster, both node groups, the optional bastion and RDS are still created for you: only the network is yours.

create_network     = false
vpc_id             = "vpc-0a1b2c3d"
private_subnet_ids = ["subnet-0aaa", "subnet-0bbb"]
public_subnet_ids  = []       # only for an internet-facing load balancer

Your subnets must meet the requirements in AWS: before you install. The plan stops early if a subnet is outside vpc_id or if the private subnets cover fewer than two availability zones. Each private subnet also needs a default route out, which preflight-aws.sh checks: without egress, nodes cannot pull images.

The load balancer controller finds subnets by tag, so the installer adds the three tags it needs: kubernetes.io/role/internal-elb on private subnets, kubernetes.io/role/elb on public ones, and kubernetes.io/cluster/<cluster_name>: shared on both. It manages each tag on its own and leaves everything else on your subnets alone. A tag your subnet already carries stays yours: the installer skips it, and a later terraform destroy leaves it in place. terraform output subnet_tags_left_as_yours lists those tags. The installer records the tags it owns in a fourth tag, marshal.codes/installer-owned-tags, and removes it on destroy.

Images

Nodes pull Marshal’s images directly from registry.marshal.codes with your pull token. That is the default, and it works with no extra setup. For production, add a pull-through cache so space starts stay fast as usage grows: the installer can run Harbor in the cluster, or you can point image_cache_registry at a mirror you already run. AWS ECR cannot be the cache, because its pull-through cache accepts only a fixed list of public upstream registries. See Registry options.

Reaching a private cluster

The cluster’s API endpoint is private by default, so terraform, kubectl and helm run from somewhere that reaches it. Choose this once: every later upgrade runs from the same place.

Use the access you already have

A VPN, Twingate, Tailscale, Direct Connect or a jump host all work, and most organizations running a private EKS cluster already have one. Keep the default:

create_bastion = false   # the default

You run terraform apply once, from wherever you normally work, and every later upgrade is a single command from the same place.

Your tunnel must route the private subnet ranges, not only the hostname. A private EKS endpoint still resolves in public DNS, to addresses inside your VPC. In Twingate and Tailscale, that means the connector or subnet router runs inside the VPC and advertises that CIDR. Test it before install day:

aws eks update-kubeconfig --name <cluster> --region <region>
kubectl get nodes

The installer opens the API endpoint to your VPC’s own range, because the security group EKS creates admits only the cluster itself. cluster_private_access_cidrs defaults to the VPC CIDR. Set it to narrow which ranges may connect, and include the range your tunnel presents. Reaching the endpoint still needs an AWS identity with an EKS access entry and Kubernetes RBAC.

No route in? Open the endpoint to your own address

Without a VPN, give the endpoint public access for your own address only:

cluster_endpoint_public_access = true
cluster_endpoint_public_access_cidrs = [
  { cidr_block = "203.0.113.7/32", display_name = "platform-laptop" }
]

The cluster stays private to everyone else. Nodes stay in private subnets, the ingress stays internal, and only that one address reaches the API, with an EKS access entry and Kubernetes RBAC in front of it. Run the install as a single terraform apply.

Keep it this way for the life of the install. Every later terraform plan, apply and destroy, upgrades and teardown included, connects to the cluster API before it changes anything, so it runs from an address that reaches the endpoint. If your policy requires a private endpoint afterwards, set cluster_endpoint_public_access = false as the last step and run Terraform from inside the VPC from then on: through a VPN, or the installer’s bastion.

When your address changes, open the endpoint to the new one with the AWS API, which does not go through the endpoint, then put the same CIDR in cluster_endpoint_public_access_cidrs so the next apply keeps it:

aws eks update-cluster-config --name <cluster_name> --region <region> \
  --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs=<your-ip>/32

No network access at all: the installer’s bastion

create_bastion = true creates a bastion with no public IP and no inbound ports, reached only through AWS Systems Manager, so there is no SSH key to manage. Every upgrade then runs from the bastion as well, so prefer the options above when you can.

aws s3api create-bucket --bucket acme-eks-tfstate --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1
aws s3api put-bucket-versioning --bucket acme-eks-tfstate \
  --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket acme-eks-tfstate \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

terraform {
  backend "s3" {
    bucket       = "acme-eks-tfstate"
    key          = "marshal/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true    # needs Terraform 1.10 or newer
  }
}

terraform init -migrate-state

terraform init
terraform apply

terraform output bastion_instance_id
aws ssm start-session --target <bastion-instance-id> --region <region>

git clone --branch v<chart_version> https://github.com/marshal-org/marshal-install.git
cd marshal-install/aws-eks
# Recreate backend_override.tf, and fetch your tfvars from where your team keeps secrets:
aws secretsmanager get-secret-value --secret-id marshal/tfvars \
  --query SecretString --output text > terraform.tfvars
 
terraform init      # picks up the shared state
terraform plan      # review
terraform apply     # finishes the rest, in AWS and in the cluster

The bastion’s role has AdministratorAccess by default, so it can create whatever is still outstanding. Set bastion_install_permissions = false for a read-only role that does only the in-cluster work. Delete the bastion when the install is done and the permission goes with it.

Move the tfvars through a secrets store, not through an S3 object or a paste into the Systems Manager shell: a versioned bucket keeps old versions after aws s3 rm, and Session Manager logging records what you paste. When the install is done, rotate the tokens, because they now exist in your tfvars, the state and the bastion’s disk.

TLS certificate

An ALB terminates TLS only with a certificate from ACM in the same region. It does not read a Kubernetes TLS Secret, so cert-manager and your own PEM files do not apply here. Pick one of four options:

The certificate covers ingress_host, every entry in extra_ingress_hosts, and the preview wildcard when preview_domain is set.

terraform apply    # with acm_request_certificate = true

terraform output acm_validation_records

terraform output acm_certificate_status   # wait for ISSUED
terraform apply                           # attaches it to the ALB

RDS options

create_rds = true creates the database for you when the installer also builds the cluster (deploy_cluster = true). The defaults keep the bill low and make an accidental delete hard.

Keeping the bill down

At rest, the default install runs one m5.2xlarge space node and one m5.2xlarge platform node. The Cluster Autoscaler (enable_cluster_autoscaler, on by default) adds space nodes when spaces are waiting for room and removes empty ones after about 10 minutes, between node_min_size and node_max_size. A node that holds a running space is never removed. The autoscaler owns the node count after the cluster is created, so node_desired_size only sets the starting size. Running Karpenter or your own autoscaler? Set enable_cluster_autoscaler = false and change a running group with aws eks update-nodegroup-config.

Cross-AZ data transfer between the nodes, the ALB and RDS is the main per-GB charge in a steady install. Pin both node groups to one AZ to remove it:

node_availability_zone = "eu-west-1a"

The cluster still needs subnets in two AZs, because EKS requires it. This only decides where nodes run. An AZ outage then takes the whole install down rather than part of it, which is usually the right trade for a developer tool. Changing this value later replaces both node groups, because a node group’s subnets cannot change.

See compute cost per space

Every space and run shows its compute cost next to its LLM cost. To price it from your own AWS bill, give the API a read-only role with one permission, ce:GetCostAndUsage, activate the aws:eks:cluster-name cost allocation tag, and set cost.billing.source: aws. The steps are in Billed compute cost.

Storage buckets

Turn on both. Space storage saves every space’s files, so people can stop, resume, fork and archive spaces. Audit export streams the audit log to a bucket in your own account.

create_spaces_persistence_bucket = true   # space storage
create_audit_persistence_bucket  = true   # audit export

Access is keyless. The installer creates an IAM role, <cluster_name>-storage-role, with object access to each bucket, and binds it through IRSA to Marshal’s service accounts: the api for both buckets, and the supervisor that runs inside each space for space storage. No access key is created or stored. The buckets live in your cluster’s region, which is also the Region you enter in Settings > Space Storage.

The installer creates these buckets when it builds the cluster (deploy_cluster = true). On a cluster you already run, create the buckets and an IRSA role for them yourself, and pass the role through api_service_account_annotations, plus supervisor_service_account_annotations for space storage:

api_service_account_annotations        = { "eks.amazonaws.com/role-arn" = "arn:aws:iam::123456789012:role/marshal-storage" }
supervisor_service_account_annotations = { "eks.amazonaws.com/role-arn" = "arn:aws:iam::123456789012:role/marshal-storage" }

After the install, an owner or admin connects each bucket in Settings > Space Storage and Settings > Auditing. See Object storage.

Teardown

Run these steps in this order. A default install takes about 12 minutes to remove. If you reach the cluster through a public endpoint open only to your address, make sure it is open before you start.

force_destroy_buckets   = true    # if you created the storage buckets: empties them
rds_deletion_protection = false   # if you used create_rds
rds_skip_final_snapshot = true    # if you used create_rds; leave it out to keep a final snapshot

terraform apply

kubectl delete namespace marshal marshal-spaces

terraform destroy

These resources stay after the destroy, because Terraform does not own them:

• The final RDS snapshot, <cluster_name>-postgres-final, unless you skipped it. A new install under the same cluster name needs a different snapshot name, so delete or rename the old one first.

• EBS volumes for PersistentVolumeClaims, and the load balancer, its target groups and the controller’s security groups, only if the namespaces were not deleted first. Each carries your cluster name as a tag, so you find exactly this install’s resources:

  aws ec2 describe-volumes --region <region> \
    --filters Name=tag:KubernetesCluster,Values=<cluster_name> \
    --query 'Volumes[].{id:VolumeId,state:State,size:Size}' --output table
  aws resourcegroupstaggingapi get-resources --region <region> \
    --resource-type-filters elasticloadbalancing:targetgroup \
    --tag-filters Key=elbv2.k8s.aws/cluster,Values=<cluster_name> \
    --query 'ResourceTagMappingList[].ResourceARN' --output text
  aws ec2 describe-security-groups --region <region> \
    --filters Name=tag:elbv2.k8s.aws/cluster,Values=<cluster_name> \
    --query 'SecurityGroups[].GroupId' --output text

Subnet tags you already had before the install stay on your subnets.

Upgrades

Check out the installer tag for the new version and bump chart_version to match. Then work through the installer’s capabilities.yaml before you plan: answer every entry newer than your version in terraform.tfvars, pinning your current node sizes so the spaces node group is not replaced. Run terraform init -upgrade and terraform plan, check it replaces nothing you did not choose, and terraform apply, from wherever reaches the cluster API: your own machine if you reach the VPC directly, or the bastion if you installed that way. The full steps are in Day-2.

The recommended on-demand vCPU quota is 72, so the Cluster Autoscaler can reach its default maximums. If you sized an earlier install for 40, request the increase before you upgrade, or the quota sets how many spaces run. scripts/preflight-aws.sh prints the quota and what is already in use.