AWS: before you install
Settle these AWS items a week before you install, and the install itself is one
terraform apply. This page adds the AWS layer on top of
Prerequisites. Every item is quick to settle by email ahead of
time, and two of them (a quota increase and an organization policy change) can take days
because other teams own them.
Run the preflight script first
MRPT_FILE=$(umask 077; mktemp) # a new file only you can read
read -rs MRPT; printf '%s' "$MRPT" > "$MRPT_FILE"; unset MRPT # paste the pull token
scripts/preflight-aws.sh --region eu-west-1 \
--vpc-id vpc-0123 --private-subnets subnet-a,subnet-b \
--ingress-host marshal.acme.com \
--deployment-id dep_ab12cd --pull-token-file "$MRPT_FILE" --chart-version <chart_version>
rm -f "$MRPT_FILE"It is read-only, needs no write permission, and reports every problem in one pass. It answers most of this page automatically: region capability, quota, subnet and DNS facts, and whether your credentials work. Run it again on the day of the install, because quotas get consumed and subnets get re-tagged in between.
What it cannot answer is the one in the next section.
Permissions for the identity that runs the apply
Nothing read-only can prove the apply will be allowed, so this is the item worth checking by hand. The EKS installer creates resources in these services:
| Service | What for |
|---|---|
eks | The cluster, two managed node groups, addons, access entries |
ec2 | VPC, subnets, route tables, internet and NAT gateways, an Elastic IP, security groups, launch templates, the bastion instance |
iam | Roles and policies for the cluster, the nodes, the addons and the load balancer controller, plus an OIDC provider for IRSA |
rds | The Postgres instance and its subnet group, only with create_rds = true and deploy_cluster = true |
acm | Importing or requesting the load balancer certificate |
route53 | Validation records, only when you let Terraform validate a requested certificate |
s3 | The optional audit and space storage buckets, and the Harbor cache bucket if you enable it |
sts, ssm | Reading the account id, and the optional bastion’s Systems Manager connection |
Recommended approach. Run the first install with an identity that has administrative access in the account you are installing into, ideally a dedicated account. Then narrow the permissions afterwards using CloudTrail, which gives you a policy derived from the calls your install actually made rather than a generic one.
If your organization cannot grant that even temporarily, raise it with your Marshal contact before the install date so the required actions can be worked through with your security team.
Check for organization policies above the account
An SCP attached to your AWS Organization is a ceiling over the whole account. It beats IAM,
it beats an administrator role, and it beats the root user, so an identity that looks fully
privileged can still be refused, for example on iam:CreateOpenIDConnectProvider, with no
sign of it in any IAM policy.
Confirm with whoever owns your organization that none of these is denied in your account and region:
eks:CreateClusterec2:RunInstancesrds:CreateDBInstanceiam:CreateOpenIDConnectProviderelasticloadbalancing:CreateLoadBalancer
Two related cases, both of which produce the same confusing AccessDenied:
- A region restriction. Some organizations pin an account to one region. The installer creates everything in a single region, so this is fine as long as it is your region, and fatal if it is not.
- A spend limit. On accounts that have one, going over it denies the expensive calls above, and operations that worked before start to fail.
Quotas
The default node size is m5.2xlarge (8 vCPU) for both the space nodes and the platform
node group. With the default scaling limits that is:
| vCPU | |
|---|---|
| At create (1 space node, 1 platform node) | 16 |
| With the optional bastion | 18 |
| Fully scaled out (5 space nodes, 2 platform nodes, bastion) | 58 |
| One extra node while a node group updates | +8 |
So ask for an on-demand standard instances quota of 72 vCPU in your region. 16 is the floor: the cluster cannot be created below it. Between 16 and 72 the install works, and the quota becomes the limit on how many spaces can run at once. A brand new AWS account can start as low as 5. The quota is shared with everything else running in the account and region, so the preflight prints both the quota and the vCPUs already in use, and checks the free headroom against these numbers. Raising it is a support request that can take from minutes to days, so ask early.
The NAT gateway needs one free Elastic IP. The preflight counts the addresses already allocated against your Elastic IP quota when the installer builds the network.
Network
By default the installer builds its own VPC, with public and private subnets, one NAT gateway and the subnet tags the load balancer controller needs. Nothing on this page applies to that path beyond the quota above.
You can also install into a VPC you already run, by setting create_network = false and
naming the VPC and subnets. The cluster, both node groups, the bastion and RDS are still
created for you. See Bring your own VPC for the inputs.
When you bring your own subnets, each of these is required. The preflight checks all of them. Two are enforced again by the installer, which stops the plan rather than failing partway through an apply: that the private subnets cover at least two availability zones, and that every subnet belongs to the VPC you named.
| Requirement | Why |
|---|---|
| Subnets in at least two availability zones | EKS refuses to create a cluster otherwise. This holds even when you pin the nodes to one AZ for cost. |
A default route (0.0.0.0/0) out of every private subnet | Without NAT, a transit gateway path or a proxy, nodes cannot pull images and the install dies partway through. |
| Private subnets that are genuinely private | A subnet whose default route is an internet gateway is public. Handing over public subnets while believing they are private is the single most common mix-up. |
kubernetes.io/role/internal-elb tagged on the private subnets | Without it the Ingress never gets an address and emits no events at all, which is slow to diagnose. The installer can add the tag for you unless your own Terraform owns those subnets. |
enableDnsSupport and enableDnsHostnames on the VPC | Required by EKS. The installer cannot set them for you. |
| Free IP addresses, a lot of them | The VPC CNI gives every pod a real subnet IP, so free addresses set the ceiling on concurrent spaces. Below about 60 free, ask for a bigger subnet before starting. |
Reaching the cluster
The EKS endpoint is private by default (cluster_endpoint_public_access = false), which
means whoever runs terraform apply has to be inside the network. Decide which of these you
are doing:
- Your own VPN or zero-trust access (Twingate, Tailscale, Direct Connect, a jump host you
already run). This is the default (
create_bastion = false) and the easiest to keep for upgrades. - A public endpoint locked to your own addresses, by setting
cluster_endpoint_public_access = truewith an explicit CIDR list. The quickest to set up, and the least isolated of the three. - A bastion the installer creates, with
create_bastion = true. No public IP, no inbound ports open, reached only throughaws ssm start-session. Terraform, kubectl and helm run from it. This needs Systems Manager reachable from the subnet, through egress or through the SSM VPC endpoints.
See Reaching a private cluster for each option.
Hostname, DNS and certificate
- Decide the hostname now, for example
marshal.acme.com. It is an input to the very first apply, before any load balancer exists. - Say who owns the DNS zone. If it is in Route53 in the same account, the certificate can be requested and validated in one apply. If it is anywhere else, the certificate takes two applies with you adding a CNAME in between. See TLS certificate.
- An ALB can only terminate TLS with an ACM certificate in the same region. It cannot read a Kubernetes TLS Secret, so cert-manager and your own PEM files do not apply here. Either bring an ACM certificate ARN, or let the installer request one.
- You point DNS at the load balancer after the first apply, since that is when its address exists.
Database
Two options, to decide before the install:
- Let the installer create RDS with
create_rds = true. This works when the installer also builds the cluster (deploy_cluster = true). You get a small single-AZ Postgres 16 instance with deletion protection on. See RDS options. - Bring your own Postgres 16, reachable from the cluster over private networking.
Requirements for a Postgres you bring yourself
- Postgres 16.x. Not 15, not 17.
- Reachable from the cluster’s private subnets, with
sslmode=require. - The
pgcryptoextension available, since the credential store is envelope-encrypted. - Password authentication.
- A dedicated database and an owner-level user on it.
- Your backups are yours. The installer does not configure backups for a database it did not create.
postgres://<user>:<password>@<private-host>:5432/<database>?sslmode=requireEgress
Outbound HTTPS to four hosts, with no inbound access into your environment ever required:
registry.marshal.codes, keys.marshal.codes, tokens.marshal.codes and
marketplace.marshal.codes. The full table, including what each one is for and the optional
fifth, is in Networking and Egress.
Node egress is the one thing you cannot test before a node exists. Check it from a node right after the cluster comes up, not from your laptop:
aws ssm send-command --document-name AWS-RunShellScript \
--targets Key=instanceids,Values=<node-instance-id> \
--parameters 'commands=["curl -sS -o /dev/null -w %{http_code} https://registry.marshal.codes/v2/"]'Optional: per-space compute cost
Marshal can price each space and run from your own AWS bill. It needs a read-only role with
one permission, ce:GetCostAndUsage, and the aws:eks:cluster-name cost allocation tag
activated in Billing, which AWS takes up to 24 hours to apply. Activating the tag early means
the numbers are ready on day one. It also needs outbound HTTPS to ce.us-east-1.amazonaws.com.
See Billed compute cost.
What to send us
Fill this in and send it back, and your Marshal contact can confirm whether the install is ready to book:
region: e.g. eu-west-1
on-demand vCPU quota there: (preflight prints it)
account setup: dedicated account? part of an AWS Organization?
known SCPs or spend limit: yes / no / unknown
apply identity: what it is, and whether it has admin in this account
hostname for Marshal: e.g. marshal.acme.com
DNS zone owner: Route53 in this account / Route53 elsewhere / another provider
certificate: existing ACM ARN / want the installer to request one
database: installer creates RDS / existing Postgres 16
cluster access: installer bastion / your own VPN / public endpoint + CIDRs
preflight output: attachedChecklist
- Preflight run, with no failures
- Apply identity confirmed, and checked against any organization SCP
- vCPU quota of 72, or an increase requested (16 is the minimum to create)
- Region offers EKS 1.36 and Postgres 16
- Hostname decided and DNS owner known
- Certificate path chosen
- Database decided
- Cluster access method decided
- Egress allowed to the four hosts
Next: install onto EKS.