Skip to Content
Installing MarshalAWS: before you install

AWS: before you install

Settle these AWS items a week before you install, and the install itself is one terraform apply. This page adds the AWS layer on top of Prerequisites. Every item is quick to settle by email ahead of time, and two of them (a quota increase and an organization policy change) can take days because other teams own them.

Run the preflight script first

MRPT_FILE=$(umask 077; mktemp) # a new file only you can read read -rs MRPT; printf '%s' "$MRPT" > "$MRPT_FILE"; unset MRPT # paste the pull token scripts/preflight-aws.sh --region eu-west-1 \ --vpc-id vpc-0123 --private-subnets subnet-a,subnet-b \ --ingress-host marshal.acme.com \ --deployment-id dep_ab12cd --pull-token-file "$MRPT_FILE" --chart-version <chart_version> rm -f "$MRPT_FILE"

It is read-only, needs no write permission, and reports every problem in one pass. It answers most of this page automatically: region capability, quota, subnet and DNS facts, and whether your credentials work. Run it again on the day of the install, because quotas get consumed and subnets get re-tagged in between.

What it cannot answer is the one in the next section.

Permissions for the identity that runs the apply

Nothing read-only can prove the apply will be allowed, so this is the item worth checking by hand. The EKS installer creates resources in these services:

ServiceWhat for
eksThe cluster, two managed node groups, addons, access entries
ec2VPC, subnets, route tables, internet and NAT gateways, an Elastic IP, security groups, launch templates, the bastion instance
iamRoles and policies for the cluster, the nodes, the addons and the load balancer controller, plus an OIDC provider for IRSA
rdsThe Postgres instance and its subnet group, only with create_rds = true and deploy_cluster = true
acmImporting or requesting the load balancer certificate
route53Validation records, only when you let Terraform validate a requested certificate
s3The optional audit and space storage buckets, and the Harbor cache bucket if you enable it
sts, ssmReading the account id, and the optional bastion’s Systems Manager connection

Recommended approach. Run the first install with an identity that has administrative access in the account you are installing into, ideally a dedicated account. Then narrow the permissions afterwards using CloudTrail, which gives you a policy derived from the calls your install actually made rather than a generic one.

If your organization cannot grant that even temporarily, raise it with your Marshal contact before the install date so the required actions can be worked through with your security team.

Check for organization policies above the account

An SCP attached to your AWS Organization is a ceiling over the whole account. It beats IAM, it beats an administrator role, and it beats the root user, so an identity that looks fully privileged can still be refused, for example on iam:CreateOpenIDConnectProvider, with no sign of it in any IAM policy.

Confirm with whoever owns your organization that none of these is denied in your account and region:

  • eks:CreateCluster
  • ec2:RunInstances
  • rds:CreateDBInstance
  • iam:CreateOpenIDConnectProvider
  • elasticloadbalancing:CreateLoadBalancer

Two related cases, both of which produce the same confusing AccessDenied:

  • A region restriction. Some organizations pin an account to one region. The installer creates everything in a single region, so this is fine as long as it is your region, and fatal if it is not.
  • A spend limit. On accounts that have one, going over it denies the expensive calls above, and operations that worked before start to fail.

Quotas

The default node size is m5.2xlarge (8 vCPU) for both the space nodes and the platform node group. With the default scaling limits that is:

vCPU
At create (1 space node, 1 platform node)16
With the optional bastion18
Fully scaled out (5 space nodes, 2 platform nodes, bastion)58
One extra node while a node group updates+8

So ask for an on-demand standard instances quota of 72 vCPU in your region. 16 is the floor: the cluster cannot be created below it. Between 16 and 72 the install works, and the quota becomes the limit on how many spaces can run at once. A brand new AWS account can start as low as 5. The quota is shared with everything else running in the account and region, so the preflight prints both the quota and the vCPUs already in use, and checks the free headroom against these numbers. Raising it is a support request that can take from minutes to days, so ask early.

The NAT gateway needs one free Elastic IP. The preflight counts the addresses already allocated against your Elastic IP quota when the installer builds the network.

Network

By default the installer builds its own VPC, with public and private subnets, one NAT gateway and the subnet tags the load balancer controller needs. Nothing on this page applies to that path beyond the quota above.

You can also install into a VPC you already run, by setting create_network = false and naming the VPC and subnets. The cluster, both node groups, the bastion and RDS are still created for you. See Bring your own VPC for the inputs.

When you bring your own subnets, each of these is required. The preflight checks all of them. Two are enforced again by the installer, which stops the plan rather than failing partway through an apply: that the private subnets cover at least two availability zones, and that every subnet belongs to the VPC you named.

RequirementWhy
Subnets in at least two availability zonesEKS refuses to create a cluster otherwise. This holds even when you pin the nodes to one AZ for cost.
A default route (0.0.0.0/0) out of every private subnetWithout NAT, a transit gateway path or a proxy, nodes cannot pull images and the install dies partway through.
Private subnets that are genuinely privateA subnet whose default route is an internet gateway is public. Handing over public subnets while believing they are private is the single most common mix-up.
kubernetes.io/role/internal-elb tagged on the private subnetsWithout it the Ingress never gets an address and emits no events at all, which is slow to diagnose. The installer can add the tag for you unless your own Terraform owns those subnets.
enableDnsSupport and enableDnsHostnames on the VPCRequired by EKS. The installer cannot set them for you.
Free IP addresses, a lot of themThe VPC CNI gives every pod a real subnet IP, so free addresses set the ceiling on concurrent spaces. Below about 60 free, ask for a bigger subnet before starting.

Reaching the cluster

The EKS endpoint is private by default (cluster_endpoint_public_access = false), which means whoever runs terraform apply has to be inside the network. Decide which of these you are doing:

  • Your own VPN or zero-trust access (Twingate, Tailscale, Direct Connect, a jump host you already run). This is the default (create_bastion = false) and the easiest to keep for upgrades.
  • A public endpoint locked to your own addresses, by setting cluster_endpoint_public_access = true with an explicit CIDR list. The quickest to set up, and the least isolated of the three.
  • A bastion the installer creates, with create_bastion = true. No public IP, no inbound ports open, reached only through aws ssm start-session. Terraform, kubectl and helm run from it. This needs Systems Manager reachable from the subnet, through egress or through the SSM VPC endpoints.

See Reaching a private cluster for each option.

Hostname, DNS and certificate

  • Decide the hostname now, for example marshal.acme.com. It is an input to the very first apply, before any load balancer exists.
  • Say who owns the DNS zone. If it is in Route53 in the same account, the certificate can be requested and validated in one apply. If it is anywhere else, the certificate takes two applies with you adding a CNAME in between. See TLS certificate.
  • An ALB can only terminate TLS with an ACM certificate in the same region. It cannot read a Kubernetes TLS Secret, so cert-manager and your own PEM files do not apply here. Either bring an ACM certificate ARN, or let the installer request one.
  • You point DNS at the load balancer after the first apply, since that is when its address exists.

Database

Two options, to decide before the install:

  • Let the installer create RDS with create_rds = true. This works when the installer also builds the cluster (deploy_cluster = true). You get a small single-AZ Postgres 16 instance with deletion protection on. See RDS options.
  • Bring your own Postgres 16, reachable from the cluster over private networking.

Requirements for a Postgres you bring yourself

  • Postgres 16.x. Not 15, not 17.
  • Reachable from the cluster’s private subnets, with sslmode=require.
  • The pgcrypto extension available, since the credential store is envelope-encrypted.
  • Password authentication.
  • A dedicated database and an owner-level user on it.
  • Your backups are yours. The installer does not configure backups for a database it did not create.
postgres://<user>:<password>@<private-host>:5432/<database>?sslmode=require

Egress

Outbound HTTPS to four hosts, with no inbound access into your environment ever required: registry.marshal.codes, keys.marshal.codes, tokens.marshal.codes and marketplace.marshal.codes. The full table, including what each one is for and the optional fifth, is in Networking and Egress.

Node egress is the one thing you cannot test before a node exists. Check it from a node right after the cluster comes up, not from your laptop:

aws ssm send-command --document-name AWS-RunShellScript \ --targets Key=instanceids,Values=<node-instance-id> \ --parameters 'commands=["curl -sS -o /dev/null -w %{http_code} https://registry.marshal.codes/v2/"]'

Optional: per-space compute cost

Marshal can price each space and run from your own AWS bill. It needs a read-only role with one permission, ce:GetCostAndUsage, and the aws:eks:cluster-name cost allocation tag activated in Billing, which AWS takes up to 24 hours to apply. Activating the tag early means the numbers are ready on day one. It also needs outbound HTTPS to ce.us-east-1.amazonaws.com. See Billed compute cost.

What to send us

Fill this in and send it back, and your Marshal contact can confirm whether the install is ready to book:

region: e.g. eu-west-1 on-demand vCPU quota there: (preflight prints it) account setup: dedicated account? part of an AWS Organization? known SCPs or spend limit: yes / no / unknown apply identity: what it is, and whether it has admin in this account hostname for Marshal: e.g. marshal.acme.com DNS zone owner: Route53 in this account / Route53 elsewhere / another provider certificate: existing ACM ARN / want the installer to request one database: installer creates RDS / existing Postgres 16 cluster access: installer bastion / your own VPN / public endpoint + CIDRs preflight output: attached

Checklist

  • Preflight run, with no failures
  • Apply identity confirmed, and checked against any organization SCP
  • vCPU quota of 72, or an increase requested (16 is the minimum to create)
  • Region offers EKS 1.36 and Postgres 16
  • Hostname decided and DNS owner known
  • Certificate path chosen
  • Database decided
  • Cluster access method decided
  • Egress allowed to the four hosts

Next: install onto EKS.