Troubleshooting

Fixes for common install and cluster problems. Each entry starts with what you see, then gives the cause and the fix. For problems that users see in the web app or the CLI, see Help and fixes.

Registry and images

`helm registry login` fails with “invalid pull token”

You see:


Error: authenticating to "registry.marshal.codes": response status code 401:
unauthorized: invalid pull token

Cause: The registry accepts only pull tokens that Marshal issued for your deployment. The value is wrong, cut short, or revoked.

Fix: Confirm the value starts with mrpt_ and contains exactly two dots. If it does, request a new token from Marshal. One token covers the chart, the product images and the tool images, so if one of those pulls works, the token is fine.

`terraform plan` fails at `provider "helm"` with “unauthorized”

You see:


Error: could not login to OCI registry "registry.marshal.codes": unauthorized
  with provider["registry.terraform.io/hashicorp/helm"],
  on providers.tf line 59, in provider "helm"

Cause: The Helm provider logs in to the registry when it starts, during plan, before Terraform reads any resource. deployment_id or registry_pull_token in your tfvars is wrong, empty or expired. The login runs even when the chart or images come from a local file or your own cache.

Fix: Check the pair outside Terraform, then run plan again.


read -rs MRPT            # paste the registry_pull_token, then Enter. Nothing is echoed.
printf '%s' "$MRPT" | helm registry login registry.marshal.codes \
  --username <deployment_id> --password-stdin

An ECR pull-through cache rule for the Marshal registry is refused

You see: ECR refuses the upstream registry URL when you create the pull-through cache rule.

Cause: ECR pull-through cache works with a fixed list of public upstream registries, such as Docker Hub, Quay and the GitHub and GitLab container registries. It checks the upstream URL against that list.

Fix: On EKS, nodes pull from the Marshal registry directly. That is the default and needs no setup. For a shared cache, install with create_harbor_cache = true (together with deploy_cluster = true), which runs a Harbor proxy cache in front of the registry. See Install on EKS. On GCP, an Artifact Registry remote repository mirrors the Marshal registry directly.

Cluster add-ons on EKS

Bundled Redis stays `Pending` because its volume never binds

You see: The Redis pod stays Pending and its PersistentVolumeClaim stays unbound.

Cause: EKS does not turn on the EBS CSI driver by default, so nothing creates the volume. GKE ships its disk driver turned on.

Fix: Install the EBS CSI driver as an EKS add-on before you install Marshal, or install with deploy_cluster = true, which installs it for you.

The Ingress never gets an address

You see: terraform apply or argocd app sync succeeds, but kubectl get ingress shows no address.

Cause: The AWS Load Balancer Controller is not installed, so nothing creates a load balancer for the Ingress.

Fix: Install the AWS Load Balancer Controller before you install Marshal on an existing cluster, or install with deploy_cluster = true, which installs it with its own IAM role.

Ingress and TLS

The browser shows a certificate warning

You see: A certificate warning on every page of your Marshal address.

Cause: The default TLS mode serves a self-signed placeholder certificate, so HTTPS works from the first minute. Its subject marks it as a placeholder.

Fix: Switch to a certificate you provide inline, or to a Kubernetes Secret you manage, for example with cert-manager. Nothing else changes. See Ingress and TLS.

The wildcard preview certificate never issues

You see: The certificate for *.<preview_domain> stays pending.

Cause: One of two things:

A GKE-managed certificate was used. Google-managed certificates do not issue wildcards.
A cert-manager ClusterIssuer uses the HTTP-01 solver. HTTP-01 proves one exact hostname per challenge, so it cannot issue a wildcard. Wildcards need DNS-01.

Fix: Use cert-manager with a DNS-01 solver for the preview domain’s zone, for example your cloud DNS provider. The default placeholder certificate already covers the wildcard, so this matters only when you replace it with a trusted certificate.

Secrets

Every user is signed out and stored credentials stop decrypting

You see: After a sync, all users are signed out, API tokens stop working, and stored credentials fail to decrypt.

Cause: A GitOps-managed install (for example ArgoCD) uses secrets.provider: generated. ArgoCD renders the manifests itself and never runs helm install, so the chart cannot reuse the secret it generated before. Each sync creates a new signing key and a new encryption key.

Fix: Set secrets.provider: existing or externalSecrets, and create a stable secret once, outside ArgoCD. Every GitOps install uses one of these two providers. Users then add again any credentials that were stored under the old key. See Secrets management.

Node pools and quotas

A control-plane node stays `Ready,SchedulingDisabled` during a pool change

You see: While Terraform resizes or replaces a single-node control-plane pool, the node is cordoned and drained, then nothing moves.

Cause: The chart keeps at least one replica of api, gateway, web and marketplace running at all times, through PodDisruptionBudgets with minAvailable: 1. With one node, the drain has nowhere to move the last replica of each.

Fix: Scale the control-plane pool to at least two nodes before you change it. The drain then moves pods to the second node and the change completes.

Pods stay `Pending` with “Insufficient cpu”

You see: Control-plane pods stay Pending, and kubectl describe pod shows Insufficient cpu.

Cause: The control-plane node pool’s machine type is too small. The chart runs two replicas of each core control-plane component, and their CPU requests add up to several vCPU before any autoscaling.

Fix: Use the recommended default machine type for the control-plane pool. Size the space node pool on its own; it can use smaller nodes.

Spaces fail to start with “too many open files”

You see: A space’s bootstrap log ends with failed to create fsnotify watcher: too many open files, often once a few spaces run on the same node.

Cause: The node’s inotify limits are used up. They apply per node and every container on the node shares them. The kernel default (128 instances) is small for coding agents and the language servers they start.

Fix: The chart’s node-tuner DaemonSet raises these limits on every node and is on by default. Check that it runs:


kubectl get ds -n <namespace> -l app.kubernetes.io/component=node-tuner

Its init container logs what it changed. If you turned it off, raise fs.inotify.max_user_instances and fs.inotify.max_user_watches in your own node configuration. See Node tuning.

Node pool scaling or a rolling update stops with no error

You see: New nodes never join, or a rollout waits, and kubectl describe shows no clear reason.

Cause: A regional CPU or disk quota in your cloud project is used up.

Fix: Check the project’s compute and disk quotas against what your node pools need: machine_type × max_node_count × disk_size_gb. Raise the quota, or size the pools to fit it. When disk quota is the limit, cap the node count rather than adding larger machines.