Billed compute cost

See what a task really costs. Every space and every run shows its compute cost, CPU, memory and disk, next to its LLM cost, with a combined total. Connect your cloud bill, and the Costs page and every space are priced from what your company actually pays for compute on AWS or Google Cloud.

Compute cost per space works on every install, from the same node and disk prices the Compute tab of Costs uses. Billed prices are available when your administrator connects AWS Cost Explorer or the Google Cloud billing export.

What you can do with it

• Answer “what did this task cost?” Open a run and read its LLM cost, its compute cost and the total on one page.
• Price from your real bill. Savings Plans, reservations, committed use discounts, sustained use discounts and spot prices are all in the numbers, with no discount to type in by hand.
• Report to finance. The Costs page, its node costs and every space’s compute cost come from the same prices, so the figures you share agree with each other and with the bill.

What you see

Open the Costs tab of a space or a run. It starts with the Total and a bar that splits it into LLM and Compute, with one line such as “Compute is 4% of this space’s cost.” Below it, the Compute card shows the amount with its CPU, Memory and Disk parts, then the LLM detail follows. Under the compute amount, a line says where the price came from:

A figure that can still change carries an estimated label: list and custom prices, a billed day the cloud has not settled yet, or a day priced from an earlier billed day.

On the Compute tab of Costs, a node pool priced from the bill carries a billed badge. In the Prices panel each machine type shows its billed price per hour next to the list price, with the days it covers, for example list 0.192 · billed 0.12, Oct 28 to Nov 3.

How it is computed

1. Node prices. Each machine type is priced in this order: a custom price an admin entered, then the billed price, then the list price. A billed price already includes your discounts, so the Discount % and Spot discount % settings do not apply to it. Spot and on-demand nodes of the same machine type are priced apart.
2. The daily rate. For each day, Marshal prices the cluster’s nodes and works out one price per vCPU-hour and one per GiB-hour of memory, weighted by each node’s size. Google Cloud bills vCPU and memory separately, and Marshal uses those two prices as they are. AWS bills a node as one amount, which Marshal splits 9 to 1 between CPU and memory, the weighting AWS uses for EKS split cost allocation.
3. The space. CPU and memory are priced by what the space requested multiplied by the time it was running, at each day’s rate. A stopped space uses no CPU or memory, and a running space is priced up to now.
4. The disk. A space’s disk is priced by its size and its disk type’s price per GB-month, from the moment the disk is created until it is removed. Time the disk sits detached while the space is stopped counts too, because the cloud bills for it. A space with no disk of its own has no disk cost.

A run is priced the same way, for its own time window. Its disk counts from the run’s start until the disk is removed or the next run on the same space starts.

Each day of a space uses that day’s billed prices. The Compute tab of Costs shows one monthly figure for the cluster, so it prices each machine type from the last 7 settled billed days, weighted by usage. Until a day settles, the newest days are used and the price is marked estimated.

Idle capacity on your nodes is not charged to any space, so the total across your spaces is lower than the compute line on your bill. The Costs page shows that difference as Idle: the headroom your cluster keeps ready for new spaces.

A space resized while running is priced at its current size for its whole life, and its figure is marked estimated.

How fresh it is

Both clouds publish a day’s cost about a day later and keep adjusting it for a while. Each time the API starts, Marshal fetches the last 14 days. After that it fetches the last 5 days every cost.billing.refreshInterval, once a day by default. One API replica refreshes at a time, and a pass is skipped when the bill was fetched less than half an interval ago, so a restart or several replicas do not add Cost Explorer or BigQuery charges.

• A billed day becomes final when Marshal fetches it at or after 00:00 UTC three days after the day started. Until then it is estimated.
• A day with no billed data yet, such as today, is priced from the newest earlier billed day, up to 14 days back, and marked estimated. Older than that, list or custom prices apply.
• If a refresh cannot reach the bill, the days already stored keep pricing, and the views keep answering. Each refresh writes a billed_cost.refresh event to the audit log with its outcome, its days and, if it failed, the reason.

Who sees billed prices

Compute cost follows the space: whoever can open a space or a run sees its cost. The Costs page is for org owners and admins.

On an install with several orgs, every org is priced from the bill and sees the billed badge. The billed price per machine type and the days it covers, in the Prices panel, are shown to owners and admins of the install’s primary org, the oldest org on the install. On an install with one org, its owners and admins see them.

Configure it

An administrator of the install sets this up once, for one cloud. The API reads the bill with its own workload identity: IRSA or EKS Pod Identity on AWS, Workload Identity on GKE. No key is stored in values, settings or the database.

Changing the source is a Helm change, so it is not in the audit log. The next billed_cost.refresh event shows the new source.

AWS

Before you start, check that:

• The cluster runs on EKS with EC2 nodes, and every node carries the aws:eks:cluster-name tag. EKS managed node groups add it for you. For self-managed nodes or Karpenter, add the tag to the nodes yourself.
• Cost Explorer is enabled in the account.
• In an AWS Organization, the tag is activated in the management (payer) account, since that account owns the cost allocation tags.

Setup has three steps.

1. Create a read-only role. It needs one permission, ce:GetCostAndUsage. Cost Explorer has no narrower resource, so the resource is *.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ce:GetCostAndUsage",
      "Resource": "*"
    }
  ]
}

With IRSA, give the role this trust policy. Replace the account id, the OIDC provider id and the namespace. The service account is the API’s, <release>-api, so marshal-api for a release named marshal.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
          "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:marshal:marshal-api"
        }
      }
    }
  ]
}

With EKS Pod Identity, give the role this trust policy instead, and associate it with the API’s service account. The cluster needs the EKS Pod Identity Agent add-on installed.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace marshal \
  --service-account marshal-api \
  --role-arn arn:aws:iam::111122223333:role/marshal-billed-cost

2. Activate the cost allocation tag. In the AWS console open Billing and Cost Management, then Cost allocation tags, and activate the AWS-generated tag aws:eks:cluster-name. AWS takes up to 24 hours before the tag reaches Cost Explorer. To use a tag of your own, activate it the same way and set cost.billing.aws.tagKey and cost.billing.aws.tagValue to match.

3. Set the Helm values.

cost:
  billing:
    source: aws
    aws:
      tagValue: my-cluster
api:
  serviceAccount:
    annotations:
      # IRSA only; leave out with Pod Identity.
      eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/marshal-billed-cost

Marshal reads the cluster’s EC2 running hours and their amortized cost per instance type and purchase type, so Savings Plans and reservations are spread the way your bill reads them. Credits and refunds are left out, because they are one-time money rather than a price.

Google Cloud

You need a GKE Standard cluster and the Cloud Billing export to BigQuery. The standard usage cost export is enough. Someone with rights on the billing account turns it on in Billing, Billing export, BigQuery export. History starts from that moment.

Before you start, check that:

• The BigQuery API and the IAM Service Account Credentials API are enabled in the job project (cost.billing.gcp.jobProject, the cluster’s project by default).

• The dataset that will hold the view is in the same location as the export dataset, for example both in US or both in EU.

• The API’s Kubernetes service account is linked to a Google service account through Workload Identity. The Kubernetes service account is <release>-api in the release namespace. Set the link in the Helm values:

  api:
    serviceAccount:
      annotations:
        iam.gke.io/gcp-service-account: <api-google-service-account>

  and let the Kubernetes service account act as it:

  gcloud iam service-accounts add-iam-policy-binding \
    <api-google-service-account> \
    --member="serviceAccount:my-gke-project.svc.id.goog[marshal/marshal-api]" \
    --role=roles/iam.workloadIdentityUser

  If your install already stores data in Cloud Storage through Workload Identity, this link is in place.

The export holds the whole billing account. The steps below give Marshal a view that holds only your cluster’s project, and a service account that can read only that view.

1. Create a view of the export for your project. Create it in a dataset of its own, and expose the partition time so queries read only recent days:

CREATE VIEW `billing-admin.marshal_billing.compute_costs` AS
SELECT *, _PARTITIONTIME AS partition_time
FROM `billing-admin.billing_export.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX`
WHERE project.id = 'my-gke-project'

Then authorize the view on the export dataset: open the export dataset in BigQuery, choose Sharing, Authorize views, and add the view. The view can now read the export, and whoever reads the view sees only your project’s rows.

2. Create a billing-only service account. Give it read access to the view and the right to run queries:

gcloud iam service-accounts create marshal-billing --project my-gke-project
 
# Read the view only (grant on the view, or on its dataset).
bq add-iam-policy-binding \
  --member=serviceAccount:marshal-billing@my-gke-project.iam.gserviceaccount.com \
  --role=roles/bigquery.dataViewer \
  billing-admin:marshal_billing.compute_costs
 
# Run query jobs in the job project.
gcloud projects add-iam-policy-binding my-gke-project \
  --member=serviceAccount:marshal-billing@my-gke-project.iam.gserviceaccount.com \
  --role=roles/bigquery.jobUser

Keep this account for billing alone. The Google service account the API already uses for storage is also used by space pods, so billing access does not belong on it, and neither does roles/billing.viewer, which covers the whole billing account.

3. Let the API act as it. Allow the API’s Google service account, the one in the iam.gke.io/gcp-service-account annotation, to create tokens for the billing account:

gcloud iam service-accounts add-iam-policy-binding \
  marshal-billing@my-gke-project.iam.gserviceaccount.com \
  --member=serviceAccount:<api-google-service-account> \
  --role=roles/iam.serviceAccountTokenCreator

4. Set the Helm values.

cost:
  billing:
    source: gcp
    gcp:
      table: billing-admin.marshal_billing.compute_costs
      partitionColumn: partition_time
      project: my-gke-project
      clusterName: my-cluster
      clusterLocation: europe-west1
      impersonate: marshal-billing@my-gke-project.iam.gserviceaccount.com

Marshal sums the Compute Engine vCPU and memory charges of the VMs labelled with your cluster’s goog-k8s-cluster-name and goog-k8s-cluster-location, per machine type and spot, so it finds the real price per vCPU-hour and per GiB-hour. Committed use discounts, sustained use discounts, negotiated discounts and subscription benefits are subtracted. Promotional and free-tier credits are not, because they are one-time money rather than a price. A bill in another currency is converted to US dollars with the exchange rate the export records for each row.

GKE Autopilot bills pods rather than node VMs, so an Autopilot cluster keeps list or custom prices.

Good to know

• Network access. Allow these from the platform nodes:
  • AWS: ce.<region>.amazonaws.com, where the region is cost.billing.aws.region (us-east-1 by default). With IRSA, also AWS STS (sts.<region>.amazonaws.com). With EKS Pod Identity, the Pod Identity Agent on the node.
  • Google Cloud: bigquery.googleapis.com, the GKE metadata server for Workload Identity, and iamcredentials.googleapis.com when the API acts as the billing service account.
• What it costs. AWS charges $0.01 per Cost Explorer request, and a daily refresh makes about one. On Google Cloud each refresh runs one query, capped by cost.billing.gcp.maxBytes, and reads only the partitions of the days it fetches.
• Nodes from the bill, disks from your prices. Billed prices cover node compute. Disks, on the Costs page and in every space’s compute cost, use their custom or list prices. The cluster management fee, managed databases, data transfer and load balancers are not part of the estimate; add them as other costs to see the whole monthly bill. They are not part of a space’s compute cost.
• Currency. Costs are shown in US dollars.

Try it

Set cost.billing.source, wait for the first refresh, then open Dashboards > Costs and the Compute tab: node pools priced from your bill carry the billed badge. Open any space and read Billed prices under its compute cost.

The audit log shows a billed_cost.refresh event a few minutes after the API starts. If none appears, check the API logs: they say why the refresh could not start, for example a missing required value.

Related

• Costs
• The run page
• Compute sizes
• Install onto EKS
• Install onto GKE