Runs and deputies

Fixes for runs that fail, stop early, or end with an unexpected status. The status text below is what the run page shows.

Available when your administrator enables runs and deputies.

The run ends out of memory

You see: Out of memory

Cause: The run’s container used more memory than its compute class allows, and the system stopped it. The transcript can end in the middle of a sentence. Large test runs, linters, and parallel builds are common causes.

Fix: Run the same work with a bigger compute class, or split the task into smaller parts. An admin sets the compute classes in Settings, Compute. With the CLI, pass --compute-class <size> to marshal run, for example --compute-class large. See Compute sizes.

The run stops with no reason given

You see: It stopped without reporting why

Cause: The agent process ended before it could report a reason. A memory stop shows as Out of memory instead.

Fix: Read the last steps in the transcript to see what the agent was doing, and check the Resources tab for memory or CPU near its limit. Then start the run again, or use Continue. For heavy work, use a bigger compute class.

The run ends at its time limit

You see: It ran out of time

Cause: The run reached its time limit. A run started without a timeout gets your org’s default time budget, or 15 minutes when the org has none. With the CLI, marshal run prints the budget when the run starts, and you can set it with --timeout.

Fix: For a live run, add time or pause the clock. Pausing stops only the timeout clock, and the run keeps working. A run can never run longer than 24 hours in total, pauses included.

marshal runs timeout <run-id> --add 15m
marshal runs timeout <run-id> --pause
marshal runs timeout <run-id> --resume

On the run page, use Pause timeout or Add time. For a new run, set a longer --timeout.

In the CLI, a run that the time limit ended shows this line, with the budget the run had in brackets:

timed out (15m) — use --timeout, or marshal runs timeout --add

To pick the work up where it stopped, use marshal runs continue <run-id> --message "...".

A timeout change is refused

You see: One of these messages when you add time, pause or resume:

The time limit passes with a silent pod

You see: It ran out of time, and its pod never reported back or It ran out of time, and its pod was already gone

Cause: The time limit passed while the run’s pod was silent, so Marshal ended the run. In the second case the pod was already gone.

Fix: Start the run again, or use Continue on the run page when it is offered.

The run waits for an answer

You see: Needs input, or It asked a question and nobody answered

Cause: The agent asked a person a question and nobody answered in time. The run ended, but the work did not go wrong. A parked run says Nobody answered before the wait ran out.

Fix: Answer the question, then use Continue. Marshal keeps your answer and gives it to the next run. See Timeouts and parking.

The run environment could not be created

You see: Its environment could not be created

Cause: The space for the run never came up. The run page shows a longer line such as the run's environment could not be created, so the run never started, followed by the reason.

Fix: Read the reason in the Error panel. For a credential reason, see Credentials and integrations. To look inside a failed run, start it with --keep-on-failure. Marshal then leaves the space up for 24 hours after a failure.

The run environment never finishes starting

You see: Its environment never finished starting

Cause: The space stayed pending past the start limit. The run page words it as the run's environment did not finish starting within a limit, so the run never started.

Fix: Start the run again. If it repeats, an admin checks that the cluster’s space node pool has room for a new space of this compute class, or picks a smaller class.

The run succeeded but its output is empty

You see: The run shows succeeded, and its output is empty.

Cause: By default Marshal reads the run’s result from the file output.json in the workspace. If the agent did not write that file, the output is empty.

Fix: Tell the agent in the task to write its result to output.json in the workspace. A deputy can also use a different output mode, set on the deputy or for one launch. See Deputies.

The output shows a redacted marker

You see: [REDACTED:github_token], [REDACTED:marshal_secret] or another [REDACTED:...] marker in a run’s output, transcript or Slack reply, and a badge such as 1 secret was hidden on the run page.

Cause: The agent printed a credential, and Marshal replaced it before the text left the space. The marker names the kind of secret it found.

Fix: If the badge is red, the run printed a value Marshal injected: rotate that credential. To keep secrets out of the output, tell the agent in the task not to print environment variables or config files. A run that needs a key uses it through its tool or an HTTP secret, so it never has to print it. See Output redaction.

The model hits its limit or declines

You see: It reached the model’s output limit or The model declined to answer

Cause: The model stopped. In the first case it hit its limit for one reply. In the second it refused the request.

Fix: For the output limit, give the agent a smaller task. For a refusal, reword the task.

The run is refused permission

You see: It was refused permission

Cause: A request from the agent was refused by the run’s permission policy.

Fix: Open the Blocked tab on the run page to see which call was denied. Change the task so it does not need that call.

A launch stays queued

You see: Queued, and the mission does not start.

Cause: A mission has a limit on how many runs can be in progress. A launch over the limit waits. The API says launch queued at position N: the mission is at its concurrency limit. The oldest waiting launch starts first. If the queue is also full, the launch is refused with this mission is at its concurrency limit and its queue is full. Try again after some runs finish.

A launch also waits when your install is at the concurrency limit in its license. The Runs page row and the deputy’s Waiting to start card then show waiting: install concurrency limit. Launches start in order as slots free up. See Licensing.

Fix: Wait, or cancel the waiting launch. In the CLI, a waiting launch has an id that starts with ql_. marshal deputies launch prints it when the launch is queued, and marshal runs list shows it. Cancel it with marshal runs cancel <id>. To start the run as soon as a slot opens and watch it, launch with --follow.

Messaging a deputy run is refused

You see: marshal deputies message is refused.

Cause: The message goes to a run that is still in progress. If the run has finished, Marshal refuses the message. The argument is also a run id, not a deputy name.

Fix: Use a run id from marshal runs list. For a finished run, start a new run or use Continue.

Continue is not available

You see: The Continue button reads Can’t continue, with one of these explanations:

Related

• Runs
• Deputies
• Run page
• Schedules and triggers
• CLI reference