Skip to Content
Use casesTriage alerts from your tools

Triage alerts from your tools

An alert fires at 03:00, and the first half hour goes to the same questions: which service, what changed, is it real. Let a deputy answer them before anyone opens a laptop. Every critical alert starts an investigation, and the on-call engineer gets a diagnosis, the suspect change and a proposed next step.

Available when your administrator enables deputies and webhooks.

The result

  • A head start on every alert. The investigation is underway seconds after the alert arrives, and the findings are waiting when a person looks.
  • Evidence, not guesses. The run reads the code and the recent merged pull requests, and points at files, lines and commits.
  • One investigation per alert. Your alert pipeline can retry as often as it likes. A repeated delivery with the same id starts the mission once.
  • A person decides on changes. The deputy opens an issue with its findings, and asks before it proposes a fix.

How it works

Your alert pipeline posts each alert to a Marshal webhook endpoint named alerts, signed with the endpoint’s secret and the event name alert.fired. Marshal checks the signature and records the delivery. The event trigger on the Alert Triage deputy matches it, checks the condition event.severity == "critical", and maps the alert’s title, service and link into the mission’s inputs. The run starts in a fresh space with your code. It reads the alert, searches acme/web for the code behind the failing service, checks the pull requests merged in the last day, and writes up the cause it finds with the evidence. It opens a GitHub issue with the findings, then asks the on-call engineer whether to prepare a fix. The answer arrives from Home or a Slack button, and the run carries on.

What you use

PieceIts role in this story
Webhook endpointalerts: a delivery URL and a signing secret for your alert pipeline.
Event triggerStarts the mission on alert.fired for critical alerts only.
Input mapPasses the alert’s title, service and link into the prompt.
DeputyAlert Triage, owned by the platform team, with its own manifest and memory.
Manifestclaude-code, github and your toolchain, with a team GitHub token.
RunOne investigation per alert, with a transcript and an audit trail.
Home and notificationsWhere the on-call engineer answers the deputy’s question, on the web or in Slack.

Build it

  1. Create the endpoint. On Personal settings > Webhooks, select Add endpoint, name it alerts, set the Owner to a team you lead, and choose a Signature format. Copy the Delivery URL and the Signing secret. See Create a webhook endpoint.
  2. Point your alerts at it. A sender that signs with a supported format posts straight to the URL. Otherwise, add a short step to your alert pipeline that signs each alert with Marshal’s format and sends X-Event-Type: alert.fired. See Sign a request.
  3. Create the manifest and the deputy. Add claude-code, github and your toolchain, with your Claude credential and a team GitHub token. Then create the “Alert Triage” deputy for your team. See Create a deputy and a mission.
  4. Write the mission. Create triage-alert with the prompt below. In the Inputs panel, make alert, service and link optional, each with a default such as unknown. The trigger then fills them from the event. Select Dry run, then Activate.
  5. Add the trigger. Open the mission, select Event under Triggers, set Source to Custom webhook, Endpoint to alerts and Event to alert.fired. In When, enter event.severity == "critical" and select Test against real events. Then select Create.
  6. Map the alert into the inputs. With an API key, list the deputy’s triggers with GET /api/deputies/alert-triage/triggers, then add the map to the new one. See Map event data into mission inputs.
  7. Send a test alert. A run appears on the deputy page with the trigger webhook and the event alert.fired.

An example alert body:

{ "id": "al_4821", "severity": "critical", "service": "checkout", "title": "Checkout error rate above 5%", "link": "https://alerts.acme.dev/al_4821" }

The input map for step 6:

curl -X PATCH https://marshal.example.com/api/deputies/alert-triage/triggers/<trigger-id> \ --header "Authorization: Bearer $MARSHAL_API_KEY" \ --header "Content-Type: application/json" \ --data '{"inputs": {"alert": {"path": "title"}, "service": {"path": "service"}, "link": {"path": "link", "default": "none"}}}'

The mission prompt:

A critical alert fired: "{{.alert}}" on the {{.service}} service. Alert link: {{.link}} 1. Clone acme/web. Find the code that serves {{.service}}. 2. List the pull requests merged in the last 24 hours with gh, and read the ones that touch that code. 3. Write the cause you find, the evidence (files, lines, commits) and a suggested fix. 4. Open a GitHub issue titled "Alert: {{.alert}}" with your findings. 5. Ask me with the choices "Prepare a fix" and "Leave it to on-call". If I pick "Prepare a fix", push a branch and open a pull request. Never merge.

Make it yours

  • Widen the net. Use the condition event.severity in ["critical", "high"], or bind to alert.* to take every alert event.
  • Read your monitoring data. On a personal deputy, add a monitoring package such as sentry-mcp, datadog-mcp or posthog-mcp, so the run reads stack traces, logs and traces too. Their sign-in acts as you. See Integrations.
  • Choose who is asked. A mission can name the people it asks, so the question goes to the on-call group. See Who is asked.
  • Poll instead of push. Swap the trigger for a schedule every 15 minutes, and have the run read new alerts through a monitoring package. See More recipes.

Next

Create a webhook endpoint for your alerts.

Related