Join the waitlist →

Sentry Integration

Connecting Sentry lets SetForth check your production errors as part of an agent's work. Once a project is connected, agents can attach Sentry checks to a task — automated follow-ups that run after the task's change ships and confirm the change did what it was supposed to do in production.

This guide walks through the whole setup: copying your project's webhook URL from SetForth, creating an Internal Integration in Sentry, granting it the right read permissions, and connecting it in SetForth. It takes a few minutes, runs in a single pass — copy one value out of SetForth, do all of the Sentry configuration in one visit, then come back to SetForth once to connect — and only needs to be done once per project.

What the Sentry integration enables

With Sentry connected, three validation skills become available when you plan a task. Each one is a production follow-up check: it runs server-side after the change deploys, reads Sentry, and passes or fails on real error data rather than a guess.

  • sentry-no-new-errors — asserts that a change introduced no new errors after it shipped. It looks at the issues first seen in an observation window after the deploy and passes when none are attributable to the change (or the count stays at or below a threshold you set).
  • sentry-error-reduction — asserts that a change actually reduced a specific known error. It counts that issue's events before and after the deploy and passes when the drop meets or exceeds the amount you required, either as an absolute count or as a percentage of the baseline.
  • sentry-no-regression — asserts that a previously resolved error did not come back. It reads the issue's status and event volume and passes only while the issue stays resolved with no new events.

These checks show up during planning whenever a Sentry connection exists for the project, so an agent can pick the right one for the work it is doing. Each check resolves on a normal polling schedule; connecting the webhook (described below) simply makes them resolve faster.

Before you begin

Sentry connects per project, from that project's own settings — not from the org-level Settings page. In SetForth, open the project, go to its Settings tab, and find the Integrations section. The Sentry card lives there.

You need to be a project admin. A project's Settings tab only renders for project admins. If the project has no Settings tab for you, ask a project admin to run this setup or to grant you admin access.

Setup runs in a single pass: you copy one value out of SetForth first, do all of the Sentry configuration in one visit, then return to SetForth once to connect. No step sends you back to Sentry after you connect.

Copying the Webhook URL from SetForth

Start in SetForth. Open Project → Settings → Integrations and find the Sentry card. It shows a per-connection Webhook URL — unique to this project and available before you connect. Copy it now; you will paste it into Sentry while setting up the Internal Integration in the next steps.

Creating an Internal Integration in Sentry

Sentry's Internal Integration is the credential SetForth uses to read your issues. Create one from your Sentry organization's developer settings.

Go straight to:

https://<org-slug>.sentry.io/settings/developer-settings/

Replace <org-slug> with your Sentry organization slug. Use the direct URL rather than hunting through the sidebar: the placement of developer settings moves between Sentry UI versions, and the direct URL lands you in the right place regardless.

On that page, choose New Internal Integration. Give it a recognizable name (for example, SetForth) so you can find it later. Leave the other fields on the details screen — Webhook Headers, Alert Action, Schema, Overview, Authorized JavaScript Origins — at their defaults; SetForth does not need them.

Permissions

An Internal Integration starts with No Access on every resource. SetForth only reads from Sentry, so grant the minimum read scopes it needs and leave everything else at No Access.

Set these three:

  • Issue & Event → Read — the core scope. It lets SetForth list issues and read their status and event counts, which is what every check relies on.
  • Project → Read — lets SetForth resolve your project and confirm the connection points at the right one.
  • Organization → Read — lets SetForth read organization-level metadata and resolve releases.

Leave everything else — Team, Distribution, Member, Alerts — at No Access.

Leave Release at No Access. Sentry only offers No Access or Admin for the Release resource; there is no read-only option. SetForth does not need write access to your releases, so do not grant Admin. The checks work without any Release scope.

Webhooks

Webhooks are optional but recommended: they let Sentry notify SetForth the moment an issue changes, so checks resolve promptly instead of waiting for the next poll.

In the Internal Integration's Webhooks section, under Issues, check:

  • Created
  • Resolved
  • Unresolved

Leave the per-event boxes under Errors (and everything under Comments and the other resource groups) unchecked — SetForth only processes issue-level events, so subscribing to raw error events would add noise without adding value.

Note: the webhook checkboxes stay greyed out until you set Issue & Event to Read in the Permissions section. If you cannot tick them, go back and grant that scope first, and they become selectable.

Webhook URL and Client Secret

The webhook needs two values: the Webhook URL you already copied from SetForth, and the Internal Integration's Client Secret.

  1. In Sentry, paste the Webhook URL you copied from SetForth into the Internal Integration's Webhook URL field.
  2. Copy the Internal Integration's Client Secret (shown on the integration's overview). You will paste it into SetForth's connection form in the final step, alongside the auth token — there is no need to come back to Sentry afterward.

SetForth uses the Client Secret to verify that incoming webhook deliveries really came from your Sentry integration. Once you connect with the secret, the connection reports that the webhook secret is configured. The webhook is optional — if you leave the secret blank when you connect, SetForth simply falls back to polling — but wiring it up now keeps setup to a single pass.

Minting the auth token

The auth token is what SetForth uses to authenticate its read requests to Sentry.

On the Internal Integration's page, choose New Token. Sentry generates a token (it starts with sntrys_). Copy it now — you will paste it into SetForth in the next step. Treat it like a password: it grants the read access you configured above.

Connecting in SetForth

Back in SetForth, return to the Sentry card in Project → Settings → Integrations and fill in the connection form:

  • Auth Token — the sntrys_… token you just minted.
  • Organization Slug — your Sentry organization slug (the <org-slug> from the URL above).
  • Project Slug — the slug of the Sentry project you want checks to read.
  • Webhook Client Secret — the Client Secret you copied from the Internal Integration. Leave it blank only if you chose to skip the webhook.

Choose Connect. SetForth validates the token and slugs immediately and shows the connection's status:

  • Connected — the token, organization slug, and project slug all checked out. The integration is ready and Sentry checks will be available in planning.
  • Error — the token was rejected — the auth token is wrong, was truncated when copied, or lacks Issue & Event read access. Re-check the token and its permissions, then update the credentials and try again.
  • Error — we couldn't find that project — the organization or project slug is wrong. Slugs are the short URL identifiers, not display names; confirm both against Sentry and re-enter them.

You can re-run the check any time with Test Connection, change values with Update Credentials, or remove the connection with Disconnect.

Verifying the setup and troubleshooting

Confirm the integration end to end:

  1. The connection shows Connected. On the Sentry card in Project → Settings → Integrations, the status badge reads Connected. If it shows an error, use the status descriptions above to fix the token or the slugs.
  2. Sentry checks appear in planning. Plan a task and confirm the Sentry validation skills (sentry-no-new-errors, sentry-error-reduction, sentry-no-regression) are offered. They only appear when a Sentry connection exists for the project, so their presence confirms the connection is wired up.
  3. A webhook delivery is accepted (only if you set up the webhook). Trigger an issue event in Sentry — for example, resolve or unresolve an issue — and open the Internal Integration's delivery log in Sentry's dashboard. A correctly configured delivery is accepted (HTTP 200). A rejected delivery (401) means the Webhook URL or the Client Secret does not match: re-check that the URL is the one SetForth shows for this connection and that the Webhook Client Secret in SetForth equals the integration's Client Secret.

A few common snags:

  • Webhook checkboxes are greyed out. You have not set Issue & Event to Read yet. Grant that permission and the Issues webhook boxes become selectable.
  • Checks never seem to run. The webhook only speeds things up; it is not required. If polling is your only path, checks still resolve — they just take until the next scheduled poll. Verify the connection is Connected first.
  • "Project not found" even though the project exists. You are almost certainly entering a display name instead of a slug. Use the slug from the Sentry project's URL for both the organization and the project.