Fix Production Errors While You Sleep

We shipped a bad null check on a Friday in June. Sentry caught it at 11pm, some 40 users hit it over the weekend, and the person on call fixed it Monday morning in about six minutes once they actually sat down with the stack trace. The fix was never the expensive part. The expensive part was the 60 hours between the alert and a human having six minutes.

This tutorial closes that gap with an automation. Sentry keeps catching the errors. A Hoplite agent picks them up on a schedule, reproduces the failure in a sandbox, writes the fix, and opens a pull request that links back to the issue. You still review everything. You just do it with the work already done.

Before you start

• A Hoplite workspace with your repository connected and setup working (if threads run green, you're fine)
• A Sentry project receiving errors from that codebase
• Permission to install a Sentry integration, which usually means being an admin on the Sentry org

Step 1: Connect Sentry

In your workspace settings, open Integrations and install the Sentry app. You'll bounce through Sentry's consent screen and land back in Hoplite. The connection is read-only. Agents can list issues, read events, and walk stack traces, and that's it. Nothing in your Sentry project gets resolved, assigned, or commented on unless a person does it.

Once it's installed, try it in a normal thread first: ask an agent to summarize your top unresolved issue. Thirty seconds of sanity checking here saves you debugging the integration through an automation later.

Step 2: Write the automation

Go to the Automations tab and create one. There are two fields. The schedule is easy: hourly is right for most teams. Tighter than that and runs start racing each other for the same issue; looser and you lose the while-you-sleep property that makes this worth doing.

The prompt deserves more thought. You are writing standing instructions for a competent stranger, so put in the judgment calls you'd otherwise deliver over Slack. Ours looks close to this:

Look at unresolved Sentry issues that first appeared in the last hour. Skip anything already linked to an open PR. Take the one affecting the most users, reproduce it with a failing test, fix it, and open a pull request that references the Sentry issue in the description. If the root cause is bad data from an external service, or the fix would touch more than a couple hundred lines, don't write code. Comment your findings on the issue instead and stop.

The last two sentences are the load-bearing ones. An automation that knows when to do nothing is one you can leave running. One that always produces a diff, whether the situation calls for it or not, gets muted within a week. Give it a way out and it will take it when it should.

Step 3: Watch the first run, then stop watching

Every automation run is an ordinary Hoplite thread. It gets its own isolated sandbox with your repo cloned and your setup applied, and you can open it and watch, the same as any thread you started by hand. The first run is worth watching. You'll see the agent query Sentry, pick an issue, trace it to a file, and write the reproduction before the fix.

That reproduction test is the thing to check in review. A fix that comes with a failing test proves the agent understood the cause; one without is a guess. If the agent skipped the test, say so in review and tighten the prompt; it takes instruction well.

The run ends at a pull request, never a merge. Agents don't get merge rights in Hoplite.

Tuning

• Split by surface. One automation for the API, one for the frontend, each with its own rules. Mixed prompts drift toward vague ones.
• Set a user threshold. 'Only issues affecting ten or more users' keeps the agent off one-off noise from a single weird client.
• Route the result to where triage happens. Have the PR description mention your on-call channel so fixes show up next to the alerts they answer.
• Watch the first week of output before trusting it. Ours needed two prompt revisions before the judgment matched the team's.

Errors are the obvious queue for this because the payoff is so legible, but the same shape works for anything that accumulates: flaky tests, dependency bumps, lint debt, TODOs past their expiry date. If you'd assign it to a junior engineer with a checklist, it schedules.