An AI on-call for a single server: what I would hand to Grafana AI and what I check myself

In one week Grafana released agents that watch a service and investigate incidents, plus a sandbox where an agent prepares a code fix. The CNCF measured how much faster an agent finds the cause when you give it context. Percona published a tool that answers the question “so what happened on Thursday.” For a business owner it is all one question: who is on call at your server at night, and what they are allowed to do without you.

What came out

Grafana Labs on October 20 announced three parts of Grafana Assistant. Watchers are standing agents: you describe in words what worries you about the service, the Assistant picks a baseline from the telemetry history and then checks the signals periodically. If it finds something critical, it starts Investigations on its own, an agent that investigates through metrics and logs. Managed Sandboxes, in public preview for now: the agent gets the repository code and live telemetry in a separate cloud environment and prepares a pull request describing what changed, why, and which checks passed. Grafana stresses that confirming the recovery in production and deciding what to roll out stays with the team.

The CNCF on October 21 published a measurement I liked more than any of the announcements. The same agent, Claude Code on the same model, went through 50 broken Kubernetes clusters from the open SREGym benchmark twice. The first time with a regular shell and kubectl, the second with read-only tools that immediately return the list of current failures, the change history and the error logs. Accuracy barely moved: 45 correct diagnoses against 46. What changed was time: 191 seconds on average against 64, and a correct answer within two minutes in 60% of cases against 86%. The authors write that the tools are their own and that both variants missed the same three failures.

Grafana Labs on October 14 described the path of an incident in Grafana Cloud IRM: the alert page shows the service and its firing history, from Slack you can ask the Assistant what else is on fire, and the investigation in the Assistant Workspace can be edited by hand: add a hypothesis, mark one as disproved, look at the sources.

Percona on October 16 showed mongo-health, an open tool under Apache 2.0. Every mongod writes its metrics to diagnostic.data once a second, but on an Atlas M10 under production load those files live two to five days. mongo-health keeps them for three months, runs 68 checks once an hour and gives one verdict for the cluster. From the examples: a query without an index read about 10 million documents for every one it returned. Percona itself writes that this is no replacement for monitoring: it reads once an hour and wakes nobody up.

Grafana Labs on October 19 reported how it improved the Assistant itself: over roughly 10 weeks, the error rate in dashboard mode fell from 2.47% to 0.65%, 95th percentile latency from 63 to 28.8 seconds, and the cost per request by 18%.

Why a one-server business should care

Take a car rental company. On Thursday evening the booking site was slow, a couple of customers gave up waiting, and the manager noticed only on Monday from a complaint. The alert didn't fire because the site was responding, even if it took ten seconds. The server has already overwritten Thursday's logs, and the load graph shows a flat line.

The conclusion is the same: keep history longer than you expect to look at it, and the first question will be “what changed right before this.” An investigating agent needs the same records you do, and without them it won't find anything either.

How I do it

At the PDR company in the US I have been the technical partner since 2021, and I monitor the infrastructure myself: a server on Debian in Google Cloud, MariaDB, VPN, telephony on our own PBX. Since 2022, when we got our own server, there have been no serious failures that needed a recovery.

At the hydroponic farm I have worked with since 2023, one network holds 25 controllers across 2 sites, a server on Debian in Google Cloud and a MariaDB database. There is no active development on my side right now, but support is fairly active: updates, debugging, working through problems as they come up. This is where I could use an on-call that gathers context at night.

AI helps me with testing and debugging, and I check the result. I would accept an automatic “proposed fix” in the spirit of Managed Sandboxes only with evidence: which metrics point to it and what changed before the failure. And with a veto: the change reaches the server through the same order of checks as any other.

What I would do

Keep history longer than the defaults. Metrics and logs for several weeks.

Keep a change log: when what was rolled out and which config changed. In the CNCF measurement, change history was the agent's second request on 35 of the 50 failures. Without it, both a person and an agent spend time on reconstruction.

Start with one service and read-only access. Give the agent the metrics, logs and change log, ask it to find the cause of a past incident, and compare with what you found.

Accept a proposed fix as a pull request with evidence, and roll it out by hand through the usual checks.

Check that backups restore. An on-call that can only diagnose won't replace a recovery.

Related materials

I wrote about the server as the center of a system in the article The server: data, access and process execution. Direction: infrastructure you can ship and restore. Cases: technical partner to a PDR company since 2021 and a hydroponic farm with 25 controllers. From other reviews: how much observability a one-server system needs, the backup nobody ever restored and what I allow an AI agent on a production server. Earlier I wrote about observability graphs and testable hypotheses. If you have a server nobody watches at night, tell me what runs on it and how you learn about problems today. Discuss your task.