How to cut alert noise so the alerts that matter get read

A team that gets fifty alerts a day reads none of them. The channel is muted, the emails are filtered into a folder, and the one alert that mattered sits between a slow-response warning and a certificate reminder for a domain that was sold last year.

Alert noise is not an annoyance. It is the way real outages get missed. The fix is fewer alerts, each of which someone acts on.

How it works

Every alert should pass one test: when it arrives, does someone do something? If the honest answer is "usually not", it is noise, whatever it is about.

That gives three kinds of signal, and each belongs somewhere different.

Page

Someone must act now.

A phone, through whoever is on call.

Notify

Someone must act today.

A team channel or email, read in working hours.

Record

Nobody has to act.

A dashboard. It goes nowhere.

Most noisy setups send all three to the same place. Sorting them is most of the work.

Step 1: Sort what you have

Take last week's alerts and put each one in a column:

• Page. Someone had to act now. An outage of something customers use.
• Notify. Someone had to act today. A certificate two weeks from expiry, a disk filling up.
• Record. Nobody had to act. A brief slow response, a single failed check that recovered.

Be strict. If nobody did anything when it fired, it is in the third column, however alarming it sounds.

Step 2: Send each kind somewhere different

• Page goes to a phone, through an on-call rotation: see how to set up an on-call rotation for a small team.
• Notify goes to a team channel or email that is read in working hours.
• Record goes nowhere. It stays on a dashboard for when someone is looking.

A channel that only ever carries things worth reading gets read.

Step 3: Confirm before firing

Many alerts are true for thirty seconds. Remove those at the source:

• Check from more than one place, and fire only when two or more agree. One bad network path is not an outage.
• Retry once before counting a failure. A single slow response is not one either.
• Hold back repeats. A service that goes down and up five times in ten minutes should produce one alert, not ten.

Step 4: Give alerts a severity, and mean it

Two or three levels are enough. What matters is that the top one is rare. If a third of alerts are "critical", the word means nothing and people stop reacting to it.

Tie severity to what happens: the top level pages at any hour, the next waits for the morning.

Step 5: Silence planned work properly

Planned work is a large source of noise, and of the bad habit of muting channels by hand. Use a maintenance window that ends by itself: see how to plan a maintenance window.

Step 6: Review, and test

Once a month, list the alerts that fired and ask of each: did someone act? Delete or demote the ones where nobody did. An alert that has fired ten times with no action is not a safety net; it is training people to ignore alerts.

Then test what is left. Make a check flap, down and up a few times in ten minutes, and count the messages. Make one fail and stay failed, and see who is told, and how fast.

Common mistakes

• Everything in one channel. The urgent and the trivial look the same.
• Muting the channel instead of fixing the rule. The noise stops, and so does everything else.
• Alerts with no owner. If nobody is responsible for acting, nobody will.
• Paging on warnings. The phone should only ring for what cannot wait.
• Keeping an alert "just in case". If it has never led to action, it will not the next time either.

Doing this in saysOK

An alert rule says which monitors it covers (all, by tag, or one by name), what condition fires it, and what to do. Three of its settings are the steps above.

Rule preview

Whena monitor tagged production goes down (confirmed from 2/3 regions).

ThenPost to Slack (#alerts) · Send SMS (On-call phone)

SeveritySEV-2 · dedupe 5m

• Confirmation fires a down alert only when a number of the regions checked agree, two of three unless you change it.
• Dedupe window, which you set between 5 and 30 minutes, holds back a new alert for a monitor that fails again within that time of its last alert, and says nothing if it has recovered again by then. A rule alerts once per outage per monitor and tells the same channels when it recovers.
• Severity runs from SEV-1 to SEV-4. A rule can be limited to weekday working hours (in UTC) or to times someone is on call, and held back during maintenance windows; SEV-1 rules ignore all three and always fire.

Test rule sends a test alert, marked as one, to the rule's channels, which is a quick way to see where each kind lands.

Alert rules are part of every plan; see pricing for the channels each plan includes.

Next

After an incident, the review usually finds an alert that was missing or one that was ignored: see how to write an incident postmortem.