How to plan a maintenance window without a night of false alarms

Planned work goes wrong in two ways that have nothing to do with the work. The first is noise: the database is restarted on schedule, forty alerts fire, the on-call phone rings, and customers email to say the site is down. The second is worse. To avoid the first, someone mutes everything, forgets to unmute it, and a real outage the next morning goes unseen.

A maintenance window is the agreement that avoids both: for this time, for these things, failures are expected, and afterwards everything is back to normal.

How it works

Three things happen around planned work.

• Customers are told, before it starts and when it ends.
• Expected alerts are held back, for the parts being worked on and only for the time of the work.
• Unexpected alerts still arrive, because planned work is exactly when something else breaks.

Step 1: Decide what, when and how long

• What will be affected. Name the services. "The database" becomes "sign-in and checkout will be unavailable; the website stays up".
• When. The hour with the least traffic for your customers, which is not always the middle of your night.
• How long. Your estimate, plus the time to roll back if it fails. A window that ends early is fine. One that overruns looks like an outage.
• The point of no return. The time by which you either finish or roll back, decided beforehand, not at 3 a.m.

Step 2: Tell customers before, not during

Say when, for how long, and what they will notice, in their words. Put it where they look: the status page, and an email to the people who subscribed to it.

A few days ahead is right for work that takes the product away. An hour is enough for something they will not notice.

Step 3: Silence only what you expect to fail

List the checks that will fail because of the work, and hold back alerts for those and nothing else.

Keep the checks running during the window. You want the record of what happened, and you want to see them turn green again before you stop.

Step 4: End it deliberately

The end of the window is the part most often skipped.

1. Check that everything in scope is passing again.
2. End the window, or let it end, so alerts are live again.
3. Tell customers it is finished.
4. Watch for a few minutes. Anything still failing should now alert, and that is the system working.

If the work finishes early, end the window early. Every extra minute of silence is a minute in which a real failure is hidden.

Step 5: Rehearse it once

Before the first real one, run a ten-minute window on one unimportant check.

1. Start the window and make the check fail. No alert should arrive.
2. Leave it failing and end the window. The alert should arrive now.
3. Look at what a customer saw on the status page at each stage.

Common mistakes

• Muting a channel by hand instead. Nothing turns it back on.
• A window that is too short. The work overruns, alerts fire, and customers see an outage after you said it was planned.
• Silencing everything. The unrelated failure at 02:30 goes unseen.
• Announcing only when it starts. Customers who were about to do something important find out too late.
• Recurring windows nobody remembers. A weekly silence set up a year ago hides failures every week. Review them.

Doing this in saysOK

A maintenance window has a title, a start and an end of up to seven days, and can repeat daily, weekly or monthly. Its scope is the monitors and tags you pick, or everything.

While it runs, checks keep running and are recorded. Alert rules with Suppress while a maintenance window is active ticked are held back for the monitors in scope. SEV-1 rules always fire, for the monitors in scope too: if the work takes down something a SEV-1 rule covers, it still pages, so expect that page or adjust the rule beforehand. Heartbeats are not silenced either, so pause one that the work will stop. A monitor that is still failing when the window ends alerts on its first check afterwards, and End now ends a running window early.

Scheduled maintenance

Database upgradeOct 17, 02:00 UTC – Oct 17, 03:00 UTCSign-in and checkout will be unavailable for up to an hour. The website stays up.

Scheduled

With Post to status page on, the window appears under Scheduled maintenance on the pages that show its monitors, from 14 days before it starts. With Email subscribers on as well, subscribers get a message when it starts and when it completes, not beforehand: the advance notice is the entry on the page.

Maintenance windows are part of every plan; see pricing for what each plan includes.

Next

Planned silence is one way to keep alerts meaningful. For the rest, see how to cut alert noise. For where the announcement appears, see how to set up a status page your customers will trust.