Nagios Notifications and Escalations

Introduction¶

Getting Nagios to notify the right people, once, at the right time is a matter of understanding how several settings interact: notification options, intervals, time periods, escalations, and dependencies.

When Nagios sends a notification¶

All of these must be true:

1. Notifications are enabled (globally, for the host/service, and for the contact).
2. The state is one the object notifies on (notification_options w,u,c,r,f).
3. The check has reached a hard state (max_check_attempts consecutive fails), not soft.
4. The current time is inside the object's notification_period.
5. The contact is inside their own service_notification_period.
6. Not currently in scheduled downtime, not suppressed by a dependency, not acknowledged (for repeat notifications).

Then the first notification goes out. Repeats follow notification_interval (minutes) until recovery or acknowledgement. notification_interval 0 = notify once, never repeat.

Core config¶

define service {
    ...
    notification_period        24x7
    notification_options       w,c,r          # warning, critical, recovery (not unknown/flapping)
    notification_interval      30             # re-notify every 30 min while not-OK
    first_notification_delay   5              # wait 5 min after going hard before first page
    contact_groups             app-team
}

first_notification_delay is the anti-noise workhorse: a service that self-heals in 3 minutes never pages anyone.

Time periods¶

define timeperiod {
    timeperiod_name  workhours
    monday           09:00-18:00
    tuesday          09:00-18:00
    ...
}

Use workhours as a notification_period for non-urgent checks so they don't page at 3 a.m.; keep 24x7 for anything that genuinely needs immediate action.

Escalations¶

Notify tier 1 first; if still broken after N notifications, add tier 2, then management:

define serviceescalation {
    host_name               app01
    service_description      Job queue depth
    first_notification      3          # from the 3rd notification...
    last_notification       5          # ...through the 5th
    notification_interval   15
    contact_groups          app-team-leads
}

define serviceescalation {
    host_name               app01
    service_description      Job queue depth
    first_notification      6
    last_notification       0          # 0 = forever
    notification_interval   30
    contact_groups          management,app-team,app-team-leads
}

Escalations add contacts; the original group keeps getting notified unless you structure it otherwise.

Dependencies — suppress downstream noise¶

If the database host is down, you don't want 40 "app can't reach DB" pages:

define servicedependency {
    host_name                       db01
    service_description             MySQL
    dependent_host_name             app01,app02,app03
    dependent_service_description    App health
    notification_failure_criteria    w,u,c      # don't notify dependents if MySQL is w/u/c
    execution_failure_criteria       c          # also skip running the dependent check
}

Host parent/child relationships (parents directive) do the same for "unreachable vs down".

Scheduled downtime¶

Before maintenance, schedule downtime (UI or EXTERNAL COMMAND) for the host — Nagios keeps checking but suppresses notifications, and child hosts inherit it if you tick "propagate".

Verification and troubleshooting¶

# Why didn't I get paged? — the log has the reasoning
grep 'SERVICE NOTIFICATION' /var/log/nagios/nagios.log
grep 'app01' /var/log/nagios/nagios.log | grep -i 'notification\|SUPPRESSED'
nagios -v /etc/nagios/nagios.cfg

• No notification at all — check the chain: is it a hard state (max_check_attempts)? Inside the notification period? Contact's period? Notifications enabled at all three levels? The log line for the state change says SUPPRESSED and why.
• Paged for a 30-second blip — no first_notification_delay, or max_check_attempts 1. Add both.
• Re-paged every 5 minutes forever — low notification_interval and nobody acknowledged. Acknowledge to stop repeats; consider notification_interval 0 for checks that only need one page.
• Whole team paged when one upstream fails — missing service/host dependencies.
• Recovery notification but no problem notification — the problem happened outside the notification period; the recovery landed inside it.
• Escalation never triggers — first_notification counts notifications, not minutes; with notification_interval 0 there's only ever notification #1.

Related tools and reading¶

• On-site: Crontab Generator.
• Related posts: Alert fatigue and how to fix it, Nagios active vs passive checks.