Situation & Duty Manager Escalation
This table outlines guidelines for an outage situation. The elapsed time listed for escalations are for a general reference, quicker escalations can be carried out as determined in each situation.
|
Elapsed Time |
Outage or Severe Risk Situation & Escalation |
|
10 minutes |
SNCC escalates to appropriate support tech identified for the service or component. If there is no after hours support defined in the CMDB or in a KB, or no after hours tech available, SNCC escalates directly to the situation manager for the service affected. |
|
2 hours |
SNCC staff ensures that the situation manager is contacted not only for the primary service experiencing the outage, but also situation managers for all affected services are contacted. For example, an LDAP authentication outage will cause a Microsoft365 & other outages. The Microsoft365 situation manager should be contacted along with the LDAP situation manager. |
|
8 hours |
|
|
12 hours
|
COO & core directors Note: the duty manager will usually be responsible for escalation to the COO & core directors although a reminder to the duty manager by SNCC would likely be appreciated. The SEO duty manager may request that SNCC triggers additional escalation & notification on their behalf. See below for additional notification. It is the responsibility of the chief operating officer to escalate outage notifications to campus and other leadership as necessary and appropriate. The severity of the outage and the breadth of impacted service(s) will determine who should be notified and how often updates should occur. |
| At discretion | Any outage may be escalated prior to the referenced times if the situation warrants, the times are maximums for when an outage must be escalated. |
Situation Manager Contact Information
SNCC should ensure that the Situation Manager is contacted within 2 hours for an outage situation (ie a service is not available or not functioning as intended) or if SNCC or a Tech have reason that a severe enough situation exists to warrant management escalation or involvement - including elevated risk (for a significant service) situations, such as a Data Center or super-network node site running on generator power or backup cooling.
Refer also to the SEO on call calendar for appropriate escalation to on call resources, ie. for a Data Center power degradation or outage, or cooling issue, escalate to the DC Team on call, as well as to a situation & duty manager.
A Situation Manager (aka primary service lead) for technical services are listed on the "COOP info" tab in a WiscIT CMDB record for the service.
Additional Management SMS Notification (WiscIT Escalation Button)
Additional communication & notification (escalation) can be triggered by using the 'Vertical Escalation Notification' button or link on the left side of a WiscIT problem ticket.
This button will trigger SMS notifications to be sent to the core directors & deputy CIO simultaneously. Use of this button will usually be performed by or at the direction of a duty manager.
SNCC can exercise their own judgment and use this without waiting for duty manager approval.
The SMS notification will include the category from the WiscIT problem ticket, as well as a link to the problem. The expectation is that the directors will use the link to obtain more information, but SNCC staff should be prepared for any of the directors to contact SNCC directly for more information.
Note that this function is intended to be a notification which summarizes an outage (probably during it's initial troubleshooting) situation that is communicated. It's for notification only, there's no requirement for directors to take action. The situation & duty managers should not presume any of their responsibilities are relieved from this communication function being carried out.
Communication Flow Diagram