• Old OIT Home
• About
• Service Levels
• Policies/Standards
• Get Help
• New OIT Website

Procedure for
Handling an Outage or Disruption Event Affecting Campus
Definitions

• Outage or disruption event:  an outage or system degradation that impacts a significant group of users
• Significant group of users:  a defined group of some size (e.g. everyone in a department, everyone in a building, or everyone that uses Banner), or all of a certain type of customer (e.g. all students), or the entire campus
• Technology Support Center (TSC):  the OIT front-end support groups consolidated under our Support Services unit
• TSC supervisor:  anyone in a supervisory role in the front-end support operation; this includes fulltime staff leads and managers, and student supervisors or those acting as supervisors on evening or weekend shifts
• Technical area:  a unit providing support for some specific enterprise or other technology, e.g. the network, Banner, the phone system, e-mail
• System Status Page:  OIT Web page showing information about the availability of key systems

Responsibilities

• First person hearing of or realizing there is an issue:  report it to the Technology Support Center directly.
• Technology Support Center:  send campus alerts and updates via e-mail; post notice and updates on System Status Page; create ticket
• Support personnel working on the issue:  provide timely and informative updates to Technology Support Center personnel so they can send useful updates to those impacted by an issue
• Technical area:  identify and resolve technical events, work with TSC to communicate technical events, log technical events
• Operations:  update the System Status Page after hours, upon the direction of a technical area representative
• Directors:  ensure that procedures are followed in a consistent and timely fashion so that OIT customers can rely on our communications when technical events occur

Concept

When an event occurs:

• someone discovers it;
• someone knows what is going on – what the problem is, how to begin investigating it;
• someone decides that it should be communicated and what should be communicated; and
• someone communicates it. 

In our environment, a technical area or the TSC may discover an event (1).  The technical area is typically the group that knows what’s going on (2).  The technical area and the TSC must collaborate to decide that an incident should be communicated and what information to provide (3).  The TSC is designated as the area to send communications (4).

Steps – During Business Hours of the Technology Support Center:

1a. If a technical unit discovers the event:  as soon as the event is discovered, call or arrange for a person to call
the TSC and report:

• The general nature of the problem and
• Who is or is likely to be affected

Parties to this interaction should arrange a time for a second contact if the event persists for 30 minutes.

1b. If the TSC begins taking phone calls and other contacts (5 or more) that suggest a problem is occurring, a
supervisor will contact the technical unit that is implicated in the issue based on what is known (i.e. if it appears that the network is down, the network support team will be called).  The Operations Call List is to be used to identify office and cell phone numbers for the first and second contact.  If neither can be reached, begin calling up the chain of supervisors, using office and cell phone numbers, as well, until someone is reached who can contact the appropriate technical personnel. 

Once a person is reached, the TSC and the technical contact will establish:

• The general nature of the problem and
• Who is or is likely to be affected

Parties to this interaction should arrange a time for a second contact if the event persists for 30 minutes.

2. The technical unit involved proceeds to troubleshoot and attempt to resolve the problem.

3a. If notified by the technical unit within 15 minutes (approx.) of the initial communication that the problem is being or has been resolved, the TSC will create an outage ticket.

3b. If not notified by the technical unit within 15 minutes (approx.) of the initial communication that the problem is being or has been resolved, the TSC makes an initial communication that there is or may be a problem:

• Post initial message on System Status Page (see Appendix 1)
• Send out an initial message to OIT Alerts using same text as System Status Page
• Create an outage ticket

3a. At the 30-minute point (approximately), a TSC supervisor will call the technical contact to check the status of the problem.  If the problem is not on its way to being fully resolved, the TSC will

• Gather additional information (if necessary) from the technical area necessary to post an announcement and
• Update the message on the Systems Status Page, if necessary
• Create an Outage/Disruption E-mail using the standard template, and send via Connect
• Appendix 2 shows the standard format
• Writer may change headings as appropriate to communicate adequately with customers

4. If the problem continues, updates will be posted on the System Status Page at specific points:  7:30 a.m.,
            11:30 a.m., and 3:30 p.m.

5. Immediately upon resolution of the problem, the technical area will contact the TSC.  The TSC will:
a) post a resolution message on the System Status Page
b) send an OIT alert that the problem has been resolved
c) send an e-mail using connect communicating resolution of the problem

Steps – Outside Business Hours of the Technology Support Center:
Technical area:  upon becoming aware of a technical event, or upon being informed of such an event by
the Operations staff, do the following:

• Perform initial troubleshooting and begin resolution processes
• if the problem is not resolved within 15 minutes, direct the Operations staff to update the System Status Page as indicated in Attachment 2

Operations staff (if directed to post a System Status Page update as noted in b above):

• Enter a ticket to record the incident
• Post the System Status Page notice
• Send an OIT alert

Steps – Campus Event Impacting All Network or All E-Mail Access
            We need to rethink how we will communicate is no one can get to the Internet or to e-mail

Steps – Campus Event of Significant Importance
This includes the following levels of service outage:

• All network
• All wireless
• All e-mail (student OR faculty/staff)
• All internet
• Banner
• Blackboard

Actions to be taken, in addition to the appropriate actions above:
Either the TSC (or the responding technical area

• Contact AD – Customer Support and Director of Support Services by cell phone
• Contact CIO by cell phone
• CIO will contact institutional senior leadership as appropriate

Responsible Office: OIT
Contact: OIT Help Desk, (812) 237-2910
Website Address for this Procedure: None
Related Documents: None
Keyword Index: None
Appendices: 2

Appendix 1
Sample Wording for the System Status Page and the OIT Alert
Initial message:
                  [Date of Message]            [Time of Message]
                  [Basic problem description]
[User group(s) impacted.]
[State a workaround if one is available.]

An update will be provided [at 11:30 a.m./3:30 p.m./7:30 a.m.] if the problem persists at that time.

Update message (11:30/3:30/7:30 a.m. as necessary)
                  [Date of Message]            [Time of Message]
[Basic problem description]
[User group(s) impacted.]
[Describe at high level any actions taken by OIT to address]
[State a general expectation for when the problem will be resolved, if known]
[State a workaround if one is available.]

                  An update will be provided [at 11:30 a.m./3:30 p.m./7:30 a.m.] if the problem persists at that
time.

Post-resolution message:
                  [Date of Message]            [Time of Message]
                  Problem resolved. 

                  From [date/time] to [date/time], describe the problem and the user groups impacted.
                                    Note:  this can be a repeat of the initial message text, if that works.

                  If you believe you are still experiencing this problem, please call the Technology Support Center at
                  812-237-2910.

                  Note:  Generally speaking, a post-resolution message should be left in place for five (5) business days.

Appendix 2.

Template for Creating a Technical Event Alert

General guidelines:

• Use the headings below almost all the time.  Most of these headings will be used with every message, however some may need to be changed, or headings may need to be added, based on the nature of the technical event.  Use the standard most of the time, but use your judgment about what headings/information make sense!
• Use wording that will resonate with the customer.  Avoid technical terms.
• Think about the information the customer needs about the event.  Remember, the customer is impacted in a certain way, for example:
• He/she may need to plan to do something else besides use the impacted system for a while . . . but how long is that?
• He/she may need to know when to try to use the system again . . . when will that be?
• He/she may need to know that a workaround exists . . . what is that?
• He/she may need to know how to get back into the system once it comes back up . . . what steps should he/she take?
• Etc.

Standard Template:
What is happening?
            Describe the issue and its impact:  what is happening in non-technical terms

Who is impacted?
            Describe the group(s) of customers that are experiencing the problem, or will experience it if
            they attempt to use the system that is having a problem.  For example:  all students, all Banner users,
            all faculty who attempt to enter grades

How does this impact me?
            Describe the result of the issue, in non-technical terms.  For example:         

• guests on campus cannot access the wireless system using ISU-GUEST; or
• students in Mills Hall are unable to access the network using wireless

Is there a workaround?
            Describe what to do while the issue is occurring, for example:
            There is no known workaround.
            Students who have an Ethernet cable may connect to the wired network on the ports in their rooms.

When and how will I be updated with a status on this issue?
            Let our customers know how to get information on what is going on.  Two paths, generally:
            “For status on this technical event, see the OIT System Status Page at: [URL]”
            An e-mail will be sent when the issue is resolved.
           
What do I do if I have questions or problems?
Contact the OIT Technology Support Center @ 812-237-2910 if you require more information or assistance.
For status on this event, check the OIT System Status Page at [URL].

All systems normal.

Check Systems >>

Show Alert History >>

Security News and Issues

OIT Security Home Page

Events & Training

Computer-based self-paced training for students, faculty and staff >>

Main Contact

Gillum Hall 103
Indiana State University
Terre Haute, IN 47809

Get Help:
(812) 237-2910

Offices:
(812) 237-8439

Submit a Service Request