During your course of being a SysAdmin, you will come across many unplanned outages. Some of them will impact BAU (Business as usual) and others will just be minor service outages. Do you know what to do in the event of these outages?
For planned outages, see Outage - Do you have a planned outage process?
Below is a process for these types of outages. Some amount of common sense is required here, an outage would be if services that would affect BAU work are disrupted and/or some hardware has failed.
Hardware Outage:
Service Outage:
Many services can be used for device monitoring e.g. WhatsUp Gold, Solarwinds, SCOM. You would do the following in any of them:
After you have determined what services have been disrupted it is time to call your SysAdmin team and organize a quick conference call. This will allow you to have a discussion prior to making any changes/fixes that could cause the outage to become worse.
Key discussion points:
What if you cannot reach anyone?
If you cannot reach anyone move on to the Email section.
If from the previous discussion you have determined that an email needs to be sent to your entire company, or you have decided this is necessary if you cannot contact anyone above, send an email in the following format:
We are experiencing an outage and the following services have been affected:
We are working on restoring these services and will keep you updated.
Thank you, SysAdmins
A separate email needs to be sent to SysAdmins outlining what was discussed on the call. If no one was contactable, please proceed with what you have determined on your own.
As per our conversation,
The following services are disrupted:
The impact of these services disrupted are:
We have decided that an email to ALL is/is not required.
The next steps to resolving this are:
Thank you, SysAdmin
If you have completed your tasks but the issue has not resolved, please try to make contact with the SysAdmin team again and send an updated 'To Myself' email.
If your actions have resolved the issue, please notify ALL of the services being restored and update your 'To Myself' email.