Network Outage 2016-12-17 Post-mortem
Issue Summary At approximately 8:28 AM PST (1628 UTC) we lost all our network connectivity in our Corvallis data center affecting all of our customers hosted in the data center. Network connectivity was restored at approximately 11:15 AM PST (1915 UTC). The root cause was due to our ISP losing power to their switches which provide our Internet connectivity due to a problem with the generator.
Timeline Time Description 8:20AM PST (approx) OSU electricians start a planned power outage affecting the building our systems are housed in. 8:28AM PST OSL received a page regarding several networks being off-line. 9:12AM PST “OSL arrives on-site and confirms our primary data center is powered on and that none of the systems had been rebooted.” 9:30AM PST| “OSL confirms the root cause is a UPS problem in the basement data center which houses our ISP network equipment.” 11:07AM PST| Power restored in basement data center. 11:15AM PST| OSL Network was restored.