🔥

Status

Events and outages will show and be recorded here.

Current status is OK

Past Events

Web services down on N6

Oct 21, 2021 / Service back online

Due to a LE chain problem related to a new certificate the web server was unable to be started automatically. The system gave errors related to this certificate causing Nginx and Apache to not load. We will have more information on this page later today as we get all the info down. A lot of the error was due to us internally not getting alerts on time and the people able to respond having to wait for another person with access to the systems to make the fixes. We are going to be working on our internal alerts and making it easer for issues like this to be fixed.

Full breakdown

2:39 AM - Got automated outage alert when nobody was online.
5:54 AM - Got customer alert that sent to a person and the person was alerted.
7:30 AM - Server manager contacted to fix the error when previous attempts by other staff where unable to resolve the error.
7:45 AM - Services restored.

DNS services down on N6

Aug 21, 2021 / Service back online

DNS bind server failed to restart causing DNS lookups to fail for sites using N6. DNS is picky and has problems with config files getting written too fast or having overlaps. Fix is easy and quick as long as we get alerted. Due to this error and the time it took from first message we are expanding and working on our automated alerts.
2:01 AM - Got first outside alert (missed).
2:26 AM - Got message about outage.
2:32 AM - Services restored.

Network error on N9 & N11

Aug 13, 2021 / Service back online

Network error that has broken outside connectivity that has resulted in web traffic requests failing.
1:38 PM - We are looking into it now.
1:50 PM - everything is online and healthy again.