Events and outages will show and be recorded here.

Current status is OK

Past Events

Server failure on N10, N11 and N15

Jan 7, 2022 / Containers mostly restored and back online on N12

Late at night we had a major failure due to a bug in a package that caused folders to get corrupt and deleted. We are working on a full breakdown but at this time we are more working on getting sites back online first. If you have a site that was on the affected servers they have been moved to N12 and you might need to update your DNS if you are not using our DNS or using a CNAME record for DNS. If you do you CNAME records the domains in question have all been pointed to N12. You can view updates on this thread. https://hey.cafe/conversation/eg9221hdv01m4hdnvvqac7hgs

Web services down on N6

Oct 21, 2021 / Service back online

Due to a LE chain problem related to a new certificate the web server was unable to be started automatically. The system gave errors related to this certificate causing Nginx and Apache to not load. We will have more information on this page later today as we get all the info down. A lot of the error was due to us internally not getting alerts on time and the people able to respond having to wait for another person with access to the systems to make the fixes. We are going to be working on our internal alerts and making it easer for issues like this to be fixed.

Full breakdown

2:39 AM - Got automated outage alert when nobody was online.
5:54 AM - Got customer alert that sent to a person and the person was alerted.
7:30 AM - Server manager contacted to fix the error when previous attempts by other staff where unable to resolve the error.
7:45 AM - Services restored.

DNS services down on N6

Aug 21, 2021 / Service back online

DNS bind server failed to restart causing DNS lookups to fail for sites using N6. DNS is picky and has problems with config files getting written too fast or having overlaps. Fix is easy and quick as long as we get alerted. Due to this error and the time it took from first message we are expanding and working on our automated alerts.
2:01 AM - Got first outside alert (missed).
2:26 AM - Got message about outage.
2:32 AM - Services restored.

Network error on N9 & N11

Aug 13, 2021 / Service back online

Network error that has broken outside connectivity that has resulted in web traffic requests failing.
1:38 PM - We are looking into it now.
1:50 PM - everything is online and healthy again.