Events and outages will show and be recorded here.
Main services status is OK
Node status is OK
Past Events
Network outage on N6
Oct 9, 2022 / Back online
Network outage lasting aprox 8 hours Everything is back online and sites should be accepting network traffic again. Certificate problem that caused web services to not resume, it also caused monitoring to not work correctly.
Network outage on N8
June 13, 2022 / Back online
Network outage lasting aprox 30 minutes. Everything is back online and sites should be accepting network traffic again.
Server failure on N10, N11 and N15
Jan 7, 2022 / Containers restored and back online on N12
Late at night we had a major failure due to a bug in a package that caused folders to get corrupt and deleted. We are working on a full breakdown but at this time we are more working on getting sites back online first. If you have a site that was on the affected servers they have been moved to N12 and you might need to update your DNS if you are not using our DNS or using a CNAME record for DNS. If you do you CNAME records the domains in question have all been pointed to N12. You can view updates on this thread. https://hey.cafe/conversation/eg9221hdv01m4hdnvvqac7hgs
Web services down on N6
Oct 21, 2021 / Service back online
Due to a LE chain problem related to a new certificate the web server was unable to be started automatically. The system gave errors related to this certificate causing Nginx and Apache to not load. We will have more information on this page later today as we get all the info down. A lot of the error was due to us internally not getting alerts on time and the people able to respond having to wait for another person with access to the systems to make the fixes. We are going to be working on our internal alerts and making it easer for issues like this to be fixed.
Full breakdown
DNS services down on N6
Aug 21, 2021 / Service back online
DNS bind server failed to restart causing DNS lookups to fail for sites using N6. DNS is picky and has problems with config files getting written too fast or having overlaps. Fix is easy and quick as long as we get alerted. Due to this error and the time it took from first message we are expanding and working on our automated alerts.
Network error on N9 & N11
Aug 13, 2021 / Service back online
Network error that has broken outside connectivity that has resulted in web traffic requests failing.
