At 3:37PM PST, we had a power blip in one of our datacenters. In those two seconds, over 1,000 systems blinked offline. As a non-profit, we don’t have all of those niceties such as hot-hot datacenters or those new fangled UPSes. Instead, we do it the old fashioned way, which means we are susceptible to power failures within the building our core systems reside.
Due to the time it takes to bring it all back online and the fact the outage took place during our daily report runs, it will take us a couple days to work through the backlog of reports. What this means is that if you receive our reports, you may receive them up to 48 hours late this week. Don’t worry, we’ll eventually catch back up and everything will be back to normal soon!
We’re not sure what caused the outage, but my money is on malicious raccoons, colluding with criminals.