Well, this happened:
#
# java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
#
# Internal Error (gcTaskThread.cpp:41), pid=3510, tid=139652483825291
# Error: Cannot create GC thread. Out of system resources.
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM
# An error report file with more information is saved as:
# /mount/reports/hs_err_pid3510.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
For those of you who receive our reports, that’s what the latest notifications looked like. We were all excited about our new reports, and something went awry with the back end generation systems. The engineering team is aware and on it, so please bear with us. I can’t remember the last time we had our reports blow up in our face like this, but we will get it fixed as soon as possible.
UPDATE (02:31 PM March 29, 2014 PDT): We have identified the problem and put in a temporary fix. For now, we have stopped all reports on our scanning activity (who knew that having thousands of new report recipients light up would be an issue?) until we get in the permanent fix. The real fix should go in on Tuesday, when we’ll turn back on the scanning reports. Unfortunately, we are not able to re-run the reports that were impacted by the bug. We apologize for this inconvenience.
UPDATE (16:00 April 02, 2014 GMT+1): Reporting has been fixed and all data going out in the reports again.