WARNING:
This blog entry is very, very, very, very, very long. It has a huge amount of images, so be forewarned.
Introduction
Well, as many of you know, we have always been kind of the underground heroes of the Internet. We have scraped by with used gear, volunteers, and parts that fail often. We have told stories about our failures and the tribulations of being a non-profit. While none of that has really changed, we have finally gotten something nice. Something very nice. Our main corporate sponsor stepped up and really built something that would be impressive for just about anyone.
Our new Data Center is beautiful.
All the other issues still exist. We are perpetually short of gear, bandwidth, money, and many other things. But we do have something nice now.
The Old Data Center
The building that we had our old data center in was sold and we had to move. Our old space was a re-purposed lab that was made for benches, not 80 racks of gear. The number of times we blew circuits and almost set the building on fire occurred often enough that the electricians only showed up in full body protection suits:
So it was time to find a new space before we were trying to steal wifi from Starbucks for more bandwidth, and power from the poles directly.
The Old Space
Yeah. It was ugly. It was crowded and noisy. It frequently had a temperature of 90F. It was a pit. But it was our pit. Our beloved pit of gear, blood, sweat, and tears.
Ah yes, we do still have P3’s running…. almost 190 of them in fact.
The beautiful orange disk trays are ones that we designed and had printed because when we received some blades none of them came with disk trays. And of course this specific vendor does not allow any third-party vendor from selling trays which means you can only pay the full amount for a tray with a disk. So we took the easier route and just made our own out of printed plastic:
Which of course stand out nicely all over our server systems:
In the background you can see the very large HVAC unit that is the additional cooling we had to add in to achieve something less than 90F.
Truly impressive enterprise cooling, honest!
And of course more orange disk trays in the back…
Our main HDFS storage cluster which is most of the row. It currently sits at about 3 petabytes of capacity.
On the left is an elastic search cluster, with our OSX known file hash whitelisting systems in the middle.
Then there are the brutes of the Data Center. Our blade servers which house the minions that do all the work for us:
Stuff in the Cupboard
Because we had run out of space and power, we actually could not turn on all the gear we had available. As die hard techies and data hoarders, this made us very unhappy. The new data center space will help us reach our goal of actually powering up all of the storage and compute capabilities that we currently have.
Our storage room mostly emptied, but there always seem to be some lingering boxes.
Some additional blade servers we might be able to turn on. Assuming we actually have all the parts. We have a lot of chassis, but not all the power supplies or blades. The joy of being an impoverished non-profit. But we can dream.
Getting Ready To Move
After lots of preparation and hard work, the big day would final arrive. But first that meant loading hardware onto pallets. Lots of pallets:
The New Data Center
There are a lot of different statistics that can be mentioned. Most can be meaningless. But we’ll give the overview:
- 750kW of power – They brought in a new 500kW transformer to the building for us in addition to us taking half of the building power as well for that total. We have been strictly instructed that we are not actually allowed to utilize 100% of that or we will burn down the building. The power is broken down into buses where the total load is spread out over the row of racks it services. So we have to balance the high-powered gear over several buses as to not blow anything up. Not that we have ever done that before…
- 84 racks – The racks are not 100% filled, but will be crowded. We will need to more seriously plan to do more in-place upgrades instead of just adding in new gear. At this point, despite the move we will be very much still at the limits of capacity to grow larger.
- 18 Inter-Row Cooling Units (IRCs) – three per-row. Water cooled units that are part of a Pod cooling system to try and maximize the cooling. We should have sufficient cooling for 800kW of power.
Images of the new racks with the IRC’s:
Here you can see the Pod cooling design. The idea is the make more efficient mini-rooms of heat that can be better controlled.
You can almost see all three of the pods:
From inside one of the pods:
The front of a managed IRC:
Holy cow Batman, Dr. Freeze must be here to keep it that cold!
The Move
It was a lot of hands on deck. We kidnapped helpers off the street, we dragged friends into dark rooms, we did what it took to get everything going and moving. No children or small animals were hurt in the process.
Preparation
Nectar of the gods:
Beverages of the lessor deities:
Ambrosia for the masses:
Now that we are properly caffeined and sugared up what could go wrong?
Packing and Shipping
Okay – our packing abilities are not the best.
It slowly is getting emptied.
Hmm, someone needs to fill those.
And off more gear goes. Ideally to the right location.
The entire storage cluster has moved. Hopefully it arrives.
All those carefully packed pallets then need unloading again:
Hey guys… I think we could have packed those a little more?
Sugar is a very important ingredient to heavy loading.
I am sure not a single person helping thought about doing this… Until they saw the warning sign:
Lift with your back!
You are going on a journey. A long, dangerous journey…
And Into the New Space
Mounting all those rails is never amusing or fun.
Lift with your back!
And it Continues:
As of 2016-10-18 we are only on the third day of the move. We believe that all the gear that is needed for our operations will be moved by Friday and hopefully also brought online. The clusters we already moved have survived and are working. Score! We still need to:
- Mount it all in the new racks
- Wire it
- Pray it all powers up
- Power it up
- Sacrifice some chickens
- Bring up the clusters
- Rinse and repeat until everything has been moved
What does this mean for you?
After over a decade of continually expanding our capabilities and giving daily free of charge remediation reports to thousands of constituents around the world, this has been our first ever week of downtime for a major data center move. Once we are fully back online and serving you all again, we hope to be coming back stronger than ever – with firmer foundations in place to power us through another few more years of growth. Expect to hear more in the coming months.
What can you do to Help?
As always, we love to get more data in to help victims, but we are always seriously in need of bandwidth – particularly in our new data center. And donations help pay for the beast that is now our backend (we add another 2+ terrabytes of storage per day, just to keep up with new data arriving). The Data Center costs are covered for now, but there is a lot more that needs your help.
As a US 501c3 non-profit organization that does not charge for its services, we survive through tax deductible donations, sponsorships, as well as funded project work to expand out what we are able to do.
We share our data for no cost with direct network owners and National CERTs all over the world. We do not ask for credit, only the occasional support. If you think you might be able to help us out, please contact us by email at admin*at*shadowserver*dot*org.
