Of Vacations and Armageddon

Start Conversation

• Chief Architect – Excuse me.
• freed0 – lalalalalalala
• Chief Architect – Hey, excuse me!
• freed0 – lalalalalala
• Chief Architect – You cannot just keep ignoring me, I am going on vacation and you cannot stop me
• freed0 – LALALALALALALA!
• Chief Architect – Putting a blanket over your head and running in the data center away from me will not work nor matter
• freed0 – LALALALALALA!
• Chief Architect – Well, I am departing Monday
• freed0 – What?  You are going somewhere?
• Chief Architect – We already discussed this.  I will be gone for two weeks.
• freed0 – Two hours?
• Chief Architect – TWO WEEKS!
• freed0 – You know that you should not say that anywhere near the data center.  It is a sensitive beast.
• Chief Architect – What?  You think it is a living breathing things?  It has feelings?  HA!
• freed0 – That is not very nice – it gets sad when you are gone.
• Chief Architect – It will be sad, and nothing will happen.
• freed0 – You mean like this?
• Chief Architect – That was an anomaly, it cannot happen again.  Everything will be perfect, we have worked out all the kinks and everything is running very well now.
• freed0 – Well that is certainly a death flag

Reality

2019-06-02 – 0820 UTC-7 – It seems that the power company “accidentally” turned off all the power to the building where our data center resides for about 20 minutes.  This of course took everything out.  The Chief Architect and The System Administrator both came in and worked to late in the night filled with bad food and coffee.  The Chief Architect still left today for his holiday while leaving the poor System Administrator scrambling to finish bringing everything back up.  We lost several pieces of critical network gear as well as several fabric modules in our blade servers.  The litany of hardware failures was not short.

What Happens Next?

Most of the gear is back up again, some reports are slow eking out of the clusters.  Everything should be back to full utilization in the next couple of days.

End Conversation

• freed0 – Told you so….
• Chief Architect – ….