During the week leading up to January 12th 2012, I was working at the Printing Office.  There was some issue with the Netapp boxes and Snapmirror working that the network team and myself had been working on.  The network team had some ideas, but when I left the office on the 12th, it was still not working.

I worked a so-called “Alternative Work Schedule” where I worked 9 hour days, and had every other Friday off.  And Friday the 13th was my AWS day, Making Thursday the 12th was my Friday.

At the time I had a two hour one-way train ride.  I often slept on the train as I never got enough at night.  So, on the train ride home I was out, and had a dream that the network team had crashed the filer.  And exactly how the filer was crashed.

I woke up thinking what a nightmare.  Went on about my day. On Friday the 13th, we were the store, and I got a text message saying FDsys was down.  Yep, I knew exactly what the problem was.  I responded saying I was at the store and will look at it when I get home.  When I got home, I called my counter part and told him I know what it is.  Logged in and got my VPN up.  And logged into the only host I knew I’d be able to get into that would let me log into the filer, the manage server.

And sure enough, I was right,  The filer’s routing table was empty.  Just as in my dream.  Someone flushed the whole damn routing table on the Filer!  All the hosts that relayed on NFS were ( just about everything ) were hung!  So, I looked at the boot time config file that had the routing table, and one-by-one re-added them, and slowly things started coming back to life. If I remember correctly, we had to reboot a couple of servers to get them to recover but for the most part, everything was back up.

The aftermath was a major shit storm. Lots of finger pointing. Someone even tried to suggest that was me since I was able to fix it so fast, but it was trivial for me to show 1) I was at the store when it happened, and 2) I wasn’t even logged in at the time.  We did figure out who it was, but I think it was just chalked up the “human error” in the end.

And yes, a couple of people suggested I was clairvoyant.  Of course I don’t believe that.  I just figured in all the emails going back and forth, I subconsciously picked something up, and the dream was my consequence warning me.   But who really knows.