Years ago I was supporting the sales effort of pursuing a very large hospital chain. In fact the main hospital was the only Tier 1 trauma center in its region. I was a customer facing engineer at the time and we had met with the IT staff and management of this hospital several times. We had assessed their issues and they knew that they have serious problems with their IT infrastructure. It was an overworked, and underserved hodgepodge, a total mess. It had security holes you could drive a large Russian Train full of hackers through. But they were stuck with this one outdated vendor that was no good for them and were not about to change. The customer was not interested in working with us. Then one-night disaster literally struck them. The entire infrastructure collapsed! They could not do anything on their network including mission critical functions required to run a hospital.
This Was Serious
Imagine you are a hospital and serving an area of 3 million people. And with hundreds of admitted patients and you cannot look up a single patient record. It was almost as bad as having a major physical damage to the hospital. Worse the problem was so severe that they could not repair them. So after two 24 hour days with no luck trying to fix it they called my company and asked us to help. Keep in mind they had their own IT staff, the vendor of the current failed system, and even consultants on site all being paid and they are now asking us for help. My company had never done a cent of business with them. But we went out there anyway, myself and a peer and we worked for the next 24 hours straight. We even brought in Demonstration equipment to replace the failed system. We put it on a cart in the middle of the data center. With all kinds of makeshift cables feeding into it to strung across the room, like some kind of emergency patient grasping for life on a make-shift life support system. And we got them going again. We would run tests for another 24 hours to improve stability, and periodically we would get a visit from a few upper-level executives checking on their patient. They were very somber and serious and were all in nice suits with dour looks on their faces, they would chat a bit thank us and then leave.
Lots Of Attaboys
After the dust had settled I received major accolades from the hospital, and these went directly to the CEO of my company. It was quite a feather in my cap. Needless to say the hospital was so appreciative and was very very amenable to looking at our thoughts on a replacement system. The sales people in no time had a quote out to them. And the hospital asked that we keep the Demonstration equipment in a little bit longer so they could recover. Of course, we said yes. They even bought our solution and placed the order. Then something amazing happened. For some unforeseen reason, they backed out of the PO and purchased a new system from the failed vendor. And they even asked to keep our demo equipment in place just a little bit longer until they could get the new system from the failed vendor installed properly! There is a technical term for this: Chutzpah.
Don’t Miss The Signals
Within 8 months that failed vendor who had been struggling for a long time was completely out of business and were not even supporting the newly installed system anymore. We never did do any business with them. The message here? There was every signal needed to warn the people at this hospital that they were in a precarious position before the system failed. What did they do with this information? Nothing! Then after it failed they were handed a valid solution to fix it. But for some cultural reason, they could not bring themselves to deal with it. See the signals, take the action and move forward. Anyone with access to a browser knew that their system vendor was in financial trouble. They chose to ignore it or never looked. Don’t ignore the signals. Look for them, be ready to react to them. Have a plan in place for catastrophic failures and by all means, act.