The earth was formless and void, and darkness was over the surface of the deep waters, and the Spirit of God was moving over the surface of the waters.
Then God said, "Let there be Light"; and there was Light!
One Monday morning while we reported to work, we had more than half of the INDIA reporting network failures…the sites were down and the teams were in total darkness nobody knew what went wrong!
The operations team gave up saying “network down”!
The maintenance guys said, “ISP cards failure”
In turn, service engineers said “High down time “meaning high “Mean time to Repair” (MTTR)–more than six hours..! (The time when the Service was down...)
Also, business heads said, “High Customer Dissatisfaction...!”
Total disaster….Darkness…!
Everyone was searching for a ray of hope...!
The Centre of Excellence team came to the rescue. On investigation they theorized that the problem was due to the temperature rise, close to the cards, owing to improper functioning of Precision Air-conditioning (PAC).
The team along with the equipment manufacturer implemented the solution. They removed the root causes by reducing the hotspots in the server rooms and thereby saving the failure of cards. And, finally, restored the site operations and network...!
In the next three months the solution was implemented at all the 59 large facilities, resulting in zero card failures hence Zero MTTR, due to the card failures.
The results were amazing in terms of improved customer satisfaction; the power utilization factor has reduced from 2.0 to 1.5 now, and resulted in high energy cost savings. More specifically, more than 64.24 crores/annum (only for 59 facilities).
This was a winner for the long term. It will give optimum results even if we expand the network capacity.
Thus as God said, "Let there be Light" and……there was Light…!
Lessons Learned
- What looks like the primary failures may not be the root causes of the actual problem
- The air-conditioning operation is a major energy source which consumes 48 % of the overall energy and can be optimised by running the same at manufacturers specified operating parameters
- A proper / technically accurate installation of any power/electronic equipment gives optimised results
- High service availability is key to high customer satisfaction