Main Menu10/5/2005
Advanced Metering Data Systems (AMDS) still has its headquarters in New Orleans, but for now, instead of housing its network operations, the AMDS floor of the multistory building sits dark and empty. All of the hardware, software, papers, and people are scattered across the country while the company management and employees weigh their options. Will they move back? How did their disaster management plan, fortunately well established before the evacuation order was issued, hold up? Did their customers lose any data?
No Data Lost
The short answer is that no data was lost, even though one tower gateway was overcome by storm surge on the Alabama coastline. Meter data was successfully captured and delivered to the utilities, with some delays, but intact. The road to success was not smooth nor without drama but AMDS feels very fortunate all of their employees are safe and they were able to climb back up to "normal" operations as quickly as they were able.
Most companies are fortunate to never have to exercise their disaster management plan but disasters are not limited to hurricane prone areas. All companies face the potential of disasters, and many come with little or no warning, such as terrorist attacks, tornadoes, extended communication or power outages, earthquakes or volcanic eruptions. If a company is going to evacuate, the recent experience in Houston shows that employees could be trapped on exit routes for days unless the decision to evacuate is made very early. Not evacuating can lead to even more serious problems, such as total destruction and loss of life if the disaster hits where predicted or comes without warning. As AMDS found, having a disaster management plan was critical in their recovery.
Employees and Families First Priority
Britton Sanderford, CEO of AMDS, explained that their first concern had to do with their employees. Were they safe? Did they need help? The uncertainty was very difficult for the families, and rather than get better over time, initially it only became worse. Cell phones didn't work once the power went out and land lines stopped working when the local phone carrier lost a switch along the Gulf coast. Loved ones could not be reached, and in many cases, the living conditions were untenable except for very short periods of time. People had stopped as soon as they found a hotel room, often sharing with friends and neighbors, resulting in too many people in a room along with various assorted pets.
Once it became clear that a return to New Orleans would not happen for weeks or even months, the top priority of AMDS was to provide some certainty to their employees in the face of overwhelming change. First, they had to find their employees. People had scattered in all directions, and their ultimate destinations were unknown. AMDS had no idea where most of the evacuated employees were, and could not contact anyone in the normal manner because of the communication outages in cellular and landline networks, and the internet.
Fortunately, their disaster plan had alternate phone numbers for family members or friends living outside the area who could relay messages to the employees. AMDS put an employee who lived outside of Louisiana in charge of recovery of communications. Within three days, he was able to set up a company wide conference call with nearly every employee. These conference calls were held every two or three days to handle getting back to work and resolve any new issues that came up. The day after the storm, AMDS was able to re-establish banking relations through remote site banks, and began searching for longer-term housing for its employees. Equally important, employees were reassured they would continue to be paid by direct deposit.
AMDS Network Recovery
The second concern was evaluating the fixed network, known as FlexNet with AMDS Connect, and initiating recovery operation of the data center. I interviewed Gregg Larson, Director Network Deployments of AMDS, at the recent AMRA convention in Long Beach about the steps AMDS took to ensure the reliability of their network in the face of hurricanes and other potential disasters. The AMDS disaster plan was typical for a "class A" data center: redundant fiber lines, redundant power choices, and AMDS had just implemented an alternate backhaul using USA Mobility, a nationwide paging company. The main flaw in the ability of AMDS to recover was the existence of only one data center, something that AMDS had been in the process of addressing, but in hindsight, not quickly enough. Now there are two: a new one in Dallas, Texas and the original one in New Orleans. Per the evacuation plan, Larson transferred the source code backup from New Orleans to Kansas, where Larson and his family are temporarily located. This resulted in a major glitch because part of the source code was missing from the backup. Though this didn't result in any data loss, it did slow the return to "normal" operations for two days while the team managed to piece the missing code back together.
Larson was pleased with the operation of the network during the hurricane. There's not much one can do to prevent a storm surge of over 30 feet of ruining a piece of equipment. However, even though the primary backhaul communication network was down, the alternate backhaul communications through USA Mobility's paging network allowed all of the data to be successfully retrieved.
As Larson reflected back on the disaster, he noted that the simplicity of their network may explain in part why there was so little damage to the network. The network consists of the gateway towers, the meter modules embedded in the meters, and the data center operations. Since there are relatively few nodes on the system, AMDS can cost effectively provide for alternate communications in the link between the gateway towers and the data centers. The Sensus iCon meters and meter modules communicate directly to the gateway tower. Even with the cellular and landline communications down, the network continued to operate more-or-less normally. The battery backup on the gateway towers allowed the data collected up to the time the power went out to be transmitted to the data center after the power failed.
While the utilities using AMDS system were aware of the scope of the disaster, especially since two of the utilities, Alabama Power and Mississippi Power, were facing the very same disaster, the utilities were concerned about reaching a semi-state of normalcy as soon as possible. This was an unscheduled test in a way of what might happen if all of the utility customer meters were read using this technology: could the utility continue to bill customers unaffected by the hurricane? The results show that with a few changes, yes, data would continue to arrive as before.
Looking Ahead
The changes required would be two fully operational "class A" data centers rather than one, an investment in satellite phones, and complete backups of the source code. What wouldn't change would be the redundant communications between the gateway towers and the data centers, and the dedication of the employees to overcome unexpected problems. In the short time since hurricane Katrina struck, AMDS has two new offices, a new data center, and everyone is living in new apartments or houses. AMDS was at the AMRA convention talking to current and potentially new customers, and life is beginning to feel a little more normal. AMDS is grateful that a brighter future lies ahead, perhaps even brighter than before Katrina made landfall.
UtiliPoint's IssueAlerts are compiled based on the independent analysis of UtiliPoint consultants. The opinions expressed in UtiliPoint's IssueAlerts are not intended to predict financial performance of companies discussed, or to be the basis for investment decisions of any kind. UtiliPoint's sole purpose in publishing its IssueAlerts is to offer an independent perspective regarding the key events occurring in the energy industry, based on its long-standing reputation as an expert on energy issues. Copyright 2005. UtiliPoint International, Inc. All rights reserved.
