CASE STUDIES                                     

“We now have visibility of all the information moving over the network. For the end user this application has movement flexibility. We can deploy a PSAP where we need to. It gives us the functionality of any other packet switched network, whether we reach end-points over wireless circuits or fixed circuits.”
       ~Bob Jones, KPB Systems Manager

Next Generation 9-1-1
in Kenai Peninsula Borough, Alaska

The Kenai Peninsula Borough Office of Emergency Management (KPB) in Soldotna, Alaska maintains a 9-1-1 system tested by one of the most demanding physical environments in North America. Winter temperatures routinely dip below minus 20 degrees, and avalanches can block the narrow two lane roadways. This is the state where major manufacturers come to perform rigorous product testing and where Scott Walden, Emergency Management Coordinator for KPB, must oversee a state-of-the-art 9-1-1 center. For more perspective on the environment Walden operates in, here are some facts on KPB to consider:

• KPB’s population consists of over 53,097 permanent residents, which doubles with the tourist and commercial fishing industry personnel during summer months
• The 9-1-1 agency receives over 20,000 9-1-1 calls per year
• It encompasses an area of over 25,600 square miles, with 7000 foot glacial mountains running along the southeast spine of the peninsula
• Earthquakes are a regular occurrence: a magnitude 3.0 quake was recorded as recently as September 8th, 2008
• The region contends with tsunamis and floods, in addition to the occasional avalanche

In late 2007, Walden, his team and microDATA embarked on a systematic overhaul and upgrade of their legacy 9-1-1 system. Beginning with the transition to an onsite, ALI Database Management System and GIS Database System, it ultimately culminated in deployment of an ESInet (Emergency Services IP Network) including an IP-based call handling system inter-connecting four primary PSAPs with 12 call taker positions. Soldotna, the largest PSAP, houses the ECDC (Emergency Call Data Center). This is connected to the KPB, Homer and Seward PSAPs over a Wide Area Network spanning, in some cases, 80 miles and traversing challenging terrain. The upgrade was a bold enterprise that demanded state-of-the-art technology to enable this transition to an integrated Next Generation 9-1-1-ready system.

KPB’s Challenges
Two common challenges initially compelled KPB to seek a replacement to their E9-1-1 system: The need for a system that met the requirements of Wireless Phase I and II, as well as the need to transition to a Next Generation-ready 9-1-1 system. Furthermore, KPB needed to provision for system diversity and enhanced data sharing between all touch points of its 9-1-1 system. Nature is not forgiving in Alaska; hence KPB’s new 9-1-1 system features multiple redundant databases and multiple IP connections between PSAPs.
KPB’s previous legacy 9-1-1 system consisted of many separate systems, like other larger 9-1-1 agencies across North America: an ALI Database maintained off site by their largest TELCO, a GIS Database maintained by KPB at their IT and GIS office, and a circuit-switched telephony system distributing the 9-1-1 calls between the respective four PSAPs. Managing mutually dependent systems that were not integrated proved not only inefficient, but at times impossible to maintain according to industry best practices.

ALI and GIS Database Challenges
Updating the ALI database once involved an arduous process, and with the ALI Database off site, oversight of the system proved challenging. In the past, one TELCO managed the database and received updates from three other TELCOs. “Prior to the transition to our new system, daily emails were sent to me and the Office of Emergency Management from the carrier maintaining our ALI database, and we had to upload those at an office located about a mile away from the State Trooper building in Soldotna,” states Tammy Goggia, Communications Center Manager. “In addition, the data was under contract with the carrier, and was not conducive to accuracy. We had double digit error rates. It could take weeks to correct an error,” says Walden. KPB’s goal was to have a system that could promptly update subscriber information in the ALI Database, ideally with a one day turnaround between the TELCOs.

Costs for maintaining the ALI database with their contracted provider were also increasing dramatically. In fact, a new contract indicated a rate increase from 9 cents per record to 25 cents, an approximate annual increase from $50,000 to over $140,000. Such an increase was enough to compel KPB to seek an alternative, and specifically to pursue the option of an onsite maintained database.

As for the GIS Database, Walden states, “prior to the new system we loaded a CD every week.” Chris Clough, Senior GIS Technician for KPB elaborates on the old updating process. “Before, we were using an in-house Arc GIS engine application on a workstation that was developed for our addressing department. I would come over to the Soldotna PSAP and hand a co-worker a CD containing the latest database, and then load it on each PSAP’s 12 call taker positions (sneaker porting). This was on an irregular basis to say the least,” states Clough. Given the terrain and distances between KPB’s four PSAPs, the sneaker porting method was not conducive to maintaining an up-to-date map. As with the ALI Database, KPB knew they had to streamline this process, eliminating the sneaker porting and reducing the update window.

Similarly, the telephony system, including an aging Selective Router, demanded upgrading to an IP-based system in order to provide system diversity and redundancy. Walden acknowledges that aside from lacking system redundancy, it was failing several times a week. Since it was a circuit-switched system and outside KPB’s internal network, Walden’s team was reliant on the vendor to fix breakages. There was no ability to monitor it, a fact that frustrated the IT Department. By Walden’s estimation they were experiencing system failures weekly with repair costs running into tens of thousands of dollars. “To make matters worse, little local expertise existed with this old equipment. We had to go to the mayor and explain we had an emergency to proceed with a Selective Router replacement,” states Walden. It was clear to Walden’s team that their system would have to make a fundamental transition from a circuit-switched network maintained by their carriers to an interconnected IP-based 9-1-1 system: an ESInet (Emergency Services IP Network) maintained by KPB and a more collaborative provider. Furthermore, their old 9-1-1 systems, the ALI Database, GIS Database and Call Handling system required a transition to IP-based technologies that share information between all stakeholders.

IT Challenges: A Circuit-switched Network
Maintaining a reliable network is all about visibility of the network and its traffic. For Bob Jones, KPB’s Systems Manager, it is essential that he be proactive, and posses the ability to predict issues before they affect the 9-1-1 network, as opposed to reacting to a problem he cannot see or isolate. KPB’s prior circuit-switched network from the selective router in was maintained by the TELCO . “Until we did the network upgrades, we relied on the phone carrier to maintain our system. It was costly and did not give us any in-house control. We were at the mercy of the carriers,” states Walden. “When a PSAP experienced a network outage the drill involved calling IT in Soldotna who would then call the circuit carrier, who in turn, would then dispatch a field tech to the problem spot,” says Walden acknowledging the cumbersome system of support. It was clear to Walden and Jones that they needed visibility into the system, plus command and control.

Establish a New Network Backbone: an ESInet
KPB recognized that building such an ESInet necessitated adoption of NENA’s standards for the IP Capable PSAP (08-501), as well as adhering to national standards laid down by accredited bodies, including ATIS. Today, redundant IP links connect each of the four PSAPs to the KPB WAN (see Fig 1 - The KPB ESInet), and the system provisions for a spare (back-up) PSAP that can be brought online at a moment’s notice. The Soldotna PSAP houses an ECDC where 9-1-1 trunks convert analog calls to IP via TDM to IP Gateways, enabling smooth distribution of all calls over the ESInet as VoIP traffic. Another ECDC resides at KPB’s Prism facility. Redundant IP links connect all PSAPs, all critical 9-1-1 systems and the call taker stations. “We had no redundancy with the old system. This new system has given us trouble-shooting ability and a safety net of redundancy,” states Walden. Jones echoes Walden’s observations and asserts, “By consolidating this technology on our network wire we have a knowledge base in place and a foundation to support it.”

Fig. 1 The KPB ESInet

Instead of the old analog selective router, 9-1-1 calls now enter from KPB’s five carriers on 9-1-1 trunks, but with added system diversity. Today, half the trunks are brought into Soldotna and the other half to the back-up Prism facility. The
9-1-1 trunks converge into TDM to IP Gateways where they are converted to IP as they enter the KPB ESInet and onto the ECDCs. Redundant xSwitch servers, two at Soldotna and two at Prism, perform tabular routing as before with NPA-NXS routing for landline calls and ESRK (Emergency Service Routing Key) routing for wireless calls. However, the software-based xSwitch servers also perform spatial routing. Through the IP link to the x9GIS Database in KPB’s IT and GIS department at Binkley Street, the system completes spatial look-ups to determine call location. Walden’s team is thankful for this, especially given the dramatic increase in nomadic callers during summer months.

Unlike the old system which had a single point of failure, with xSwitch in two geographically separate locations, dropped calls due to one system failing are eliminated. What is more, unlike many proprietary redundant systems, the xSwitch at Soldotna and the xSwitch at Prism are both online simultaneously, in effect, load- balancing the calls as opposed to one live server in operation with a cold spare coming online in the instance of primary failure. In the event that the Soldotna facility should be cut off or damaged by a natural disaster, all current
9-1-1 calls into the PSAP would be redirected to the secondary ECDC at Prism. In an emergency situation this means a call is redirected seamlessly, as opposed to a dropped call, as in the past.

In contrast to the previous proprietary call handling equipment, KPB’s new system is built from the ground up on COTS (common off the shelf) hardware. All servers within the ECDCs (xSwitch, xStore Servers and Monitoring Servers) are Intel-based with serviceable hard drives, memory and controller cards. Likewise, the network architecture consists of Ethernet routers and switches based on COTS. As Jones explains, “the COTS approach allows our 9-1-1 provider, microDATA, to focus on the software application, not hardware, and the market place serves to support that with the hardware.”

Supporting the Network
The new ESInet eliminates the frustrations that Walden and Jones have experienced in hours of costly service calls to the circuit providers. Now the demarcation point for circuit providers is at the selective router. Furthermore, support has evolved beyond a reliance on dispatching field personnel to a site to diagnose and resolve an issue. In fact, Jones and his IT team can now resolve issues, in many cases, remotely over the network from their IT and GIS Office at Binkley Street. In addition, the KPB network is now monitored and maintained by remote support from microDATA via an encrypted VPN connection into KPB’s ESInet. Since most of the core systems are software-based, updates and any service items can be completed by technicians back at the microDATA offices over a wire. “Using this remote support system becomes a more efficient way of doing business. In the winter time it is not uncommon to have an avalanche block a highway. This system eliminates the need to travel back and forth,” states Walden.

An In-house ALI Database Management System
As part of their E9-1-1 system overhaul in late 2007, Walden’s team collaborated with microDATA to put in place an in-house, IP-based ALI Database Management System (xALI DBMS). The system consists of a software application, xALI DBMS, residing on a server at the IT and GIS facility at Binkley Street and a redundant system in Soldotna. Updates from KPB’s four TELCOs are transmitted via a secure web interface (xALI Web) to the IT and GIS facility. These are validated against the MSAG at KPB. Any updates that fail are instantly sent back to the submitting TELCO for resolution.

xALI DBMS also enables the call taker stations running the xT911 CTI to log and transmit address discrepancies over the ESInet to the ALI Database at the IT department, where Goggia receives a report. In this way, the IP-based system closes the loop, augmenting the rate of updates from the TELCOs outside the ESInet and at the call taker stations from inside the ESInet. “The network has streamlined the process. We now have a 24 hour turnaround for the ALI Database,” cites Goggia.

In addition to streamlining the ALI Database update process, Goggia now has a system that she can oversee. Reports can be generated instantly, producing statistics on the rate at which each TELCO is correcting their errors or on other criteria. Costs for maintaining the database have also been contained, with the ALI DBMS in-house. KPB is no longer subject to rate increases from a contracted provider. The new process levels the playing field, while eliminating intermediary steps and enabling the TELCOs to submit changes directly to KPB (See Fig 2).

Fig. 2 ALI Updates

A Streamlined GIS Database Management
The new GIS Database Management System consists of microDATA’s x9GIS Database on an Intel-based server. x9GIS runs the database on a Microsoft SQL Server back-end, coupled with ESRI’s Arc GIS Enterprise Server 9.2. These reside at the IT and GIS facility with a workstation running a duplicate system from where Clough applies the GIS updates daily. Once applied to the GIS Database Server, these updates are replicated over the KPB ESInet to the 12 call taker work station’s xTrakker maps in their respective PSAPs (See Fig 3). x9GIS enables an efficient transaction-based replication process whereby only the changes (deltas) to the database are transmitted to each of the call taker work stations running the xTrakker map with Arc SDE. For Clough, this is a far cry from the labor intensive sneaker port process. Plus, as Walden notes, “Today the updating is performed every 24 hours.”

Fig. 3 GIS Updates

As with any implementation, there are always areas that can be enhanced. For KPB, with both the ALI and GIS Database running in-house on microDATA’s integrated applications there is the capability to have the ALI Database validate against the GIS Database. Such an approach ensures synchronized map and ALI data, and increases efficiency with one single entry point for address data.

Enhancements to the CTI
Establishing an ESInet and enhancing the back-end resources means a radical new system for each of KPB’s 12 call takers. All the call taker stations have the microDATA map display, xTrakker, that is integrated with the call handling interface, xT911. The system runs on COTS hardware, and provides redundancy with a companion SIP phone (xTPhone) that sits alongside the work stations. In the event that a call taker station should fail, the IP-based SIP phone will not only receive the 9-1-1 call but will also present the call taker with ALI data on its integrated LCD screen, as it would through the xT911 CTI.
Goggia acknowledges that the new feature-rich CTI does come with a learning curve for her call takers, but, she says, “The learning curve is quick. Before, we did not have these capabilities. It is a simple program to use.” For an administrator, xT911 has changed the way Goggia obtains 9-1-1 call recordings. Instead of sifting through a tape reel, Goggia can access the digital recordings by date and time range on a digital voice recorder. “It has made it much easier and faster to retrieve our 9-1-1 calls,” cites Goggia. Through administrator login, Goggia may copy recordings to a thumb drive from any of the 12 call taker workstations for admission in a criminal case or call taker training.
“From an IT perspective I see more of a team work approach, sharing information,” states Jones. “Now all the other PSAPs can see what each PSAP is doing,” concurs Goggia. Much in the same way Jones’ IT Department has visibility of the network, each of Goggia’s call takers now have visibility of calls answered by fellow call takers at other PSAPs on the xT911 screen.

The Upshot of NG9-1-1 for KPB
The solution at KPB resulted in an integration of all the primary touch points of the agency’s 9-1-1 system (Call Handling, ALI Database Management, and GIS Database Management over an ESInet). Systems have been streamlined and a sense of empowerment pervades the agency:

“We now have visibility of all the information moving over the network. For the end user this application has movement flexibility. We can deploy a PSAP where we need to. It gives us the functionality of any other packet switched network, whether we reach end-points over wireless circuits or fixed circuits,” states Jones.

“The network has streamlined the process. We now have a 24 hour turnaround for the ALI Database,” states Goggia.

“Today, all of our PSAPs are on the same sheet of music and they can be updated from one location (my office). No more hand carrying a CD of map data to each PSAP and loading it on all the call taker stations,” says Clough.

For KPB’s Emergency Management Coordinator, there is a palpable sense of momentum and limitless possibilities for system growth and enhancement. Not content to rest, Walden sees further enhancements he would like to make. For instance, he would like better tracking of technical support tickets, and later in the year the addition of a new CAD, which Walden acknowledges will be smoother to implement today because of the open standards architecture of the microDATA applications. Most of all, when he compares the new system with the legacy system it replaced, Walden notes: “On the legacy system we never progressed or moved forward. This new system has given us an opportunity to move forward each day; it has given us system evolution."