Utility communications

Modernising utility communications – Oncor’s experience with IP/MPLS

Critical legacy systems should not be compromised when migrating to a modern communications infrastructure.
Published: Fri 01 Feb 2019

Communications is fundamental to the modern utility. The evolving energy system requires networks that are able to support the latest grid applications now and into the future. Equally importantly, these networks need to support the critical legacy systems such as SCADA that have long lifetimes in utility operations.

These demands are forcing utilities to modernise their communications, often involving multiple networks developed in a piecemeal way, migrating to an infrastructure that can deliver the required performance across both the new and legacy systems. A popular choice is IP/multi-protocol label switching (IP/MPLS), which has been deployed by many tens of utilities across the globe.

“It is necessary to deploy a converged network that can support multi-service,” says Hansen Chan, Marketing Manager at Nokia. “IP/MPLS is a good companion for utilities on the modernisation journey, providing flexibility and security and extending out to the field area network, offers the foundation for the future as intelligence is pushed out to the grid edge.”

In an Engerati webinar, John Alberti, Senior Network Architect at Texas utility Oncor Electric Delivery, sixth largest in the US, and Chan present insights on IP/MPLS and the utility’s experience in planning a smooth migration from its time-division multiplexing (TDM) solution and support of legacy systems – a work still in progress with time to run.

Planning IP/MPLS

IP/MPLS is not a new technology for Oncor and about 10 years ago the utility put in a backbone between the major operations centres to support the smart metering and smart grid initiatives under way. This has been operated alongside the older TDM network, which was built for SCADA and other critical services such as mobile radio. However, over time as the TDM network aged it became harder to support and at the same time, the telcos were discontinuing services that Oncor needed into the future.

“We looked at lots of options and the requirement that we wanted to maintain as much control as possible drove us to refresh the TDM network with a private network and as before, to do so on a packet-based infrastructure to give us future flexibility,” Alberti explains.

He says that a key to the project was the planning. “You need to understand your pain points and where to focus when upgrading systems and we also worked extensively with vendors in lab and field tests to confirm we could do what we wanted.”

Key planning elements he mentions included identification of the scope and top priorities, an inventory of all existing circuit types and locations, a proposed topology for the new network and a least risk plan for the migration of services – in Oncor’s case on a regional basis.

“Knowing the different options, we could categorise which sites would be using which services and then group them to optimise the migration plan. And one doesn’t have to do everything all at once and we are at a ‘hybrid’ stage of still running the legacy channel banks.”

Alberti comments that a key challenge has been around time synchronisation, which Chan identifies as one of the three building blocks for a modern communications system (the others being interoperable network services that can be connected to IP/MPLS and a quality of service policy to ensure high priority traffic classification and consistent delay and jotter).

“This is a learning area and we chose to use Sync-E as its easier to configure and understand,” he says, adding that PTP is being investigated as a backup.

“During the initial build-out, we found different equipment had different Sync-E capabilities and so one needs to work closely with the vendors to configure it correctly. There also are hop count limits before the clock degrades and our goal is to not exceed 20 Sync-E device hops, including routers and radios in the path.”

Utility lessons learned

Considering the variety of sites such as large and small substations and towers that can be supported with IP/MPLS infrastructure, Oncor’s solution has been built with modularity and flexibility in mind, Alberti says.

“One can add or take away components that can be reused for other sites and even though there are different types of sites, architecturally it looks like one seamless network,” he comments. “Then when you map the services, they can be cross-connected to wherever they need to go.”

Alberti adds that vital tools to building and operating an IP/MPLS infrastructure include router and switch management, transport encryption and capacity management to track bandwidth usage and avoid congestion.

“If there is a failure one needs to know there is the capacity to support traffic when it reroutes.”

Turning to the benefits gained in the IP/MPLS migration, Alberti highlights operational benefits including greater visibility and remote work capabilities as well as cost savings by moving away from some leased telco services.

“The flexibility is a big thing and for example we have been able to offset some of the leased line costs by using cellular and it is also more reliable than the services over analogue circuits from the telcos. So, we are both gaining reliability as well as saving costs!”