The Scoop V2 – Empowering Utilities with Better Communications

Seamless operations with secure networking and resilient timing

Wide area networks have always been critical to energy companies. But the solutions applied today might not be sufficient for the demands of the future as legacy technologies become obsolete and new protocols must be supported. The digital transformation is also impacting networking technologies and forcing a convergence to IP and Ethernet. What’s more, with distributed energy production, there is a need to push advanced control technologies from core to edge. This extends the coverage of the wide area network but also requires precise synchronization to be delivered to remote sites in the power grid. The migration of the wide area network is a highly complex task, which becomes even more complex with the changing threat landscape and the increasing number of sophisticated attacks.

This article highlights proven methods for converging legacy communication networks and making them future-proof and secure. It explains the relevance of resilient, accurate timing and shares best practices for robust synchronization networks. It also highlights some solutions from ADVA that can help you make the transition.

Towards a future-proof WAN architecture

Historically, energy companies have installed a variety of application-specific solutions. Supervisory control and data acquisition (SCADA) systems, industrial control systems (ICS), solutions for tele-protection, and intra-office systems use specific communication protocols. Proprietary protocols and purpose-built hardware have successfully worked in the past, but this approach will not meet future needs. Figure 1 outlines the technology convergence as it happens in power grids today.

While SDH/PDH technology is still used in the power utility WAN, many operators have complemented, or substituted, those legacy networks with IP and MPLS technology. In many cases, MPLS-TP is being applied as this specific flavor of MPLS uses centralized provisioning of label-switched paths, and is operationally aligned with SDH/PDH networks, simplifying its introduction and adoption.

The International Council on Large Electric Systems (CIGRE) Study Committee D2, December 2021 recently performed a survey on the current state of telecommunications in power utilities. The results shown in Figure 2 give a good insight into the state of migration from TDM to packet:

While roughly a third of this traffic is transported over MPLS, there is still a large proportion using legacy technologies.

Technology migration happens gradually. A connectivity network will need to transport legacy TDM but also MPLS traffic. MPLS-TP may be used for a tactical smooth transition from TDM to packet network technologies. However, as it isn’t widely supported by suppliers and it is getting competition from segment routing, this technology is unlikely to continue. Segment routing combines the best of IP and MPLS while including TP management practices.

Vertically integrated solutions are difficult to migrate and align with changing requirements such as MPLS technology evolution. In consequence, we should architect networks in a disaggregated and open way with the possibility to select best-in-class technology for MPLS and DWDM transport. As shown in Figure 3, open optical transport technology can easily be combined with hybrid TDM/packet technologies such as those from our partner OTN Systems or a pureplay ADVA FSP 150-XG400 packet solution.

Secure transport in the WAN is essential for protecting critical infrastructure from cyberattacks.

In many countries, regulatory obligations require the encryption of mission-critical communication to assure business continuity. The packet network and optical network devices should be able to encrypt network traffic at line rate. Our FSP 3000 open optical transport platform supports line-rate encryption of wavelengths while MACsec+ provides end-to-end encryption at the FSP 150 packet layer.

Failures of network components or fiber breaks cannot be avoided, but the impact can be minimized with resilient and well-monitored networks. Protection mechanisms such as optical path protection, Ethernet ring protection, or redundant MPLS paths can instantly repair network failures in an automated way. However, it is still necessary to identify the failed component and initiate immediate repair. This is done through sophisticated OAM capabilities at the optical, Ethernet, and IP/MPLS layer.

A failure in the passive fiber network can be detected and localized within milliseconds with the help of OTDR-based fiber monitoring solutions. What’s more, with the help of fiber sensors for security and environment applications, any threat to the physical infrastructure can be instantly detected.

Protecting sub-stations from cyber threats

With the emergence of distributed power generation, automated control is moving from primary to secondary substations and towards decentralized solar power plants and wind farms. Distributed power generation creates a need for visibility and control of a high number of remote energy production sites.

As operational technology (OT) is moving to the edge of the power grid, standardized and innovative solutions will be applied, converging the previously applied operational protocols on a common IP and Ethernet transport layer.

The digital transformation at the substation will not happen with a forklift upgrade but in a seamless migration, leveraging proven technologies and migrating towards a converged IEC 61850 automated substation architecture. During this transition period, a diverse set of technologies will need to be supported at substations. Established asynchronous and synchronous interfaces such as RS232 or STM1 or application-specific interfaces such as C37.94 for teleprotection will be applied while TDM network interfaces migrate to Ethernet over DWDM.

Most legacy technologies have not been designed with security in mind. In some cases, operating systems are now reaching end-of-life and are suffering from a lack of security patches. This results in the need for a security architecture at a substation that prevents vulnerable components and sub-systems from any access.

With IEC 62443 a comprehensive framework is available, guiding the implementation of cybersecurity in industrial control systems (ICS). Based on a risk assessment, it specifies a zoned architecture to protect the most critical assets with multiple layers of defense.

National regulators are also requesting critical infrastructure to implement controls for assuring business continuity. This makes general security controls such as encryption of the connectivity network, firewalls, and intrusion detection systems a requirement.

In Germany, for instance, the Information Security Act 2.0 creates a need to identify and report attacks to critical infrastructure. This isn’t an easy task, as present substation applications do not provide an easy way to do this.

ADVA is responding to this need with a software-centric security solution, optionally supported by a secure and environmentally hardened edge compute node. Monitoring of ICS/ SCADA traffic can be done most efficiently by software probes. Several suppliers provide the appropriate software applications, but they do need to run in a protected, security-hardened environment.

Ensemble Connector is a hosting platform and network operating system with comprehensive protective controls. It can be installed on an FSP 150-XG118 (CSH), which is an edge compute node featuring advanced encryption for securely connecting a substation to other sites of the power grid. Ensemble Connector can also be installed on x86 based COTS servers; for utility use cases the servers will need the appropriate security features.

ADVA is working with leading suppliers of software for monitoring substation traffic such as Nozomi or Palo Alto to identify any malicious activities targeting both the IT and OT network.

Making synchronization networks robust and secure

Synchronization and timing have been a requirement for the PDH/SDH communication network in the past. Many protocols applied at substations such as DNP or GOOSE require timing, but only with moderate millisecond accuracy. With the IEC 61850 substation automation standard and technologies for precise localization of network faults, sub-microsecond timing will need to be assured. While frequency synchronization is delivered from the connectivity network by means of SyncE, time of day (ToD) information is frequently provided by Global Navigation Satellite Systems (GNSS), such as GPS, located at each and every site.

In consequence, many power utilities apply a high number of GNSS receivers at their substations for access to the precise time. There are growing concerns about disturbances caused by jamming and spoofing attacks, solar events, weather patterns, etc. The operational complexity of thousands of GNSS receivers also adds considerable cost. What’s more, these receivers are often not integrated into central management systems. Consequently, problems, failures, and attacks against timing networks at substations might pass unnoticed until major failures in the power grid occur.

There is an urgent need for a highly resilient and accurate synchronization architecture in power utilities. Delivering accurate synchronization with the packet network is a sensible approach to initially backup GNSS-delivered timing and eventually become the main timing source. This requires timing-aware switches and routers supporting SyncE and PTP in combination with highly stable core grandmasters backed up with ultra-stable cesium atomic clocks to meet even the most stringent enhanced primary reference time clock (ePRTC) specifications.

As mentioned before, multiple protocols need to be supported at substations. This is also true for the synchronization network, which needs to support legacy interfaces such as IRIG-B at substations or NTP.

ADVA’s Oscilloquartz division provides the most comprehensive assured positioning, navigation, and timing (aPNT+™) technology for resilient power grid timing. With multi-technology substation grandmasters, modular and redundant core grandmasters in combination with proven, ultrastable cesium clocks, this comprehensive solution portfolio can be adapted to any power grid timing need. What’s more, the unique AI-empowered Ensemble Sync Director management solution assures accuracy of timing at any site, even analyzing data from any third-party GNSS receiver.

Technology experts are joining forces

Communication networks for power utilities are covering applications in very different competence domains. There are IT applications that need standard IP connectivity over fixed and wireless networks but also utility OT applications such as SCADA and tele-protections with domain-specific protocols and interfaces. Suppliers with OT experience and products will work with suppliers addressing the IT and service provider market in order to combine their solutions and meet both current and future communication requirements.

This is why OTN and ADVA have joined forces to pre-integrate and test a network solution for power utilities. Both companies have a leading portfolio in their home markets, and their combined solution perfectly meets the most demanding communication requirements of critical infrastructures.

OTN Systems provides a market-leading network solution for specific industrial segments, including public networks, light rail, oil and gas, power utilities, and mining. With its new product line, XTran, the company is well-positioned to offer market-leading solutions to those enterprises.

What’s more, ADVA is joining forces with independent software vendors providing software applications at the substation to monitor and protect the IT and OT network.

Jointly with our partners, ADVA offers a comprehensive, secure, and resilient wide area network for power utilities, featuring accurate and robust synchronization and supported by a powerful end-to-end management system.

The ADVA solution for wide-area networks

ADVA’s carrier-grade network products are a perfect solution for communication networks with power utilities, combining operational simplicity with high availability and resilient designs. In addition, ADVA offers infrastructure assurance solutions for passive, in-service monitoring of fiber and sites. Oscilloquartz, a fully-owned ADVA company, is a market-leading supplier of synchronization solutions. Its portfolio was recently extended with PTP power profile, NTP server, and IRIG-B interfaces for the seamless migration of legacy synchronization solutions toward highly accurate PTP technology. ADVA’s unique aPNT+ platform has been specifically designed and optimized for robust, secure, and resilient timing for critical infrastructure.

The wide area networks of power utilities combine DWDM transport with IP routing and Ethernet, as well as MPLS switching. Disaggregated architectures create flexibility and simplify the complex migration from today’s architecture to next-generation power grid networks.

However, this does create the need for a strong management system to integrate all components with central control. ADVA’s Ensemble Controller provides comprehensive end-to-end management for the packet, optical, virtualization, and synchronization network.

While the digital transformation of IT and OT networks is making network operations robust and future-proof, it is a very complex undertaking. Operators will move carefully and will initially focus on those components with the most significant need for action. We believe that the synchronization and timing network is an area of underestimated risk; the threat landscape is changing rapidly as nation-state actors engage. In addition, component obsolescence or shortage due to the silicon crisis is triggering a transition to open, multi-vendor technologies with unique benefits but also transformational challenges. ADVA is an experienced supplier of market-leading communication solutions that understand what is needed to operate a network. We can be your reliable partner for making communication networks secure, resilient, and future-proof.

This article was kindly contributed by AVDA