Smart grids demand and generate lots of data, so optical fibre networks are a big part of the make-up of every large scale installation.
Whether it is optical fibre that is buried, aerial fibre on poles, or optical fibre nested into OPGW (optical ground wire) does not matter much, just that the connections are made and the network meets the performance criteria for the applications.
These days, with the NERC CIP rules in flux, the pressing question is the level of integration between IT (information technology) and OT (operational technology). Some say they must be two discrete networks, others argue that they can be fully integrated. Some think that reality is a place in the middle between two disparate networks and one holistic solution. Whatever your point of view, suffice it to say that optical networks are a big part of any smart grid strategy, no matter how it is designed.
Cisco smart grid network designed with RAD statistical multiplexers 2011
Another burning question is the way that the signals are transmitted. Do we continue to use SONET, or do we look to IP/MPLS or Carrier Ethernet? All have merit. The manner in which the Utility is organized plays a big part in technology selection. Most agree that SONET is beyond its useful life so the debate turns to IP/MPLS versus Carrier Ethernet. When the Utility is driven by the OT department, then Carrier Ethernet is preferred, however, if the IT team is leading, then IP/MPLS is desired. It is a matter of perspective and priorities based upon the leadership’s background. Of course, this is not always the case, but it is usually. So, understanding who is driving the adoption helps to predict the desired solution. Is this the best way to approach technology selection, not really, but it is a reality of the industry. The best solution should be selected based upon a number of technology and business parameters built upon qualitative and quantitative metrics. Personal bias should not be totally ignored, but it should not be the dominate rating and ranking judgement factor either. So, let’s consider the options.
SONET
SONET (synchronous optical networking) has been the mainstay of Utility networks for decades and has served their needs well. Often, it was just used by the OT team and is mostly found on OPGW. However, as new applications evolved, especially from the IT side of the house, SONET was not able to support them or was expensive to scale as necessary. It is a clocked network so that makes it ideal for the OT folks who demand deterministic delivery of datagrams. But, it is challenged to meet all of the needs for all applications. As the global demand for SONET products shrinks in favour of IP/MPLS and Carrier Ethernet, availability of supply and high cost make it hard to justify.further investment in this technology.
IP/MPLS
IP/MPLS (internet protocol / multi-protocol label switching) is a leading technology with great ability to scale and impressive data handling capacity. There are many quality vendors and several equally impressive bolt-on standards to help mitigate some integration challenges for the OT side of the house. The IT folks can embrace it fully. But, depending upon the current and future approaches to smart grid, it may not be the best choice for OT. There are a few issues that make IP/MPLS difficult for OT. For example, it is a layer 3 technology so it has the overheads and delays of TCP/IP. it can use different paths in the forward direction versus the reverse direction meaning that the latency in both directions can be different. The clocking needs to be added with technologies like ITU’s SyncE (synchronous Ethernet), IETF’s NTP (network time protocol), or the ever popular IEEE 1588 version 2, however it is still non-deterministic even with clocking added. Numerous protocols need to be set and maintained within the IP/MPLS networks, so it can be complex to operate.
Carrier Ethernet
Carrier Ethernet has advantages over IP/MPLS and is often preferred by the OT side of the house since it is the natural outgrowth of SONET in the packetized world. Carrier Ethernet is generally considered to be lower cost to implement compared to IP/MPLS. While IP/MPLS can scale to thousands of sites, Carrier Ethernet can scale to hundreds of sites, still very respectable and normally not a barrier for Utilities (compared to telecom carriers that service more nodes). Carrier Ethernet is the preferred technology by telecom carriers to connect data centres and other mission critical network nodes. Carrier Ethernet offers lower latency compared to IP/MPLS. It is a Layer 2 solution. Carrier Ethernet is an all Ethernet structure so it can connect end to end with Ethernet, it is better than IP/MPLS in this regard which needs to convert between MPLS and Ethernet at the network edges.
Wave Division Multiplexing
Wave Division Multiplexing (WDM) is a means to carry multiple wavelengths or frequencies of light simultaneously over a single strand of optical fibre. WDM is able to provision bidirectional communications over the same strand of glass. The first systems used just two wavelengths of light but it is possible to multiplex up to 160 frequencies today, although lower number of frequencies are more common today such as 16 or 24 wavelengths. Two flavours of WDM are used, Course Wave Division Multiplexing (CWDM) and Dense Wave Division Multiplexing (DWDM). Other variants of these two types are seen, but these two are the most popular. In the Utility world, we normally just use DWDM. This approach uses the 1550 nanometre (nm) band. So, the glass used must have this wavelength window available, Normally, glass is purchased with three windows for the greatest flexibility, 850 nm, 1310 nm, and 1550 nm. Reconfigurable Optical Add-Drop Multiplexer (ROADM) is the main termination device used at fibre nodes to deliver or add traffic to the network. ROADM devices operate on WDM networks and can selectively manage individual wavelengths. When the optical fibre network is configured as a mesh architecture, then Optical Cross Connects (OXC) matrices are used to map the fibres to each other. For long distance transports of signals, IP/MPLS and Carrier Ethernet ride over these DWDM networks and can use ROADM and OXC at termination points or network nodes.
Other considerations include legacy solutions like OTN and propriety GigE links.
Optical Transport Network
OTN (optical transport network) is an ITU standard for wrapping a variety of signal formats like SONET, Gigabit Ethernet, and Fibre Channel, It is used to wrap these different signalling formats into a common Layer 1 optical foundation to improve the functionality of transport, multiplexing, switching, management, supervision and survivability of optical channels carrying client signals. It is used with WDM (wave division multiplexing) and applies each signal to its own wavelength. Rather than using a form of encapsulation into a common carrier format like Carrier Ethernet or IP/MPLS, OTN permits the discrete signal formats to be individually mapped to the OTN structure and are carried in parallel.
Image sourced from Microsemi on December 12, 2014 from http://www.microsemi.com/products/timing-and-synchronization/otn-timing
While OTN is cost effective, it is limited in flexibility and usability for smart grid needs. OTN varies its latency depending upon the source signal being wrapped and therefore different signal types will flow over the same optical pathway in different time durations.
Gigabit Ethernet
Gigabit Ethernet is a common access connection solution in baseband formats over copper wire. But, it can be delivered in five physical layer standards for Gigabit Ethernet using optical fibre (1000BASE-X),using copper on twisted pair cable (1000BASE-T), or shielded balanced copper cable (1000BASE-CX). Sometimes it is used in intra-city links as point to point connections to connect sites. It is not scalable and should be replaced when data rate demand a better solution.
Ring versus Mesh Architecture
One of the more interesting aspects of optical network design, is the approach to the network architecture pertaining to redundancy and network robustness. Specifically, should the network be designed as a ring or a mesh? Historically, rings have been the most popular approach but that was driven by the legacy SONET networks that used counter-rotating rings to add robustness to the design. However, now with IP/MPLS and Carrier Ethernet, we can continue to employ the ring approach or consider the mesh model. Both have merit and both work well for IT applications. But, for OT applications, it is important to know the latency of the pathway and to appreciate the convergence time delays when the primary path fails and the secondary pathway is switched into circuit. If the existing fibre strands are already deployed in a ring model, then it is likely that a ring model architecture will continue in the future designs. However, some aspects of the mesh model may be incorporated into the new design either in part or whole. When using OPGW, the ring model may be the only option as well, since the nature of the transmission corridor is in long trunks that facilitate rings instead of mesh architecture.
Network Management System
A surprising majority of smart grid optical network projects neglect the NMS (network management system) requirement. Sometimes, element managers are purchased to support the optical network, but that is not always the case and all to often they do not include integration with other network aspects like point-to-point microwave links, point-to-multipoint wireless networks, and baseband routing and switching to name a few. It is critical to view all of the network elements as one seamless network fabric. Therefore, a manager of managers (MOM) is needed to orchestrate the many element managers into one harmonious solution that interacts well together and “makes and breaks” connections upon demand. This MOM should also include an out-of-band diagnostic system that feeds troubleshooting back to the NOC (network operation centre). The MOM often supports network upgrades and pushing new images to the end devices, such as firewalls.
Multiplexers and Legacy Interfaces
How the traffic connects to the network is always challenging for smart grid networks since there are so many legacy interfaces that need to be respected in the design. Therefore, encapsulation / de-encapsulation is used for IP/MPLS and Carrier Ethernet. This process can add latency to the link. So, great care is needed to respect the latency issue for some smart grid applications, such as teleprotection. There are several ways to interface. Some baseband routers and switches can accommodate a direct connection and other need to be connected via an intermediary device, most often a statistical multiplexer. There are many good statistical multiplexers available today that are well suited for the smart grid applications and are loaded with features to make traffic flow management and grooming easier. These devices need their own element manager in the NMS.
So, the world of optical networks within a smart grid environment does not need to be overly complex. Once the traffic types are known, and properly quantified, then the best solution can be evaluated and identified. Deep knowledge of the applications and their expected traffic flows, regardless if they are IT or OT, will make the task of selecting a technology solution much easier. So, begin with the needs first, then look to the architecture, and finally consider the technology to make it all work. Too often, we see clients start with the technology first, this is wrong. A perfect network is one that is invisible to the applications. Business is run on the applications, then networks are built to serve them. Do not get caught in the techolust arguments that we see far to often. Engineers love their technology, but it is the business of running a Utility that is paramount. Technology is the foundation upon which the business is operated. So, while it is important, it should not dictate how the business is run, the best solutions underpin the business.
——————–MJM——————–
Michael Martin has more than 35 years of experience in broadband networks, optical fibre, wireless and digital communications technologies. He is a Senior Executive Consultant with IBM’s Global Center of Excellence for Energy and Utilities. He was previously a founding partner and President of MICAN Communications and earlier was President of Comlink Systems Limited and Ensat Broadcast Services, Inc., both divisions of Cygnal Technologies Corporation. He holds three Masters level degrees, in business (MBA), communication (MA), and education (MEd). As well, he has diplomas and certifications in business, computer programming, internetworking, project management, media, photography, and communication technology.