Building secure wastewater management in the cloud

Water/wastewater wireless networking upgrades led to a supervisory control and data acquisition (SCADA) system upgrade, cloud capabilities, improved cybersecurity and physical security, more information from instrumentation and other benefits for Waterford Township Department of Public Works (DPW).

Formally organized in 1834, Waterford Township is located geographically in the center of Oakland County, Michigan, with more than 72,000 residents. With 360 miles of water main and 355 miles of sanitary sewer, water management in Waterford is no small task. The DPW operates and maintains 19 production wells, 3 storage tanks, 11 treatment plants and 63 sewer lift stations.

To run these systems, they invested years ago in integrating core applications including geographic information systems (GIS), asset management systems (AMS), enterprise content management (ECM), and supervisory control and data acquisition (SCADA).

That system delivered a lot of value, but nothing lasts forever.

Time to upgrade, networking, SCADA

In 2017, Russell Williams, director of public works, and Frank Fisher, engineering superintendent, at Waterford DPW started on a project to upgrade their core SCADA infrastructure. At the time, they used a serial polling program to request updates from their many sites through a network of remote telemetry units (RTUs) that communicated over licensed radio frequency (RF) transmitters, and they began replacing these with newer remote input/output (I/O) and industrial cellular modems communicating through a private Verizon network.

However, the next year, they attended a conference about edge controllers and became excited by the potential of eliminating some systemic limitations after being introduced to MQTT Sparkplug.

With more than 90 controllers on the network, the polling mechanism they used, combined with the limited bandwidth of the radio network, meant data from each site would update only every 3 to 5 minutes. Sometimes a lift station would run briefly in between polling cycles, creating reporting gaps and inhibiting operators’ ability to detect issues until alarms eventually made their way through. For each I/O point they added to the system, this latency worsened.

Figure 1: Waterford DPW’s legacy infrastructure relied on a network of RTUs and RF transmitters communicating to SCADA workstations in the office. Courtesy: Opto 22

Report-by-exception publishing rule eases network traffic

It seemed clear MQTT (formerly IBM’s MQ Telemetry Transport, now available in open source) could reduce bandwidth usage and ensure delivery of important system actions. That’s because, as opposed to cyclic polling, MQTT follows a strict report-by-exception publishing rule. Instead of waiting to be commanded by the central server (called, in MQTT parlance, the broker), field devices and other MQTT network clients send data on their own, if and only if there is a change in a monitored value.

“We have many lift stations that will spend most of their time sitting,” Williams said, “[so] why transfer data all the time?”

Without dependence on a central polling program, they also saw the possibility of eliminating a systemic bottleneck and potential point of failure.

Figure 2: A comparison of one of Waterford DPW’s lift station control panels showing the old RTU layout and new layout with the Opto 22 groov EPIC controller and DIGI modem. Courtesy: Opto 22

Upgrade proof-of-concept to production

To help execute their vision, Waterford DPW engaged Perceptive Controls, a Michigan-based system integrator specializing in industrial and process control applications for the water/wastewater, food and beverage and oil and gas industries. Building an MQTT system for the first time came with a learning curve, according to Kevin Finkler, software engineer at Perceptive.

“This was the first time I had done something that wasn’t strict client-server,” he said. “The topic system and how you can subscribe to a particular topic is pretty different.”

Figure 3: Waterford DPW’s modernized infrastructure publishes data from Opto 22 groov EPIC controllers to a cloud-hosted Ignition SCADA and MQTT broker over a Verizon 4G LTE cellular network. Courtesy: Opto 22

Five ways publish-subscribe differs from polling networks

MQTT’s publish-subscribe communication model is a definite departure from that of traditional industrial protocols in a few key ways:

1. Field device connections originate from the device, not the broker.
2. Each field device connects only to the broker, regardless of where its data needs to go.
3. When using Sparkplug payloads, each device publishes (sends) a list of its available data items, called topics, upon establishing a connection to the broker.
4. Other MQTT clients may also connect to the broker, see the available topics, and then subscribe to (request) updates on those topics when published by field devices.
5. When a field device publishes an update, the broker forwards that update to all subscribing clients.

The first challenge Waterford faced was integrating this new communication model into the existing SCADA system, but this proved to be a nonstarter. Their system simply lacked the ability to work with the MQTT protocol, so they experimented with a few newer systems and chose one that offered very tight MQTT integration, including the ability to serve as an MQTT broker itself.

Even though understanding the MQTT communication model took some work at first, establishing communication was straightforward once Finkler had the right tools.

“It kind of happens automagically,” Finkler said. “You basically define a few parameters to set up the broker. And each of the [edge controllers] was pretty simple. You just point it at the broker and it starts sending tags.”

“I love that both of these sides have embraced MQTT,” Fisher said. “It makes the connection seamless.”

Figure 4: One of Waterford’s new lift station control screens. The station security panel is shown at center left. Courtesy: Opto 22

Building defense-in-depth cybersecurity

As Fisher searched for the components to build Waterford’s new SCADA infrastructure, he experimented with hosting an MQTT broker on Amazon Web Services (AWS). With the cellular network already under construction, it seemed like a good opportunity to leverage cloud computing for greater fault tolerance and scalability.

Having successfully tested the concept, Fisher decided to deploy the SCADA system directly on AWS, and he and Finkler began building out the mechanisms to secure the new infrastructure.

First, Fisher configured the firewall on AWS to accept traffic only from his edge controllers and specific SCADA clients in Waterford’s and Perceptive’s offices. Firewalls on the cell modems were also configured to accept only trusted IPs.

Fisher then installed a client SSL certificate on each edge controller so the SCADA could authenticate and encrypt the connection, protecting against human-in-the-middle attacks that could expose data or permit unauthorized control.

Every authorized user is required to create strong passwords to access any edge controller or SCADA client in the system. In addition, every user login is tracked and reported throughout the system.

Fisher and Finkler even integrated physical site security into the new SCADA. Each lift station is secured with an outer door under lock and key, and a physical switch on the door is connected to the local edge controller. The SCADA monitors the switch state to detect when someone enters. If a user login with access privileges for that specific room is not registered within a specific time, the SCADA generates a global alarm.

Figure 5: Waterford’s new infrastructure increased the speed of data updates from minutes to less than a second. Courtesy: Opto 22

Benefits for SCADA, networking wastewater project

After completing upgrades on all 63 of its sewage lift stations and six of its clean water sites, the new edge-oriented infrastructure has reduced field updates from multi-minute cycles to sub-second event-driven publications. Waterford never misses a system action or alarm notification anymore and operators can stay connected from anywhere through cell-enabled tablets.

Because of MQTT’s report-by-exception behavior, in combination with analog I/O deadbanding in each edge controller, the new infrastructure has also reduced bandwidth consumption, allowing Waterford to publish even more data than before. They have access to communications and controller diagnostics (such as update latency, connection timestamps, message size and firmware version), which were impossible in the old system.

Waterford’s cloud-based infrastructure also enables greater flexibility and reliability. Perceptive can perform controller updates over the air, which has reduced travel time and allowed them to continue project development unabated throughout the COVID-19 pandemic. If there is an issue connecting to the data center in Ohio that hosts the new SCADA server, Fisher can have the system up and running in a different data center in 30 minutes. In time, he has the ability to set up full server redundancy.

A recent internet outage at the Department of Public Works offices provided an unexpected test of the new system, which continued working.

“We only lost the old system,” Fisher said. “Our internal [clients] couldn’t reach out, of course, but our [Apple] iPads could connect through Verizon… and I was able to get back in touch. In a situation like this, the old system couldn’t send out alarms because it depended on a local connection. The new system didn’t even notice or care because it’s not running anything local.”

Figure 6: Waterford DPW’s control screens provide an overview of operations. Courtesy: Opto 22

More upgrades, more data from instrumentation

Waterford will continue to manage a few sites through their legacy SCADA system until the end of 2022, by which time they expect to complete all remaining upgrades.

With huge increases in bandwidth, the low administrative overhead of MQTT Sparkplug, and edge controllers providing spare data processing in the field, Waterford can continue expanding its system for a very long time. Each new device or application they add only needs a connection to the MQTT broker to produce or consume data for/from the whole system.

“We are still trying to figure out what else we can do with this,” Fisher said. “We have a lot of other instrumentation that we want to be able to pull data from out in the field that wasn’t really feasible before… not just at our lift stations and our treatment plants but throughout the organization. Where can we use [MQTT] with flowmeters? Where can we use it throughout all of our assets to give us a better overview? We’re just beginning that journey.”

Josh Eastburn is director of technical marketing, Opto 22. Edited by Mark T. Hoske, content manager, Control Engineering, CFE Media, mhoske@cfemedia.com.

KEYWORDS: SCADA, MQTT networking, wastewater controls

LEARNING OBJECTIVES

CONSIDER THIS

How can modern wireless networking, edge computing, and SCADA upgrade help your applications?

ONLINE

www.perceptivecontrols.com

https://waterfordmi.gov

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.