In 2015 Snohomish County Public Utilities District (SnoPUD) in Washington state invited a trained team of hackers from the Washington National Guard to test its ability to detect, deter and thwart cyber-attacks as part of a two-week ‘penetration test’. The penetration test was designed to assess SnoPUD’s detection and response procedures to attacks in real time, exposing weaknesses in the utility’s contingency plans. During the exercise, SnoPUD’s smart grid test lab was the target of the attack, as it replicates actual systems, from EMS, SCADA and substations, as well as battery storage devices and distribution automation systems. The hackers sought to attack the utility’s operational systems, such as substation controls, along with administrative systems, such as human resources and customer service.
The infiltration was achieved using publicly available, off-the-shelf tools found at any computer store. In fact, we could not use tools that were exclusively available to the US Department of Defense. While described in some news outlets as “groundbreaking,” I’ve actually participated in a number of these types of assessments. The truly innovative element is that SnoPUD is the first facility to actually go public with its results.
The end result was that, despite some pretty sophisticated systems and a number of defensive actions, the hackers were ultimately able to breach SnoPUD’s defences within 22 minutes.
From the reports circling around, the sentiment seems to suggest that SnoPUD did not have the appropriate cyber defenses in place. This couldn’t be further from the truth. In fact, SnoPUD was not only in complete compliance with NERC CIP, but it also followed best practices that aren’t enforceable. The facility even went above and beyond by investing heavily in cybersecurity protections.
In fact, I was surprised at the level of defense they had installed. In its corporate network, they had firewalls, endpoint protection and antivirus software from some of the most reputable security companies. On the operational (OT) infrastructure, SnoPUD separated its supervisory control and data acquisition (SCADA) network from the corporate network. Because of the robust protection in place, SnoPUD owners and operators were completely blown away with how quickly our team was able to infiltrate the systems.
The SnoPUD assessment
There were two pieces to the assessment. First, we had to gain access to the corporate network, which, predictably, wasn’t difficult at all. Then, once we had this access, our goal was to jump over the smart grid network. I was quite surprised at how quickly we were able to perform this manoeuvre.
Despite having robust security in place on the corporate network, we did not see any security tools in the SCADA network. It was clear that investments were heavily placed in the corporate network in order to make it difficult to access the separate, SCADA network. However, once that jump is made into the SCADA system, there is no way of detecting, mitigating or stopping an attack. It was quite eye opening to see the disparity in the amount of money invested in corporate versus SCADA security. As we made our way through the systems, we found security protection up until the last machine. However, once we successfully made the leap from the corporate network to the smart grid components – there was nothing there. Unfortunately, the truth is there aren’t a lot of solutions available to defend SCADA and other industrial control systems (ICS).
The issue with IT-OT convergence
The OT side of an organization is traditional, with old school systems that are run by old school operators. However, IT connectivity is being bolted on top of these OT systems, forcing a convergence of two historically disparate entities. Which, as you can imagine, has created many challenges within organizations, like the Snohomish Public Utility District.
You see, ICS engineers are still working from a non-connected perspective. They have the mindset that the solution is to take the network offline and remove the IT element altogether. The problem is, you can’t do that anymore. SnoPUD, for example, is running power distribution for a significant part of Washington, and it is simply not possible to do so without connectivity. If there is a crisis in two or three locations, you must have remote management capabilities or risk significant outages. The line of IT-OT demarcation has come and gone, and professionals on both sides must understand the reality of cyber risks.
While describing possible scenarios and conducting tabletop narratives may start the conversation, there had yet to be real data brought to the table until SnoPUD. I have been in this industry a long time, and most of the work I do I can’t talk about to anyone. Yet, general scenarios as to what might happen to a power facility goes right over people’s heads. SnoPUD is the largest public utility in the state of Washington, and it is recognized across the nation; so when it comes forward with evidence of the robust security it had in place – and how it didn’t work – people will listen. This openness allows for better conversation, as it comes from within the community. Looking back, one of the really positive pieces of this assessment is that it was directed by the governor. For the most part, these types of initiatives are reactive and don’t take place until after a major event occurs. Yet, the governor realized that the power plants, ports and critical infrastructure in the state support the entire nation and need to be protected.
State-funded cyber evaluations are not limited to Washington. South Carolina and Maryland have both conducted similar assessments, and Texas is looking into it, but that is still only four out of 50 states. While I expect more information to come out about the SnoPUD assessment, I also predict many states across the nation will conduct similar penetration tests.
After all, it’s truly a win-win situation: the assessment team gets to enhance their cybersecurity skills and the organization being assessed receives the ammunition it needs to better defend its systems and assets. If this test accomplished one thing, it’s that it reinforced the immediate need for very specific ICS cybersecurity to be adopted at our nation’s critical infrastructure.
This article first appeared in Metering & Smart Energy International Issue 2, 2016.