Vulnerability Management in Substations and Power Plants

With increased digitalization, the vulnerability of power utility systems to cyber attacks, blackmail and blackouts is on the rise. In this episode, you will discover how power utilities can best manage cybersecurity risks and protect their systems from intrusion.

It is essential that power utilities take proactive steps to ensure the security of their systems, such as implementing vulnerability management processes and tools in substations and power plants to effectively identify, prioritize and handle risks.

In this episode, OMICRON Power Grid Cybersecurity Expert Andreas Klien explains how vulnerability management can lead to more effective cybersecurity in power grids, including the advantages of enhanced system visibility, a comprehensive asset inventory, intrusion detection and proper risk prioritization.

Andreas also describes the challenges of vulnerability management and how OMICRON’s solution, called StationGuard, overcomes these challenges and effectively secures the power system networks it is implemented in.

Listen to our podcast

Scott: Can you tell us about your personal involvement in the protection of substations and power plants?
Andreas: Scott, I have been working with substation automation protocols since 2005. In 2010, the Stuxnet incident triggered a big discussion about power grid cybersecurity. That’s when OMICRON started researching in the field of cyber security in substations and power grid SCADA systems. The hype cycle went down a bit in utilities, until it picked up again in about 2016. I stayed with the topic for all these years.

Scott: What makes you an expert in this field?
Andreas: I have been conducting security assessments in substations and power plants all over the world for many years now. Now I am leading a team of engineers who keep doing this and who work on developing solutions tailor made for these networks. I keep learning new things, every time we think we saw it all in terms of how power grid automation networks are implemented and what could go wrong, we see a new substation or SCADA architecture. For example, how the network architecture in power plants can be different in different countries, which brings new cybersecurity implications.

Scott: Vulnerability Management is a big buzzword. Can you explain to us what this means exactly? What aspects of a system’s cybersecurity are covered by it?
Andreas: Vulnerability Management has been state of the art in the IT domain for many years now. It is the continuous process of “dealing with security vulnerabilities of the software you use”. Vulnerability management thus covers finding vulnerabilities in the software you use, evaluating their impact on your business, and treating these vulnerabilities. One option for treatment is patching the software, the other option would be to just accept the risk and not patching it. There is obviously also a lot of software running in the devices that control and protect our power grid infrastructure. These devices also have security vulnerabilities. There is no software without bugs. When you want to assess the cyber risks of your power grid OT network, then the security vulnerabilities of your devices are one aspect of it. When you do vulnerability management in the power grid OT domain, there are many things different than for classical IT systems, though.

Scott: Why is an effective vulnerability management so important for utilities worldwide? It already works well in the IT domain – why should that be different in the power grid?
Andreas: Vulnerability management is important because you cannot protect yourself against cyber risks which you don’t even know. You need to know about your assets and how their vulnerabilities could be exploited - and on that basis you can protect them. The other reason for implementing vuln. mgmt. is because it is required by several standards and regulations, such as by the ISO 27k series. The fundamental difference between IT and in the power grid OT domain is that you can’t patch the devices in the latter. You can’t shut down a device which is protecting a power line or a generator while the system is energized. So, you must shut the power down. 

How to deenergize when the power grid is overloaded anyway, or when the load situation changes rapidly over the day?
o    You must wait for weeks or months to get approval to de-energize.
o    Until you are done, already the next vulns are published in the meantime and you should start over again.
-    Second problem: How do you know that everything still works in the same way as before the update? Protection tests can be done easily, but what about your custom programmed logic?
o    You cannot really test these functions as you would have to operate breakers and other equipment which you simply cannot do in a live substation.
o    So, the risk of applying a patch can be higher than to not apply that patch.
-    Therefore, you cannot just apply every software patch. You must manage your risks and only apply those patches where the effort makes sense.
-    For this you need an effective vulnerability management.

Scott: Where do you see the biggest obstacles for substation and power plant owners who wish to implement a proper vulnerability management in their systems?
Andreas: The challenge is that you need to know a lot about your devices to be able to assess if a certain vulnerability affects you or not. For example, one vendor issues vulnerabilities for a certain communication module, which has been used in several different protection relays over the last years. The person doing the vulnerability management needs to know that this module is used in many of their relays, then they need to know the firmware version of that module, etc. Therefore, always OT experts are needed in the vulnerability management process. So, IT and OT experts need to work together a lot.

Scott: What challenges do you think need to be overcome for a risk management to be successful? Are there checkboxes on the to do list that need to be checked off? Are some of them more important than others?
Andreas: First, you need to establish a very precise asset inventory to have that knowledge about the devices and their firmware versions.
-    Some utilities already maintain an asset inventory which includes the information needed to do vulnerability management.
-    There are multiple options to get to such an asset inventory and to keep it up to date.
-    One option is to manually do that; some utilities do this in their ERP system or other databases, some of them maintain spreadsheets for each site, or maybe even one big spreadsheet for all assets of an organization.
-    The problem with this is that it is a lot of work and error-prone.
-    The other option is to automatically do this using systems which scan the network either once or periodically.
-    OMICRON provides such a system with StationGuard. You can run it once to scan the network, but you can also install it permanently to scan permanently for assets.
-    That can also be done together with an Intrusion Detection System.

Scott: Why is a comprehensive asset inventory so uncommon? Shouldn’t it be common sense for engineers and IT specialists to know what assets are in their systems and which firmware is installed on them?
Andreas: Well, you would be surprised that a comprehensive up-to-date asset inventory is also not always common in classical IT, especially if you look at the server networks of organizations.
-    The challenge is that all involved people need to correctly document the changes they made to the configuration and the firmware installed.
-    The problem is also that there are not just protection relays in a station, there are also a lot of other devices like switches, SCADA equipment, IP cameras etc.
-    All this equipment from different teams needs to be listed in one common, searchable database.

Scott: How does a perfect vulnerability management look like?
Andreas: Continuously scans the assets in the networks. It alarms you if there is a new asset communicating
-    it reads out the firmware version directly over IEC 61850 protocol and it detects configuration changes and reads out the firmware versions directly.
-    Needs knowledge about the specific devices and their components.
-    The vulnerability management tool needs to understand, which vulnerability matches to which device. Therefore, it needs to understand also which firmware version and which communication card belongs to which device types.
-    We once calculated the effort that asset inventory and vulnerability management creates. Assuming that all these databases have been established already, it would still take around 1000 hours for a small utility to just go through all security advisories that come in every month and to keep the asset inventory up to date.
-    It is our goal to save this time.

Scott: For over ten years now, OMICRON energy has been researching and developing solutions in the area of power grid cybersecurity. What help can OMICRON offer to utilities?
Andreas: OMICRON provides solutions which have this power grid knowledge built in.
-For example, our vulnerability management solution already knows which communication modules are installed in which protection relays and which firmware versions are affected by certain vulns.
-This built-in knowledge ensures that the IT security officers are equipped with the OT knowledge they need. This saves time of the OT experts. On the other hand, our solutions speak the language of protection and SCADA engineers, which improves the collaboration of these two parties.
-We do also provide cybersecurity consulting specifically for the OT networks in control center, power plants, and substations.
-For example, we are also conducting Security Assessments to quickly see what the risk factors in my substation network are.

Scott: What distinguishes OMICRON’s solution for vulnerability management from other solutions on the market.
Andreas: Over the last years we created a curated database of security vulnerabilities in power utility automation products, which includes protective relays, SCADA, and networking equipment.
-We work together with vendors to add all the most recent vulnerabilities to this database, and we add metadata written by power grid experts to it.
-Our system also uses a purpose-built asset type database which knows which components and firmware versions are used in which asset type and how this comes together with the security vulnerabilities.
-Our StationGuard sensors on-site scan for new assets and they are even able to actively read out the nameplate information of each device.
-With this we can precisely match vulnerabilities and show you only the vulnerabilities which are a risk for you.

Scott: Where do you think vulnerability management will have to evolve to remain secure? What steps can already be taken in this direction?
Andreas: There are certainly many improvements necessary by the vendors to improve their security advisories. It should be easier for security practitioners to assess if an advisory matches their devices or not.
-    Over the last two years we found many examples how these can be improved, and we are already working together with some of the vendors to improve security advisories for the power grid.
-    A big step was that several vendors are now publishing their security advisories in a machine-readable format, the CSAF (Common Security Advisory Framework).
-    We are already using this format and are automatically scanning all vendor websites for changes in the security advisories they publish.
-    On that basis and with the current demand for improving vulnerability management in the power grid, I am sure that many interesting things will come in the future.