Cybersecurity in the Power Grid 
A 360° View | Part 7

Welcome to the 7^th episode of our Energy Talks miniseries, Cybersecurity in the Power Grid, in which we provide a 360-degree view of how power grids can best protect their infrastructures from cyber attacks. 

In this episode, we learn how an OMICRON team of cybersecurity experts approaches the analysis of incoming alarms and how even the smallest indicators can reveal hidden dangers that threaten the security and operation of an entire system.

This was put to the test by Norwegian utility Glitre Nett. They conducted an Intrusion Detection System (IDS) evaluation exercise at a live substation to compare solutions from several vendors, including OMICRON.

OMICRON cybersecurity experts Ozan Dayanc, Christoph Rheinberger, and Lukas Schneider join us in this episode to describe their involvement in the Glitre Nett exercise and discuss how they successfully completed it using OMICRON’s StationGuard intrusion detection system. 

The project involved executing live attack simulations in the substation to compare the detection mechanisms of the various IDS solutions. Glitre Nett also wanted to test the detection response and support capabilities of the cybersecurity experts at each vendor company. 

How well did we perform?

Scott Williams Welcome to Energy Talks, a regular podcast series with expert discussions on topics related to power system testing, data management and cyber security in the power industry. My name is Scott Williams from the podcast team at OMICRON and I will be your host. 

Hello everyone! In this episode, which is part seven of our mini-series called Cybersecurity in the Power Grid, we will learn how an OMICRON team of cybersecurity experts goes about analyzing the incoming alarms and how even the smallest indicators can reveal hidden dangers that threaten the safety and operations of an entire system. This was put to the test with the Norwegian utility company Glitre Nett, they conducted an intrusion detection system evaluation exercise at a live substation to compare the solutions of multiple intrusion detection system vendors, including OMICRON. The project involved executing live attacks simulations in the substation to compare the detection mechanisms of the solutions. Glitre Nett also wanted to ensure the detection, response, and support capabilities of the cybersecurity experts at each of the vendor companies. Three on the part of the cybersecurity experts joined me in this episode to describe their involvement in the project with Glitre Nett and how they successfully completed it using OMICRON, the StationGuard intrusion detection system. Let me introduce them to you. First of all, we have Ozan Dayanc. He is the technical product manager for OMICRON StationGuard and GridOps Solutions. Ozan has been a guest before on EnergyTalks. Welcome back Ozan. 

Ozan Dayanc Hi, Scott. Nice to hear you again. 

Scott Williams  My next guest is Christoph Rheinberger. He is an OMICRON cybersecurity analyst who has also been a guest on energy talks. Thank you for joining me again, Christoph. 

Christoph Rheinberger Hello. Nice to be here again. 

Scott Williams And my third guest is Lukas Schneider. He is a full stack developer in the OMICRON cybersecurity team. Lukas, welcome to EnergyTalks. 

Lukas Schneider Hey. Thank you. 

Scott Williams Well, great. I would like to get deeper into this topic. Ozan, can you describe the complete situation here? Why did Glitre Nett approach OMICRON for this attack simulation test? 

Ozan Dayanc  Yeah. Before I answer that question, a little bit information about Glitre Nett. Glitre Nett is one of the biggest electricity distribution system operators in Norway. They provide electricity to around 317,000 people. Glitre Nett was already an OMICRON customer for our other business units. So, they knew us already, and we knew them. And as you know, OMICRON specializes in power grid environments in primary, protection, and in the networks as well. And this project sounded innovative. And you know how OMICRON likes to be involved in innovative projects. So therefore, we were up for it, and they were also confident that we were the right candidates for this project. 

Scott Williams What was the real purpose behind this test? 

Ozan Dayanc Indeed, that is a pertinent question. The objective of this test is to ensure the security of the substation automation systems, which represent the critical infrastructures for this DSO, distribution system operators. To achieve this, a series of measures must be taken. Monitoring is one such measure. The objective was to identify the optimal monitoring solution, or intrusion detection system solution, for their critical infrastructure. Consequently, they initiated an evaluation project to identify the optimal solution for this environment. This involved seeking out ideas, solutions, and conducting attack and simulation tests with vendors. This is how the project began.

Scott Williams Very good. Well, thank you for that explanation. So, can someone explain the test set up to me? How did you initially receive the data? 

Christoph Rheinberger I can answer this. So, our sensor was sitting in the live substation and gathering the data during the conduction of these attacks. So, after the attacks were done, we got the configuration files for StationGuard, the event logs, and all the PCAP that were created on our system to analyze what happened. So, we could either use direct access to the installation on site or a local replica of our system. All additional information regarding the collected data we needed was provided on demand during this evaluation. 

Scott Williams How can attackers infiltrate systems and who are they targeting? 

Ozan Dayanc First, we must examine the profiles of these attackers to gain insight into the motivations behind these attacks. In this context, we can identify three distinct profiles. The first is comprised of individuals who are relatively inexperienced. They are engaged in exploratory activities, attempting to learn through the use of new scripts and attacks. While these individuals may not pose a significant threat, they could potentially evolve into more dangerous actors. The second profile is comprised of professionals who engage in this activity for financial gain. In light of the recent spate of ransomware attacks, it is evident that there is no political motivation behind many of them. However, last but not least, the most sophisticated attacks are those carried out by state actors, who are the most dangerous of all. In terms of attackers, we can identify these three distinct profiles. Christoph, would you like to add something to this? 

Christoph Rheinberger When discussing who the attackers are targeting, we focus on the entire OT environment and critical infrastructure. Yes, they primarily target critical infrastructure. The specific targets can vary, but our expertise lies in protecting critical infrastructure. To understand the attack scenarios conducted, we need to consider the different types of intrusions an attacker might use. These include physical intrusion, remote connections that can be exploited, and portable devices that are dropped on the premises and plugged in by an unsuspecting or careless employee. There are also device vulnerabilities that can be exploited or equipment that a regular employee might bring into the substation for daily use, like testing equipment and engineering notebooks. There are many ways to infiltrate these systems. Returning to the specific Glitre Nett simulations, Ozan, you organized the entire project. Could you provide some insights into how they approached these attack simulations?

Ozan Dayanc Sure. While listening, I noticed some of the attack vectors used in this project, and they are indeed commonly targeted. Regarding our project, Glitre Nett took a unique approach by conducting the attack simulation in a live substation. I have traveled between utilities worldwide and have never seen such an approach before. Three IDS vendors participated, including our own StationGuard. The goal was to find the best solution through this attack simulation. Glitre Nett, along with their service provider, executed sophisticated attacks on the live system without disrupting operations. After the attack, each vendor analyzed their system to find solutions that required less effort for a rollout across substations and offered a tailor-made solution. They evaluated not only the IDS systems but also the capabilities of the teams behind them, focusing on incident response as well as detection. This was the approach Glitre Nett took. Christoph, it would also be beneficial to mention the composition of our cybersecurity team at OMICRON, as that is where much of our value lies.

Christoph Rheinberger I will follow the same approach we used earlier. I will start with our OT engineers and those knowledgeable about protocols like IEC61850 and 104. These experts provide essential knowledge to me and other cybersecurity analysts, enabling us to make informed decisions about incidents or alerts in these systems. Additionally, we have team members like Lukas, who assist with device development and offer insights into how we detect and analyze various issues. Our team possesses an extensive knowledge base that can be leveraged at any time. This is a significant advantage. Additionally, it would be beneficial to discuss the tools we utilized. Lukas, would you like to elaborate on that?

Lukas Schneider Sure. So, for this simulation, we used two of our products. One is StationGuard, which provides clear and precise information about the system's status and any security events. This clarity is very crucial for quick decision-making and effective responses to incidents. And moreover, we used StationGuard's central management system GridOps, which is an absolute game changer. It allows analysts to efficiently analyze and mark incidents. This centralized approach means that insights and data are easily accessible and streamline the process of incident management and reduce response times even more. As Christoph already said, together, our team of experts and our solutions create a powerful set up to react to incidents effectively and accurately in this attack simulation. 

Ozan Dayanc I mean, I've seen how you all distributed the tasks and how you allocated the responsibilities. This also played quite a crucial role in this project, right? This planning of things. Lukas, you were there as well as with Christoph. You can extend that a little bit. 

Lukas Schneider  Sure. So first we created a clone of the customer's system using backups and configuration files that were provided us, as you two already mentioned. This allowed us to be very flexible and work with all tools like our customer support. Next, we organized the collecting of the events into specific areas that seemed most interesting using GridOps. This helped us to systematically identify several types of potential attacks, categorizing them by patterns and anomalies simultaneously. We conducted preliminary checks for any obvious indicators, such as the presence of unknown devices connected to the network. This initial scan allowed us to quickly address the most apparent attacks, such as TCP or UDP port scans. And the man in the middle attacks through ARP poisoning. Finally, after our rough initial analysis, we shifted our focus to more puzzling incidences. This requires deeper investigation, as they were less straightforward and often involved complex scenarios that weren't immediately clear from the onset. 

Christoph Rheinberger  So yes, I'll jump in here again. We had to rely on the expertise of various of our specialists because there were issues I couldn't identify immediately. For example, there were functional problems in the substation triggered by certain interactions, whether unprovoked or provoked. We consulted our experts in protocols like IEC61850 to address these issues, such as unsuccessful control commands or timeout requests. There were interactions like MMS interactions that were not allowed, such as the HMI reading values and writing to control blocks while the system was not in maintenance mode. In addition to these issues, there were also more direct attacks. During the simulations, there was significant reconnaissance activity in the substation, including UDP trace routes, ping sweeps, and port scans. Interestingly, the network devices were well-configured to prevent such reconnaissance right away. The attackers had to reconfigure the devices to disable security measures and then enable them again to carry out their attacks. One significant incident was when a physical device was replaced with an identical clone. We detected this because the device wasn't sending data at certain times, and there were suspicious activities in between. For example, right after reconnecting the device, it sent a Gracious ARP, checking if its IP address was still valid by broadcasting it to the network. This was combined with the fact that it had an automatic private IP address, indicating an attempt to connect to the DHCP server while disconnected from the substation. These clues led us to conclude that the device had been replaced, which was indeed the case. The compromised device also attempted reconnaissance activities, such as pings and trying to connect to web servers in the network. All other activities by this server, like MMS connections, had to be scrutinized since the device was compromised. These were not the only attacks we analyzed. We documented many others in our report. As Lukas mentioned, there was persistent DNS poisoning, functional problems like disappearing GOOSE messages when a device was disconnected or denied, NetBIOS reconnaissance, and RTP brute force attacks to access a remote host using the RTP protocol. Additionally, we conducted Asset Vulnerability Matching, although this evaluation wasn't part of the project's scope. Looking at the complete report, we achieved a 99% success rate in meeting our goals overall.

Ozan Dayanc It's worth mentioning that the main tool that helped us detect all these problems was StationGuard. StationGuard identified all these attacks, and the rest was just analyzing, digging into the details, and uncovering hidden issues. Our solution was incredibly helpful during this analysis, which is clear. I must also say that Glitre Nett conducted very sophisticated tests, including diverse types of attacks. It was a good exercise for our team. At the end of the day, as you mentioned, Christoph, we detected 99% of these attacks and provided a detailed report with a timeline of when these attacks occurred. This outcome gave Glitre Nett confidence in our products and our team, as we delivered the report, explained each attack, and demonstrated the depth of our analysis. This project once again proved the intelligence of our solution as the primary detection system. Another crucial point is the collaboration between IT and OT. It's our role to bring these two domains together to secure environments. Our IT experts have extensive knowledge of cybersecurity, having implemented it for a long time, while our OT experts have deep knowledge of the operational environment. By leveraging expertise in both domains, we were able to detect and analyze these problems effectively.

Scott Williams Well, thank you all for that interesting description and discussion. It surely isn't easy for solution providers like OMICRON to continuously react to this variety of attack possibilities, but this is why we continue to improve our products and solutions. Am I right? 

Ozan Dayanc You're right. We can say that we are in an era of agile development. Digitalization is everywhere. We can see it in the substations, control centers, and critical infrastructures. The market is too dynamic, requirements are keep changing because it's like an accelerated development. The regional requirements, between continents, for example make it even harder because yes, the requirements are similar to each other, but there could be slight changes between, for example, European security requirements and American security requirements. But, at the same time, these market requirements and regulations help us through the developments because they are like a guidance. We closely monitor these global security standards and listen to our customers' needs. By merging this knowledge and feedback, we continuously enhance our solutions. Of course, the main challenge here, let's say, what plays the most crucial role is prioritization because that gives us the momentum to implement all these requirements. For example, out of this exercise there was a missing 1% of the attacks, right? We keep saying 99%. And this was due to a minor role to behavior issue at our solution. And this was really a minor thing that would have been nipped in the but immediately. And then we improved on that and installed it into the StationGuard for a even quicker response time. So that's why it was good exercise for us. There were a lot of takeaways from our side as well. So, if you would participate in this attack simulation again, we would detect 100% of these attacks that have been conducted by the system owner in time. And for the future, of course, we have a lot of plans we want to improve. For example, if I go into a little detail, we want to improve our signature base detection and visibility in the network, with more functional security features. Another important topic is improving the asset inventory collection. But we want to improve our product. Because I said inventory collection is a topic for every customer that we visit. Also, it is worth mentioning this valuable exercise can improve our knowledge in the field of incident response. These are our plans. 

Scott Williams Very good. Well, thank you for that summary. Is there anything else that anybody wants to mention before we say goodbye? 

Ozan Dayanc  Yes. I have one last message. The OT security monitoring is a very important topic and becoming a very hot topic for the especially for the target environments. It is important that the solutions provided into these environments need to be tailor-made, focusing on specific domains. That's the first takeaway I want our audience to have. The second thing is that the IT and OT collaboration is extremely important when it comes to the incident response in these kinds of environments. If you have a solution that it can work together with, that's given an additional advantage. And that's what we are promising in OMICRON with our solution StationGuard and with our experts in the team. 

Scott Williams Very good. Well, Ozan, Christoph and Lukas, thank you very much for joining me for this episode of Energy Talks, for this interesting discussion about the exercise with the utility company in their way. 

Ozan Dayanc Thank you very much, Scott. This was fun for me. 

Lukas Schneider Thank you. 

Christoph Rheinberger  Thank you for having us. 

Scott Williams Great. Well, thank you all. And a big thank you to our audience for listening to this episode of Energy Talks. We always welcome your questions and feedback, so please send us an email to podcast@omicronenergy.com. OMICRON has several years of experience in power system testing, data management and cybersecurity in the power industry and offers you the matching solution for your application. For more information, be sure to visit our website at omicronenergy.com. Please try this to listen to the next episode of Energy Talks. Goodbye for now everyone.