Cybersecurity in the Power Grid – A 360° View | Part 3

Welcome to the 3rd episode of our Energy Talks miniseries, called Cybersecurity in the Power Grid, in which we provide you with a 360-degree view of how power grids can best safeguard their infrastructures from cyber attacks.

In Part 3 of our cybersecurity miniseries, we take a closer look at power grid security from the perspective of cybersecurity analysts. Join us as we dive into the common problems faced by customers and the dedicated efforts involved in finding practical solutions.

OMICRON cybersecurity analysts Christoph Rheinberger and Eric Heindl share their personal experiences, recommendations for IT and OT security officers, as well as valuable insights into how to navigate the ever-evolving landscape of cyber threats.

They also provide us with a deeper understanding of power grid cybersecurity and how to enhance protection in this vital industry.

Stay tuned for upcoming episodes in our Cybersecurity in the Power Grid miniseries.

Listen to the podcast episode

Scott Williams, Host [00:00:07] Welcome to Energy Talks, a regular podcast series with expert discussions on topics related to power system testing, data management, and cybersecurity in the power industry. My name is Scott Williams from the podcast team at OMICRON, and I will be your host. 

Hello, everyone. Welcome to our special Energy Talks miniseries called “Cyber Security in the Power Grid”, in which we provide you with a 360-degree view of how power grids can best safeguard their infrastructures from cyber-attacks. 

In this third episode of our cybersecurity miniseries, we take a closer look at power grid security from the perspective of cybersecurity analysts. Join us as we dive into the common problems faced by customers and the dedicated efforts involved in finding effective solutions. 

My guests in this episode are OMICRON cybersecurity experts Christoph Rheinberger and Eric Heindl. They share their personal experiences, recommendations for IT and OT security officers, as well as valuable insights into how to navigate the ever-evolving landscape of cyber threats.

They also provide us with a deeper understanding of power grid cybersecurity and how to enhance the protection of this vital industry. 

Hey, welcome, Christopher! Welcome, Eric! Could you please introduce yourselves and share the experiences that led you to become cybersecurity analysts at OMICRON? Christoph, could you start? 

Christoph Rheinberger, Host [00:01:37] Yeah, sure. So first off, thanks for the invitation. My name is Christoph, and I have been working for OMICRON as a cybersecurity analyst for the last three years. I started right at the beginning of 2020 as a master’s thesis intern, and I wrote my master's thesis at OMICRON about network topologies and how to detect network topologies. After the internship, I continued to work as a cybersecurity analyst, and that was also my first real work in the cybersecurity field. Before that, I studied computer science, and I found my way into the whole topic and field. I have really enjoyed the journey so far. 

Scott Williams, Host [00:02:27] Okay, great. Hey, you mentioned that you wrote your master's thesis in computer science about network topology detection. What is that exactly? 

Christoph Rheinberger, Guest [00:02:37] So the network topology explains how a network is built up, how devices in a network communicate with each other, and network topology detection. The idea was to figure out the topology just by listening in to traffic so that we know the logical way the network is built up, like which devices are connected to each switch, what switch is connected to other switches such things. We managed to figure out a way to do this and utilize the information. 

Scott Williams, Host [00:03:14] Okay. So, you're pretty much mapping out the network. 

Christoph Rheinberger, Guest [00:03:17] Exactly. 

Scott Williams, Host [00:03:18] Interesting. Eric, please describe your background. 

Eric Heindl, Guest [00:03:22] I'm Eric. I started working here at OMICRON as a cybersecurity analyst in 2022, Before I studied information technology and system management at the University of Applied Science in Salzburg. So, I'm coming from the outside and currently learning my way into OT, so into the OMICRON products and environment. 

Scott Williams, Host [00:03:44] So Christoph and Eric, what sparked your interest in power network security, and how has it shaped your professional journey? Christoph, what about you?

Christoph Rheinberger, Guest [00:03:52] I always was interested in the cyber security topics, so back during studies, I always dove into the cybersecurity stuff you could read up on and also join, like courses and lessons at the university, but usually that was more like the classical IT-related things you would learn and hear about. So, the whole OT specifics I dove into at OMICRON for the first time, but I think that was really nice to do so because we had a lot of or we still have a lot of OT specialists, and their OT environment were able to help us a great deal to get into the topic, to get into the whole world of what OT is in comparison to IT. So yeah, it was an interesting journey, and it will probably also be an interesting journey in the future. 

Scott Williams, Host [00:04:44] So Christoph, I've been speaking with other guests and prior episodes about this. Could you briefly describe these differences between IT and OT and their different cybersecurity approaches? 

Christoph Rheinberger, Guest [00:04:56] So I mean, I was just without a few major ones. The first thing is when I first saw an OT network, and I got that question from many other IT people during my last three years working at OMICRON as well. Why isn't stuff encrypted? Stuff like that. In OT, we do not encrypt like normal control commands like, you know, 6150 mms, stuff like that. And for an IT person, in the beginning, this looks weird. Also, on the other side, for example, vulnerability management. So, in IT, you just have your “Patch Tuesdays.” You just apply the patches, and that's it in IT. I mean, it's very simply put. Yes. But, in OT, you get the patches, and then you can't apply them because you cannot just shut off the device, for example. These are two major differences. There are a lot more, but I would say to these two. 

Scott Williams, Host [00:05:52] Very good. Okay. Well, thank you, Christoph. Eric, what sparked your interest in power network security? 

Eric Heindl, Guest [00:05:58] Well, cybersecurity has always been an interesting topic and gets even more exciting with all the latest cybersecurity news and activities. Like every day, there's a new vulnerability, there's a new exploit. And yeah, what really surprised me in the last year was how many companies still kind of undervalue cybersecurity, especially in the OT sector. It's like they're not fully aware of all the risks and consequences. So, I'm happy and excited to be part of this journey in this cybersecurity landscape and try to help make the network more secure. 

Scott Williams, Host [00:06:33] Very good. So, Eric, what do you think leads to this lack of awareness of cybersecurity risks and the consequences in the OT world? 

Eric Heindl, Guest [00:06:41] It's a difficult one. As Christoph already mentioned, there are very old devices out there, devices from which didn't see a patch for ten years. So, it's a lot of work for OT guys to keep track of all the vulnerabilities and all the versions, so it's not so easy to patch and later new versions on it. So, I think that's a problem. It's too much. 

Scott Williams, Host [00:07:06] Well, thank you. Could both of you walk us through a typical workday as a cybersecurity analyst? How do you encounter ad hoc tasks or unexpected challenges? Christoph, may I ask you to start? 

Christoph Rheinberger, Guest [00:07:19] Yeah, sure. So, I mean, in the last three years, I worked as a cybersecurity analyst, and I had many different tasks to complete, like problems to analyze, and it's very hard for me to pinpoint what a typical day looks like for me. So, I could just give a few examples. So first, of course, we have customers requesting, like, what is this all about? Just sending us some issues they had with the network or some traffic they found in the network, asking us what it's all about, or if it's dangerous, if this should be prohibited, if we have any type of hints on how to interact with a problem. That's one or another one is just giving different kinds of people, employees from customers or just colleagues at OMICRON, like training and educating them to help them to get into the topic easier or just giving them an overview, like what is pen testing? I mean, for example, how do you match vulnerabilities to a device? So just a very broad area we must cover. But to be fair, I think my workday or our workdays, Eric's and mine, are just a little bit different than the workday of a normal cybersecurity analyst in, for example, a Security Operations Center or SOC, which is or usually you would expect to cybersecurity analysts to analyze a lot of things, and which we do. But as I said before, it's not just that. It's a lot more so if, for example, we do not have anything to analyze right away from a customer request. For example, we try to help improve our product, help the development team to investigate scenarios and functionalities, to do proof-of-concept in that direction, or just help with the overall security of the product. Just to try to attack to the device, just break in, stuff like that. I mean, to come back, primarily, our job is to react to customer requests related to our ideas for StationGuard and, for example, vulnerability management and grid ops. And yeah, this vulnerability management is also another big topic. Eric and I work on a lot to improve the database to improve matching, and this is a lot of maintenance to be done to stay up to date with all these topics. 

Scott Williams, Host [00:09:57] Okay, So the customers you are working with, they are actually using OMICRON cybersecurity products, is that correct? 

Christoph Rheinberger, Guest [00:10:04] Yes. So, for example, proof-of-concepts are actually customers using the device. So, in their everyday work, they have all the possibilities here. 

Scott Williams, Host [00:10:14] Eric, what is your opinion? 

Eric Heindl, Guest [00:10:16] Yeah, I think Christoph already mentioned nearly all the tasks of our working day, but there's no such thing as a typical day, as you call it. Each day is different. One day, we have to perform network attacks in our labs, and the next day write code for our vulnerability management. But there's one thing I typically do every day. That's like reading through all the newly published vulnerabilities and exploits. This is kind of frightening but also fun to see. So, all in all, our days are filled with a lot of different interesting tasks. 

Scott Williams, Host [00:10:51] Okay, So Eric, you mentioned that you're doing a lot of research into new vulnerabilities. Are you building ways to protect against these vulnerabilities and the products as you go? 

Eric Heindl, Guest [00:11:02] Yes. So, what we do we crawl on our vendors, like the vendors, which are typically used in OT networks and see if they have new vulnerabilities. If they have, we write them in a machine-readable format and build them into our products. 

Scott Williams, Host [00:11:17] Very good. So that's just like with any other software program; there are vulnerabilities that come up time and time again, and with each new software release, they build in those protections.

Eric Heindl, Guest [00:11:29] Yes, Basically, yes.

Scott Williams, Host [00:11:30] Okay. Could you both share some notable experiences you've had onsite with customers? What are your findings, and what personal learnings or challenges did you encounter? Christoph let's start with you.

Christoph Rheinberger, Guest [00:11:43] The site's visits have been quite uneventful, which sounds bad, but it isn't because they were remarkably interesting. So, with uneventful, I mean there was nothing malicious going on in the network when we were there. There was nothing too out of the ordinary. There were functional problems and stuff like that. And we've seen a lot of different networks. So very simplistic ones through to very sophisticated, complex ones using like various techniques to make an attack cause life more difficult. One of the highlights I’ve seen too is one of these very sophisticated networks, very sophisticated substations and we were tasked to get a day and were able to try to attack this substation, which was de-energized, like in real. So, we were allowed to connect to the network and try to perform attacks on devices, which was very difficult because it was such a sophisticated network design. But in the end, they also just returned, for example, the IEDs, the devices in the network, back to their pre pen test configurations, just to make sure that nothing was changed because although it was still de-energized, we could still damage some of the equipment if we just to control commands or stuff like that or just changed the configuration of these devices. So, they had to return them to their pre-attack state to make sure that nothing was changed so that it could go online afterwards. So, one of the other highlights was we were tasked to do an analysis of some simulated attacks and malicious activities, which was requested by a utility in northern Europe. And they performed cyber-attacks on the realistic power grid automation network. And it was our job to detect the attacks and explain what they did. So, they did this as a contest between different companies. And we were one of these companies participating. So, this was actually a very nice opportunity to witness someone else performing malicious activities in a monitored network. Usually, Eric and myself are the ones who showcase the capabilities of our products, doing attacks on our network and showing how our devices would recognize them. This time around, it was a lot more fun to do because we had to check our wits with no previous insights on what they actually did on a network. So, we just had our knowledge. We had the events StationGuard alerted, we had the visualization of grid ops, and then that's it. And we had to check what they did during that time. And this was just a lot of fun because they performed quite an array of different attacks on a network, ranging from reconnaissance like port scanning very simply or just physical man-in-the-middle attacks and to just straight up replace a device. Our goal or our job was to identify these things. And I might add, we were quite successful in doing so. So it's just a lot of fun when you see them trying to perform an attack, for example, and when it fails because a device in the middle blocks it because it was set up properly, and then you see them connecting to the blocking device and just changing some configurations and afterward the attack is successful and you just know that that's exactly the issues we usually run into because it's just a lot of fun to analyze the stuff and figure out what happens in a network. And it's just like a “I know what you did there feeling”. It was a very interesting experience for us as well. It turned out that we have been the fastest and the most thorough of the participating companies, which is also very nice.

Scott Williams, Host [00:15:54] So okay, so they must have had a list of attacks that they performed. How many would you say that you were able to detect?

Christoph Rheinberger, Guest [00:16:02] I would say about 99%. There are just a few things we didn't. We're not sure what they did, but their response and their feedback to us was very positive. So, all the things they actively did, we found. Maybe some things like that just happened in a network in parallel we didn't.

Scott Williams, Host [00:16:23] So okay, Eric, do you have anything to add? What was your experience with that whole exercise?

Eric Heindl, Guest [00:16:29] It's really nice to see our products in action on the customer side, and to see how well it worked was amazing. Ans analyzing the attacks was challenging but also fun. It's like playing detective to uncover hints and clues and find the correct attack mechanism.

Scott Williams, Host [00:16:46] Interesting. So, going back to your day-to-day tasks, would you say that's very similar to what you've always put up against, identifying new possible approaches for attacks and building blockers for them into your protection devices?

Eric Heindl, Guest [00:17:00] The example described by Christoph was not a typical task for us, but yes, we analyzed a lot of networks and traffic, but mostly to find possible attack vectors and issues in the system. For example, we often see devices broadcasting system information into the network. This makes it easy for an attacker to choose the correct attack to be successful. So, during our analysis, we tell the customer how to minimize the attack vectors for their networks.

Scott Williams, Host [00:17:27] So what are the common reasons customers approach you, or why do you approach customers?

Eric Heindl, Guest [00:17:33] Yes, so the cybersecurity sector is always changing. Every day, there can be new vulnerabilities, and it's very, very difficult to know every threat in your company. That's why often the experience of experts like us is needed to find all those security issues, especially if you don't have a big team, a security team doing this for you. In the OT network like each port, not your services or unknown devices, can be an attack. If you don't have the knowledge and experience of security and critical IT and OT protocols used for those attacks, tt's very, very difficult to find those attack vectors and to close them.

Scott Williams, Host [00:18:15] All right, very good. So, how do you go about developing this knowledge and experience? Is it just with your day-to-day exposure to all the different types of attacks that are taking place?

Eric Heindl, Guest [00:18:26] Yeah. So, we see when we analyze networks or also perform attacks, we see how it's done. I also like researching or reading the news. See other people or other hackers, as I call them, to listen and try to see how it's done and get all the knowledge and give it to the customer.

Scott Williams, Host [00:18:46] So, as cybersecurity analysts, what recommendations do you have for our listeners? Why is your work important, and what can IT and OT security officers and engineers do to mitigate hacker threats?

Christoph Rheinberger, Guest [00:19:01] So I'll just start here, and I'll jump start with a very simple one. Train your employees about cybersecurity. I think this is a very crucial step, although the road to a proper cyber-secure setup is one with a lot more steps than this when you consider that the breach usually has some kind of human interaction in the initial stages of the attack, and additionally to that, creating a proper asset inventory, defining your DMZs, your demilitarized zones, and utilizing just a general defense in depth strategy or some keywords that all of us have heard so far, maybe one or two times too often already, but there are still very valid points to make. And the interaction between IT and OT has to be a corporation, if that makes sense.

Scott Williams, Host [00:19:53] How can IT and OT technicians develop better cooperation?

Christoph Rheinberger, Guest [00:19:57] I mean, for us with our products, for example, StationGuard, we try to talk to the language of both worlds. So that's an OT engineer knows what an alert is in functional terms. For if something happens to OT, the engineer just knows, “Hey, this is that thing doing the other thing”. And for the IT Engineer, that alert or alarm just must be informative. What's going on? But without the knowledge of the functional foundation of what, for example, this protocol does, it's tough for an IT officer to do something to make a proper decision on that topic. So, it's crucial for both of them to interact so that the whole process is designed in a way that these two parties work together well.

Scott Williams, Host [00:20:50] So it sounds like you are developing solutions that both IT and OT can relate to.

Christoph Rheinberger, Guest [00:20:55] Yes, exactly.

Scott Williams, Host [00:20:57] OK. Thank you. Eric, do you have any recommendations?

Eric Heindl, Guest [00:21:01] Yes. A big recommendation from my side is to know your network and have an emergency plan for cyber-attacks as well in IT but especially in OT networks because every new device in your system can be a potential hacker. It's nearly impossible to perform an attack without an attack undetected. So, if you detect and act fast, you can minimize the impact to a minimum.

Scott Williams, Host [00:21:27] Very good. So, you mentioned having a plan for cyber-attacks. How do you go about putting together such a plan?

Eric Heindl, Guest [00:21:34] There are great incident response guidelines available for such a scenario. Like from NIS or ISA, which cover this topic. The first step is always to prepare for a cyber-attack. This includes educating the employees about such an event or implementing tools to limit the attack surface. Next is that detect and analyze the attack type. A good approach is always an IDS system, which shows you where the attack is coming from and if it's already spread in the network. After identifying the attack, it's very important to stop that attack and to recover the system. It can be very trivial, like changing the password, or it can also be very, very complex, like changing the hardware or even decrypting the system. For recovery, mostly backups are used. And the last step is post-incident activities. In this phase, countermeasures should be defined and implemented so that similar incidents are handled better in the future. As well as testing the whole process from time to time is really important. Like a fire alarm in a company.

Scott Williams, Host [00:22:40] Okay. Well, thank you. Are there any specific actions or practices you wish more people would adopt to enhance cybersecurity in the power provider industry? Christoph, what do you think?

Christoph Rheinberger, Guest [00:22:52] So, as I said before, adapting the workflow of cybersecurity topics to involve specialists of both worlds is crucial. So, the networking and security knowledge of IT, combined with the knowledge about the OT systems in question, should be the basis for building a proper defense. So, on the other hand, what is good? What good is an analysis of a functional operation? If you do not know what is required in the system, if that functional operation is benign or malicious, and what if the operation is performed non-maliciously but without information OT can provide? You spent a lot of time investigating something that may be seldom but still required in a network. This is also the false positive, usually called, So, this is why we try to provide interfaces to both the CS analysts and the OT specialists. So, in a way that both can work with it properly. I also need help from people who understand the industrial protocols far better than I do when I'm analyzing stuff a customer just sends in as a request or a problem they encountered. So, as in the previous podcast from Orsan and Thomas Wolfe, they're usually the ones I go to if I have questions about some specific industrial protocols. You're pretty much lost if you try to do this alone without help from people who experts in the specific fields are. So that is, I just can't drive this point home too often, just the cooperation. It's not an OT versus IT. It's an OT in cooperation with IT. That's just not a way we can do without.

Scott Williams, Host [00:24:43] Very good point. Eric, do you have anything to add?

Christoph Rheinberger, Guest [00:24:46]: Yeah, I think the most important thing is to be aware of the consequences and the risks. So, a lot of companies or customers we also see are not aware of how big the consequences can be if the attack was such an attack was successful. So, I think the awareness would be higher than it is right now.

Scott Williams, Host [00:25:05] Very good. Okay. Looking ahead, what do you envision for the future of power utility cybersecurity? Are there any emerging trends or technologies that will shape the landscape?

Eric Heindl, Guest [00:25:18] Yeah, I would start here. I think one thing we observed especially is the increasing use of artificial intelligence. So, AI tools are now employed both in crafting cyber-attacks and in creating defense strategies. The dynamic interactions lead to a cat-and-mouse game. By staying up to date, cutting-edge technologies become crucial to avoid successful cyber-attacks. As the cyber landscape continues to involve robust cybersecurity measures, proactive defense will become essential to protect critical key infrastructures.

Scott Williams, Host [00:25:55] Very good. Okay. So, what developments do you anticipate for the future of power network security? Are there any emerging challenges or opportunities that power providers should be prepared for? And how can OMICRON support them in navigating this evolving landscape? Christoph, do you have anything to say to this?

Christoph Rheinberger, Guest [00:26:15]: Let's just look at one of the topics we think is our expertise. Like, for example, vulnerability management. How vulnerabilities were published before, like. If they were published at all, and how the Common Security Advisory Framework, also known as CSF, starts to be adopted more and more by vendors of devices in the OT world. I think that we can provide our customers with the needed knowledge and the proper tools to utilize this. So, I mean, vulnerability management always has been and always will be a big part of how you can secure any network, not just power networks. So, with all these things being standardized with CSF, for example, this gives us the opportunity to provide the customers with all the needed two worlds to actually do the manual work they had to do before matching vulnerabilities to devices in an automated way, just to help them get a lot of the workload done quicker so they can focus on other topics and other fields, which also need a lot of attention. So, it's just one way OMICRON can help with our tools and the knowledge we have.

Scott Williams, Host [00:27:41] Eric, what developments do you anticipate?

Eric Heindl, Guest [00:27:44] As Christoph already mentioned, ability management is a big topic currently. One thing is to publish vulnerabilities, but another one is to match them to the network devices. The standardized framework described already by Christoph helps, but it's still not used by all vendors, so implementing a new device into the network requires days of checking vulnerabilities for this specific device. Therefore, a tool matching vulnerabilities to devices is really needed in the industry in the future. And that's a topic that we are heavily developing right now. To wrap it up, I think the future in the security sector is an interesting one with a lot of innovation and development, and I'm really looking forward to it.

Scott Williams, Host [00:28:26] Well, thank you both for joining me for this third episode on cybersecurity in the power industry. And I thank you very much for your insights and your recommendations for solving common cybersecurity problems in electrical power networks.

Christoph Rheinberger, Guest [00:28:39] Thanks a lot for inviting us. It was interesting and nice.

Eric Heindl, Guest [00:28:42] Thanks very much.

Scott Williams, Host [00:28:43] And a big thank you to our audience for listening to this episode of Energy Talks. We always welcome your questions and feedback. Simply send us an email to podcast@omicronenergy.com. OMICRON has several years of experience in power system testing, data management, and cybersecurity in the power industry and offers you the matching solution for your application. For more information, be sure to visit our Web site at www.omicroncybersecurity.com. Please join us to listen to the next episode of Energy Talks. Goodbye for now, everyone.