We're going to have a panel now and we'll also invite to the stage Frank Fischer from DHL. We're going to continue the conversation about IoT, OT, and ICS.
Frank, would you like to go ahead and introduce yourself? Sure. I'm happy to do that. I'm the CISO of the DHL Group since about 15 months and take care of both IT and OT and report up to the CEO. Great.
Well, Sarb, you started off talking mostly about IoT. Maybe we should start with some definitions of how we see all these different acronyms. So I guess I look at OT, operational technology, as being the bigger umbrella term that includes things like IoT, industrial IoT, industrial control systems, and under that you find PLCs, programmable logic controllers, and HMIs, and SCADA interfaces, and things like that. How do you see the definition of OT and ICS and IoT, and are they converging?
Yeah, I don't think they're converging and there's no reason why they should. There are lots of similarities, but it's the differences that are important. One of those differences is that when it comes to OT, you normally know who's responsible. You know in most organizations that use that OT-type technology, they're going to be the same right across the many different enterprises, and I think that's a key differentiator that's worth thinking about.
IoT, it could be anything from the electric kettle to the fridge, it could be the lighting, it's almost anything and everything, and it will be different people that are responsible across one enterprise to another enterprise to another enterprise. Excuse me a second, sorry.
Okay, yeah, I switched that off. So probably from a practical point of view, I guess we rather look at the OT side of things, so in wrapping in all the sensors and actors and what have you, whether or not they have an internet connection or they have proprietary bus systems, it doesn't matter.
But essentially, I mean, if you talk about the internet of things, you kind of suggest it has a connection to the outside world, which makes it obviously more reachable and therefore more vulnerable, while we currently wrestle more with the legacy of infrastructure that is, or used to be kind of a shrink wrap off the grid and now becomes a part on the grid and is actually not built for it, right? So building systems like bus systems are not intrinsically secure. So you need to think around the corner to make them actually a little bit more secure than they are actually.
Yeah, I think a lot of organizations like manufacturing companies have a mix of both, they're using SCADA, PLC, HMI, but they've also outfitted all their buildings with smart building kinds of technology, various kinds of sensors. And so they've got this whole host of different kinds of devices in play with different protocols and different, maybe insufficient means of trying to secure them. Any thoughts on that?
Well, let's start with an organizational pattern, right? And some of you who are probably into OT, you know, the Purdue model, right? Which structures kind of the devices a little bit more into different, let's say, realms. Not exactly layers, but you could also look at it in a layered model. So it has a large portion that is what we would call IT for OT. So essentially everything up to the control plane, right? So where your control PC sits that has a PLC attached to it, right, for instance.
And then you have the actual controls plane, right, which is below and which really gets down to the factory floor. And those are the ones that are really hard to maintain, to lifecycle, and to protect because they usually have lifecycle times. We mentioned that earlier. I would rather say they are in the vicinity of 20 to 50 years, depending on which industry you look at, and they're just stubborn, right? So and you have a lot of, and it's a vendor-driven environment. So large vendors dominate the space.
They tell you what they think is right and what you need to be doing in order to operate your environment. And they would have clear ideas what they want to have in terms of access, monitoring, delivery, whatever, right? You're sometimes held at gunpoint, right, in order to keep your operation up and running.
Yeah, it's very interesting. You mentioned vendors, and you're absolutely right. Compared to IT and cybersecurity technologies and systems, when it comes to IoT and OT systems, we are at the mercy of the vendors in the sense that most of these things do not easily integrate to the things that we've talked about over the last two or three days. Everything is proprietary. They want to keep it proprietary. They don't want to open it up, and that's one of the challenges that we have got in this sector.
Yeah, in some earlier sessions, there was talk about vendor lock-in. There's probably no stronger vendor lock-in than in the OT world because they provide the support. They provide all the updates, and they severely limit, in many cases, what you can do with the equipment that you're given, which is kind of problematic, too, because if you try to apply traditional IT security controls like endpoint security, you find in many cases that no, the vendor won't allow you to put an endpoint agent on this machine. So then what do you do? Have you encountered situations like that before?
Yeah, surely. I mean, you end up ring-fencing the stuff, right? So if you can't protect it, I mean, you can't update it. You can probably look at it from a timeframe's perspective, right? You have stuff that sits there, and you just inherit it, like marriage would be probably a similarity, right? You get another family you have no influence on, right? So that's what happens when you all of a sudden have OT in your system, right? And then there's a part that really defies updates because it's so old, right? You just need to keep it running. It's long depreciated, right?
And then you just keep it running and running and running, and thereby, there is nobody even around anymore. Maybe the developers have long died because the stuff is so old, right?
Seriously, I mean, there are folks that, I mean, my former employer, we were looking for Windows 3.11 administrators because some of the OT could actually run on it, right? So find some people that are capable of doing that. If they haven't changed profession, they maybe have other preoccupations, right? And so the only way you can actually handle that is to ring-fence it and then put it off the grid, right? So kind of structured networks, literally creating also fenced off networks that are either having data diodes or other stuff, right?
It prevents, let's say, easy access from the outside, right? That's the only way that helps. And the slight difference with IoT systems, even of yesteryear, is that how they moved from non-IoT into IoT, and those systems may still be running today, that the vendor or someone else produced some middleware that would enable that IoT functionality connecting to the internet and so on. So there's a lot of middleware around in some organizations, although the kit behind it is still 20, 30 years old, as you're saying.
But there is, at least in that field, there has been someone somewhere that has looked at updating it slightly, and that's the middleware part, not the end part. So they're updating just that bit, and that's dangerous because it's still sitting on old code that may not have been updated for a long time. And also there's so-called embedded systems. I guess you're familiar with those, right? They can either come as a part of your microcontrollers, right? They run a software stack, right? And that's the nasty part because you can't get to them. It's somewhat shrink-wrapped, right?
You don't even have easy means to access it. Sometimes it's done locally, so you can at least guard the people that actually show up with a USB stick, right? Also a nasty way of dealing with those items. But you also have a sense of, is that embedded system still capable to be upgraded, or do we just have to put it in a coffin, right? My experience of many of those embedded systems is that they were developed with a backdoor from the developer, and many of them do have some sort of backdoor that someone will know about, and they'll be able to take advantage of.
When I was looking way back in 2004 to about 2007 at these systems, at the time, there were around 250 embedded operating systems, and interestingly enough, a high percentage in those days were written by master's students as proof of concept, never believing that someone was going to be smart enough, and I use that word carefully, to then use it into a production system.
And it wasn't just the embedded operating systems, but web servers, again, there were around about 250, there could have been maybe 10, 20 more, whatever, but the embedded web servers, again, similar number, and again, they were written by people who thought, I just want to prove that I can write something that can be useful in this particular way, not imagining it would go into a production system. And it was crazy, and some of those systems still exist today, and they're operating today.
Sorry, I know you're laughing, but yeah, I finished. Yeah, things like that happen.
Well, let's talk about some of the unique kinds of threats that OT environments face. There have been a number of stories in the news over the last few years of OT systems being hacked into, or maybe saying it's being hacked into isn't quite right. In some cases, using insecure remote access to get in and mess around with some of the controls, in some cases, it sounds like the attackers didn't really know what they were doing, they were just kind of probing things.
But other things are, these OT devices have to be updated very carefully when you do get around to doing it, it's firmware on a USB stick, usually, there have been a few attacks that involve contaminated firmware. So what do you all see as other kinds of threats that are specific to IoT, OT, ICS environments? And I think the thing is that if you look around today, in cybersecurity, there's always someone somewhere that would have talked about something being a threat, and nobody took them seriously.
And what I mean by stating that is that there are many, many risks that people have been talking about for a long time. They're not new. Although the risk isn't new, something wasn't done. So in terms of the threat, quite often, the threat is that the risk was not taken seriously for whatever reason it might be. It's not that there's new risks or new threats. It's the existing threats that we've known about for quite some time. They're the ones where we didn't respond, that they're the ones that I'm often most worried about.
I mean, the example I was talking about the embedded systems being around, and those things haven't been updated. The brilliant session we had earlier on resilience about keeping your pets and your cattle separate and looking at things like that.
I mean, it was a really, really good session. We need to look at how we can take away as many of our old systems as possible right across the board. And those old systems, however we try to ring fence them, chances are that either if the technology is isolated, the people of the process may fail. And that may be what the cause of the problem is. It's not the technology that was a problem.
Yes, it was vulnerable, but nobody attacked the technology. They got to the people of the processes, and that's how they got in. I'll probably just shed some light on threat modeling there, because when you look at threats, probably there are different categories of threats in an OT environment. So part of that you just inherit from the connection point to the IT. So everything you see in IT actually applies to OT as far as the control plane is related, because anything that can be reached for a network is obviously subject to potential vulnerabilities and also compromise.
So any of the SCADA systems that run on Windows or Linux operating systems are equally affected. That's one group of threats you have to watch out. So usually you could contain that if you understand the access path. It becomes nasty if your vendor has a backdoor or has other means to get to its equipment, maybe with a proprietary web service or other APIs he produced and exposed in the internet. So that needs to be shut down. So perimeter knowledge is quite essential.
Then there's an emerging number of threats where people got very conscious about how to actually deal with different groups of vendor families. So you probably have half a dozen, maybe a dozen large vendors that dominate those environments. Siemens is usually very much among the popular ones. You have Johnson and Johnson Controls. You have Mitsubishi. You have Hitachi, Thales sometimes. You have a bunch of vendors, Omron in the Far East. So you have a whole host of vendors.
And the tricky part is once you've compromised one of their devices or one of their control systems, usually that equally applies to a whole family of those. So you could argue once you compromised it effectively, you could rewrite the code to apply to the same family. So that happened to Omron a while ago. It also happened to Siemens PLCs. I guess that's the most infamous event. We all watched the centrifuge threat. But irrespective of that, that group of threats, they keep proliferating.
You have, when you look at it, and you may have talked to companies like Dragos, they have a whole threat landscape. And we work very closely with folks like those to figure out what's the CTI we need for that space because it equally applies to our environment. So the trick is you need to have reasonable threat intelligence and it's establishing a similar infrastructure or industry as you see on the IT side right now. People get more knowledgeable about those control systems and they know how to explore it and exploit it.
And we had some staggering events that were really related to control system and then most notably everything around energy infrastructure you see right now in the Ukraine is usually sided with cyber related attacks as well. So those folks know their way around. Just on that, I mean, the IoT assurance framework, if you do get a chance to look at it, it's worth looking at because the assurance framework is technology agnostic, but you can look at anywhere along almost any product, the software and the hardware part and model threats on that.
But also, I mean, talking about software, the Cyber Resilience Act, it does talk about the software bill of materials and that's quite important because many of these are built on someone else's software or software libraries. And software libraries are a big source of vulnerabilities that really can have amazing and astounding impacts. A couple of years ago, I used to talk about Apple phones. In one year, there was a problem with one Wi-Fi library and that Wi-Fi library was used by so many applications.
And that particular library in one month in the same year, it affected a couple of hundred applications. Another month, it affected a couple of thousand. And then another month, it affected another couple of thousand.
So, in a single year, we're talking not five or six users. These are applications that are downloaded by millions of millions of users who are relying on that Wi-Fi library being correct. And that's just one of many libraries that could be used for developing applications. And when it's coming to embedded systems, you have got libraries again there and you're relying on these libraries not having the vulnerabilities.
So, the source of threats and vulnerabilities, you know, the bill of materials that we're being asked to check from the CRA actually is quite an important thing and keeping updated with that and making sure you are on top of those vulnerabilities. Yeah. What are some of the top challenges that you're facing in your environment today? What – is it managing security software in conjunction with OT systems or is it just the somewhat outdated nature of what you've got to do?
Or are you, like a lot of other organizations, being forced into or encouraged to open up access so you could get all this rich data out of your OT environment that could then be processed by big data analytics programs in the cloud? We've been hearing about initiatives like that going on, you know, wanting to open up OT for its data so that it can be used for great things like predictive maintenance and whatnot.
Yeah, I guess that comes back to the vendor link, right? So, clearly the vendors would love to have more of their telemetry and the more, let's say, intelligent they create their – or make their systems, the more data they would like to see. And predictive maintenance is certainly one of the situations.
I mean, when you think about it, there's probably three or four areas we are currently focusing on. One is basically remote access, right?
So, how do we ensure that not the wrong people or at the wrong time can access those systems, right? Well, we're talking about safety systems, right?
So, it's a different design principle, right? That's the one part. And it doesn't pertain only to the primary control plane. It's also the fail-safes, right?
Usually, people make the mistake that the fail-safes are weaker than the primary ones. So, easy way is you create a fail-safe situation and then you compromise the fail-safe, right?
So, that must be observed as a holistic view, right? So, that's kind of access, right? The other part is network segregation. And as we talked about earlier, you need to come to a point where you can actually either isolate or at least create choke points so you can quickly get into a resilient architecture that can defend itself and then you don't have uncontrolled access, right? Not whatsoever, which creates some challenges because your traditional, I open a firewall and I get access to data, it won't work anymore, right?
So, you kind of push back into a territory where you need to go kind of asynchronous, right? You potentially have master-slave configurations, those type of things.
Then, third part usually is telemetry, right? So, we obviously need telemetry because you only have to have, can act properly if you know the health status. And then last but not least, it's actually asset inventory.
So, I have to know where things are. There's very little chance that in a production environment, you can automate discovery, right?
So, you end up doing a fair amount of floor walking depending on how fast your production states are. That could be quite a, well, actually, it's a healthy task because you make a lot of miles per day, but that obviously isn't the objective, right?
So, you need to find other ways to observe it. So, you're talking about complementary surveillance systems, right? It could be CCTV, it could be drones, it could be anything, right? If you have a real estate that doesn't really have a perimeter, right?
So, some of us have production plants, they have a perimeter. But if you talk about rails or pipelines or stuff like that, so networked infrastructure, you can't easily see it. And some of those infrastructure out there are subterranean, right?
So, if you look at energy, a large proportion of that doesn't are over, they're in kind of power lines, so you have them underground. So, you need to be conscious that there are other means required, so more tricks to get telemetry. I know we've run out of time, so I'd only add one other thing to that, which is shadow IoT and OT or whatever, because whether it's CCTV or anything else, you're always going to get someone who's going to install their own things.
Electric kettle, doorbells, we've seen, you know, like the Amazon ring or whatever it might be, those things being installed, just connect to the network. Oh, it's just useful for us sort of thing. It's that part of shadow IoT creeping into IoT.
So, we are close to the end of time. Any parting comments on what you see for the future of OT security? I think the CRA Act is good, and I know I've mentioned it a few times. It's going to change the larger vendors to an extent to make sure that they are trying to get their acts together better than they have before. It's not the be-all and end-all, and it doesn't affect the smaller vendors, but I think it's a positive move in the right direction, and I think that I'd rather look positively, and I think that's a positive thing.
It's been a long time coming, and I just look forward to see how it works out. I second that. I would probably add, we need to encourage the production teams to have more of a cyber acumen or essentially a coverage of the topic, right?
So, we do massive education. We create kind of people in the field that upgrade to have more cyber knowledge, but also responsibility in the field rather than kind of trying to insert cyber people in production. That doesn't work.
So, you need to train your people to a level that there can be at least those additional pair of eyes that look out for anomalies, right? So, as long as we don't have a situation where we can automate the system, we need to have those folks who make it a part of their daily safety routine.
So, securing the environment is actually becoming a safety feature because security protects safety, right? If that's the pattern, we need to really push for that and then get people out there that know what they're seeing and then they can act appropriately. And if somebody does electrical maintenance, they probably can upgrade the embedded system as well, right? And I guess using that and then train people and have more knowledge and awareness of what needs to be done and what is constituting an anomaly I need to follow up upon. That definitely helps in the meantime. Great.
Any questions from the audience? Yeah, go ahead. Thank you very much.
So, one of the issues that I see in research is the increasingly problematic, let's say, interconnection between cyber risks from IT and TOT, of course, but also to the link of risks, concrete risks, as the cascading effects. And also the problems of, let's say, opportunistic, I would say, we call them opportunistic cyber risks. For example, having an explosion or a natural event, some cyber risks are more likely than before.
And this is happening, for example, I'm working in a big harbor IT, TOT systems in which we are exploring this type of connections between risks of explosion or physical hazards and cyber risks. So, I would like to know your opinion on this.
I'm sorry, I didn't quite get the question. I heard what you said.
Yeah, the question is, which type of solutions or evolutions do you see from your point of view in managing the cascading risks of attacks starting as a cyber attack, ending to a physical attack, and integrating these modules? Because the evaluation of cyber risks is something that is known, even in OT. The evaluation of risks is something that is known, even if it is a separate branch of risk evaluation. The connection between these two worlds is somehow, by my point of view, still problematic.
So, what do you think about? I would say, in the same way in cyber, you know, the attackers attack people process technology. It's the same with IoT and OT. They'll attack whatever they can.
So, to get from one to the other isn't a big deal. When attackers are attacking, I was having this conversation yesterday with the two speakers that started the day off, the two hackers. And I was saying to them, the interesting thing is when attackers are attacking, they don't think to themselves, oh, do you know what? That's a physical security system. I must get someone who's a physical security expert to do this because, you know, I only do cyber. I don't do physical. They don't do that. They sit there and they find a solution.
Whereas us as defenders, we try and separate IoT, OT, physical, cyber, as if they are completely separate things and they should be kept separate. Attackers don't think that way.
So, in response to what you're suggesting, I think we need to be defending our systems from a single point of view. And that single point of view is having a converged view of all risks to these systems that we're talking about. There are not enough solutions out there that enable you to have that view at the moment. And if we do start to get those technologies, they will make a difference in that, in all the movies from yesteryear, you will have seen when they've got a command center, that command center has got access to everything.
The doors, the locks, the this, the that. They've got access to everything. They haven't got one team in another place, another part of the world that looks at physical, another one that looks at logical.
No, it's all, everything's there. And that's the way that our, you know, SOCs and our systems defending what we've got should be as well. Probably with that, you have to figure out what are they really interested in. When you become a target, what's their action on objectives, right? What are they trying to achieve? And what is usually the attack path they're following? You could use Mitre for that as an example, right? To figure out, okay, what is my pathway to achieve my objectives as an attacker, right? So it's really about the crown jewels, right?
No matter whether it's a piece of information, it's process or it's a facility. If I want to impair a target, right? I obviously try to do that as in as many ways as possible. And therefore I need to get to a proportional response because if I have a large real estate, they can't possibly defend at all. So I need to really figure out where are the choke points, where I can force the attacker to pass through and basically choke that area to have an effective response. So I need to level the playing field, otherwise I'm not in a position. And that also pertains to the risk.
So when I do the risk modeling, I certainly have to figure out how much knowledge can be transferred over from the knowledge about the IT attack vectors, right? Into OT. And if that is just a little bit, they probably source it in, right? They probably bring in consultants, right? To help them develop that attack framework. And in fact that happens, right? So we need to be conscious what's their capabilities or understanding the capability of the attackers is quite essential and then model the risk around that. Well said. Okay.
Well, thanks everyone. That's the end of the panel, end of this session. We'll head over to Plateau and finish out the day there. Thank you very much.
