Panel | AI Unleashed: The Battle of Wits in Cybersecurity - Attackers vs. Defenders

Let's just begin with a short round of introductions, where I would like our panelists, and by the way, they asked me to be one as well, to introduce themselves and make a short punchy statement just to initiate the discussion. What's your view on this whole battle of wits of AI and cyber security? Ladies first. I want some time to breathe after you. Thanks. Hi everyone, I'm Beverly McCann. I'm a Director of Analysis in EMEA for Darktrace. What does that mean? I am leading a team of cyber analysts here in EMEA that protect our customers, work in their SOC and support them when they need it.

That's sort of my major role. I look after our roughly 9,000 customers at the moment. Who are we at Darktrace? In case you haven't come across us yet, we are an AI-based cyber security company. I know AI is in every vendor's name at the moment.

However, we started with this already 10 years ago and our fundamental approach was to use AI to defend and secure businesses. Maybe as a punchy statement, I know there have been plenty of talks already this morning and I really enjoyed listening to them. There were all these graphs about what gen AI can be used on the attacker side as well as the defender side. It's not just a concept anymore. We do really see those attacks emerging. We can see the use of generative AI and that's something that we've been predicting for many, many years now.

Now, with the rise of gen AI, it really is starting to show quite heavily and I'm more than happy to go into that in more detail. Cool. I'm also looking for...

Sorry, you want to... Sebastian, please, just to remind our new viewers, maybe those who will be watching this in the recording, so what's your relationship with AI and cyber security? My background is originally in systems engineering and security also. Machine learning, I went into this. I'm a side-joiner, I could say, but gained a lot of practice now and I see the potential a lot, of course, for both sides, sure. But my provocative statement is I'm always puzzled. As I tried to explain in the talk also, it still creates a lot of effort.

Also, you need first... I mean, there's this no free lunch theorem, right, which says at the end, if you don't have any assumptions, then every machine learning model or approach more or less has the same performance, right? So it always causes a lot of effort.

First, you need to get clear what kind of problem you have and I'm always a bit sceptical if... I don't mean your company or something, but if some companies claim, okay, we're doing a lot of AI and more or less give the impression that it works out of the box and I think I'm sceptical about this. I want to understand the detail. I don't know, but the potential is there, but there's still a lot of work. It's not just switch chat GPT on or a security version of that and then it solves your problem, right?

Okay, great. As for me, my name is Alexey Blagansky, I'm a lead analyst at Kubernetes Coal, the host of this conference, and I am by no means an AI expert in the traditional sense like you both are, but I do remember over 30 years ago, my first exposure to this whole problem was I read an article about the Chinese problem discussion and maybe since then I've been the kind of, if not an outright AI sceptic, but at least the one who goes around talking to vendors, poking them with a stick and saying, okay, you say you use AI, show me, prove it.

So my statement probably would be, there is no such thing as the AI. It's not magic, it's just at the best case, a lot of math and at the worst case, it's some stuff outsourced to cheap labor in India and those use cases really have happened before.

Okay, so let's maybe start with, again, kind of going back to the elephant in the room. Everyone is talking about generative AI, but surely AI did not start with GGBT, right? So it's been around for years and you are, I mean, what I've seen before actually does not actually touch that area. So what's AI for you? Is it new? Is it old? How it's been evolving and what's really there, like a threshold event which suddenly turned AI from an obscure sophisticated technology tool, something everyone suddenly wants to have and use?

Yeah, I'm very happy to go for that first. I mean, yes, AI has been around for over 40 years. Large language models have been around for almost 10 years now, the chat GPTs and stuff. So it's nothing new. We've been using these techniques and these tools and the AI in so many different applications and ways for a very long time. It is really, and maybe to go on top of your point, it is something that you as a business, as an industry, as a team, yeah, really need to challenge when a vendor comes up to you and says, oh, we use AI.

Well, what is it? What type of AI? In which context? Which data are you using? So it's really crucial to make sure that you have this understanding of, okay, when we talk AI, what is it? Are you using machine learning? Is it supervised? Is it unsupervised? Are you working on my data or are you working on a group of data? What data are you using to feed that AI with and where the data source is?

And also, how do you store my data? And that's, I think, a big discussion, especially in Germany, where data protection is obviously really, really crucial about what happens to my data. Where do you store it? Where do you use it?

So yes, AI isn't something new. We have been using it for 10 years.

And again, yeah, we as a vendor, we don't just use one AI. We're in our sixth generation of using different types of AI techniques. So we use a combination of unsupervised machine learning. We use graph theory. We use large language models. We use natural language processing. We use Bayesian statistics.

So yeah, it's really important to understand what is it that is being used and how it's being used and with what data. Yeah, absolutely.

Yeah, AI, a term, I think it goes back to the 50s. I think Turing and people like this, they framed this first. And of course, back in this, it was all rule-based. Like Eliza probably heard about this. You can program this in Prolog and then it chats with you. But of course, the performance is not so well. And then we had in the 80s, I think, neural networks were invented in the 80s already. But all of that couldn't fly because the computational power was not there, the data was not available, the bandwidth and the storage was not cheap enough and so on.

And then, of course, so it's an old story. And a lot of this is based on statistics. Absolutely. Sometimes also data science statistics, very, very close to each other.

And then, yeah, there were some graphs in some previous talks where suddenly it was more or less linear, the power, and suddenly it went up exponentially with the availability of the data. And of course, this in turn has amplified the development of new techniques, deep learning, particularly natural language processing, as you said, and many, many things. But for instance, Jürgen Schmidt, you probably heard about him. He invented the LSTM networks, for instance. He often uses to say he invented this back in the 90s.

And he says sometimes in conferences which take place these days, he says, why is this new? We did this already in the 90s and something.

So many, many background theory was invented already there. And yeah, some people have, it's still, where does it go next? I don't know. So it's all very exciting, sometimes a bit overhyped maybe, but of course we are still on the verge to understand the potential and so on. And some people like Jan LeCun, he's the Turing Award winner. They say we are in a dead end a bit because we do not know what is next. So all this, we need a lot of data to train them. This is also a sustainability problem, not only to power consumption, but also it's not how humans learn still.

They say we can do a lot of involvement there, but somehow we do also something wrong, something incorrectly. And therefore maybe we also need a fundamental change. In general, there's this observation that this deep learning, it performs so well.

However, the theory lags behind. So we are not fully clear why, to a certain extent at least. I don't know how you see it, but yeah.

Yeah, it's really interesting. I mean, everyone probably remembers that we used to have a thing called big data, which was really expensive and required a special hardware to run. It came with a lot of limitations. Only the biggest companies could afford it. And now we just don't talk about it anymore because it's commoditized to such an extent that it just became data, right? And the same, I guess, goes with AI. Now you just have, I mean, anyone can run AI in the cloud. Maybe in a few years you could run JGPT on your mobile phone.

So yeah, it will eventually become cheaper. And I wasn't joking, by the way, about this story of outsourcing quote-unquote AI to India, because that was a real thing. A few years ago, there was a camera with object recognition capabilities, which was actually not using real AI, but it was cheaper and easier, I guess, back then to use people to do that job. Not anymore, but we now, of course, have other challenges, and lots of other challenges. Data protection and compliance, you already mentioned.

But yeah, I guess we have to go back to this question, the battle of wits. Do you really see that the attackers are overtaking us, the defenders, in that regard? Do you see practical usage of AI of any kind in this?

Yes, we definitely started to see some indications of, especially with the rise of Gen AI, the usage of these tools on the attacker side. And I think if you've been to Sergey's sessions earlier, you could have seen a really nice overview of potentials of where an attacker could actually apply AI in the way of attacking, like automating running of attacks and codes. But what we've definitely seen that already is a huge increase in sophistication of phishing emails.

And obviously, yes, they have been around before, social engineering techniques to tailor a phishing email towards the target that they want to compromise. But now that can obviously be elevated in a huge way, rather than having to do it really one by one.

I have to, as an attacker, do my research in the social platforms to find out information about a target. And then I craft a very specific email that relates to a social media post that has been made. Now I can just task the AI to go away, grab that information for me, tailor that email, and send it off. Maybe create a domain that sounds very similar to the company's domain, buy that domain for me, put that into a phishing email, and send it off. And you can do that at scale.

So it's not also, to some extent, obviously, it's that sophistication of really targeted attacks, but it's also then to really scale that, and thereby getting to your goals really, really quickly. And we've also seen, obviously, the speed of attacks increasing, and that is due to the usage of automation and of tools that help you with automating these. And that's sort of ways where we can see this already emerging, and obviously the next things are to come. So we'll just have to, yeah, be getting ready for that, and being able to, yes, be prepared for this to come.

So whereas, obviously, a big approach was, previously, is awareness training, and trying to train your workers of trying to look for grammar mistakes, and spelling mistakes, and look out for, don't click any links. But the way that emails look like these days, they look so convincing. You wouldn't be able to distinguish anymore whether an email has been generated by a human or by an AI.

You need to be able to have tools in place that will detect these changes, so that you cannot just put all the pressure on your employees to be able to say, yes, of course, we're always going to, we're never going to click on links, and we're never going to open up an attachment that contains some malicious code. So yeah, it's really being prepared for these changes to come, and having tools in place that will detect those changes. And by the way, I wanted to follow up on that specifically for Sebastian with a question for you.

So to fight those AI threats, you have to understand, you have to research them first. Do you think you already have enough tool, enough expertise to do this as an independent researcher or scientist? Or do you think you have to collaborate with other stakeholders, like the vendors, or maybe even on the authorities and the government?

Of course, research is always interdisciplinary, and it makes it better, of course. That's also where I look always for collaborating partners. For instance, the company where you have the real data, and it's not like researching in your dark room and trying to do something on paper anymore.

Of course, you collaborate a lot. And yeah, it would be awesome to also do it with companies that actually...

I mean, you do the same, right? When we met today, you started as a research company, and you also have research in this. It's not so much different. Besides that your researchers just do research, I also do teaching.

But yeah, of course, absolutely. It makes absolutely sense. And I see it similarly, as Beverly said, with the other part there. It speeds things a lot up, right? So it gets this AI, the productivity boost is enormous. And I see it also as an interaction with the hacker. And you can use this as a tool, and it's the next step of automation. Do you develop, maybe? Everybody uses GitHub Copilot, maybe? It's really amazing. You just write some code, and then it suggests you how it continues. That's awesome, right? And you can also use this to craft some bots, for instance, of course.

Or during the Capture the Flag session, one team used JTPT to solve the crypto-puzzle, because it's pretty well in detecting patterns. These are cool, tangible examples of how easy the productivity boost is. But of course, I'm always looking for partners to collaborate with in research.

Okay, awesome. So I guess one maybe even final question for today.

So yeah, you are like the developers, the researchers at the forefront of this AI cybersecurity research. You have a lot of things to do, and you deliver a lot, probably even a little bit too much sometimes. We have this feeling there is so much AI being thrown around, and how do we understand which kind of AI is actually good enough and which isn't? How do we measure?

I mean, we as the laymen in this industry, the customers, the engineers, how do we understand, how do we tell apart the good AI tools from the bad ones? Is there any way to independently measure the efficiency of cybersecurity tools, or is it something you have to research as well?

I mean, you can always measure the performance of a machine learning classification problem or regression problem or something. You always have the, for instance, by classification you can measure false positives, false negatives. So you have also key performance indicators then, like precision recall. You have this curve, the so-called receiver operator characteristic, where you see if the area under the curve is big, then the performance is better. And I mean, okay, this is maybe too specific for a completely uneducated guy, but still this is not complex machine learning stuff.

You could compare those maybe, but still you need some people who understand something, right? I guess that would be my guess, yeah, to distinguish how does it really work well. So also you need to measure, I mean, false positives, false negatives. You can also measure directly in this approach that I presented, right? You get feedback from the customer support, and if there are a lot of complaints, maybe the tool was not so bad.

Okay, maybe a bit early in the stage you want to distinguish it before, right? But yeah, you need some people who know how to measure the strength of such an approach, I suppose, which can help you, I suppose. I don't know, Bepali, what do you say?

Yes, I think it is generally really hard for like a layman, and I don't understand the depths of AI as well, but I think a big part of it is obviously, and that's with every new technology that comes out, is that you need a level of trust. You need to trust the technology to be able to deliver what it says it delivers.

And I think that's why there are a lot of conversations going on in the world of politics as well, to see whether we need to put some guardrails around it, some regulations around it, just to be able to showcase, okay, within the realms of this concept, yes, AI can be and should be used in this way, and this is where the limit is. And also have the ability for the technologies that the vendors, that they have to showcase, yes, these are the technologies they're using. I think it's really hard to, I don't think there is any good AI or bad AI as such.

I think it is really key about applying the right AI to the right problem, because I don't think JetGPT can be put on all the problems out there. Like to maybe have a quick final example, if we're looking around these chatbots coming out, these prompt-based AI solutions that promise that they can help your security teams to investigate certain anomalies, if you as a security analyst want to look at the problem, you always look at it from a very malicious sort of everything is bad angle. So if you ask the chat GPTs to investigate a unusual connection, you ask it whether it's malicious.

So obviously it will go from a biased angle of trying to see and find out whether something is malicious, whereas if you're an IT engineer and you want to obviously, the goal for you is to always keep everything up and running and keep everything as good as possible, your question might be to that JetGPT, is this a legitimate activity? Is it a legitimate connection?

And again, so you based on the questions you're asking, you create a bias to your answer. So it's really important to keep that in mind, whatever you put in, it always needs to be looked at with a deferred human set of eyes again.

So yeah, it's all about trying to apply the right AI to the right problem, I think. Okay, awesome. So I guess let's finish this panel with one final takeaway per panelist, and I hope you'll excuse me this shameless plug to say always kind of ignore the labels and always look for specific capabilities. And if you cannot understand those capabilities yourself, always look for a neutral second opinion from a company like Kuping & Co. Thank you very much. Nice one.

And well, Sebastian, what would be your takeaway from this? Well, okay, yeah, a lot of potential AI, we are just at the start, but there's also a lot of research going on, maybe, so this can evolve in every direction. And we are living in exciting days, but don't, do not trust every AI label, really try to understand what's really in there. Right.

Yeah, and for me, I guess, yeah, just being aware that AI attacks are on the rise, AI attacks are coming, they're there actually already, it's not just a future concept anymore. And to be able to attack AI, you cannot throw more and more humans at it, you need to have AI in your security tech stack as well, to be able to defend against those attacks. That will be from me.

Okay, awesome. Well, thank you very much, Beverly and Sebastian.