Agent SMITH Entered The Game

I want to introduce you to Sergej Epp. He is a global CISO, currently no company given, so in transition. And he has a title for all of those who have cultural references and like them, Agent Smith Entered The Game. So let's wait and see what Sergej and Agent Smith have to tell us. So please welcome Sergej Epp.

Yes, I think we just need to configure that. One second. Is that working? Amazing. Thank you very much.

And hello, everybody. I think it has not been even a year since, you know, we've got some Janyar agents from saying hello world to really us asking the agent, hey, can you please take care of this specific meeting for me? There's literally a disruption happening right now in a lot of software industries where software, existing software is being rebuilt, where on the other hand, a new categories have been created and a lot of hype has been created as well sometimes.

And for us, from a cybersecurity point of view, it means, first of all, we need to understand the technological disruptions happening because we need always to secure these disruptions. And sometimes we need as well to understand where's too much hype so we can just avoid this being happening. And first of all, I'd like to introduce myself. So Sergej Epp, CISO, and also happy to announce that I've joined SysTick as global chief information security officer. I think the announcement is going to be sent out also earlier today.

And yeah, let me share my personal experience when I've met the agent first myself, and it was not two years ago. It was 2001. And this was the guy I've met. If you remember this very nice game called Counter-Strike, who have you played this one? Amazing. Wow. Wow. Version 0.7 was the best one. So I've spent a lot of nights playing this game. And all of a sudden, one guy appeared in the game called Agent Smith, right? And this guy was just really killing everybody. It was pretty obvious he is cheating. And I've tried to reach out to this guy and find out, hey, what are you using?

Like, how is it possible to do it this way? And he pointed me to this bot, JoeBot, which was written by a guy same age as me, 16 years old at that point in time. And all the paper and the code for this specific Counter-Strike bot. Very simplistic bot, just two neural networks. One was one layer, second was two layers. I've started to spend a lot of time myself building neural networks back then and found this very amusing. I think we had this period of time in games where cheaters were advancing and where they are sort of leading as well.

But very quickly, the gaming industry found a way how to block these different cheaters, right? And we came back to calling the software in games without cheaters. I think times are changing for gaming industry right now. My son is playing Minecraft. And based on this new agents, GUI agents, you start to see a very interesting pattern as well happening in this space, which is that you can simply just task your agent to do something, right?

So agents are evolving from this rule-based, low-code to objective-based agents, where you can say, hey, play now Minecraft for me or play this other game for me. And in the same way, like playing games, there's tons of other agents, as you see here, which were released just during the last two years to work with your Windows computer, with your Mac computer, with your games, with your programs, much more better than us humans in certain areas. And this is just one small part of all the agentic framework.

So I think for us, what is really important is to understand, first of all, how this agent is going to work, how they're working today already, and how can we try to prepare from a security point of view to secure that? Let me try to answer this here in a very, very simplistic way and then also draw down how this is going to look like from a threat modeling point of view if we want to secure them. The obvious first, yeah, LLM, ChatGPT or Blahma or anything like that is very easy.

We just send a prompt, we get an answer, and we very well understood the different type of threats associated to that. So with prompt injection, you can create hallucinations, data leaks, model leaks, you can influence the model itself and so on, right? So we don't need to talk about that. Second one, companies as they apply GenAI and LLMs want to do more.

They want to really extend the context to their internal data and the best technology for that is either fine tuning or RAC, retrieval augmentation generation, where they try in a very simplistic way, if you send a query, they go to a database and then try to get relevant data from this database and then send this together with your query to the LLM, right? And based on that, there's a lot of different new attack types such as indirect prompt injection, poisoning, and so on.

So in a very, very easy way, we shouldn't trust this channel because if there's a permission data set, if somebody is controlling what kind of data is being used now for RAC, it can get compromised. And then the last step of that is, obviously, that's what we call agentic today, is instead, or in addition to these features, we get also tools. So for the first time the LLM can also use tools, APIs, executing command, executing tools, reaching out to somewhere. And in addition, we're going to talk a little bit about the reasoning components, so doing this really in an iterative way.

And here we're adding up to these different threats we saw already, some other threats like plugin hijaction, like API misuse, code execution, everything we know from traditional application security, but now being done in a different way, right? So I think that should settle where we stand in terms of agentic behavior and also what threat could look like.

Now, looking at LinkedIn, Twitter, or whatever, we see there's many more discussions around AI security and it's very hard just to keep track of all them. So I was inspired by this guy on GitHub to draw this, you know, curve of excitement because it's very hard to distinguish what is going on today. And now let me tell you the truth. So if you don't understand prompt injections today, you're really old school because this is already adopted. Nobody's talking about that anymore, right?

So we've got all this interesting topics like sleeper agents and all this excitement topics we're going to talk a bit about, but not too much because we don't have so much time. I think what I want to make sure with this slide is just to say you one thing. Everything we're going to talk about here is not yet working because according to Benedict Evans, AI is whatever doesn't work yet. When it works, it's just software, right? As we find also with our Counter-Strike game.

So not everything is working, but a lot of these patterns are going to work and we need to secure them now as they move on to production. All right. So with this, I'd like to settle our questions and really try to answer a couple of foundational questions, which are going to be important for us. And the first question is, first of all, how do we evaluate capabilities of an AI agent? Because we can talk a lot of marketing material. We can look at all those different agents, but how then do we evaluate agents?

If we can't evaluate agents, we can't really try to distinguish between a marketing hype and what is it that we're doing. So we can't protect that. And I was trying to look for that already for a lot of months and weeks. And I think the best way to do it today, unfortunately, is only CyberSec ever. Who of you heard about this evaluation framework?

Of course, Sunil. Otherwise I would be very surprised. So this is a framework which was developed by the NETA team in order to specifically ask questions, how secure is their model and what can you do with this model? And this is an amazing framework because it's open source and we can repeat the, like with MITRE ATT&CK and other frameworks, evaluate what are the risks currently, the different agentic capabilities, but also gen AI is posing to us.

We don't have time to dive into that, but let me just share a couple of observations because I have like 40 slides for 20 minutes, so bear with me on that one. First of all, prompt injections.

Yes, prompt injections are working and all the models are affected by prompt injections. Without any guardrails, 20 to 40 percent of prompt injections are possible. With guardrails, they are still possible. And as you see, if you look at Mixtro, Lama, the open source models are more prone to prompt injections than anything else.

Next one, can you execute also code? Yes, you can execute code. What is possible is you can, of course, create a lot of guardrails. Like with GPT-4, you see that code execution is not possible, or most not possible. Sometimes it's possible. With open source models without guardrails, it's very easily possible. So if your LLM is controlling code execution, it will be quite dangerous. The other one is, well, what's about code generation itself, right? All the nice copilots like GitHub copilot and so on.

You see, these models are also prone to generate a lot of bugs. According to this model, it's around 30 percent of code, which is being generated, which is insecure.

So again, another data matrix. But can you also help me facilitate attacks? Referred earlier today from Bill, I believe, talking a bit about how we can use GenAI to prepare certain malware components for phishing attacks.

Yes, we can in quite a lot of different fashion, especially with open source models like Gemini or Mixtro. This is very easy.

Well, can we go one step further? Can we just now plug in these Lama models or LLM models to, for instance, Kali Linux, an offensive virtual machine with a lot of offensive tools, and just let it hack? Not quite.

Well, the first steps like network scanning, finding the right vulnerabilities, finding the machines, that's working fine. But then trying to really get the right exploit and exploit this was pretty difficult.

But then, perhaps if we cannot find the right exploit, let's build the exploit, right? Let's try to find vulnerabilities ourselves. They have looked at this problem as well. And apparently, it provides the same type of capability like regular Fuzzer or other type of tools you would use to quickly find zero-day vulnerabilities. So that was not really exciting. But what is really important is to understand specifically the conclusion.

And the conclusion is only related to Lama, the meta model, not the other models, where they clearly say the offensive capabilities, so how AI generally can be used for security tools, were not so dramatic, right? With Lama 3.0, which is the previous version of Lama. But what is more dramatic is really how you secure these models. If you're starting to use these models in your production environment, how do you really make sure you secure them? And how you vendor securing them? Because remember what I said at the beginning, everybody is deploying these agents right now in the state.

So let's look at that. I think the call here is very clearly we need more benchmarking, more standardized benchmarking. And this has to be repeatable. It has to cover all those different models and all those different use cases. What I'd like to share next is really introduce now this reasoning capability.

Because all these tests were made based on the old models of Chet-GPT, GPT-4, and other models, which didn't really have this interactive capability like Chet-GPT-01, is introducing, for instance, the chain of thought capability, which is the most common capability where you start to break down specific tasks in multiple steps and then try to reduce the steps. So here, unfortunately, we don't have right now cybersec evaluation benchmark.

But looking at the first research from other department units, like here, for instance, Alan Turing Institute, we see that there's a 3x type of improvement with the different tests they have performed. So there's definitely an upside capability. And I've talked just last week to the Meta-RED team again. They're seeing this as well with the new models being released.

Again, this test I was just talking about was from July and the one that was released in September. Going one step further, we see even that some other capabilities are possible. You've potentially heard already that Google discovered a zero-day alert, a zero-day vulnerability, sorry, in SQLite. The DARPA Cyber Grand Challenge also identified one, but I think this is one which is a bit more documented. A German startup, Code Intelligence, discovered one. So these reasoning capabilities are constantly improving how we can use AI for cybersecurity.

What I found very fascinating is also this type of conclusion, which I'd like to reflect. Google is saying or assuming that offensive capabilities, finding vulnerabilities, could really increase the defensive potential and even turning the tables and achieving an asymmetric advantage for defenders. We had this discussion with 20 years ago, right, Sunil? So I think that's not a new one, but I'd like to reflect on this one and see what it can really look like. Because even Bruce had ringed the bell that Bruce wrote a similar statement on his blog a couple of weeks ago.

Remember, like fast-forwarding to the future, remember this early decades of this century when software was full of vulnerabilities? That's before the AI found them.

Oh, wow, that was a crazy time. Are we going to live in such a future? I don't know. But let's put this as a question and try to understand, will agents provide an asymmetric advantage for defenders? So are we going as defenders to have some sort of a bullet time capability where we can stop all the attacks? Do you believe in that? And to do that, we can just simply go back to our conversation we are having since 20 years, like offensive capabilities are dual use words.

And yes, they can first of all create more transparency. They can help identify vulnerabilities before they're being released. But on the other hand, we also see that they can be misused. And once again, I have tons of data on that. But just two of my favorite tools, Minicats, was invented and developed by an analyst at Bank of France as a good tool to identify actually if you can, once you compromise a specific PC or specific device, if you can own the entire company by carving out all the different credentials from the memory.

And meanwhile, it's being used by so many APT groups, it became the standard tool for penetration testing. Another one, CobaltStrike is one of the tools which is used by the most pen testers out there, but as we see also by a lot of attackers, right? A lot of cyber criminals just looking at the report from Trellix, they found in the last two years more than 12,000 command and control servers based on CobaltStrike. So clearly cyber criminals are misusing these capabilities very, very heavily.

Now, but what we can do is perhaps we can go back and just use this offensive capabilities to hack back, right? Also something we're discussing for a while with LLMs, there was already very interesting research. If you run an LLM agent and he spots that somebody is trying to lure him or compromise him, he'll just start to, you know, put a honeypot in front of him or to hack him back basically and try to find out what are you doing, right? To increase the cost of attack.

Very interesting concept, but in the end we have to conclude here that all these offensive capabilities are not going to change the symmetric challenge we're having. We have all these available solutions like formal verification, like application security, we can build secure software ready today, but we still, and this is an example, you know, from vendor, we still have kernel, Linux kernels, which are 20 years old, you know, in the center of our data centers. So that's the reality. Cyber security is once again not a problem of a lack of solution, but a problem for lack of execution adoption.

We have to understand that. And I think with this, we can sort of, going back to our agent analogy, we can accept this AI illusion that we oversimplify AI and hope AI is going to solve everything for us and believe that code will be vulnerable, without vulnerabilities, sorry, or we just assume breach as we've been doing in the past all the time. And I think this is a more fair version of our future. So the question is just, if you assume that, how can we secure our code in future? And how can we secure our EIs in future as well?

Because this is the next attack surface which is growing right now. And to do that, I've tried to simplify the history of application security. I'm not sure if you're going to agree with me or not, please throw an X on me or whatever. But I feel we have the sort of waves we went through. The first wave was we were not really sure how to secure our application. There were not many attackers out there. So we've tried to secure as much as possible our production, our systems running around. Then a guy called Larry Smith, agent Smith, Larry Smith, invented the shift left concept.

He came up with the idea, hey, because our releases are taking 40 weeks, let's put the controls much more early in the development chain so we can save costs. And we don't need to reship the entire software back then on OCD. And then out of the sudden, a lot of companies recognized, especially in the tech sector, hey, that's slowing down our development process because all these tools are throwing 85% of false positives. So Netflix paved, Netflix created this paved path type of analogy where you shift very heavily left.

So you introduce threat modeling, security by default, and you protect right quite heavily by introducing more runtime security controls to understand really what is happening in production and feed all this information back and sort of flattering the line. Now, what do you think, where do we stand with securing GenAI systems today? Do you understand how we can fix GenAI systems today during the development phase? Are you doing that? I don't think that anybody understands this today, right?

So with security GenAI, we are more on this first wave where we're trying to understand what the heck is happening in production, secure this as much as possible, running red teaming against it, trying really to understand what is happening there. And if you just take this analogy, like for traditional applications, workloads cloud, where somebody compromised a supply chain or the SQL injection, then there's a letter movement, container escape, moving to the cloud. The agents is nothing else than just another application, right?

This is how you can also escape to container and from a container to your cloud or even outside. It's similar. There's only one difference. AI agents do not work as software. So if you trust today our software, because we were able to secure this in a better way, we can trust our agents. And this is my conclusion from that. We should really consider like all the LLM outputs, all the actions as not being trusted and just refocus base back again on our basics, which is every time there's an activity, you try to harden this activity.

You try to limit this activity, parameterized requests, hardening with runtime, your workloads where AI is running, ensuring having proper API security, less privilege. All those basics are going to help you assuming this component is bridged to have a better security and then starting really to understand a bit better how LLM guardrails can be implemented.

Now, with this, my last question, which I'd like to address is, do we need also to become AI agents to be able to fight AI? That's a question which was many times addressed and I think we need definitely to discuss that because it can provide us a lot of superpowers. But looking back at the first implementation of AI, Gen AI and Gen AI agents, we've just heard from the lady before talking about the Air Canada case, there was just this funny agent being deployed on the blockchain, which, you know, we can steal money if you just compromise this agent and they were able to get 50,000 out of that.

We understand that the topic of prompt injections in general is not really 100% solved. We're still trying to solve that. So the question is just, how can we cope with this? And the industry in general is just looking at three use cases where this error rate is sort of accepted. The first one is heavily focused on coding because in coding, we can very easily test and get feedback if the code is not working. I'm not talking about the feedback on the errors, on cyber errors or cyber vulnerabilities, but on issues. In marketing, there are no wrong answers, so you can use Gen AI and agents.

And also, customer support is potentially the most adopted third use case. So what could such use cases be in cyber security? And I've tried to map them back here on two different, you know, two different perspectives. The first perspective is, where is Gen AI really providing a value? And the second one, where can we introduce human in the loops? And just looking at the use cases, like having spent a lot of time in forensics, that's a perfect use case.

GRC, security trainings, assessments, code security. This is where you can, where you have always a human in the loop, and this is where you can tolerate errors, mistakes, and so on. And I think our industry now, taking the first slide from the marketing from SysTick as well, our industry is also heavily focused on that, because specifically looking at the forensic use cases, looking at the cloud, right, you have a lot of complexity bringing all these locks together.

Even having worked in the cloud space for 10 years, it will be very hard to understand all the configuration, all the lock formats, and so on. So having an agent which is available only to trusted individuals would be very, very helpful to use it. And with this, I think we're running over time, but just to conclude the three takeaways. The first one is definitely we need to have better evaluation for general AI and agents in general, which are repeatable and which we can introduce everywhere. We need to understand that multi-step reasoning is going to improve capabilities, that's clear.

And I think we have to accept the fact that offense capabilities will be stronger for a while than defense capabilities. We can't really get rid of that. The biggest takeaway is we have to go back to basics and treat the output of general agents as not trusted and then double down on how do we really secure the integration to the AI, right, the ecosystem around the AI when it's running with all the traditional tools before we implement AI agents.

And third, understand really what's happening under the hood. The agents are not just another blockchain. I think there will be use in production, a lot of vendors implementing them. So try to understand and familiarize yourself with the security and threat landscape there, and also with the offense capabilities and not just defense capabilities, because this is where we have currently a big advantage and where we can test before the attackers are going to start using them.

With this, thank you very much. Sergei, as usual, when somebody uses more time, it adds up, but let's wait and see. We have a panel coming up. I will see you later at 10 to 6, and then we can see if you can beat the AI with a bit tongue-in-cheek, of course, but let's try this. And with this, thank you very much again, Sergei. No time for questions, of course, but reach out to him. He should be there.