Hi. So this is a, a good application of ai. This is not gonna be a, a terrifying talk hopefully, but welcome to where the bots are today we are going to try to stop bots while minimizing friction with ai. So what do I mean by friction? And what is my viewpoint? So my name is Beatrice Mosac. I am a staff researcher, AI researcher at AU Zero by Okta. And today I will be presenting our research with my colleague Matt Woodard, who is a threat intelligence researcher from the point of view of AU Zero by Okta.
So a zero by Okta is a customer identity access management, which means think of the login box when you're trying to log in anywhere like a retail shop, et cetera. We were previously called AU zero and we were acquired by Okta in 2021. So imagine us as a giant honey pot.
Everybody wants to get in, so, so the bots are trying to log in. That's what I mean, know your enemy first. So in order for us to understand and to stop those bots, we have to understand what the attackers need with those bots. So those bots are actually the mean that the attacker are using to get to their objectives.
But what are those objectives? So the first objective is financial gain. The second objective is personal data for financial gain. And the third one is account for sales for financial gain. So it's always kind of always money related. It's not that they want to know your birthday date, it's that they really want to get something out of you. So usually financial gain. Think about the type of attacks that we are gonna see. It's gonna be attackers trying to steal cryptocurrency by taking advantage of you having reused your password multiple times.
And so they're gonna take over your account and steal those cryptocurrencies. Of course, you have a lot of organizations nowadays that are turning on the multi-factor authentication MFA. If you are part of academia or the government or some very large organizations, you probably are already used to using those security layers where you have to answer a push on your phone, et cetera. But the attackers still can bypass by MFA bypass type of attack, those security control in order to get your personal data inside those more secure environment. And then the third type of attack for accounts.
What is accounts sell? What does that mean?
Well, so you have a lot of websites in which you can create accounts for free, like a lot of retailers, Amazon, et cetera, that will have you create an account because there is an incentive for new users. So oftentimes the attackers will create a ton of accounts very suddenly and in large volume in order to either exploit this incentive that exists for new users or create those users as sock puppets.
They will wait and age those accounts such that they look a little bit more legitimate and then sell them to other attackers that will use them for other nefarious purposes.
And if you would like to know more about the terrifying world of bots everywhere on the internet, I invite you to read our state of secure identity report. It's a report that we publish every year based on the research that we see from the data that we have from our customers of all of the customer in identity access management that we have. So if you wanna see it breaking down by industry and type of attacks, it's all in there. And today I am only condensing very, very quickly the very this very large and detailed reports. So how do they do it, right?
So now we know what they want, they want money, but how do they do it?
They do it mainly in two ways. The first is to use abuse prone networks, and the second is to use residential pro residential proxies to create to mount botnet networks. So what is, what is an abuse prone network?
Well, it's usually a network that actually abuse prone is a funny word because sometimes it's part of their business model to have those abuse running on their server and it benefits them to have all of those attackers using their servers because what does the attacker gain out of it? It gains a very, very large pool of IP addresses that they can use for the attacks. And so especially if they have I IP V six, then they have tens of millions of IP addresses that they can use to rotate such that any type of attack detection that is IP based will eventually have just a whack-a-mole problem.
Like I fingerprint two, oh, you're bad, I'm gonna block you. And then poof, you, you come out again with a different IP time times 10 million times.
The other type of network abuse that happens is if they don't want that to happen to their network, the network owner just lack the capabilities of stopping those abuse. So it's also something that we see that we have this weird a SN from, I don't know, Croatia who is attacking us, but the guy just can't stop it.
And so for the second one, for the botnet using resident residential proxies, this one is really the hardest one to to deal with because essentially you have attackers that have downloaded malware on devices or had you use free VPN in order to have you do activity. Remember, if it's free, you are the product. So with Residential Pro, it's even harder for us to be able to stop it because you look human, you have an IP from the middle of wherever, misery for instance, and you're connecting where you should be connecting.
So it's very hard to to to differentiate that from malicious traffic because you have all of the markers, all of the attribute of a proper connection, especially through the ISP. Alright, so those are the bad news. Now the good news is that we can fight back. So today I will talk about three methods, no controls, which is crazy. Don't do that. But imagine what if all of your endpoints were available on the internet? Imagine for us it's a login flow freely available, no security whatsoever.
Then we have the traditional cybersecurity approach, which is rule-based in that particular context, we would have WAF rules implemented at the edge that triggers based on activity threshold. So you know, if you hit my authorization server more than 10 times every five milliseconds, you're probably about. And then finally we have the ML based. So I wanna take you a minute to highlight the difference between rule based and ML based, because in both cases we are talking about rules being generated in the rule-based approach.
The rules are generated by a security researcher that says, Hmm, okay, if I have those ips coming from that we are the SN from Croatia, we are gonna block it, right? ML based, ML based on the other hand also uses this security expertise. But the way the rules are generated is based on a statistical process and a probability distribution based on the data. So that allows us to have more flexible rules that deals a little bit more into the gray area that is less obvious. So let's see how that perform.
So it'll surprise, hopefully absolutely no one in this room that having no security control on your end point is a terrible idea. A recent example in on our end, we had a cross origin authentication, which is a machine to machine login protocol for those who don't know that was abused by attackers. And so one small example of this, we had one attacker on one customer for 24 hours generated more than 9 million calls on to the authorization server.
So it's very, very large. And that was just one example. It's actually much bigger. So don't do it rule based.
Once we have some rules that any security researcher should be able to generate, we were able to detect 18% of the bots in a given sample traffic, which is, it's okay, but it's not much to write home about. But when we do the ML based, we get up to 78% of detection of the bots in the dataset. So now note that I have said detection, not response because the detection for us is you're about to log in and say, Hmm, you look like a bot, I'm gonna show you a response. In that case, very often it is a capture and you're gonna say, but bot can pass capture. That is true.
So our detection is only as good as our response is gonna be because I can detect 78%, a hundred percent of all the, the bots in the dataset.
If my response is inefficient, my detection is weak, is useless. So what can we say about that?
Well, let's, I'm gonna introduce a concept that as an AI researcher, I found very obvious, but I have found it's a little bit more difficult for security researchers to to, to accept, to, to discover because it's not like the traditional way of thinking about things. But there is what I call the deterrent effect, like a lock on a bike regardless of the performance of the response, having a better detection incrementally improves the overall response because it creates more friction for the attacker, even if the attacker has some motivation to pass the capture. So what do I mean by this?
When we look at the tenants as our, at our customers who turned on but detection, we are gonna look at the data 90 days before, 90 days after they turned on bot detection for all of the cohorts of very large customers that we have.
When we look at the undesirable traffic, which is all of the traffic that has the parameters that correlates with credential staffing attacks, we can see a reduction of 58% of that traffic overall in that cohort. It means 58% less likely to have attacks because all of that bad traffic is gone.
We also see that the attacks that do happen happened once we have bad detection in place, have less intensity. So what is the intensity of an attack? It's essentially the duration as well as the amplitude of it. How many millions of invocation of the server did they actually do? So we have fewer traffic susceptible to create attacks. Those attacks that are created are less intense.
Meanwhile, our customers were still able to grow their traffic. So the friction added to that traffic was not so cumbersome that that didn't grow their own customer bases.
So 100% it means that the traffic doubled, but in the meantime, the share of that traffic that is undesirable decreased by half, more than half. So that is the deterrent effect. I don't have to fight bots, I don't have to have a great response for bots that don't even show up. And that is the best security is the security I don't have to use.
But of course friction, that's the bad word in product friction is when I have to show you a capture and you're like, oh my god, this is so hard you can't read it. Whatever that letter is, you know when the letters are mumbled, et cetera. So it's it, it is very hard. We have customers that are very sensitive to that. So we have to be, we have to be careful with the friction. And it's complicated for us because, you know, so we are a platform that has phone apps, computer apps, their own apps.
Like everything is different. So even the user, the user experience of it is very different.
So it's hard to know whether it's us or it's their app that it's not working well. But what we see overall is that there is a faster completion time. And why is that? I have a suspicion that it is, once you remove this 58% of undesirable traffic, you remove all of the script kitties, which were trying to look humans by having maybe a delay in between each step because they didn't want to do everything in a micro millisecond that would be too obvious that throw a bot.
So instead we remove those and now it looks like the, the user is faster, but the the human user was probably not doing any difference. It's just that now we have a truer image of our user, which is great for our customers because they are relying on this data to make decisions about their UX processes as well.
So having security protection adds a better performing security protection adds to other areas of your product, which is your UX for instance, your user experience. But as I said, it's very, very touchy subject for a lot of customers.
Some of them are very sensitive to friction, they don't want any friction, but we have seen that no control is a terrible idea, so we can't do that. And then some others are very sensitive to security, they want zero bot at all because for them it's a high, too much of a high risk. So we have to balance security and friction. But on the other hand, I myself, I have, you know, thousands of customers and only one machine learning model. I can't make 1000 machine learning models for all of them. It's not gonna work well.
Fortunately the world of mathematics surrounding machine learning gives us a beautiful tool, which is to look at the FPR and FNR.
So FPR false positive rate, an FNR, false negative rate, which is something that balance each other. It's a zero sum game between the two where they are pulling one from the other. If I move the threshold that triggers the model to say this is a bot, this is not a bot, I can decide to have more FPR or more FNR more FPR will say that more humans are likely to be challenged. I want more security, so I will be more sensitive.
But when I trigger, I'm likely to show a capture to a human or I want less friction. In that case, I reduce my, my sensitivity threshold, I increase my FNR, which my false negative rates, which means that there is more bots that are likely to not be challenged, but for my other client, that's okay. And then we can create a product where the client can slide left and right and decide what sensibility they have regarding our model, rather than us having to change our model.
So we are able to balance security and friction by kind of offloading them, establishing their own sensibility regarding the model. So I have talked about security, I've talked about friction, I have talked about the balance between the two and we will always have to balance the two of them, but I want to leave you today with this image that we always have to think about security and friction.
But very soon, actually now we have to think of a third element to balance and that is regulation, especially here in, in the EU since a few months ago you have now the EU AI act that kind of changes things a little bit for systems like this one. And so the, the same landscape is also quickly changing in the US and in other countries like Japan and India. So that gives us things to consider in addition of the usual security versus friction. Thank you very much. We have a few minutes of questions.
Thank you so much. This one was certainly an insightful presentation. It's good.
Do we have any question for, for Beatrice?
Yeah. Can you help us understand, you know, so either the humans inside the organizations that are making decisions and with these new threat surfaces and the new tools that are available for them, what are some of the ways and the framings to help people understand the nature of the risk that it's not yesterday's risk, and what are some of the ways that we can help them understand which tools will be most effective in their organizations?
Yeah,
It's, and it kind of linked to the previous discussion when we were saying we have to educate people to understand better those questions. And this is, this is my, my answer is the same as Emily. We have to educate people more as to what those technology are and they represent what is the difference between rule based and machine learning. It's a statistical model. So there is only so much we can do with a statistical model. The model cannot know what it doesn't know. It's one of the things that I'm really trying to talk with customers is to explain that it's not actually a black box.
You can't just put the data in there, shake, it gets a magic answer. You have to open it and say, well the model right now is failing because we have a lot of IP rotation, so no matter what we do, it's, this is what's happening or some other issues. Right? There is always something. So it's both the education of the, the customers, the general population as to what those tool can be. And also the education of the practitioner researcher like myself to force themselves to open those boxes.
Because I think too often it's a little bit too easy to just call the API and and defect into the, the, the black magic, the ritualistic machine learning. And so forcing oneself to have the understanding of why the model is failing helps us understand what is the risk that is not covered.
In addition to the report that you've noted before, the state of the security identity report, does Okta and other organizations that you're aware of, do they make available those kind of training resources, educational resources? Or is the report kind of the gateway to getting more information about it?
Are there other pathways that are available?
The, the report would be the first point of entry and we have a blog as well, in which the Okta blog in which we publish a few more of those parts on what is machine learning, what is artificial intelligence, what does it mean specific for identity? And we have, I think in the past year even our CEO has written a blog about it.
So yeah, everybody is trying to pitching in that effort. Yeah.
And then one last question, which is a question I've asked at every one of the sessions, 15 years out, so the year 2040, what does good look like for you?
AI systems that are properly regulated in which there is no more fear of, you know, it's just a magic black box where there is an understanding that it's a magic, a mathematical tool that can be used for, for, for good.
Please join me in thanking wonderful presentation. Thank you so much. Thank you Peter, so much. Thanks a lot.
