Rise of the Machines: The Need for Cyber Heroes in AI Development

So the talk today is called Rise of the Machines the Need for Cyber Heroes in AI Development. Before we get started I wanted to introduce myself. I'm based in Dallas, Texas in the United States but so happy to be here today in Frankfurt. I work in the industry as a security researcher. I'm an international speaker and I regularly focus on AI security both the offensive and defensive aspects. I have a doctorate in cybersecurity analytics so that means my research focus on combining AI with cybersecurity. So the best way to contact me is on LinkedIn.

If you want you can scan the QR code to contact me there. So what is AI? Probably we hear a lot about AI now and one of the first things we think about when we think about AI is generative AI. So if I look up AI it typically will tell me okay you can use AI to generate a picture to create artwork for me but actually AI is not just generative AI.

So AI encompasses many different techniques that you can use for data analytics to analyze lots and lots of data to see patterns to gather insights from data and generative AI is just one kind of AI technology where we are still using data analytics and then it generates new content based on data that it analyzed. So even though those pictures look unique it's really just training it on all these data samples of pictures images and then from that it creates something that looks unique but it's really just learning based on other data. So of course we think about AI being very powerful.

It's a very powerful tool for data analytics but one downside with AI is that AI can make mistakes. AI is never 100% accurate. No matter what algorithm you use it will always make some kind of mistake. So one technique that we use in the industry and in research is this idea of adversarial examples. So this is a concept called the evasion attack which is one way you can attack AI models. Note that this applies not only to images but also to generative AI. I'm just talking about images because it's easy for you to visualize.

So the first technique you see on the top with the panda it's called adversarial noise. So what this means is that if we look here the panda images are the exact same. Can anyone see any difference between these two images? No right? Yeah they're the exact same but the AI model thinks that this is a gibbon because you add noise to the background of this image. So this is an example of an adversarial example because even though we've even though we've contaminated the image with this background noise we don't actually see anything that is different here.

So in case you're not aware gibbon looks like that. So very very different compared to a panda but because we add this noise to this image AI thinks it's this. So very different and that's not the only way you can contaminate data samples. You can also perform adversarial rotation. So that is rotating the image in some way. So again AI thinks that this vulture image is something else entirely different. So here if we do this adversarial rotation we rotate the vulture image slightly very slightly. AI thinks it's an orangutan which looks like that.

So just by making that small modification AI is fooled and it thinks okay this is an orangutan even though it's clearly not. So that's another technique you can use. And now the final technique that we're going to discuss is this idea of adversarial photographer. So that means you take the picture or you take the data sample and you make some kind of change to the data sample itself so that the AI model is fooled in some way. So this is an example of a granola bar. It's called Nature Valley Granola Bars. So basically you know something for breakfast that you eat right.

So this is the box that it comes in. You see here but if you take a picture of just one single bar just this way then you think the AI thinks that it's a hot dog because of this orientation of the granola bar itself. See it's just one bar in this orientation so that's why it thinks it could potentially be a hot dog because it's looking at that orientation there. So because AI is making these mistakes and we can't fully rely on AI we need people to analyze and to help build these AI systems. We need to ensure that we have people overlooking what AI is giving us and what AI is predicting.

So in this talk I talk about cyber heroes but basically the idea is we need people to help us when we're creating these AI systems. So in the industry we call this the human in the loop approach and basically it means whenever we deploy, develop, or design AI we need to have people with each step of this process so that the people can help oversee what AI is doing. So here in this image you see we're involving people in all stages designing, building it, deploying, and refining and optimization.

So this first part is we need people when we're actually gathering the data right because AI is nothing if it doesn't have good data. So to actually gather this data for example images of pandas, images of hot dogs, we need people to gather this data to clean this data so we want clear images or clear data points and we want to annotate the data in some cases. So in AI and machine learning there's this type of machine learning called supervised machine learning. So in supervised machine learning all of our data points are labeled.

So in a cyber security context it would mean for example malware, not malware, benign, just a label like that right. So that would be an example of data annotation and then we need people to actually quality control this annotation like is this really malware or is someone tricking my AI to think that this is malware when it's really benign and then we need AI when we're actually developing the AI models and we want to refine it. So here whenever we see AI is giving us the result we want to validate the model.

We want to make sure yes pandas are in fact pandas and you don't think a panda is a gibbon. We want to make sure that when we're using machine learning ops or we have a pipeline so we can automatically deploy and scale our models that we optimize these models as well. So we need people throughout each and every step of this process. There should never be an instance where we completely just blindly trust the AI model because as we know AI can make mistakes. So now I'll go over some successful examples in the industry. We've seen this in all different sectors.

So the industry has demonstrated using humans in this human in the loop approach. So the first example is the Bloomberg terminal. The finance sector is really big on AI. So with Bloomberg terminal there's a way to look at all this data for example stock prices the trends that we see and then using AI and machine learning to identify this data to see okay will the stock trend upwards will there be a crash somewhere and it uses AI and machine learning to identify this data and to put this data into text tables charts some kind of visualization so it's easy for someone to see this visualization.

So instead of looking at these numbers in an excel spreadsheet you can look at this in a chart format. The second option is Tesla. So of course we know Tesla when you drive the car it has a self-driving mode but sometimes we need people in the loop if Tesla makes some kind of mistake. So we don't have completely fully self-driving mode yet that's something we're they're working on but as of right now we still need people involved when we are driving the Tesla car. And then this last example is from a company based in New York called Rain.

So they're developing a voice assistant to determine the correct intent. So what this means is if I have a voice assistant like Amazon Alexa and I say Alexa tell me the weather today is it going to rain? Alexa should be able to tell me like yes it's going to rain today or no it's not going to rain today. If I say Alexa tell me the best restaurants in Frankfurt Alexa should tell me this.

So that's what a voice assistant is doing and when I ask it a question like tell me the best restaurant or tell me the best place to eat Alexa should be able to recognize any voice assistant should be able to recognize those mean the same thing. So how is human in the loop actually being used in all these different examples? So the first thing is with the Bloomberg terminal this is from Bloomberg directly they say that they're using human in the loop with these three kinds of roles. So the first role is that we still need people to actually code and develop the AI models.

So we need machine learning engineers. The second thing is we need AI researchers and data scientists because we need AI researchers to design and implement the workflows for us to say yes you use this data in a certain way and this is how you should have the workflow like using this data we want to visualize it in a chart like this. And then finally they use journalists and subject matter experts to annotate and collect the data. So they also perform quality assurance.

So for example on the Bloomberg television show when they show these charts from the Bloomberg terminal they journalists often look at the charts themselves and say like yes this makes sense or no it doesn't make sense. So that's another example of quality assurance. And then of course with Tesla we have the case where humans take control with autonomous driving in the event of errors. So sometimes the Tesla it beeps and says take control immediately because there's an obstacle that Tesla cannot anticipate for or it cannot correct itself. So then in that case we need people.

And then finally with this voice assistant from Rain they use humans to help with data acquisition to provide better and more labeled data because of course ideally in machine learning the more data you have the better your model generally does. Just generally speaking the more data you have it is always better. So that's why humans are helping with that. And humans can also help with determining synonyms. So for example words that mean the same thing right like restaurant or place to eat they should mean the same thing.

So humans will help determine these synonyms like yes if you give this response to this question then it should be the same response for a very similar question with the synonym. So because human in the loop is so important this is very closely related to this idea of AI ethics. So AI ethics is another big thing we're talking about all over the world in the EU and the United States everywhere in the world right now. So these five principles are what we think about when we think about AI ethics. So the first principle is human centeredness. Second one is fairness.

Third one is transparency and explainability. The fourth one is accountability. And finally the fifth one is privacy and security. So now what do these principles actually mean? So the first one human centeredness. This is the idea of we want to involve people with all aspects of the problem solving process. This kind of relates to the human in the loop example that I was talking about. So whenever we have AI models like Bloomberg terminal we want to make sure it's easy for people to use. It shouldn't be that we design something and then no one can use it.

So human centeredness means that we want the products or the AI algorithms that we're using it should empower people and to help them turn all this data into information. So that's the first aspect of AI ethics is we still want humans involved in all aspects of the problem solving process. The second one is fairness. So the idea of fairness is we want to correct and minimize AI algorithmic bias or discrimination.

One important thing to note is that no matter what AI algorithm you have your AI algorithm will always have some form of bias with it because just the nature of AI no data will ever be free of bias. There's always some level of bias because as humans we are biased right? So we can't completely eliminate this but we want to minimize this bias as much as we can and this relates to discrimination as well. So for example a tech company a couple of years ago they used AI for screening resumes right?

So instead of having recruiters manually look at resumes they just wanted to use AI to automate this process and when we did this with the tech company right so they found that it was actually discriminating against very qualified women candidates that were equally as qualified as the men and that's because the resume the AI resume screener was trained on successful candidates that all happen to be men and so then when it saw something that was different than what it learned was successful or the candidate that they wanted in the company so for example women or terms in the resume related to women then it automatically discounted those candidates that had those phrases related to women because that doesn't match with what it was trained on so that's one example of what we're trying to achieve when we're talking about AI fairness.

The third one transparency and explainability so there are tools now like Lime and Shape that help us do this and this is an ongoing research problem ongoing research area is how do we make AI transparent and explainable so right now a lot of the traditional AI thinking has been or was that AI is a black box so I just give AI my data and it gives me some kind of output I don't really know what it's doing I just know it works and we want to stay away from that logic and that thinking because we want AI to be transparent and explainable so these are two closely related concepts but they're slightly different so transparency means we want AI to be transparent so whenever we're designing or developing AI systems we want the operations and the development process to be open and accessible so that is I know what's actually going into my AI it's not like cryptography where we have this idea of cryptography we want to hide our ciphers we don't want that in AI we want the the algorithm or the way the AI system is being developed to be open and accessible so we know actually how it's working it shouldn't be a black box and then explainability means we want every single decision that AI is making to be understandable so for example if I have an AI system that qualifies like this is malware or this is benign I should know why it said that this sample in particular was malware or this sample was benign what exactly did it do to come to that decision so that's the idea of explainability and then accountability is this idea that whenever we have any AI system and it gives us some kind of output if there are any bad actions then the blame should be assigned to people that are at fault parties should be accountable for their actions so for example AI is being used across industries now and as we're using AI we're now learning more and more that AI is making mistakes recently there was this there was a story where Air Canada the airline they they used AI in a chat bot to help you instead of looking at their website you can use a chat bot and it gives you a response back now what happened with Air Canada was a passenger used this chat bot and the chat bot actually gave the passenger incorrect bad advice so because this bad advice was given via the Air Canada chat bot the passenger they tried to pursue it suing Air Canada but Air Canada was saying no it's not it's not our fault AI did it we are we are not to blame at all AI is responsible for its own actions and of course going through the legal system it was found that this is not actually correct that this should not be upheld because parties like Air Canada or anyone who's using AI they should be accountable for their actions so AI should not just be responsible for itself someone should be accountable for those actions and then the last one is privacy and security so when we talk about AI it's all about data so we want to make sure that the data AI is using is secure because for example if you're using AI in hospitals you have all these patient records you want to make sure that this data is secure so if I'm a hospital and I have all this access to patients then I want to make sure that no one can just hack the hospital and gather this patient information so we want to make sure that AI is private and we also want to make sure AI is secure so that means preventing AI from being attacked so like those adversarial examples I showed you earlier we don't want that to happen and there are many ways to do this and one way you might have heard a lot about in the industry is using AI to fight AI so we could potentially use AI to analyze this data and say this is a security vulnerability but ultimately you need a defense in depth approach there's not just one technique that can secure AI for you so that concludes the AI ethics portion and my presentation thank you so much any questions from the audience thank you very much yeah thank you so first of all I'm doing this for 10 years this was the first time that I forgot to do the moderation in the beginning so this is really something new but are there any questions to Anmol in the room I think it's not we're already over the time so thank you very much it was a great presentation thank you thank you very much thank you