Cyber Defense Matrix

Hi, I'm Sounil Yu. I live in Washington, D.C. and I'm currently CTO for a company focused on solving some interesting challenges in AI. But before that, I used to be the chief scientist at Bank of America. And when I was there, I had a couple of different roles. I was a math scientist. I was an evaluator of different products. I also ran teams that tried to break into the bank. And in the process, I had a chance to really understand this landscape that we call cybersecurity.

And it's a very challenging landscape for those who are maybe struggling to understand the breadth of all things in cybersecurity. Well, that's what I had to do. My challenge was trying to figure out how to make sense of all these things that we see in cybersecurity all the time. And my goal was to find gaps. So let me ask you, do you see any gaps?

Now, you might see gaps, but it's probably hard to find because what you see is all there is. And so I said, I need a structure. I need something that's more structured to help me identify where gaps are in all the things that you see here. And so I came up with a simple structure. And that simple structure is based on two things, things that I care about and things that I do. The things that I care about are nouns. The things that I do, of course, are verbs. And so these five things that I care about and five things that I do align nicely into what I call the cyber defense matrix.

So that's what you have in front of you, the cyber defense matrix. Now, with the cyber defense matrix, I can take all those buzzwords that we saw previously and now organize them more functionally. And when I start to do that, well, gaps become a lot more apparent. So of course, I can do this with product categories, but I could also do it with vendors, too. So you'll see a lot of vendors outside, right? Do you know what they sell? Or more importantly, do you know what they sell, how they fit into your program? And if you don't, it's partly because they try to confuse you.

Now, part of what we'll do is to figure out where things like this map, and that's really what that workshop is all about. Now, there's a person by the name of George Box, and he had this wonderful quote, which is, all models are wrong. So this model that I'm showing you is absolutely wrong, okay? Because it won't solve every problem that you have in cybersecurity, but it has turned out to be extraordinarily useful. So his quote is, all models are wrong, but some are useful. And what I have found with the cyber defense matrix is that it's been very useful for a lot of things.

And so when I first presented this in the U.S. back in 2016, I came up with a lot of use cases.

Now, my original use case was just mapping vendors, but I also mapped a lot of other things. And these other things end up, you know, some of them worked and some of them didn't quite work. But nonetheless, it turned out that as an organizational system, the cyber defense matrix was really useful to make sense of the world. And there's a lot of use cases.

Now, I wrote about some of these use cases in the book, but we're actually going to go through some of these use cases as a workshop as well, okay? Now, one analogy I use to help us understand this process is food.

And so we, of course, understand food because we have to eat it every day. And the interesting thing about the cyber defense matrix is we can use it as a framework to map a lot of different types of categories, things that are very common to us in the cybersecurity practice.

So, for example, what do we have in our current set of capabilities? What's in our cupboard or pantry? What are recipes, things that we have to comply with? What are compliance requirements? What are architectural frameworks? What are good practices? We also want to be able to understand our nutritional needs. These are things like threats, like ransomware. How does ransomware fit into the matrix? We want to understand what's available in the grocery store, in the supermarket. So we walk outside, we see a vendor, where do they fit in the matrix?

And then lastly, we're also concerned about allergies. Allergies are business constraints, and we'll talk about that more as well.

Okay, so ultimately, we want to take each of these and we can map them to the matrix. And so as an example, let's look at recipes. So the CIS, Center for Internet Security, they have a set of 18 controls. Here's how they map into the matrix, all the different controls, and you can see the rough distribution of where every one of them are. You also see gaps too, right?

Well, the CIS controls, they're just 18 controls. Where's number 19? Where's number 20? Are they even covering some of these other ones that we see that we have gaps in? But anyway, that said, here's an example of recipes and where those fit. Then we have nutritional needs. We have threats like phishing and ransomware. So as an example, this is from a friend of mine, Ross Young. He has identified a bunch of different types of threats and the different types of controls that you need for each of the different types of threats.

And if we were to look at ransomware specifically, here are the boxes that are threatened when it comes to ransomware. So if you want to have coverage for ransomware, if you want ransomware controls, then these are the boxes that you need to have covered. But you can see how by layering the different parts of the matrix, you can see how this becomes more interesting, because I can take these areas of threats, add in the recipes, then add in what's available in the supermarket. And then I can highlight those particular areas, nutritional needs, threats, that are more prevalent.

So I could say this particular box here, I have a higher degree of concern. And then lastly, I want to capture allergies. So I want to make sure that we account for the fact that not all these boxes can be covered. And that's because the business will say, that will cause business impact for me if you put something in. If you tried to implement a security control, and you've had no problems whatsoever in implementing that control across your business, then you probably don't actually have a business.

Because I haven't run into a situation where the business doesn't push back and say, well, hold on, let's make sure this makes sense in the context of our business. Well, it turns out there are whether there's different profiles for different types of business units. So if you've ever, for example, one of the basic controls that we have in CIS is removing administrator rights.

Okay, removing administrator rights from your endpoints. Well, let me ask you, have you ever tried to remove administrator rights from developers?

Okay, that laugh tells you the story here, which is, you end up with an allergy. The developers say, this will prevent me from doing any work. So I cannot have that control in here. So there's allergies that developers have. Have you tried to remove, have you tried to tell sales and marketing, we need to pre-approve every application that you use before you use it.

Okay, it doesn't work. Because they want to try every new application under the sun. And if you have to pre-approve it every time, then it just slows them down.

Now, there's balances here that we need to trade off some balances that we need to work to it. But we have to understand that putting in certain security policies and constraints or security controls creates problems with the business. And so we have to recognize those are constraints. And so with those constraints, we then can layer on those on top of the matrix as well. And so now you have this more complete view of what your options are, what controls you need to meet, what threats you're trying to deal with, what vendors might be useful, what constraints you have to deal with. Make sense?

Okay. Well, with that, we could then answer questions that we've been wanting to answer for a long time. Questions like, how secure are we? How secure should we be? And for the how secure we should be, we can look at those recipes. We can also look at those threats. And then lastly, how do we get there?

Well, we look at the market and we'll also look at those business constraints or allergies. Okay. All right. So with that, oh, so okay.

Now, we're going to jump in shortly with the actual exercise, the workshop. But before we do that, I need to also just make sure that we understand this mapping. So everything seems pretty straightforward, right? Yes. Anyone confused? Any questions there? Okay. Because you'll have a lot of questions once we start to exercise. Because where things map is not so simple. It's not so straightforward. Okay. And because I know that we're dealing with, again, I'm American, so I only speak one language, American. We're going to have a language challenge.

We may have a language challenge here just because I don't speak German. But let me also just say, we have this language challenge, even amongst those in the U.S. that all speak English. Okay. Because we have these words that we use, but we don't actually know, we don't have a common definition for what these words mean. Let alone across different cultures, different environments and whatnot. And so I'm going to, it's almost, you don't have to even worry about the words. Just call them function one, function two, function three, function four, function five. Okay.

Because we get thrown off by what the words say, and we think it means something else. Okay. So for example, the word identify versus the word detect. Okay. And the English language, at least, they're relatively synonymous and we interchange them all the time.

In fact, NIST, which is one of the agencies in the United States that defines standards in their definition of identify, they use the word detect. In the definition of the word detect, they use the word identify. That is not very helpful. Okay. Because it creates this confusion around what exactly these things are.

Well, one of the ways that I try to remove this confusion is by having a very clear distinction between left and right of BOOM. So BOOM occurs between protect and detect. Everything before that is under the functions of identify and protect. Everything after that is detect, respond, recover. Okay. Now let's just do a quick quiz to see whether or not you got that. All right. So let's take vulnerabilities. Okay. Do we identify vulnerabilities or do we detect vulnerabilities? Okay. Who thinks it's identify? And who thinks it's the word detect? Okay.

So we have a couple of people who, well, so we clearly have this agreement here. Now, for those who raised their hand for detect, according to NIST, you are correct. Okay. But everyone else who said identify, you are actually functionally correct because looking for vulnerabilities is something that happens before BOOM.

Therefore, the right word is identify, not detect. However, NIST got it wrong. So in version one of the NIST cybersecurity framework, they call it detect. In version two, they fixed it. It's now identify. Because it's left of BOOM. So and when you talk about left of BOOM, we're talking about the structure of your environment. And the structure includes things like your structural weaknesses, which is like a vulnerability, versus situational awareness, which is the exploitation against the vulnerability.

So it's simple things like that that will help us figure out whether or not we're dealing with left or right of BOOM. Okay. All right.

With that, let's now dive into the workshop. So I'm going to skip exercise one. Exercise one, I left open for you all to fill in the blank, so to speak. So as we go through the exercise, if there are things that you want to map that I haven't mapped for you, that we haven't mapped yet, then just fill it in and we'll talk about that as a part of the workshop. Okay. I don't think I have enough materials for everyone here, unfortunately. So some of y'all won't need to share. For those who are sitting in the back, you will need to probably have a table or something to work with.

And I do have some materials for you guys to use to be able to do the exercise. So if you want to grab some of these. And I don't know if I brought more books. Yeah.

Oh, yeah. You guys can sit here too. Okay. So the first exercise is mapping. So the first exercise is actually exercise number two. It's sort of like your elevators here, where ground floor is not one, as it should be. But but anyway, never mind. That's a joke. So everyone should minimally have sticky notes and a whiteboard version of the Cyber Defense Matrix. If you don't have a book, I have more somewhere. All right. Okay. So the exercise that we're going to do is to map product categories.

And so this may be challenging if you're not familiar with some of these words, but I will kind of explain some of them. But the goal is to nonetheless try to figure out where they map on the matrix. So all you got to do is take these sticky notes and put them in the box that you think is appropriate. Okay. And like like that. All right. Okay. And so and you can work with your partner or some a friend or somebody next to you if you want. But the goal is to figure out where each of these things fit.

Now, one of the goals is to try to make sure that they fit in one box and only one box. Okay.

However, you will find that that's a challenge. Okay. And I want you to struggle through the challenge because that's actually part of the learning. But that said, if you find that you have to put it in multiple boxes, so be it. But there's a reason why I only gave you one sticker and that sticker is supposed to generally fit in only one box. Okay. So go ahead. Try your best. For those who don't know what some of these words mean, SBOM is Software Bill of Materials. XDR is Extended Detection and Response. And all the rest, I'll let you figure out. Okay.

So I'll give you about five minutes or so and then we'll reconvene and talk about the answers. Okay. That's yes. But what does X mean? That's part of the challenge, right? What exactly does X mean? And for those who are, hopefully everyone here is somewhat familiar with cybersecurity, like hopefully you guys are practitioners or at least have some understanding of cybersecurity. But if you're like, what are these words? I didn't make up any of these. These are all like market terms.

Oh, hey, oh, I should mention, don't use the answers in the back. Hey, hey, that's cheating. That's cheating. The answers are in the book. Or some of them are. Don't cheat. Right. Forgot about that. Okay. All right. I think most of y'all are mostly done. Okay. How was that exercise? Thoughts? Just you can start yelling them out. What were your thoughts? Easy? Hard? Medium? Okay. All right.

Well, let's see if you got it right. Because you might thought it was easy and you got them all wrong.

So, you know. Okay. And by the way, this is my opinion. Okay.

So, I could be wrong as well. All right.

So, I could be wrong. By all means, I could be wrong. But this is my opinion. And I'm always happy to be corrected by whoever has more and better information.

So, let's start with Software Bill of Materials. So, Software Bill of Materials helps me understand what are the ingredients in my software application. And therefore, I'm enumerating all the application components.

Therefore, I put it under Application Identify. Anyone have SBOMP anywhere else? Okay. Data Security Posture Management. All right.

So, this one's, well, data is obvious, right? Data security, right? But it's a question of whether or not it's only identify or identify and protect.

So, Security Posture Management, usually, most of these tools that say Security Posture, it's really just an identify function. It's trying to understand, do you have any structural deficiencies? Not actually correct them. Okay. But some of these Data Security Posture Management tools are starting to correct them as well.

And so, that's why I am okay to put it into protect. But most of them fall in under identify.

Oh, I should mention. So, one of the things that I try to do in the cyber defense matrix is I have a really clear understanding of what the primary capability is. And that's really where I map most of these things.

So, I should probably put Data Security Posture Management more under, like, shift it even more so under identify. But there's a lot of confusion that you'll get when you try to understand what these vendors do relative to their primary function versus their secondary function.

So, I'll give you an example. Let's take antivirus.

Of course, everyone knows what antivirus is, right? What box do you think antivirus fits in? Okay.

So, let's start with the asset class. What type of asset does an antivirus protect?

Oh, address. Devices. And then what function? Protect. Okay.

Now, if you read. So, device protect is the most likely place to put it, right? But if you read their marketing literature, they also claim to protect your data, right? Okay. Does antivirus protect your data?

Well, wait. You're about to say a word. They. It does.

I mean, malware getting in then stops that malware. You were about to say a word, but you kind of hesitated.

You said, I think you're about to say the word eventually. Yes. It eventually protects your data. Okay.

I just, I don't know if you were going to say that word, but you just kind of almost hinted at the word eventually. But it's not, it doesn't do it directly. It does it indirectly. Okay. But most of these products will claim protecting data as a secondary function because, well, it's trying to protect something else.

So, like a firewall protects your, what function, what asset class? Network. But they will also claim to protect your data too, right? But does it? And the answer is no, not directly. And so that's where it gets confusing. And it's particularly confusing when we get to other functions. So they'll say, we will look for all your vulnerabilities and we'll also protect you from them, or we can help you protect yourself from those vulnerabilities. And so it's what I call in English, a weasel word. A weasel word is like helps or supports or enables. They don't actually do it. They help you do it.

And usually it requires something else for you to actually be able to do that function that we're, that they're claiming. Okay. All right. So that's data security posture management. All right. So XDR. So the DNR should be pretty straightforward. Okay. Sometimes the R, the response piece, that's actually questionable. Most of the times it's only just the D, the detect piece. But the real question is, what do we mean by X? Okay.

Now, most XDR products that you'll see in the market really only cover devices and networks. And for them, that's X. But for me, X is like an asterisk. It's a wild card, which means all five asset classes. But most XDR products that you'll see do not do all five asset classes. They barely just do two. But if you look in the market today, largely XDR is in these two categories that you see, under devices and networks and under detect and respond. And that's primarily where they sit. Any questions or any other thoughts on that one? Questions? Objections? Have you seen anything else?

How about that, in terms of where XDR might fit? Yeah, I know. I cheated. You could tear the sticker in half, I guess. Don't do that. All right. Then lastly, we have identity detection and response.

Now, this is where you could have put it. But I will also take that answer in a lot of different places. And the reason why, and later on you'll see why I do this, but the word identity is one of those other confusing words in our practice. Because it implies that identity is only tied to users. And that's not true. Every one of these asset classes have an identity. And we'll talk about that a little later. So the notion of an identity detection response being largely user-centric is actually a mistake. But that's generally where most of those fit.

And we'll talk about that again a little later. But if you put identity detection and response across multiple asset classes, I'll take that answer as well. All right. Okay. How did y'all do? Anyone get all of them mostly right?

Okay, good. Okay. And part of the idea here is, hopefully, as we do these exercises, we converge on the right answer as a collective group.

Because, well, I mean, if we're all confused about what all these things do, then, you know, what hope do we have in trying to secure our cybersecurity ecosystem? Once we have a better understanding of all these things, then it helps us then make sense of what we're doing as well. Okay. All right. Okay. So the next exercise is on measurement. So in the book, you'll see that I have no, I have exactly zero mentions of the word maturity. So maturity is another word that I, in the English language, that we abuse too much.

And it's because it's a crutch for representing a lot of different things that we hope for in cybersecurity. But what I wanted to do instead was to have a much more precise understanding of what do we mean by this type of progression. And one of the ways that we can understand that is in the progression of what we're measuring itself. Okay. And so there's a perspective that I have in terms of different types of measurements of being a different quality, but also different difficulty as well. We oftentimes provide a measurement that is really easy, but is also very useless as well. Okay.

And we oftentimes see this manifest in, for example, security vendor questionnaires. Okay.

Like, for example, do you have MFA? And of course, the answer is probably yes, I have MFA because I kind of have to, right? Okay. So here we go.

Yes, I have MFA. There we go. I have MFA, right? Here's my little token. But what good is it if I just bought it and haven't really actually implemented it anywhere? And that's really kind of the perspective here. Just because I've bought it, just because I have the presence of it, doesn't mean anything besides the fact that I bought it. It doesn't mean that I've actually implemented it anywhere. It's not like I've plugged it in. Although I'm going to plug it back in so I don't lose it. It's not like I haven't turned on, I don't know if I've turned on all the features.

Do I have origin binding turned on? Have I turned on all the necessary features to make it phishing resistant? Then how well does it work? And oftentimes that's the kind of question that we really want to answer. But if you don't have some of these preceding measurements, then how good is your actual calculation of performance?

Lastly, how efficient or how cost effective is the capability? Again, those are questions that we all seek to answer, but it's oftentimes difficult to make a claim that we have those kinds of measurements if we don't have all the preceding ones as well. To make it very clear as to why that's the case, consider we have two situations where I can say I have on the, okay, so in the red on the far right, I have something that's 60%. I make a claim that it's 60% efficient and what I'm trying to accomplish. That sounds great, but you realize my coverage is pretty weak.

So my overall efficiency is only 12%. Okay. On the right, my efficiency is 30%, but my coverage is pretty good across most of the things. So my overall efficiency is actually 27%. We can use these numbers to lie all day, right? And that's what we do in cybersecurity often, you know, when we're presenting to the board or presenting to a lot of other folks. But when we're trying, when we don't want to fool ourselves, we want to have a much more systematic methodology where we have a very clear understanding of how good our program actually is. Okay.

And to do that, I think it's important for us to understand, for ourselves at least, how good our measurements actually are. Do we have all the requisite preceding values? So if we make a claim that this performs this much, this well, well, how do we know, do we have any other information on coverage? Do we have any other information on utilization? Do we have all this other information that we might be missing? And so that's our exercise. Okay. So the exercise, now we're in exercise three, and there are two parts of this exercise.

So the first part is understanding what type of measurement it is. Is this a A-level measurement, a B-level measurement, C-level measurement, or D-level measurement, or E-level measurement? Okay. And then later on, we're also going to then map it to the matrix as well. Okay.

Now, here's the thing that I want you to do. So there's a claim that quality. Okay. There's a claim that's being made, meaning we're claiming that we have, let's say, if I were to give you an example, let's say if I have an A-level measurement. Okay. If I'm claiming an A-level measurement, then what we should think about is, okay, from this claim, do I have any other evidence of performance, utilization, or coverage? Okay. If I have a B-level measurement, do I have any claims around utilization or coverage? Okay. And so on and so on.

So if you read a particular claim, then look for all the evidence before that as well. Okay. And if it's missing, just note that it's missing. The claim is still going to be that I have an A-level measurement, but you may say, okay, I don't see any performance or utilization or coverage data to be able to support this. And maybe that's what I need to look for. Okay. So that's the goal. This one's a little bit more complicated, but hopefully you'll get a sense of how this works. I'll give you guys about maybe 10 minutes to go through this one.

If you walked in recently and don't have materials, I have materials for you here. All right. So 10 minutes-ish, and we'll reconvene. By the way, this is a workshop for those who came in recently, which means that you have to actually work. I think I'm almost out. You guys can share. Yeah. What's that? Yes. We're on exercise three. Not yet.

So, oh, yeah, I'm sorry. Thank you for the clarification. So initially, before, yeah, before it, you know, you're going to have to do a little bit of Yes. We're on exercise three. Not yet.

So, oh, yeah, I'm sorry. Thank you for the clarification. So initially, before, yeah, before you see where the boxes are, get a pen and write the letter, the associated letter in the box. What kind of measurement is it? So let's say, for example, we'll walk through one of these together. Okay. So we have DLP. Okay. Because I'm sure you've run into, anyone who's filled out a vendor questionnaire, that's a question, right? Do you have DLP? Anyone see that question? And of course, the answer is yes, we do. What type of measurement is that? We have DLP. You can yell it out. E.

It's a presence measurement. It doesn't say anything else about whether or not you actually have turned it on, anyone's monitoring logs, you've turned on the features. That's all it is.

We have, we bought DLP. Okay. So what you would write in the, in the sticky note is E. Okay. In the little box. Write the letter E. So then look at the other ones and figure out what the associated quality of the measurement is. Then later on, we're going to, oh, actually not later on.

You can, you can do this as well, but just take that sticky note and stick it in the right box as well. Okay. All right. But so who here missed the intro? Who here missed the intro?

No, no. I mean the intro, intro. I know you guys, I know many people have missed it because I, I'm but you're not raising your hand. So the very beginning intro. Okay. All right. I know who you're, who's lying because I, anyway. Okay. For those who missed the intro, I'm going to, for the, if you saw the intro, keep working. But for the sake of those who missed the intro, let me go back and just share the intro again. Okay.

So again, this is just for those who missed the beginning. Well, one of the challenges that we have in cybersecurity is that we have a lot of, it's very difficult to parse through all the buzzwords that we see in the cybersecurity ecosystem. And so what I did was I came up with a simple structure. The structure consists of these two dimensions, things that I care about, things that I do. I organize this into this cyber defense matrix. It allows me to take all those buzzwords and organize them into something that's more functionally aligned.

And so once I can align them functionally, it allows me to see those gaps much more easily. Before it was hard to find the gaps. Finding gaps here is really hard. Finding gaps here is much easier. And of course I can map vendors and those vendors help me understand what they do as well. There's a lot of different use cases that the cyber defense matrix has. And there's tons of historical briefings that I have on all this. But more interestingly, the cyber defense matrix is a nice system for organizing information. And as an example, I organize different types of categories of information.

So for example, what are your current set of capabilities? You can think of that as what's in your cupboard. What are the practices or what are the requirements that you have to meet? Think of those as recipes. What threats do we have to deal with? Think of those as nutritional needs. What's available in the, what do the vendors outside do? Think of those as a supermarket. And then lastly, what sort of business constraints do I have to deal with? Think of those as allergies. Each of these layers can be mapped to the cyber defense matrix.

And so as an example, here's a mapping of all the controls that we see in the CIS, critical security controls. These are like recipes. There's different types of threats like phishing, ransomware, and so on and so forth. Here's a representation of all the controls that you need for ransomware. Now I can start layering different things. Like for example, here are controls that we need for ransomware. Here are the potential vendors that can help. I can now organize these threats based on severity and figure out which ones I need to prioritize. I also have to deal with allergies.

So different business units have different allergies, different constraints that prevent me from being able to implement all the controls I want to be able to do. So if you have, and each different business unit has different allergies. So you got to just be mindful of which ones they are. But you take all those and then you can layer them on to say, here are now, here's a composite view of my threats, my controls, my threats, my control requirements, my business constraints, the vendors that I have. And so with this composite view, I can now understand questions like, how secure are we today?

How secure do we need to be? And how do we get there? So that's what the cyber defense matrix is all about. And we are now in exercise three. Okay. Questions? All right. Okay. We have about three more minutes of my 10 minutes, but it looks like most of you are done. So I think. Yes. Anyone not done yet? All right. So let's walk through the first part of the exercise in terms of the quality of the measurement.

Actually, no, we'll do that. We'll do the second part too. Okay. If you haven't already, so has everyone mapped their math as well? Okay. Not everyone has mapped them, but we'll go ahead and go through their exercise. We'll go ahead and talk through the answers there as well. All right. So as I mentioned, the example, so this is what you would do at the end. You'd have like, if we had an example, like we have MFA again, that's an E level measurement. And that just tells you you have presence. It doesn't tell you much more else than that. And then MFA maps to user identify.

I'll talk about that a little later as to why that's the case. So let's go ahead and and then MFA maps to user identify. I'll talk about that a little later as to why that's the case. But here are the answers that I had.

Again, this is my opinion. I could be wrong. You guys can have a different opinion on things. When I marked it with the asterisks, it's where I would say it's a B type of measurement, but it's, I'm lacking the additional information to make, to have a really strong claim around whether or not it's actually correct. So let's walk through some of these. We'll start with, we've scanned 98% of our code base for vulnerabilities.

So again, looking for vulnerabilities is an identify function. It's talking about code, so that's applications. Now it's saying we've scanned 98% of our code base. That's a coverage measurement. So hopefully you have D for that, and that's under application identify. Anyone have something else for that? I'm sure somebody has something else for it.

Okay, what did you have? And you want to make an argument for why that may be different?

Oh, you thought it was a performance measurement. What if you found no vulnerabilities? If you scan 98% of your code and you found no vulnerabilities, would you think that your vulnerability scanner was actually working? Probably not, right? Just because there's always vulnerabilities in your code. So what I would say is, and here's the thing that I would say. If you're looking for a higher level measurement, then let's talk about what that measurement actually might be, right?

So for this one, I would want to look for, if I'm trying to move the needle up in terms of a better type of measurement, it would be, is it looking for, let's use some examples around like OWASP, top 10, whatever, whatever, okay? Does it have coverage? Can it cover all the different types of vulnerabilities? What about different languages? Does it cover Java? Does it cover TypeScript? Does it cover all the different types of languages?

Those are kind of questions that are left unspoken here, for which you would want to have clear perspectives on because otherwise, again, there's not much to, it's a claim that someone's making. And by the way, oh, I should say, these are all, these are not made up claims. I'm sorry. These are not made up metrics. I've seen these in various security reporting. I've seen this, some of these in board reporting, which is kind of dumb. But nonetheless, these are not, I did not pull these out of thin air.

These are numbers that I've seen people report, where I said, okay, let's see what we can do to do better here. All right, next one. Our phishing click rate is 25%. So first of all, let's see, did anyone map this somewhere else?

Okay, where? D.

Okay, actually, no, sorry, not mapping it, but not the grade, but the mapping, like where it fits on the box. Did anyone put it in a different box? What? I put it on detect.

I mean, it didn't say anything about, okay, obviously, it's mock phishing and it's real phishing. Oh, yes, yes. A very good point. Very good point. So if it was mock phishing, then it's identified. If it's real phishing, then it's detect.

Now, what's the difference? I mean, kind of what, like, so if it's mock phishing, you should think of that as a vulnerability scan of a user, okay? How do we vulnerability scan? We vulnerability scan devices, we vulnerability scan applications, and so on, so on. Think about what do we use to continuously vulnerability scan our users to see whether or not they're susceptible to phishing emails, and those are mock phishing emails.

So to see a phishing click rate of 25% of our vulnerability scans against our user is a representation of our user's vulnerabilities, or user susceptibility to vulnerabilities. I meant mock, so that's why I put it there, but you're absolutely right. If it were a real phishing click rate, then that would be perhaps more on the detect side. You could argue that that's the same thing, but it's still a test, right?

Well, it's still a test. So if it's a simulation against, it's a scan, right? It's another form of a scan to see whether or not a entity is susceptible to a vulnerability. By the way, so how do you patch a human? Training.

No, you don't fire them. Well, if they don't take, if you don't patch a device, if someone, if you find a vulnerability in a device and they don't patch it, what do you do with the device eventually? Isolate you, and eventually you say, we've got to get rid of this thing, right?

So, I mean, you could take a similar sort of view with outside of the workers council getting in the way, I'm sure you can do something like that, right? Okay.

Anyway, as far as the grade is concerned, someone said D, right? Okay. Yeah. So I agree. I agree it's useless, which is why I put it in here as an example. What are ways to make it better that actually, so I put an asterisk there to say, even though, well, again, we can debate on the specifics of where it might be in the ranking, but we hope for a better measurement, right? So what will make this a better measurement to really truly be a B, like what kind of, what additional information can you measure? Okay.

So, yeah, so let me let me make an argument that maybe this number is actually really good. Okay, let me rephrase it.

Okay, our mock phishing click rate is 25% for phishing emails that simulate a nation-state attacker with highly targeted phishing content. Okay, and if I had that as a precondition, 25%, I'd be like, wow, we're doing really well.

Okay, okay. But my point is, I've caveated it to say, it's not just that we've done a random phishing test. It's we've done a very highly targeted type of phishing test, which then get into more around the type of quality of measurement that we would see that would address like the C quality measurement or D quality measurement. So what are, if I'm, for example, in other words, another aspect I'm saying is our phishing click rate for all our admins, okay, versus our phishing click rate for our entire employee population. Very different, right?

Or our phishing click rate for our highest risk users. Again, all those are very different types of measurements, which give you lots of different value in different ways. This is not very valuable because it doesn't have any information on coverage, doesn't have any information on the sophistication of the phishing emails. And those two additional details can make this measurement way better. It would be interesting if you could measure somehow what people do with real data. Whether they just ignore them in the past and tell you what they're using, or something else.

But that would be a good measure of whether you're seriously aware of what you're using. Yeah, so the...

Yeah, the question is, or the statement is, if we have actual data on how well our users actually respond to real phishing emails, or simulated ones, that's actually additional detail that's helpful in understanding the strength of our security program. Or, let me rephrase it slightly, it's additional detail that helps us understand how well our users are protected.

So, everything that you just mentioned then falls into measurements associated with user protect. What's the actual reporting rate? If security training and awareness falls under user protect, what are the measurements to say that that program is actually working? And some of the examples that you gave fall into user protect.

Okay, all right, questions, thoughts? It's making sense? Anyone confused?

All right, okay, let's keep going. So, I think we already talked about UbiHop DLP is a pretty crappy measurement.

Now, in terms of where it fits, what do you think? Did anyone put it somewhere else besides data protect?

Okay, where'd you put it? Data recover?

Okay, and why would it be data recover, DLP? It's data loss protection, so it should be recovered. We just call it data loss?

Yeah, protection. Why wouldn't it be protect?

Yeah, but it's the process that should be taken care. So, the data has been lost, and it has been protect.

So, you need to respond it and recover it, the data. All right, okay, who thinks, let me generalize it a little bit. Who thinks DLP is a right-of-boom tool? Anyone?

Okay, who uses DLP as a right-of-boom tool? As a right-of-boom?

Meaning, okay. Okay, right.

So, for those who missed my earlier conversation about left-right-of-boom. So, right-of-boom meaning it's used for detect, respond, and recover.

And so, part of why I think you're making a claim is it's a right-of-boom tool, specifically recover. I would question that. But most of us do not put DLP in blocking mode. Why? Because we will get fired. Okay. But does that make it a detect tool? Let me ask you. We have firewalls, right? And we have firewalls. And we can turn it on blocking mode, and we do for, you know, very obvious bad stuff. But for many other things, we may actually put it in just monitor mode, right? Does that make a firewall? Does that move a firewall from network protect to network detect?

Okay, so now let's take DLP again. DLP is a protection tool. It's data protect.

That's, you know, data loss protection. Just because we turn it into monitor mode and not block, does it all of a sudden become a detect tool?

Well, I'd argue it doesn't, because what do we do? We take the log, we put the log into the seam or something like that, and the seam is the detect tool. Awesome. Exactly right.

So, again, the DLP, a firewall, AV, all these protect tools log and sometimes block. But across the board, they log. The logs go into an actual detect tool like a seam.

And, by the way, I didn't mention this, but this is actually a pretty important thing on the bottom, the degree of dependency on people, process, and technology. I don't know why I don't mention this more, but that's actually probably the most interesting part of the matrix, because we get this wrong all the time. We think we put in a seam, and the seam works perfectly fine without any user intervention, right? And we quickly realize, no, it needs constant daily feeding and tuning and adjusting, okay? And that's done by people, all right?

And so we forget to resource detect tools appropriately, making sure that there's an equal amount of people time as much as the cost of the technology as well. But, anyway, exactly right in terms of the function of a DLP tool is not a detection tool. It's a logging tool, and the logs feed into a detection tool, okay? But just because we put it into monitor mode doesn't all of a sudden make it a detection tool. It doesn't make it fall into that detection category.

And, again, we confuse this all the time. And the reason why this is important and the reason why it's good to not forget that is because if you buy DLP because it has detection abilities, or you think it does, and you don't actually buy a sim or you don't buy an aggregation tool of some sort, then you are in for a world of pain, okay? You're not going to actually be able to effectively perform your detect function just because you're going to be overwhelmed. You're not going to have any correlated logs. You're not going to have all these other sort of things.

And so that's where we, again, think, okay, just because I bought this and because they claim detection, all of a sudden you think you have detection when you don't actually have it, okay? All right. Let's go ahead and go to the next one. Our firewalls have blocked 45,000 attempted connections.

Okay, I saw this as a performance measurement as well, but, again, a very weak one, okay? A very weak one. In other words, it's saying, hey, this thing is working because it's blocking stuff, okay? That's about it, right? So the question is, what can make this better? What can make it a true ‑‑ like, what are the coverage and utilization measurements that would actually make it better? Some analysis, yes, sure.

Yeah, I mean, so it's kind of ‑‑ I'm trying to not make fun of too many people, but I've seen measurements like we've blocked 100% of all our attacks. Like, you've blocked 100% of all your attacks.

Like, okay, I'm not sure what the denominator is, and I'm not sure what you classify as an attack, but I've seen those kind of claims, and I'm sure you've seen we've been attacked 45,000 times, okay, or 45 million times or whatever. You know, these are like stupid claims that we had to come up with in the past days. I hope we don't make those claims anymore, but these are real claims that I'm sure some of you all have seen.

The real challenge for us is, one, let's not try to use these crappy measurements anymore, but, two, how do we make these better so that they're actually useful to us as practitioners? Okay, I'm going to go ahead and keep moving on. We've had seven SLA misses for patching critical application vulnerabilities last month. So I put this under application protect, and I actually like this measurement because it provided the necessary conditions of what I'm actually measuring, okay? There's a time frame. There's a scope around which applications I'm trying to patch.

There's a defined, presumably there's a defined SLA. It doesn't say which SLA misses. I could have been, oh, actually, I guess it's for critical SLAs.

Anyway, but I like this particular measurement because it had the necessary pieces to provide a perspective of coverage, meaning critical apps, and I'm not sure what the utilization measurement is, but you can hopefully see why it's a better sounding measurement than all the other ones that we've seen so far. Okay, did anyone have any questions about this one or thoughts to make it better?

Okay, all right, last one. Our main time containment is five minutes for intrusions in our DMZ. This is a network respond, and again, a B-level measurement, a performance measurement because it's saying how good our containment measures are. It's scoped to the DMZ.

Yeah, okay, anyway, all right, questions on the measurements piece? All right, so my challenge to you guys is take your measurements and try to map them to the matrix. See how you do. Some of them are, well, you'll start questioning your metrics and your measurements after you go through this exercise.

All right, okay, last one is mapping zero trust. So a big buzzword, as we all know, but I wanted to give some framing of how I think about zero trust because it provides, I think, a more systematic view than I think most will have seen. So I mentioned in the upfront portion, I shared this cyber defense matrix with everyone back in 2016, and I talked about different shortfalls where I couldn't map orchestration, analytics, and governance.

You take these five asset classes and take these three foundational layers that I couldn't figure out how to fit, and you organize it this way, and you have the system zero trust maturity model. So if you are familiar with the zero trust maturity model that the U.S. government has put forth, the foundation is actually the cyber defense matrix.

Okay, so anyone familiar with the zero trust maturity model? If you're not familiar with it, definitely just search on it. You'll find it's a great resource. But the thing I wanted to point out that you see from the change from here to here is that users becomes identity, okay? But the icon is still a user, okay? So it implies, again, that only users have identity, which is a problem because all these assets have identity, not just users. And because of that, we have to think about what is the identity of all these things as well. What's a device identity? What's a network identity?

What's an application identity and so on? And when we think about that, it will help us then map things appropriately in the matrix as well, okay? And that's actually the next exercise. So I'm going to let you guys struggle through this. And then when we talk about the answer, you'll see how everything fits together. But the struggle is the good part. So four categories that are in the zero trust area. See what you can do to try to map these four into the cyber defense matrix. And I'll give you, like, four minutes here.

For those, anyone not familiar with passkey? Okay. So I'm not going to try to give you an answer. I'll try to tell you in a way that doesn't give you the way to answer. But passkey is a passwordless mechanism that has been introduced recently. And there's been more mainstream adoption. But it's passwordless authentication where it stores an authentication credential on... It stores the authentication credentials on the things that you use so that you don't have to use passwords. And I'll give you a couple of minutes to figure it out. And then we'll start with the passkey.

And I'll give you a couple of minutes to figure it out. So I'm going to give you a couple minutes to figure it out. Okay. And I'll give you a couple of minutes to figure it out. Okay. And I'll give you a couple of minutes to figure it out. Okay. Okay. All right. Let's wrap it up in about 20 more seconds. All right. I'm not going to show you the answers right away, because I'm just curious what people actually, where people put things.

So first, 2FA token. Where did you put 2FA token? Users. And what asset class?

I'm sorry, what function? Protect? Identify? Okay. Between both. Did anyone else put, besides users, did they put it anywhere else? Devices. Okay. Others? Applications. So we think that 2FA tokens identifies users, and protects devices, and applications, and the user themselves. Right? That's where we think it fits? Okay. If I can only pick one box, which box would that be, of all those things?

User, and what? Identify. So remember the whole idea about primary versus secondary? Okay. What's the first thing that the 2FA token does, versus what can it do after that? Okay. So this is a situation where we confuse the primary versus the secondary capabilities. The primary capability of a 2FA token is to identify a user. The secondary capability is to be used as a mechanism for protecting all these other types of assets. I'm not exactly sure I agree with you on that. Because all this thing does is it says, I am a device.

And in the background, someone says, okay, as part of the identification for this user, he must show this device, or he must let this device interact with the logon mechanism. But it still doesn't say, this is alpha. It just says, I'm a device, but it's supposed to be owned by me. Okay. So let me ask you, another way to think about this is, you kind of hinted at it in the way that you described it, but they require something else to be able to perform the protection function. Okay?

And if something that, in your mapping exercises, as you go through the mapping exercise, if you see that there's something else that's required, you think it's going to protect your application, but it requires something else to actually protect that application, then, again, that's a secondary capability for whatever you're trying to map. So a 2FA token by itself does not protect your application. A 2FA token with something else protects your application. But that something else is actually what protects your application. Okay? So let's actually talk about that with API Gateway.

So API Gateway is something that actually protects your application. And it may consume a 2FA token to do that. So if you're trying to actually protect your application, that might be...

Now, API Gateway, you don't necessarily want to use a two-factor authentication with API Gateways, because they'll break. But the perspective is, there is a... If you're trying to protect a device or an application or a network or data, then that's something... That requires something else. And it's usually something like a gateway or an access proxy. So let's look at Zero Trust's network access proxy.

Network, right? Hopefully. And hopefully you got all set, put that under protect as well. Did anyone put it elsewhere besides network protect? Hmm?

Oh, devices as well? Okay. All right. So this is where... So this is...

Okay, hold that thought. And I'll talk about that in a little moment, because you are correct in how many people think about it.

Also, Zero Trust network access, people think of that also as protecting your applications as well. Okay? But you'll see a little later where I break that out into something more specific. And then lastly, passkey. So passkey is a little confusing. But where do people put passkey? User identified? Okay. And I generally will maybe accept that, except that passkeys, as designed, are supposed to be tied to your device.

Now, things like iCloud will abstract that and put it on all your Apple devices, but it's actually tied to the device. So in my view, again, my opinion, I could be wrong, is passkey is device identified, the 2FA token is user identified, and then you see where the API gateway and the Zero Trust network access proxy is.

Okay, now, you brought up a good point around Zero Trust network access and it being something that provides access to other things besides the network. And that is the category of Zero Trust access covers that. But we are oftentimes imprecise in our understanding of what Zero Trust network access is. Okay? And to be able to understand that, I thought through this in a more systematic way. So we look at the functions of identify and protect, and you can think of one side as a from and the other side as a to. On the left side, it's the authentication.

On the right side is what does the authorization. So on the left side, think about all the things that we can use to identify who you are or what it is.

Okay, so again, all these asset classes have identity. And so in the current, and how most people think about the current set of market offerings, we typically, this is what we typically do. We have some sort of device cert or passkey. If you want to authenticate the browser or the application, you can use a secure enterprise browser. A way to authenticate your network is like an IP address. And of course, username and password for the user themselves.

Now, in terms of what it accesses, okay, there's all these different access proxies that we have. For desktops, for applications. Instead of Zero Trust Network Access, what I see is Zero Trust Application Access, okay. For networks, of course, Zero Trust Network Access. And for data, we also see something similar. It goes by different names, but you have data access proxies and so on and so forth.

But the perspective here is that this is just a partial way to think about, or this is, what we see in the market today is just a partial way to think about what's available from an identity standpoint. There's lots of different types of identities we can put forth, okay. And we can go overboard here, too, and add tons of friction. But if it's a machine, in theory, you can actually ask for a lot of these things and not really care because it's a machine.

But it takes more overhead, more orchestration efforts to make sure that these systems are communicating with each other in such a way that you know exactly who and what it is. Now, the way to think about that is also, each of these have varying levels of trustworthiness.

And I can go from a scale, this is just an arbitrary perspective, but if I had three different levels, where zero is I'll take anything, to two being a much more rigorous form of authentication, I can set different levels of trustworthiness, saying, okay, I want more of these attributes on the left, but it depends on what you're trying to access. So, for example, if I'm just trying to log in and check my balance, maybe all I need is a few identity attributes. But if I'm accessing a PII database, then I need a lot more attributes on the left side to grant you access.

And the perspective here is that on the left side and the right side, we have essentially a mesh. From device A to accessing any other resource, I can have any number of combinations of things where I provide some form of authentication to an application.

Let's say, if I'm trying to access application H, I can use a form of identity from my device, a form of identity from my network, a form of identity from my application. I'm using a form of identity from the data I'm sending, a form of identity of the user themselves. And the perspective is, all these different forms of identity can be used, depending upon your level of paranoia that you have around whatever resource that you're trying to restrict access to. All right. All that might be a little bit confusing, but I can go into more detail if folks want.

At the end of the day, you have a lot of different zero-trust capabilities that make up the market. First is, on the left, you have identity. So everything on the left is about identity. Identity of the device, identity of the application, identity of the network, identity of the data, identity of the users. And then everything to the right of that are the access proxies that actually do the protection function. And this is where I mentioned zero-trust network access is only one part of that.

Right here, ZTNA, if you think of that in a very strict standpoint, that is for zero-trust access to the network, not to anything else. If you want to give zero-trust access to something else, like your application, a business application, then instead of a ZTNA solution, you should actually be looking for a ZTAA solution. Okay? If you want to get access to a specific server, you don't really care about what network it's on. You don't really care about what application it's running. Then you would have something like a zero-trust device access, and so on and so forth.

What we would love to have as practitioners is one thing that consolidates all these different types of zero-trust solutions. I don't know if that's actually feasible. But I think that's kind of where SASE is heading. I don't know if it's really going to happen, but that's how I would map something like that. All right. Any questions? Sure. Okay. Based on the risk tolerance for that organization, right?

And we all have different varying – and this is kind of represented here, again, logging into a bank account, we actually – in the past, we didn't care that much about authentication because if all you could do, all you wanted to know was your balance, then we would have a very low bar for that. If you want to start moving money, we have a higher bar. If you wanted to move a billion dollars, we'd have a much, much higher bar. It turned out, though, that we had a really low bar for account login just to check your balance. And attackers abused that to then do other interesting things.

And so we had to actually raise the bar. What you're pointing out is what we thought was a sufficient bar of trustworthiness that was needed in the past, because of new and emerging things that we hadn't considered, now we have to raise that bar. And that was kind of done initially, company by company, but eventually became more of an industry-wide sort of standard. I think that's what you're trying to get at. And I don't know if we know what that... Every company will start off with their own sort of tolerance around cost versus benefit and so on and so on.

And I should say, oh, this is just conceptual, but the cost to implement is sort of like the Richter scale. It's 10 times harder as you keep going up higher levels of trustworthiness. So it takes effort, right, to demand this additional trustworthiness from all these other types of assets.

So anyway, I don't know if I fully answered your question, but that's... Oh, yeah, yeah.

It does, yes, yes. In fact, this specific diagram was shared at least amongst financial institutions to be able to say, what should we as an industry kind of calibrate towards, right? And so to your point about, does this become a way for us to all calibrate our own expectations and measurements or calibration of trustworthiness?

Yes, it was. Because we were here, oh, you know what? We should move this from zero to one, okay? And we would debate as to why and the cost, and eventually we would realize, oh, that's a lot easier than we thought. Let's go ahead and move that notch up or whatever. Sure.

Okay, so now let me ask a question here. I'm going to show you this right here, and then I'm going to go to another slide just to show you as well. So give me an example of where we need... So data should have identity, and it's presented to the user, okay? Where data should have identity and it's presented to the user. Another way to think about it here is it's this representation here. So data to a user. So does anyone have an example of a failure of that? Where that's failing for us today? Where we wish we would have some form of data identity when it's presented to a user?

Okay, what kind of certification? Sure. So give me an example. Do we know, is there... Okay.

All right, is there a very prevalent form that's happening nowadays? Fake news, deep fakes, disinformation. All these things that we're seeing where we have unauthenticated forms of data being presented to a person that doesn't have good access controls for that data to determine whether or not that data is authentic is exactly that problem, okay? So I share that just to say this is a flexible model that helps you think about lots of interesting problems that even go beyond...

You can argue it's still part of cybersecurity, but you can see how that's something that works for something even like deep fakes. All right, so just to wrap up, again, I want to reemphasize George Box's quote that all models are wrong, and again, the cyber defense matrix is definitely one of those. But I want to also make sure that you understand this is a strategic framework, okay? And when we think about strategic frameworks, I also want you to think about this from the standpoint of how we look at the world, okay?

In the physical world, we have this nice continuous mapping from something that's very local to something that's all the way at the global level, and we can connect them all together, okay? I don't know if we can fully connect them in cybersecurity, but in cybersecurity, we have similar levels as well. The cyber defense matrix is a model that works at the 50,000-foot level. And I hope that eventually we'll be able to connect it all the way down to the 10,000-foot, 1,000-foot level, but we will struggle in representing it properly as you get further and further down.

The information that you see at the 1,000-foot level will get very, very crowded and very messy at the 50,000-foot level. Imagine if you pulled out an atlas or even Google Maps and you zoom out, but you see just roads everywhere. Nothing would be recognizable. It'd be completely useless, okay? And so that's the danger that we run into with the cyber defense matrix. If you try to jam too many things into it, it will get very crowded and won't give you the type of strategic insight that you need. And it's very similar in terms of even how we think about common words like risk, okay?

The word risk means different things at different altitudes. We use the same word, but what you mean by risk at a 1,000-foot level is very different than what you mean at a 50,000-foot level and certainly very different than even what the board of directors will expect. And the board of directors are at the, you know, 100,000, 200,000-foot level. But at the very least, we have a way to be able to, we want to know what altitude we're communicating with, and we have a way to connect the dots from above and below.

But if you try to jump two levels down, three levels down, you'll start running into issues. Okay, so just be aware of that.

Again, all models are wrong, but some are useful. I hope that through today's workshop, you see where it's useful, but you also see where it's wrong.

That said, if you find other use cases, please let me know. Please share. I would very much love to hear feedback on where you find it useful. I wrote the book because I wanted to share these use cases, but more importantly, I wanted you to share use cases with the community as well. This is how to reach me. And thank you very much. If you have any questions, I'll be around. Thank you.