Event Recording

Deploying Open Policy Agent (OPA) at Scale and in Production

Name: Deploying Open Policy Agent (OPA) at Scale and in Production
Uploaded: 2022-05-12T12:00:00+02:00
Duration: 22 min 36 s
Description: OPA is a fast rising star in the Authz market. In this deep dive we will cover lessons learned and best practice from early adopters on how to deploy OPA at scale and in production. How can you ensure consistent polices, how do you test and life cycle policies, how do connect with external data sources.

Gustaf Kaijser

Head of Sales

SlashID

Posted on May 12, 2022

OPA is a fast rising star in the Authz market. In this deep dive we will cover lessons learned and best practice from early adopters on how to deploy OPA at scale and in production. How can you ensure consistent polices, how do you test and life cycle policies, how do connect with external data sources.

Video Description

Show Transcript

Thank you. Thank you. And thanks first to John, we talked about one other use cases, the infrastructure, we had a call or session earlier today by Magnus from Solando talked more about application use cases. I'm gonna try to talk a little bit more general. What have we learned from the customers we have had so far? What are the best practice and what do you need to think about when you're deploying a bad scale in a in environment? Okay.

See, this is not real working for me. Can you me a bit Help our presenter?

Ah, oh, right. Working, everything looked you super. So first of all, what ISPA OPPE open source project soy a commercial vendor on top of OPPA that has a controlled plane. We wrote and released OPPA open source in 2018 and OPPE really at PDP with old terminology. That is a decision engine. So you send it a question in format. We evaluated question or with the policies and the data we have attend, you can send us data in the request. We can cash data and we can fetch data. I'll come back to that later. And we send back an answer in format.

It's not more complicated than that, but that also means that in a cloud native environment, you will not have one open as Magnus from Saland said, they have 4,000 microservices. So that means they will have hundreds of thousands. So it will be highly distributed. And that's the whole idea. It should be as close to the service as possible to keep down the latency. We are always looking for seal like single dig, millisecond, something three, four millisecond, or even sub millisecond, if possible. So that's where we are.

Uber, as you heard yesterday has become a great success. I thought when I joined a year and a half ago, or actually closing in in two years, we had 7 million downloads. We were like 1,700 people or something on the OPPA slack channel that we had grown a lot. And we had, but last week we passed like 130 million downloads and actually checked this morning.

We are 5,850 people now on the slack channel. So it constantly growing and growing faster and faster.

I said, we released this to the club. Now the computer foundation in 2018, and we actually graduated less than three year later among this time in 2021 SOPA really vivid community and has become the defactor standard for authorization all over the cloud stack. And what are people using on perform every year around this time? So if you are an open slack channel or interested, you will soon get a little pull to see where we are. So we see where people are using it and how it's growing. This is from the polar. We did last year in March, April, sometime.

And we see the biggest use cases are not really external and compliance, like fine tech, like PCI, like banking, things like that is more internal compliance and best practices to make sure that developers and operations team can work fast in a secure way. So that's what we see our people are embracing, but to start with little, be surprised to me when I saw it, that actually end user identity is 40 over 40%. And I think that will grow even more.

And another thing is the use cases, Kubernetes emission control is the single most common, but if you look at application and authorization, not real and powered, but if you combine them together, that will even be bigger. And if you are rather good Matthias, you see that is lot more than 100%. And the reason for that is that over 50% of the open users have more than one use case. Normally you start with one use case, but we see that already within a year or something, people are adding more and more use cases. It tend to grow and tend to grow between different organizations.

And we also see, this is rather static since we're growing at that phase VI. So a third that are in experimentation, a third pre-production and a third in production. What I haven't shown here is also how long does it take actually to get into production? And what we usually see that within six month, 50% or 50% plus are actually in production and within a year is 80, 90% are up in production.

We, we talk about one way or another. Okay. And this is normal where we as a vendor come in. And when we talking to people, because since open, open source developers and people like starting using bar, looking at the documentation, trying it out, and then they come and want to talk to someone because they say, okay, we used it for three to six months.

We really, really like it, but now we won't gonna deploy it at scale. Okay. So what are the best practice? Do we need some enterprise support? How should we write a SIS? Are there some kind of tooling? Are there some kind of training we should? And that's really, when we get into the picture, Shane Lawson is a C of extend doing point of sales and other commercial solutions. We actually have them sitting down here as one of our customer.

We had a small meetup in Stockholm two weeks ago and was, Heva said if I have a security policy and I ask 10 developers to implement this policy, they will come up with felt different, brilliant solutions, but it will be held different. And that means that live cycle lemon managed them over time will be extremely cost. And that's really why people looking at deploying oat scale, just deploying a tactical one, people use open source more or less. And this is normally where people have been. And when people's first impression of Uber, it's like what we call the rego playground.

It's a webpage where you can write simply reg policies. It's the prebuilt integrations that a community has built for us is growing while every month I think is over 50 now. And it's still past slack channel where you can ask people from Tyra or from APA vendors. It's like 5,000 people there. And we have at least six to eight people there, more or less full-time answering questions, helping out. So that's where people come from, but really to get the benefits because just decoupling authorization from a microservice or an API to a Sycor will not bring you in benefits.

It's just moving it from one place to another, the benefits and this slide a little bit in the wrong order. Come, if you can get consistent policies, that's what you need to aim for because you want to re well, sorry, write the policies once and then reuse them instead of writing the policies 10 times and not reuse them, will you earn a lot of money or a lot of time by doing that? You will own some, but that's not really the big benefit to return on investment.

What people tell us when it comes to is that if you have consistent policies and make sure that you reuse them between APIs and microservices, you can do security reviews in a central place. And that means that instead of looking at different services, you can look at the policies and the policies decisions have been taken and doing audit or security reviews that way. And that will free up a lot of time for your developers because they want to code. They don't want to see it in security reviews. So that's one or the key benefits people tell us.

The other thing is decrease the cost of change, or actually the ability to change your speed or change because authorization roads needs needs to be adopted from time to time. It's not something you do every week or every month, but politicians change your relations. That's what we have them for more or less, and your reorganization, your company, and things like that. So you need to change the authorization lot different time to time.

And instead of doing that in the application and redeploying that, because that usually take everything from like two working days to a week, you do it in a central way, push out the policies without redeploying. And if you can do that and share the policies with 2050 other microservices to return on investment is great. So that's really the driver. Our people are implementing all that scale and what you're trying to achieve, the only way to do that is to have some kind of policy lifecycle and a central control plane. That's why Uber actually comes with a couple of public APIs.

And that's what you need to do in order to create some kind of control plane that can cater for all of the life cycle. You will probably not need all of the life cycle tools in the beginning, but sooner or later, you will get that. So there are the bundle. It's a way to distribute policies and data, discover to make sure that Webpass are, and they are happy status and decision logging is on the other hand. So you can upload and have a central view on all our decisions. We at Tyra have built one that we call the Ty da and it cater for a lot of different use cases.

The star be VIPA is not just for application. Jo was talking more here. What we call our mission controller infrastructure side Salanda was talking more on the API microservices. We also see people protecting their CIC, the pipeline. We see some people now using it to do dynamic filtration on databases. And we are also going more and more into pure identity with our new entitlement system. So that's really where we are going. And I'm gonna talk a little bit about the policy life cycle, using the standard as an example, how you can do that and the functionality you need.

So this is how it looks when you log in, it's a free version. If you're going to C, so you can just click there. And as long as you're not using more than four system, it's total free here on the left hand side, the see that what we call a system. And that's a way to work with OPA could be one, OPAC could be 10 us, and here we have a decision log and of course you'll need to make sure, sure that some people can see some data. Some people can see some system. So you need to make sure that it could be secret. Some people shouldn't change the other team's data and things like that. Okay.

So now you start here and just make sure that we're on time. Perfect. The first thing you need to do is okay, how should I enforce my policies? What we see as the best start is normal. People start with some kind of gateway. The good thing with that is you don't need to touch the code so you can do core screen, or you can do your first check on the API level. We also see the same people doing more and more fine grain at the microservice level.

Today, we have pre-built enforcement points or integration for a set of policies, but everything is possible is just a and call. So every like mesh technology or API, anything that can actually send out a yay and call for external authorization will work. But you need to define that that, okay, this is how we, as a company, work with enforcement, these are approved enforcement points. And when you have come so far, it's time to write policies the most, most interesting and fun part. There it's a little advertisement. This is total free. We have the star academy.

It's how to run off place, how to write policies free courses. Now when we're adding more, so go there before you start writing policies. Then when it comes to policies, different people like writing policies in different way. This is the core way, raw ego writing policies. When we pro talking policies code, and we're talking developers, they write it this way, developers like that. They don't want to have a higher abstraction layers. A developer picks this up in a working day. Last summer, I put my 16 year old son to write some rego for me. And he was writing rego the second day.

So it's not more complicated myself. I haven't written a single line of code since I left university 25 years ago until I started at star. I'm actually cheating a little bit Ingo. Not that I can really develop, but I can change. So it's not that complicated can an order to read it. Yes. If it's well comment, it can, but you need to make sure that they comment in the right way. That way that we see is what we have also, that there are some kind of abstraction that people doing, sort of writing policies every time here is RGO in the background.

It's just another way for people to ease the righto. So this is what we see people that are not code savvy using and what we see coming more and more, but for specific system type is we call pre-built policies or policy snippets, where we have a policy and you do like in a configuration, it should be for get, it should be for a post. It should be for this API path. So we see more and more functionality that, and as the systems mature, we will have more and more pre-built policy languages in pre-built policy libraries. I see.

And also what we seeing, and that's a functionality of larger that customers would write their own policy snippets. So like, we will have a Saland policy library that they can pick from.

So you, as a customer should be able to write your policy policy library that your developers or more business people can pick from in a, like a drop list. That's what we see coming more and more. Okay. So fine. Now we have written all these policies. Yes. But the issue with the policy is, is what completely nothing without data. Cause we're talking actually attribute policy based access. So we need a way to get data into the policies. And there are a couple of things, oh, sorry. I'm going back. I have changed the slight little since I see how they did sharing policies.

We're starting with that before the data part, there are a couple of ways we see the people share policies. One is libraries as code is so it's optional.

We used, I write policies. I let someone else do it. I can like import a library. The other thing is that we might have some company policies that should always yield regardless what a local developer is trying to do. There should be some like group of policies that we need to enforce. And you can do that one way is working with stacks. So the stacks have been elevated first. And if you pass the stack roads, you will go in to the like app specific or API specifically.

So these are the way to forget identity teams or central security teams to enforce policies and put a guard reigns out for, for the developers. Now to the data side, there are three ways we can consume data in an open. What we always see is that if you have the data, when you're requesting a policy decision, send it, that's what we see. You need to know that. Of course you need to send like some kind of identifier, some kind of action. Normally we see in all application use cases, they're sending us ye token and an action. That's what we see.

But if you have additional data, send it, that's the best practice. So what we're doing here, we're getting a ye token in and we pass it up and use that as a piece of the Polish evaluation. The other thing is that we working, which we call policy policy and data bundles. That means that we load and cash data in the open. The reason we do that is that we want to have beam as fast as possible. Like we want actually, we want to be like a millisecond latency, something, if that's possible. And then we can't go any network hope.

So we need to have it open, close, and we need to have the data in OPPA. So we do that by connecting. We have some pre-built data sources, but you can actually upload any kind of data in a use format in the API. So that's couple of things to do when the third way working with data is to fetch data during evaluation. We see people using that as a lost resort. Really. We don't really see people using that because data are dynamics so often because data are not really that dynamic and you can build this. So it works 10, 15, 20 seconds or something.

What we see is people using this data is getting too big and they have big distributed environments. So if you have like a hundred that need the same data, you can't put gigabits of data in there because you build from Amazon a bit high actually. So you want to keep the data small because you want to keep it as close possible and you want to be able to scale it horizontally. So that's what we see when people doing that, but we don't see that sofa. So you try to avoid that. It's my best practice that, okay. So where are we? We have right policies. We have the data fine job done. Not really.

We need to test the policies. That's also something we need to have a central way and test the policies as soon as possible before they hit the production. And what we see is people doing is the shift left theory. They try to test the policies as to write them as to mock them up. So in this cases, someone have written a policy and we're doing two things outta the box. We have pre-written test cases because we're good developers. We start writing the test cases actually, before we start writing the code. And we also doing something that is even more important.

Some cases we're doing impact analysis. Okay. Even if the policy pass my test. Yeah. But what will happen? Will it change a lot or will it actually change nothing? I need to know that before I implement something. So this policy, well fine.

Green, nothing change may, maybe something should change. If I change, write a new policy in this case, maybe something I should think twice before I push to production, my test cases are breaking down and I see after the 600 last decisions taken by this 200 will actually get a new result. So maybe I should do a second check or maybe this is exactly what I wanted. So I can check here, actually inspect the thing. That's getting a new result to see that what I wanted. So that's really how people work with them. They take what they have.

So now I'm E bit cheating because now we have gone the wheel pro and looking at historical data to violate policies. So that's we see, okay. So now we've write the policies. We have the data they are tested and let's, let's put 'em out. Let's put 'em in production. Maybe not. So what we usually see is in here, me and grey doesn't really agree. One of a few points, developers love to have a, like a GI integration and use that as a CIC, the pipeline to make sure that you have version control and a roll work functionality question is, should we include this more inside the tool?

We haven't got that yet, yet to developers really, really like working with get, so this is the way we're doing it. So push it to review branch, someone merge it, someone approves it. We have diversion control in there. We know which policies were in place is what time? And I just make wait on time. So we have 10 minutes we time. Yeah. We was just looking at watch. Some was afraid. So that's fine. So when we have done that, we pushed them, we take them back and then we pushed them out to make sure the right OPA get the right set of policies.

Because all this OPA here will have a little bit different set of policies, different set of data, the share, same, but there will be a difference. And other thing is we make sure that, okay, which kind of policy are implemented in this Uber at this stage. So that's really where we come more to distribution and what do we need more? Okay. So now the Ubers are out there. Now they are working now, everything is fine. Now it's time to get the benefits. And that's really what I can say. Okay. What kind of decisions are it taken? So we need to be able to doing the orders and doing belonging.

And as I showed in the beginning, there is a decision log API. So what you can do is when you download the bundle to the power, pulls the bundle down, it could also give you the decision log and you should work it this a couple different way. We use it for a couple of couple of weeks to have the data. So you can work with when you write policies and work with the policies, but you should, of course, lock it down somewhere for all purposes as well, because we are not an audit. We're not the C and platform. We are a policy administration poll.

So here we have the datas, we have approvals and we have Denis and we can actually, for every different system, we can also do a deep dive. We can look in, we can look at different policies. So what you see here is the input. You see some times stamps and you see outputs, you can have that all fidelity for everything. And what I not showing you right now is that you can actually say, okay, this decision was taken two days ago. Now I changed the policy to correct something. So now I can replay against the new set of policies to see.

Has, has anything changed or are we more or less on the same level? That was all I had for today. So any questions from, from audience.

Like this?

Don't like this?

Why don't you like this?

Deploying Open Policy Agent (OPA) at Scale and in Production