Data Strategy for Generative AI Applications — Why Your Data is the Differentiator

So hello everyone. Thanks for taking the time and for joining the session today where we talk about generative AI and also data strategy and why we think this is an important topic and why it goes together. Many people like build generative AI applications and there's a lot of attention on it and lots of focus is also on LLMs and prompt engineering. But we should also focus on data because this is a differentiator between your company, your organization, and maybe other organization. So my name is Manuel, I'm a solution architect with AWS.

And with you today is Pria who will take over the second half of the presentation. And we are solution architect. That means we work with companies together in Germany from software companies and make them successful in the cloud. So generative AI has the ability to transform industries and maybe a quick show of hand.

So how many of you have used generative ai like in your personal or professional life, like chat bots, chat, GPT, quick show offense, and how many of you are trying or building or planning to build generative AI applications into the application or your software that you want to build? So also, also quite, quite a few. So when we think about generative AI applications data out there, we often focus on what this function, this feature or this application does. And maybe we also focus on the large language model or the foundation model.

So the models that are trained on a vast amount of data in order to, yeah, to inference on them. What we mostly don't see or take less attention is what's underneath it. So underneath the surface. So we need a solid data foundation. We need to think about how to govern this data, what to do with the data, which data we want to use in order to provide value to customers and how we can make it available to the alternative AI application. And this is what we want to discuss and think about today in this presentation.

So how we can put data plus alternative AI together and how we can create value for your customers. We did a survey with CSUITE executives, AWS internally and asked, so what is the biggest challenge if you want to build alternative AI applications? What are you facing? And the number one thing was data quality and the data that you use for those applications among the top ones are also finding the right use case. So what should I do? What can I do?

And also, number four, security and privacy. We've related today at five 30 also have a talk on securing alternative value applications. So if you want to join that, you're more than welcome to do so. You probably also know about the capabilities that those large language models or foundation models have. So you can generate text, you can summarize text, also classify text. So this opens a lot of use cases out there that you can build with those models.

Also, different modalities, like you can use it with text or generate images, understand images, or also use it for audio or video creation. So it opens lots of possibilities that you can use to build applications and to build value for your customers. One example application could be a conversational search application. So imagine you have company data in your organization like Slack, like Vicki, like other assets that you have. And the question is how can you search within them? Convenient day would be to just type in what you're looking for. Maybe you're looking for a policy or something.

And degenerative AI application can go in and look at the policy or provide you like a normal text and provide you the sources where it got it from and provide you the answer to it right away. So this would be an example of such a generative AI application that has to take out the underlying data and have to think about this. So when you think about data, we need a strategy, how we manage, how we access data and how we for example, also store it and make it available to alternative ai, AI applications or analytics applications. So let's have a look at data strategy and how we think about this.

So what do we want to do when we think of data? So on the one side we have the data in your our organization that we can use. And what our end goal is, we want to create value from this data. So we want to sort it, we want to, yeah, get, get something out of it. So create business value to be more specific. And the big question is how do we get from all the data that we have to creating business value and to making it available and usable for such generative applications? Let's have a a look what this entails and what you need to think about and also what a data strategy should cover.

First, value. Value is very different for different organizations. So there's no not one size fits all model, but there's yeah, different things that you want to achieve that others maybe not want to achieve. Want to optimize for cost, reduce cost or maybe optimize processes or reduce customer churn for example. And then also of course the data that you have available is not the same that other organizations have available. You have different internal external data. You have data that you can buy or that's openly available on the market that you can use in order to create the value.

So we need to work backwards from the value that we want to generate and think about the data that we have and can use. And then there's different layers that you think about when you create a data strategy. So from a technical layer, the data value chain. So how do I actually store the data? How do I make it available? How do I analyze it, transform it, and maybe also do backup? Then we have data management. So we need to ensure that there's the integrity of the data that we can actually make it available within our organization.

'cause you have data sitting in a database that nobody uses or is not available. There's no value in this. So we have to think about this as well. And on the top data governance. So what are some policies? Who's the owner of the data, who can or cannot use it? And what metrics and other responsibilities here. So you need to think about those things as well. What are some challenges? So there are different challenges that can arise, like having data silos, different databases containing different data or data warehouses, different databases that contain these data.

And there might also be people silos. So people sit on data, you probably want to make it automated in order to get the data out and to use it. And also business silos maybe have different organizational units that sit on the data. Maybe you need to think about like the legal and compliance implications on making this data available.

We see customers implementing end-to-end strategies for or end data strategies where you have producers on the one side that are domain expert that make data products available internally or to your users and consumers on the other side who know the business, who know the business value that they want to create and who know the requirements. And in the middle, middle you have a data foundations team who's, for example, the central IT who connects the two, who makes a data catalog, who who gets the data products together and who, yeah, connects consumers and produces.

So to summarize modern data strategy, what should we think about and and what is it? So one component is of course technology. So how you build it, for example, if you want to build it in a cloud, it's possi, it's possible to do it with the data lakes that you can then use for machine learning. But there's also other components like people and processes that you have to think about like also transforming the way you think and also changing the mindset. So going from people having data and also to sharing it, making it openly available.

And also technical people working together with business people in order to, in order to create the value. And now I wanna hand it over to Pria who will talk about how can we use this data in a generative AI application and how can we build such applications? Thanks Naru. I hope you can hear me well. So let's have a look on how we can actually now take the different data that we have coming from different places and apply it to a gene AI application. And before we start, so maybe a question to you who has already created their own gene AI chatbot. All right, we have few people here.

So these are the brave ones, the bravehearts. And for the rest let's have a look on how we can actually take our enterprise data and create something useful for customers. So here we have a a marketplace for insurances. So many of you might have a car insurance and typically once in a year you think about, well it's actually quite expensive, so let's look for a new one. And here we are using a gen AI chatbot actually guiding us through that. And what do you want to look into now is what type of data do we have and how can we use it together with the large language model?

So you also saw, if you saw the previous session that with large language models it's actually quite easy to add your data and with that to steer away the model's behavior and for that direction what you really want to see. So here we have a chat bot and what the chat bot does is it helps now our user, and this is a known user, so we have someone who has previously been registered here and we have some information about the person. So now the person ask about the car insurance.

And now what the gen AI model does is it has some follow up questions, things like, hey, I know last time you drove your BMW, is it still the car you're using? And I know you're living at this place. All these data that I have about you correct, we will get to where this information is coming from and how we can actually teach the large language model this type of information.

And ideally what we are expecting is something like this that the AI assistant answers a few proposals to the user in text form and then the user can decide if they want to take for example, an insurance from company A, B, C. So now let's have a deeper look into what we are actually sending over to the large language model. So the first thing is a instruction. What you can do is with a large language model, you don't actually need to take a existing model and fine tune it to teach it a specific task. You can just use a text input.

And with this text input, the model will do the type of task that you are telling it. So this is called incon learning, which means we take a text input, give it to the model, and with that we can shape the model's behavior. This is quite useful. And now what we are doing is in this instruction we are telling the model that we are a marketplace for car insurances and the model should behave as an agent and support our customers.

Alright, so with that we are setting the guards and the guard rates how the model should behave. Now next thing is what we can also add is so-called situational context. So situational context can be information that we are for example storing in relational databases and tabular forms. So it's very well structured and if we have a session now with the user where we know the person or maybe previously throughout the conversation we just gather this data, then we can use that with our large language model.

And in this specific case we are getting the data about the user and now we are putting it into a string format and telling all of this about the user to the model in this exact way as we can see it here. So we are telling the model, this is Bob, this is where Bob lives. Bob is using A-B-M-W-X three from 2021 and also his personal situation. So this is data that we can get directly from our databases or maybe through APIs. But there is also another way how we can actually during runtime, automatically fetch additional data. This is called semantic context. This works a little bit differently.

So with the situational context, we are really filtering for specific data which is well structured with semantic context. We are actually using text to search through a a space of data. And this space is a vector space, which means that for every unstructured data that we have, we have actually vector information, which are just numbers describing the data. You might ask yourself, well how do we get actually these numbers describing my unstructured data? There are machine learning models for that which you can use.

These are called embedding models, but we'll not dive deep into embedding models for this session. But what is the takeaway here is that you can actually use semantic search and apply it to your unstructured data, which can be for example PDFs or even images or presentations. And then through the semantic search you can find relevant parts or chunks of these data that you have there. And now we can also use that for our agent here guiding the customer. And if we have some specific policies that apply to the offerings, then we can find them on the fly through semantic search.

And then additionally, if we want to, we can also add the user's current requirement from the chat. So which is the person asking for a car insurance. So all this back together and it's all at the end in text form when we send it to the large language model, this is the Incon learning information that we hand over to the large language model helping us to process everything for the customer.

Alright, so let's look at a very simple architecture, how this would now look like. So we have of course the user and the user task, sorry, the user talks to the general application that we have here and this now is our orchestrator. So this can get additional data from our databases for situational context or search through our unstructured data through semantic search and then send all of it as one prompt over to our large language model. Then we will get the response back and then show it to the customer. So this is all the information that we saw now as an overview.

And now what you can also do is this semantic context can be fetched during runtime, but there is also another approach which is let's say you have information that almost every customer needs, and of course you can load it every time through a semantic search during runtime. But what you can also do is you can actually teach the model disinformation and you can either create your own model from scratch or you can take an existing model and fine tune it. And now what happens is we move from in-context learning with retrieving the data on the fly to something like this.

So now what we have is you can see from the prompt that we are sending to the model, we have extracted the semantic context and we have created labeled data. So this labeled data is typically question answering examples. You take a domain knowledge that you have and you create a lot of question answer scenarios and you can even use large language models for that to create your data synthetically to simplify the process and decrease the effort. And now with that you can fine tune a model or create a new model and now you don't need to fetch this information always on the fly.

And then we have the same approach. So what we do is we use our prompt sending to the model and the semantic context is now already inside the knowledge of the model itself. So it doesn't need to retrieve it dynamically. And this is now a combination of in context learning. So why in context learning? Because we have the situational context which we load from our databases and our APIs, and then we have the instructions which we define. So this is the in-context learning. So we don't fine tune here at all. And then we have also the semantic context inside the model knowledge.

Now it's important to understand the differences here. So if you want something very easy with the least cost needed at the beginning and also with the least complexity, then we always start with retrieve augmented generation, what you see at the bottom. So this is really the dynamic fetching of data for situational context, for semantic context. And here we don't fine tune the model and with the skill sets in our teams and also the cost that we want to invest, we will ideally move to a higher levels if we don't, if we really need to change this dynamical approach.

And then we think about fine tuning a model or even creating our own large language model for our domain. And with that you can see the higher you go the more complex it becomes. And first off, you typically start with retrieval augmented generation, which is this dynamic fetching of data. Now we have seen at the beginning that large language models support different modalities and this is also something we want to consider for our data strategy. So we know that we have different types of data can be structured, can be unstructured.

And now let's see how we can bring all together for our gen AI model. So on the left side you can see what we also saw earlier. So we have a user interacting with our gen AI application and we have now two things. So one on the bottom is the prompt template. So the prompt template is containing instructions that we have predefined and also pre optimiz, you might remember the car insurance agent. So this prompt would be something describing how we want the agent to interact with the user providing car insurances and we have pretor them somewhere, for example on a object storage.

And then we have also on top a conversation history. So let's say this is a reoccurring customer visiting our agent or it's a customer talking now for a while with the agent. And what we can do is we can memorize the conversation and we can also use this as an additional information for the large language model to help the customer. Could be things like the person just changes the his address or car information on the fly or the person was interested in another product few days ago. Then we can load all this information to the conversation history.

Now on the right side you can see the different data types that we have. So we have our structured data inside our relational data stores or no SQL data stores. And then we have also the unstructured data below, which can be for example, videos, PDFs, emails or photos. And now what we can do is we can actually leverage both of them for our general application we will need a pipeline to ingest them. So we need to load them and sometimes we need to transform the data a little bit more, maybe also clean it up. This would be more towards the structured data.

And for the unstructured data, what we need is we need to create metadata for our data because let's say we have a PDF and it's unstructured, so we need some vector information about it describing it so that we can find the relevant parts of this doc document later on. And as I mentioned earlier, we can use machine learning models for that. So at the end you will have something like this. So you use your embeddings model to take the data and create the embeddings and then you can store the embeddings inside your data lake for example.

And then later on you see you can load it as a semantic context for your gen AI application. And this is also called rag pipeline. So it's a retrieval augmented generation pipeline. And the second part which we have here is then the situational context, which you might remember, which was the structured data that we have, which we load directly, for example, from our data warehouse or our databases, which is then on top in the situational context. And then altogether. So we have the situational context, semantic context, and also our of course main component.

The large language model altogether combined is then our gen AI application. Now we saw different approaches. So we saw the incon learning, which is the loading of data on the fly. We have seen training your own model and we have seen fine tuning. So when to use which. So if you have data that you need for almost every customer, but the data doesn't change a lot, then you would actually train this data maybe once in a year and really bake it into your own large language model.

And if you have data, which is also used a lot, but changes really frequently, like things of a speed of, for example, once in a week, then you actually need a fine tuning pipeline. So this is really a pipeline that is being triggered frequently. And with that, you're teaching the model this data through a labeled data set with question answer examples. And if you have data that can change, for example, within seconds, then you would typically not think of using fine tuning or creating a large language model from scratch.

You would actually use retrieval augmented generation, which is the loading of data on the fly through incon learning. So with that, we are coming almost to an end of this session and I would like to give you a brief overview of the most important parts that we saw here as a takeaway message. So first off, when it comes to gene AI applications, we don't only look into PDF documents and load them. We think of structured data and unstructured data.

So we will also leverage our data stores from relational databases, data warehouses, and bring everything together for our large language model as semantic context and situational context. The other part is that we need tools that help us to integrate this data regularly into our solution. So it can be data for our training pipeline to fine tune a model, for example, once in a week. Or it can also be the transformation of unstructured data into a semantic context. And then of course we need to have a governance solution towards security.

So to make sure that the right people have the access to the data. So we should be having this modern mindset, so to allow people throughout the organization to collaborate with each other, but we should also be really clear about who's doing what. And we also need to really understand our data. So the better the data quality is, no matter if you're fine tuning or using retrieval augmented generation, the better results should be. So therefore we need to understand is our data clean enough? Do we have bias in our data? And bake this into our pipelines. And with that we're coming to an end.

So thanks a lot for your attention. And we also would love to share with you a landing page that we have.

Oh, it's already gone. Maybe we can just go back to the last slide if possible.

Alright, thank you so much for your, yeah, yeah, thank you so much. And yeah, see us later for questions. Thank you. Thank you. So a big thank you to both of you. We do have a quick question if we can handle quickly.

Yeah, of course. And it's having to do with internal data and putting borders on this. The question is, if you let users access corporate resources to answer questions in a chat bot, that data might be in parts only for internal use. How do you set up the borders? So I mean you can set up like Guidewire or permissions internally who can access the data and what, what you can do with the data. And then yeah, restrict it internally to to build value on top of the data and to maybe do analytics internally. Thank you for that.