Control AI with Prompt Engineering

Good morning and thank you so much for having me in today's session. My sincere apologies I couldn't make it in person to the conference, but I hope you are all having a nice time. So today I'm going to be speaking on control AI with prompt engineering and this image right here, it's about garbage in, garbage out and that's mostly the angle I'm going to be focusing on. Talking about the different techniques of prompt engineering, how prompt engineering has evolved over time and also some limitations to be aware of respect to prompt engineering.

A little bit about the future where we are going with prompt engineering. So I'll give you a scenario here. Let's assume a large bank, large corporate bank wants to build its own application using gene I to improve the productivity of relationship managers. These managers, they spend a lot of time reviewing large documents like reports, transcript of calls so they can stay up to date on the client priorities.

Now this bank decides to build a solution that could use a foundation model gen AI model through an API and these tools scans documents and can quickly provide synthesized answer to questions that are asked by these relationship managers. So in order for it to, or in order for the relationship managers to receive the most accurate response possible or answer, the bank trains them in prompt engineering.

Now the other aspect is for the bank to also use agents, what we call large language model agents, which we're going to be speaking on shortly to automate this conversation and streamline various complex processes. So this is typical use case you'll find in with respect to how organizations are using prompt engineering. So on one side you're talking about training people on being effective, it's all about getting the most accurate answers or getting the best out of AI or AI tools. That's what prompt engineering is about.

So today we're going to be looking more around things aspects, which is the corporate training people on prompt engineering. So how can people be better at prompt engineering but also the aspect of automation in order to streamline processes using what we call agents. So that's how prompt engineering has evolved over time. It can be as simple as how to craft a good prompt and we've seen a lot of these these days, this sort of training for the user but also from the developer perspective or from the enterprise perspective, how to also use it to automate conversations.

Well this prompt engineering now was the hype with the garner's hype cycle for gene I, you could actually see it's coming towards the peak of implement expectations as you can see right here. So it's become a very key and important skill respect to making the most out of AI capabilities. And we can also see that 80% of jobs can incorporate gen AI into work activities that mean we're going to be seeing a rise, we expect to prompt engineering and also the fact that 18% of global workforce could be automated. And when you hear automation also product engineering is navigating towards automation.

Now let's really focus on how from engineering, how it evolves. So first of all you've got a base model which we usually call a foundation model, but the base model is trained using tens of terabytes or even more of internet data or public data. And that forms the model. But this model has to be fine tuned to a specific task at hand. So that fine tuning could involve some human labelers labeling data. So you've got the question and the answer as a label and that trains it to become an assistant model.

So what actually makes it more useful for conversation is the fact that you have human labelers making this base model fit for purpose. And this can be, you can pick up a base model and fine tune it even on, let's say you wanted to build a finance app or a health app, this is where you could also spend some time fine tuning the model. Mostly it's going to be expensive but of course not as expensive as training it from scratch. Now we move on to prompting.

So prompting, let's assume you have the user at the user end, they want to make the most of that application and that's where prompting comes in. But there's an argument that people shouldn't really focus on learning about prompting, prompting techniques and all that. And that school of thought is coming from the fact that well if the AI app is going to be user friendly is going to be optimized for users, then users should just like having a conversation.

You ask a question or you say something and people make sense of what you are saying even when sometimes the sentence may not be complete or the grammar may not be a hundred percent but people can make sense of what you're saying. So why do people have to spend their time learning about prompt engineering going through all this prompt engineering courses? So of thought for courses on, well the other end should be for developers for the application to take that user prompt.

However it is, it might even be someone giving a general goal of what they want or someone making some incomplete sentences. But then what happens behind the same is where the other side of prompt engineering focuses on. So take that user imperfect impute and work on it behind the same to improve it and make sure the large language model can out output an answer that is fit for purpose. This is also coming from the perspective that people see. Sometimes people misuse or abuse this large language model. So there needs to be control behind the scene and that's also where prompting would come in.

So these two sites you have on one side like the developer side focusing on maybe security and governance control building proprietary products and automation. And on the user side maybe they just want ease of use, they just want the large language model to understand the tasks that they want or their goals. But both of them could have a shared goal that actually we want to maximize AI capabilities, want to improve on accuracy, clarity and precision of the answers that we receive from these models. And now looking at building LLM applications, it has never been this easy, right?

Just exposing the API of these large language models. You could already have LLM driven application but the danger with that is you don't have so much control. So control control it. When you hear prompt engineering think control because building an LLM application this way you could have unpredictable outputs and hallucinations, which is typical the model making things up. There isn't any specialized use case alignment or this is going to be very easy to replicate.

Anyone can actually build it takes a few minutes if you know what you're doing to build an LLM application 'cause it's just doing something on top of already existing foundation model API connecting to that. So it's very easy to replicate. You are going to also encounter high LLM costs if there's no control and there is no memory on stateless. So these are some of the challenges with building LLM applications this way and what you see is more sophisticated behind the same layout and architecture.

We a focus on also having a prompt manager and some other components like you've got contexts, you've got memory, you've got a model that manages response. You've also got a tax planner. So this is mostly in the case where a user has a very generic goal in mind and send that request and there is a model that plans out the tasks and breaks it into chunks for the LLM to respond to. And this way there's so many things that could be manage here like your use of multiple L LMS as well.

So that is another field altogether where you bring in multiple LMS to optimize the responses rather than focusing on using one in particular. So we're going to be looking at classify this prompts engineering and this is not exhaustive for simply broken into classifying by instruction and also by complexity and flexibility or flexibility. And for the first part, which is by instruction, you've got the base simple one zero short prompting where you just ask a question or put the prompt in no guidance. So that's why it's called zero short and getting a response.

Few short is with a few examples and chain of thought prompting is about making the LLM thing step by step in how you approach the task. And for complexity there's been that evolution increasing complexity. We started off with static prompts then moving down to prompt tuning and solve prompts from B to be focusing on the first seven for this session.

Okay, so we're going to be looking at, so these are some of the prompts and I'll take them one after the other. Static prompt refers to fixed and non changing inputs or set of instruction that giving while we prompt template, you begin to allow prompts to be stored, reused and programmed. And with prompt composition you have two or more prompt templates that are combined together at one time. So you can see here, and this would create a more advanced prompt, co contextual prompts speaks more around providing some context.

'cause we realize that with prompts engineering without more context the answer might be limited. Especially this is something that could reduce the level of hallucination that could occur using these tools or these models. So here some context has been provided but to remember as well, the more context you put into it, the more expensive it's going to get and especially if the context is going to be really, really large. So that's where things also have evolved. So for prompt chaining, prompt chaining speaks more around chaining the prompt together.

So you've got a prompt that leads onto another, onto another depending on the flow of the conversation as well. We've also moved onto prompt pipeline where you have your popular technique called retriever augmented generation that combines the strength of retrieval based and generative models to improve quality and accuracy. So you can have a user send a request out and say that the request comes from such and retrieval from existing databases, online data sources, or you could actually have that response coming from large language models or it might be a combination of both.

So the idea is that you are also providing relevant responses rather than the LLM making things up that it doesn't know or 'cause if you ask a question and that question relates to your enterprise data that is not aware of it could either make things up or give you incorrect responses or best case scenario it tells you actually, I don't know this, but to tell all those responses to your organization. That's where this poor pipeline and retrieval augmented generation comes in.

And there's a quote actually like, and this, it says having personal conversation with an AI is useful but the future really lies in building applications that automate this conversation. So rag is just a start. We'll see agents that can solve complex problems using multiple ais or by taking a simple request and generating a complex series of prompts. And that's where we are getting to with prompt engineering. That's where autonomous agents come in.

You have an LLM agent that has set of tools that works with, has got an interface, it's got some knowledge and memory and within it as well you have a prompt recipe that could consist of the tax attend, the instructions, the persona, and even the parameters that are required. So this way it's able to break tasks down, it's able to decide on the decisions it needs to make step by step in order to achieve that task. And it also has a set of tools that I could use to also perform the task.

So we're getting to that point where, you know, LLM or autonomous agents would be able to have more general goals, context, memory and response formats. So these goals might be more general, like in this case you can specify I need a system to analyze large documents and extract key client insights or some of that general goals as well that are typical application that we mentioned at the start. These relationship managers, what they want to achieve, what their goals are and the autonomous agent can then break that task down and work out the processes around responding to such requests.

So I'll show you an example now this one was typical product manager, how they can use this autonomous agent to do some research and get some key information, all autonomous, all by putting the goal that they they want. So It's gonna look at pros, cons, and the prices. And it's gonna create a written report with, Let me play that again. In this example we're creating an autonomous agent that's going to conduct product research. We're asking it to look for the best headphones on the market. It's gonna look at pros, cons, and the prices.

And it's gonna create a written report with all of its findings. This happens completely automatically. It's doing a web search to find the different headphones, different reviews from different websites and it's going to scrape all that information, clean it up, condense it for us into a nice TXT file. As you can see here, it gives us the top five headphones with the pros, cons and price. And it also gives us more information on the pros and cons. And to make your own agent just use the template I'm providing fork it and you should be good to go.

Okay, so in this example we're creating an autonomous agent that's going to connect. Okay? So we would also look at some cautionary tales beyond the hype of prompt engineering things to be aware of. So reliability users may struggle to judge the reliability of AI ampu because models sometimes can generate different responses for the same prompt. So you I not expect to have the same response all the time. And that's because of the nature of the large language models and these sort of things can create some issues.

Like you can see that in the recent reports about some US lawyers who are fine for submitting fake court citations. So usually that can affect also the accuracy of the LLM security. Gen AI is a double-edged sword. So sometimes malicious actors can exploit it and visit for harmful purposes. So this was a case of someone posing as a chief financial officer called using deep fake as also issues of explainability. 'cause gene ai, they work like complex web. Sometimes you can't understand even when it gives you an answer it's hard to understand how it came up with such answers as well.

You can see even Air Canada recently was ordered to pay customer who was misled by a chatbot. That's because of issues like this. And cost is another one as well cost, even though yes it's less conditionally intensive than training for those calls can ramp up quickly. As an example, if you were had 10 K daily usage or daily users, it can cost up to 30 K using for then using open AI applications as an example. And costs. There's also the cost around your competitive advantage when there is data leak.

When people put, especially for the public charts, applications like chart GPT, they can put in data and that data gets used for training future models. So the thing to emphasize here is the future of pump engineering in there is of course the productivity that it's, the more the automation, the more improvements in productivity. But it's also the side around the caution, how to make it fit for purpose and safer.

And we, I'm going to leave you with this quote. We are not at the moon yet. Human-like AI is still some way off and we're getting to a place with prompt engineering where automation and bringing things into more complex processes is coming in. So I would like you to connect on LinkedIn and maybe we can take the conversation for that. If you've got questions as well, you can always reach out to ask. But thank you so much for listening. I.