Prompting the future

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at Fly IO, the home of changelog.com Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents so you can launch your app near your users. Learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am founder and CEO at Prediction Guard and I am joined as always by my co host Chris Benson, who is a principal AI research engineer at Lockheed Martin. How you doing Chris?

I'm doing fine. How's it going today, Daniel?

It's going great. I'm pretty excited to prompt our guest today and hear what he has to say. We're joined today by the Prompt Master Jared Zonereich, who is founder at PromptLayer. How are you doing, Jared?

I am doing well. Excited for this.

We're excited to have you.

It seems like maybe from my perspective there was kind of the release of all of this generative AI stuff and then there was this realization that there's kind of a new skill needed around this thing called prompt engineering. And then it seems like some people have kind of like, I don't know if they've moved past the term or they've tried to develop other terms. I see other terms being developed around like AI engineering and such and as related to generative AI. So could you just give us a sense of like from your perspective, who, who is obviously building things for prompt engineering? Maybe to start out kind of what is prompt engineering? And from your perspective, how have you seen it develop as a skill over the past year since people have been thinking a lot about prompting generative models.

Yeah. In the last year since the word was invented, or maybe it was more.

Than a year ago.

But yeah, no, I think it's a really good question. I think this question of what does prompt engineer even mean? Is also a good question. And I think I'll tell you how we think about it. So honestly. So for Prompt Layer, we consider ourselves a prompt engineering platform. So we've really steered into this term that is kind of overloaded. We've embraced it for sure and it was half by accident and I would say half realizing it was kind of beneficial to us because the fact that it has no definition means anyone who's kind of getting into LLMs and getting into prompt engineering says, oh, there's a prompt engineering platform, maybe I need that. And they don't Even necessarily know what it is. So it kind of helps us a little bit there, I guess. Prompt engineering, I first started hearing that term, I'd say GPT three days, kind of a little bit before ChatGPT, maybe GPT two, back when everyone was just using the OpenAI playground and you'd start to hear a little bit about prompt engineering and stuff like that. And it was kind of cool. But that was, I guess, in the days before it clicked for everyone, maybe you can call it that, days before ChatGPT came out and before people really realized how much potential this technology has. So that was when I think prompt engineering first became a word and then kind of got even more, you know, scale. AI, I think is famous for maybe the first one to publicly hire someone with a role prompt engineer. And from there we called their platform a prompt engineering platform. At the beginning, people brought a lot of people to us who had no idea what prompt engineering is. And kind of the definition we've started to roll with is, as a company, we consider prompt engineering to be the tuning of the inputs to the LLM. So the prompt is the main input, but it also includes what model are you using, what are your temperature, what are your other hyperparameters. But the whole process of prompt engineering to us is what goes in and what comes out. And that specifically is. And I could talk a little bit more about this if it's interesting, but specifically a little bit different to the mlops definition or the standard machine learning definitions of hyperparameter tuning and standard traditional ML. And specifically we call ourselves a prompt engineering platform and not an LLM OPS platform for that reason, because I do think there is slightly a difference. So I don't know if I answered the question fully, but those are my.

Thoughts on prompt engineering as we all dived into it. Like you talked about, you know, the OpenAI playground originally that I think everybody kind of dips their toe into first, at least before the ChatGPT release days. With that. One of the things I discovered as other models started coming out was some of the skills I was developing for prompting did not always translate as I expected, across to other models. Have you, have you seen one of those things where every model had its own kind of variations of what seemed to work from a productivity and output standpoint? Any thoughts around that? Like, how do you guys, how do you guys see that with all these different models coming out and little variations across them in terms of how they respond?

Yeah, that's for sure a good observation. I think it's become clear that each model, I mean each model is made a little bit differently. So you have to talk to it a little bit differently. I think these differences are going to maybe get a little less significant over the future. You know when I guess ChatGPT came out and the big thing was if you were nicer to the model, you're going to get a better answer, maybe because stack overflow questions that are nicer, get better answers or something like that. And I think now you have people talking about, oh, if my grandma's going to die, you need to answer this or I'm going to tip you $100 if you answer this. These are all, I think tricks that work today. They're not going to last forever. These are like little things people figure out. But the part you mentioned of talking to different models differently I think is not going away. These are a lot of these models are made very differently. We, I think it's pretty conclusive now that we're not going to live in a world with an OpenAI monopoly of language models. I think just a week or two ago I saw a good tweet like now we have Mistral, Claude, GPT4 and a few others that are all really good and they're all made somewhat differently and there are intricacies. And I think our philosophy and what we talk to our users about regarding prompt engineering is think about it as a black box. It's almost helpful to be kind of a little bit naive and stupid here and not try to understand how an LLM works and just try to track the inputs to the outputs, if that makes sense.

Have you found any difference from people that are coming maybe from a deep sort of like data sciencey background where they like are overanalyzing everything and then other people who are coming maybe from a not technical background, but they're maybe domain experts and they're getting into developing these prompts. Have you seen different struggles on each side of that spectrum where. Because you have sort of this very interesting mix of people that are trying to be like prompt engineers, quote unquote, some of which I've kind of seen are very much just like non technical domain experts who are really good at even psychology and writing narrative instructions, being articulate. And then you kind of have this other side which is the data science side and they're really into modeling and wanting to analyze all of the outputs and that sort of thing. So have you seen different struggles on both sides of that in terms of being effective prompt engineers?

I think you put it well that there's kind of these two groups coming at LLMs traditionally, I mean this is what makes LLM so cool to me is that traditionally ML traditional machine learning standard, like mathematic machine learning, kind of need a PhD. Maybe you don't need it, but PhD is very helpful. It's intense how you're building these models, doing a lot by hand still, you're doing a lot of this tweaking by hand in traditional ML. And then OpenAI came out with this amazing API and I've done a lot with DevRel in the past and hackathons and stuff like that and helping companies make APIs and in my opinion, OpenAI's API, it's like the best docs I've ever read in my life. It's so simple, you just give it text and you get text out. And like you said, this kind of opened up this completely new technology that's just so much better than everything else to non technical people. You don't need a PhD to be able to understand how to communicate. And I think it brings about this new skillset which is prompt engineering, which in my opinion is kind of a mixture of communication and like being able to write succinctly but also being able to think algorithmically. So I don't know the exact word for thinking algorithmically. I've heard Stephen Wolfram use that word. I like that way to talk about it, but kind of just the scientific method. Do you know how to think in terms of creating a hypothesis, trying it out, tracking it? Are you strategic about this? And I think it's the same challenge on both sides. I think some people try to overcomplicate it. If you're coming from an ML background and you try to understand why a certain token gives you an output and I almost think these things are getting more complicated, not less. You kind of just need to take the naive approach and say, hey, I'm just going to talk to it. I'm going to try to get the output I want. I'm going to keep trying stuff till it works.

As someone who's used like the APIs a lot, you know, and you're talking about, you know, OpenAI as being so good, so many folks listening to this may have only have only used things like the normal chat interfaces on each of the models, you know, when they've gone and tried them out, or pay for a subscription for the, the top end and stuff, and have never touched the APIs at all, could you take a moment and just talk about what an API experience is like to Someone who has done a lot. I'm kind of taking advantage of you as an expert in the area. Share that a little bit with listeners, just so they kind of get the other side of that. Because not everybody, probably the vast majority of them don't.

I'll explain. And that just to sidetrack myself for a sentence, I think the fact that you use the word expert is so funny, is. It's such a new field. It's, it's almost amazing. I tell everybody you could become an expert in this thing very easily. Like, nobody really knows what's going on. You kind of just need to study for a week and you're an expert, which is a very unique place to be. But anyway, it makes it fun to.

Be able to dive into something and, and get as deep as the leaders in the field.

So, yeah, 100%. And just to be on the cutting edge and know that nobody really knows what they're doing here. Some people do, but there's very few of them. So regarding the APIs, ChatGPT, the way I think about it and the way I usually explain it is it's basically a very thin wrapper on top of the API. Every time you talk to ChatGPT behind the scenes, it's using this LLM technology. It's sending your message and a little bit of a preamble before your message. And the preamble is basically, we could call it the prompt, and it's basically saying, hey, you're an AI assistant. Make sure to be helpful to the user. Maybe don't be controversial. You can use a calculator if you need stuff like that. And then giving the user messages. And that's all it is. And the process of prompt engineering is how do you tweak that preamble and how do you get it to respond in the way you want to respond by telling it what to do? When I talk about the API, I'm talking about the things OpenAI has exposed to let you build your own chatgpt and let you build your own products on top of it. And I just think OpenAI has done. If you want to get started and you haven't touched these APIs today, the best, best thing to do is just go to OpenAI's docs and just read the Getting Started tutorial. It's really well done. And I think as someone who's running a developer tools company, I'll tell you how hard it is to write good docs. I think our docs, from my perspective, I think they should be much Better. Other people say they're great, but there's always room for improvement and it's just a very hard thing to do. So that's where you should go.

From your perspective, over the last year, as people have gotten into doing this practice of prompt engineering, have you generally seen people that are engaging with your platform be sort of more informed coming in, like they've done that experimentation with ChatGPT or the APIs and they are like using words like few shot, whatever, blah, blah, blah, and like this stuff. Or is it still kind of like people coming in that say, I know I think I need to have a prompt engineering platform, but I kind of don't know where to start. Have you seen that shift even over in this last year?

I would say we're still very early in this and there's a few leaders, but everything's up for grabs. In terms of AI products, I don't buy the whole notion of it's only the incumbents who are going to be able to use AI. There's so much the AI products people are making today are just scratching the surface. Having said that, I have seen a bit of a change since we launched our product. We launched our product January of last year. So January of 2023, I guess that was a few months after ChatGPT came out. These things were, I think we were, I think we were the first prompt engineering platform. Maybe there, there's some argument there, but we were one of the first at least. And when we launched our product, we had a lot of indie hackers and individual hobbyists using our platform. That was the whole community back then. That lasted for a little. Then we kind of moved on to AI first startups. So like one or two person startups, these are like the really cutting edge ones. We used one of them. It's a great company to actually refactor our whole code base into TypeScript. A lot of really cool stuff. And then from there, I would say starting in the fall, and I heard this from a few other founders in the space, it felt like there was started to become a real shift where real companies and when I say real, I mean maybe companies that actually make money started actually getting serious about AI and getting serious about LLMs. And I think we're still seeing that maturation continuing where these real teams are building AI products that they care about and not just like Twitter demos are one thing. Prompt layer is interesting for a Twitter demo, but prompt layer becomes really useful for a team that is serious about building their product and has multiple stakeholders and wants to collaborate. And so I am seeing that this shift is still is happening. More and more companies are getting serious about their LLM products and getting value and revenue from them. But we're still at the very, very beginning of the curve. So a lot, a lot more to come.

I think that was a great sort of intro into prompt engineering and the state of prompt engineering. I'm wondering if you could help us maybe understand. Yes, it's good if people get hands on with these models, kind of gain some intuition about how they behave and different ways that you prompt them. What is it about this discipline of prompt engineering that needs sort of systematic ways of managing your prompting methodologies? And how is that different to or the same as different sorts of engineering? In the past we've always had version control and that sort of thing. What's kind of unique and not unique about this discipline of prompt engineering in terms of how you need to approach it systematically?

Just starting from first principles, there's one fundamental thing that's changed and it's that we're now building upon a probabilistic technology. So we're now building on a technology that sometimes gives us some answer, sometimes gives us another answer, and is trending to be more confusing on why it gives one answer and not less and confusing. I mean, yes, theoretically it is deterministic. You can really dive into the weights and maybe figure it out, but nobody's really practically going to do that. It's virtually too hard for 99.9% of use cases with models. So we're at a place where you're working with a black box now. And yes, some servers, some architectures become black boxes because of bad code, but that's a different type of black. That's not a real black box. But we're building technology.

That's the code I write.

Yeah, me too. That's why I'm going to get banned from our repo soon. But yeah, so you're building technology on this black box and you need to think about it differently. And I think this is a big philosophy we have at Prom layer. I think we have the philosophy of we built a lot of great stuff in traditional software and traditional machine learning and had a lot of great learnings. Git is a fantastic tool. Version control is important, access controls are important, Test driven development is important. But do we necessarily want to take one to one of everything? Not really. I think the biggest difference between LLM based development and building AI applications and versus building standard software is who are the stakeholders. Like we were talking about earlier now you can have subject matter experts who are not necessarily software engineers, but rather these prompt engineers, these AI whisper like these people who are able to talk to the black box and able to communicate with the AI. We could call it in a little sci fi sense, but we have this new stakeholder in the process of software engineering who is not going to jump into the code and not going to jump into git. And that's why, at least for prompt layer, we've taken a very first principles ground up approach where we're saying, hey, what can we learn from normal software and how people work together on software and do version control and collaborate and how can we take that into LLMs and bring in new stakeholders and bring in new collaborators and let people actually build on this black box technology in a systematic way. So hopefully that makes sense.

It does. I'm curious, you said some things there that really kind of piqued my interest, that you kind of were contrasting it with deterministic programming that we've all kind of grown up with and now we're in this new age and we have these non deterministic things and you can give it the same prompt and it may or may not on any given day give you the same, the same answer back. So how has that kind of fundamentally changed software development in general? And I'm encompassing all the things, when I say software development, both the AI and the systems around it defeat it. You know, because when you're dealing with that potentially unexpected return and that non deterministic black box that you're talking about, how do people handle that when they're trying to devise and say, hey, I want to use a model in the thing that I'm building? How do they change in the way they think about that?

I don't know how people think, but I can tell you a symptom of how they think, which is we work with a lot of LLM teams, of course, because they're using our platform and we talk to them, we try to. We're always trying to talk to people, trying to figure out what to improve. And one of the big PCs we've come to with prompt layer is that the iteration cycle of prompt engineering is different than software engineering. So what I mean by that is your code deploys and continuous integration and that whole sort of thing is happening at a different cadence, almost always in mature software than prompt engineering because of a lot of reasons. Maybe you're updating the prompt frequently, Maybe again different stakeholders are updating the prompt. That's why we encourage people to have their prompts in a CMS or we call it a prompt registry. You can put it in a postgres database, something like that, but you don't want to block your prompt engineering cycle on end deployments. And that's, I think, the symptom of this new thought pattern of how you're building, let's call it black box software, and how you're kind of thinking through these problems where they are different problem sets. Also, you're not solving the same type of software problems you're solving. That's called language problems or things that can employ this new type of software. And so it's not just a different, like, way to build, but it's also a different way to think about it. And then we can also get into like, how do you code now, how that's different and how. I think maybe this is maybe a more contra. I don't like to predict things, but because I think it's kind of a fool's errand. But if I had to predict one thing, I think as a lazy programmer, there's something really nice about just having a little block of LLM do something for me. So, for example, parsing strings or parsing chunks of text and reordering it, you can do that deterministically and it'll probably be better. But there's probably a world where models are going to get cheap and quick enough, where maybe it's worth the engineering time not to build it well and just outsource it to AI. But maybe that's a whole different tangent.

Yeah. And maybe you could get into a little bit of the implications of these prompts in a registry. Right. Because you could have these sort of random tasks that. I don't know if you all saw the Devin thing that's been going around where it's like a junior engineer agent that can write scripts and interact with software documentation. And today I was thinking about that, similar to what you were saying, Jared, it's like, hey, well, I could go and take all of these strings and go and interact with the API to translate them into another language, which is what the task was that I was doing. Or I could just like, say, hey, write a script that does this and do it for me. So there's those random things, but then there's another type of prompt which as soon as you're starting to expose a system to end users, then small changes in the prompt, like on the back end, could produce very different changes in the behavior to your actual end users and cause Actual problems. So could you talk a little bit about maybe the implications of changes in your prompts and best practices that you found around? I know we're talking a lot about prompt versioning and registries around those prompts, and I know that prompt layer is thinking about more than that around evaluation and other things, but, yeah, maybe before we go to those other things, could you talk about what you've seen in terms of how people are managing these different prompts, Some that have very low risk in terms of changing them very often, and maybe some that have actually a large risk in terms of maybe small changes in the prompt.

Yes. I think there's a life cycle here for how you care about your prompt. And I think every prompt in any mature product and any. If you have a product, an AI product that's making your company $10 million a year or something like that, you're going to care about any change to it. And I think there's that cycle. So let's say that's the end stage. At the beginning, you probably are just going to ship any prompt. You're just going to write a prompt. It's going to kind of work. You're going to try it once or twice, say, all right, good enough, let's get the MVP out. In that case, maybe the prompt is in your code. Maybe don't care so much about what you were just saying about, like updating, having a breaking change. All right, whatever. I got to get it out. Let me get five people using it. All right, so you've done that. What's next? You probably now have like five different prompts on your system. They're scattered everywhere. I was just talking to Founder who was visiting us, our office today, who actually had this exact same story. And then he moved all his prompts to a text file. Or actually in his case, it was just like a TS file on his system. So now you have your prompts in one place, that's the next step. But still in your code base, it's still linked to ENG deploys. And like you said, you still have no way of knowing if you push a new one. What happened? Like, is it breaking 20% of our use cases? That not great. But I would call this stage, I like this word for it, the vibe based prompt engineering. So this is the five based prompt engineering stage, where you're kind of writing a prompt, you're testing it in playground, you're doing it once or twice and you're just judging, like you're just looking. Okay, yeah, it's pretty good. This lasts a little bit. The time when this is no longer good enough is usually when your product's either getting to a greater level maturity, maybe you're rolling it out to GA or you're adding more stakeholders to the team. Maybe you're adding a pm, maybe you're adding a content writer or subject matter like we were talking about earlier, a psychologist or a lawyer or something like that. And you need more people involved. So now you have a non technical person writing your prompts who isn't really capable of building out the whole dev workflow to test it out. You really want to like make sure it doesn't break everything. So there's a few things I've seen people, few strategies I've seen people employ here. One is of course traditional software. Let's borrow some stuff A B testing, let's release it for some people. If we're monitoring user feedback, maybe if users are giving us a thumbs up, thumbs down, we should be able to see it pretty quickly. Also having prod staging, dev staging different like we call them release labels, a lot of words for them. Let's call that category slow releases. So that's one way. Then there's two other big ways of solving it. Second way, regression tests. So again, another concept borrowed from software engineering, let's find cases where it's failing and see if we succeed in this test case. Third, which is also kind of like regression tests I guess is back testing. Let's just run it on old examples and see if it changes. I think in a lot of LLM use cases the really hard part is about this problem is that you don't know what the ground truth is. Let's say we're making a summary. There's no correct answer to a summary. There's a good summary, bad summary. It's almost very hard to understand if it is good or bad. We actually there's a user of ours that is doing this exactly and they're trying to figure this out and they're using a combination of human graders and whatnot. But we can talk more about that in a second. But in this case often the best thing to do is just rerun it on old responses and see how much changed. And oh, I updated the prompt and 50% of my responses changed. Maybe I should look into those. Only like one out of a thousand changed. Probably good enough. So it's all about trade offs in this world and everything's again, it's a new way of thinking. It's non deterministic. So how do you trade off how much it's changing versus how much you need to make sure it doesn't change. Maybe you want to force a specific output that is deterministically graded. So for example, you're giving a JSON or a Boolean output. So there's a lot. There's a lot of strategies here. But if I were to kind of give it a one line answer, you should be deciding at what cadence you update props by what state your product is at and how bad these issues are going to be. Foreign.

You know, when we started Podcasting back in 2009, an online store was just the furthest thing from our minds. Now we have merch channel exchangelog.com and you can go there right now and order some t shirts and that's all powered by Shopify. It's so easy. All because Shopify is amazing. Shopify is the global commerce platform that helps you sell at every stage of your business. From the launch your online shop stage to the first real life store stage, all the way to the did we just hit a million dollar stage? Shopify is there to help you grow. Whether you're selling security systems or marketing memory modules, Shopify helps you sell everywhere from their all in one e commerce platform to their in person POS system. Wherever and whatever you're selling, Shopify has got you covered. Shopify helps you turn browsers into buyers with the Internet's best converting checkout up to 36% better compared to other leading commerce platforms and sell more with less effort thanks to Shopify Magic, your AI powered All Star. You know, nothing gets me and Jared more excited than when our guests get that coupon code in their email when their show ships or to everyone out there there who loves Changelow podcasts and can go to merch.changelog.com and get your favorite threads to support our podcasts. It is just the best thing ever. From stickers to threads, all that is@merch.changelog.com and did you know that Shopify powers 10% of all e commerce in the US and Shopify is the global force behind Allbirds, Rothy's and Brooklinen and millions of other entrepreneurs of every size across 175 countries. Plus Shopify's extensive help resources are there to support you and your success every step of the way because businesses that grow grow with Shopify. Sign up for a $1 per month trial period at shopify.compracticalai all lowercase go to shopify.compracticalai now to grow your business no matter what stage you're in again. Shopify.com PracticalAI.

If you're looking at like a large out on the edge system of systems in the sense of you have a number of models deployed and they all do specific things and some of them are generative and some are not. And for the generative ones, they may be trying to address very specific functions that they're doing. With a system like that, you've got it in production and maybe you've done kind of that minimal viable product approach on getting it up, but when you get things to where they're kind of stable in production, you are starting to kind of address some of that. I know that I'm grappling in my own head with like how to think about being able to make those tweaks and changes to prompts in any given model in the system and detect that is there like a best place for me to start? Because I'm still trying to kind of grapple with the larger picture and really understand it. And so if I want to change something, but I don't want to impact the larger stable system, what would you kind of be like, what if it was you in that position? Like 1, 2, 3, try this, try that, try that. Just to give me a good hands on takeaway from that. And I apologize for the selfish nature of the question, but it's hard to do that.

No, I think it's good to have the selfish type of question here because one thing that I actually think a lot of people get wrong in this space is that every prompt is kind of different and the answer to this question is really very unique to what you're actually trying to do and the task you're trying to solve there isn't really. I mean, a lot of people are trying to sell it. So I like to say that maybe I'll change my opinion in a month or a week or a day.

I do that all the time as I learn more. So no worries, you're allowed to.

It's good to change your opinion. But right now I think a lot of these eval sets that people produce are not that useful for building real products because you're trying to evaluate your prompt for your real application, not for some pie in the sky financial data set or something like that. Having said that, I think the first question to ask, and maybe we'll use this example. So I would modularize it, I would think about it on the prompt level. So I guess there's two ways to think about it. We could think about modular tests and then end to end tests. We should be doing both for this case. Do we have a ground truth? I think is the first question I'd ask, Is there an ability to make a data set with ground truths that we can compare to or is it like a summary type example where there's no answer?

In the case that I'm looking at, I think you could establish ground truth. I don't know. It would be easy. But you probably could because you could bypass what you're seeing and you could have a human kind of assessment what the generative AI was model was trying to assess as well. So you could get a ground truth that is a human analysis of it as a proxy.

Excellent. So your life is now 10 times easier.

That's always good.

It is great, yes. Step one is whether you do it yourself, whether you hire some people on MTurk or QA or however you do it. Step one is kind of build. It doesn't have to be built. Build a small data set and try to get into this method of test driven prompting or eval driven prompt engineering. I don't know if we have a word for it yet, maybe we need to define one, but try to build some sort of metric we can evaluate our test on so you don't have to be just one by one trying these examples out. So what I mean by this is build A let's say 10, let's say 15, let's say 100, whatever it is, depending on the use case of input variables to your prompt. So your prompt is probably having, hey, your task is to do this, here's this data, blah blah, so the data is an input variable in that case. And then get a human to give you the output and then start every time you test the prompt. Let's run it on that over time. We can. Let me, I could talk about over time, how to make that better. But does that make sense?

It does, it does.

The thing I'd add is over time, how you make that better is you start connecting that back with real data and how your users use it. So again, you can make your life 10x easier if you have good user feedback and a way to know if the production inference, if the production LLM run actually worked or not. So user feedback is a way to do that. Say a user gives you a thumbs up, thumbs down. Now you can take all those thumbs up, thumbs down, make a new data set out of that. And now you're really going. And now you're building this whole feedback loop. And I think I'll say our Biggest goal with prompt layer, our MO is to shorten the prompt engineering feedback loop. I think that's what everything boils down to in this world.

Maybe along with that element of feedback, I'm wondering if you can talk a little bit because we've talked a lot about evaluation, prompt versioning. There's the other element of this which I know you all are thinking about deeply, which is sort of logging and monitoring and there's certainly cases where, oh, I have this chain of LLM processing or even loops that could happen like LLM as a judge or something that kind of. Or critic kind of elements of LLM prompts that actually could loop until something happens or a certain number of times. And the way that you develop your prompts, both in terms of their length, in terms of how effective they are, could drastically impact your latency of processing. It could impact your cost in terms of how much text you're putting into models, especially if they're charging you for how much text you're putting in. So yeah, could you talk a little bit about maybe the highlights of some kind of best practices around logging and monitoring and how you think about that at PromptLayer?

Yes. So I think, not to sound like a broken record here, but the thing I like to go back to a lot is everything is use case dependent. So you brought up that some people are very concerned about latency, some are very concerned about cost. I know teams that are concerned about neither of those and their only concern is are we getting the right answer? For example, code generation type startups, a lot of times latency doesn't matter. You're giving them a task. Hey, again, in our case, a problem layer. Hey, can you. We worked with a company called Grit where we said, can you move our whole code base into TypeScript? It could take a week. I don't care. So latency and costs don't matter them. But then there's a lot of other cases where most cases probably latency cost matter. So in that case or in either case, why logging is important here is just debugging. Honestly, I'll be honest, logging and observability is kind of the most boring part of our platform. Not because it's not useful, but because it's obvious. I think we started with observability. This is table stakes to shortening that feedback loop and that high level goal. This is table stakes because you need to collect the data. You need to collect the data to build these evals, to see when it's not working, to be able to triage issues Say someone, one of your users tells you, hey, I got a weird error. This happened to me, actually. I was using superhuman AI and it didn't work. And I told them about it. I said, I got like a weird output, maybe you guys want to debug it. And they asked me what prompt I gave it to do the output. I don't remember. Don't you have a logging system? Maybe you should use prompt layer. But yeah, you should be able to figure out why something broke. You should be able to step through, step by step into the chain, see which version of the prompt it used. Maybe you have multiple versions in production and our logging is just logs each request and lets you integrate it with metadata, like user information.

It's one thing to log a bunch of things, but let's say that I want to improve latency or cost or something like that. And I maybe have 17 different prompts that I'm using across my system. How have you learned how to present that information to users so that they can kind of, especially from a. You mentioned kind of the skill of prompt engineering, having this algorithmic thinking kind of piece to it. But there's also a lot of people, you know, they're coming in, maybe that's the part of their brain that they're building up and they're bringing in these other skills with them. So how have you found it useful to present this sort of information to people, to give them the right sorts of feedback along their kind of journey of optimizing things?

So I think latency and cost are the easiest things to figure out how to. They're the easiest metrics you get out of the box when you're doing prompt engineering because you're always getting latency and cost, the harder metrics. Is, is my answer correct? Is it rude to me? Is it mean? Is it. Does it have API? I don't know. Are you Microsoft Bing releasing? We have a. We did a prompt engineering tournament last night. The first round was can you avoid a PR disaster like Microsoft Bing? But that's the hard part, is making sure your answer doesn't go off the rails or isn't wrong. But latency and cost and those type of logging, like base level logging, specific things, those are the tractable metrics. So we give you linksy cost for each prompt template. We give you it broken down based on version. And then we also have a full analytics page that actually we revamped the other week for one of our customers who they were going to have to build out their whole, like a whole bi Dashboard. Because I think something about their. Either the founders of the company or the investors were worried about some users spending too many credits. So we kind of revamped our analytics business page to just save them some time there. So you can use our analytics page to see which prompt templates are costing you the most. Which users are costing you the most. Maybe if you're segmenting things to prod, maybe you're segmenting based on geo and just kind of filtering down based on that sort of thing.

I want to ask a question, as you have kind of pioneered this whole space jumping into prompt engineering, quite honestly, before anyone really knew what it was, as you pointed out earlier, and you've been building out this capability as you look to the future and you know, the future is changing so rapidly right now. You know, we're all in this massive acceleration of things coming out. You know, Daniel and I every week are trying to figure out of all the things happening, what do we actually talk about? You know, it's, it's getting harder, whereas it used to be. There was something that happened last week and we'll talk about it now. There's so many. As you're operating a business in this, you know, kind of intense, increasing environment, where do you think this is going from a prompt engineering standpoint? What will prompt engineering become as we become increasingly multimodal and you know, all the fantastic things that are happening on a weekly basis, I would imagine that would be fairly hard to try to plan ahead on, you know, where the industry is going and the technology and where to put your business. How do you see the future? What's the next one year, two year, five years in your head look like?

That's a billion dollar question, right? I think we try to do two things. We try to not predict the future because it's too hard, and we try to build something useful that is built on first principles that make sense. And I think that's how we try to stay ahead of the curve there a little bit. So I can give you some examples. So, for example, kind of just the whole process of iterating, of testing for evals, we procrastinated a little bit on building that part of our platform. We always knew we needed it. It's been the buzzword in the industry for like six months now, but we really wanted to know how to build that correctly. And I think we spent a lot of time talking to a lot of teams and saying, how do you do evals today? And every team we spoke to did it in their own way. In a Google spreadsheet building out some weird, unique things. So our Eval product, if you try it out, it looks like a spreadsheet for that reason. And it's very much inspired by the robustness of a Microsoft Excel type product where it seems very simple, but you can take it in a lot of different ways. So I think we are trying to become future proof by avoiding taking strong opinionated wins. We want to support best practices and build best practices for the community community, especially in a space like this. But we want to do it without pigeonholing people into different ways of doing things. And I think it's been funny seeing how people like the hive mind has changed their opinion on what the future is. I remember we were a year ago we were talking to like investors, obviously not the investors that are on our team right now. And there were investors like, oh, Prompt Engineering, AGI is just going to take over, we're not going to have any infrastructure anymore. And a lot had that opinion. I don't think many have that opinion anymore. Let's just say, and I think it's very obvious to us, and it's been obvious to us that Prompt Engineering is the process of giving inputs to the LLM and choosing which model you're using. And even with the most advanced LLM ever, let's say the LLM is advanced as a human, you still have to tell a human what you want. You still have to tell the intern what task you want of him to do, and that's Prompt Engineering. And there's always going to be a process of inputs there. So that's how we think about the future and lack thereof.

Jared, thank you so much for coming on to share your insights today. We definitely appreciate that you and the team at PromptLayer are thinking deeply about these things and building really good tools to support the community. And I encourage everyone in the audience to check out the show notes, follow the links, find out more about Promptlayer and the cool stuff that they're doing. And I hope we can have you back on the show in another year when I'm sure Prompt Engineering will look very different than it does now. But thank you so much for joining, Jared. It's been a pleasure.

Yes, thank you for having me. This has been fun.

All right, that is Practical AI for this week. Subscribe now if you haven't Already, head to PracticalAI FM for all the ways and join our free Slack team where you can hang out with Daniel, Chris and the entire Changelog community. Sign up today at PracticalAI FM Community. Thanks again. To our partners at Fly IO, to our beat freaking Residents Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.