LangChain/LangSmith Office Hours

The main thing that people are thinking about for this course is probably LangSmith, um, the like observability tool in the LangChain suite. Um, so I'm curious how you think about positioning LangSmith relative to all the other observability options. You have, you know, LangFuse, W&B Weave, you have a previous generation of tooling with Arise Gantry, Ylabs, Fiddler, um, there's, uh, like Honeyhive. There's like this broad array, and I imagine you have like a kind of mental map of that market and, and where Langsmith is, and I'd, I'd love to hear that.

Yeah, very good question, and I'm sure you're familiar with a bunch of these, so you can probably keep me honest as well. Um, I think, yeah, there are a lot of solutions. Um, there are definitely a lot similarities between them. I think they're, you know, starting off with something which I don't think distinguishes, but just to like, I mean, maybe it distinguishes, like we're very aimed at LLM applications. And so I think some of these solutions are, some of the previous ones might be aimed at more general things. Some of the standard OTAL solutions may, you know, be aimed at generic kind of kind of like logging. I think we're very aimed at LLM solutions, specifically what that means. I think you can add nice visualization of elements that are specific to LLM apps, like messages and documents, which I actually think goes a very long and understated way in terms of debugging. I think there's, there's also like the logging and testing component of this as well. So some of these are more focused on logging. We try to do both, and we try to have a nice interweave between them as well. I think those are like the generic ones that probably some of these other ones would say they do as well. I think like specifically, I think we have always been really, really focused on not just like single LLM calls, but like chains of LLM calls. I think just kind of like given, yeah, the fact that we Also build LangChain. I think that's been a big focus of ours. And I think that's useful for a bunch of reasons. You know, if you're building a more complex app, having that full sequence of LLM calls and other things like retrieval steps in there is really important. That's been a focus of ours kind of from the beginning. This also shows up in testing as well. So if you want to test not only the final result, but intermediate steps, if you have access to the full trace, you can access things like that. Things there. Um, another thing which, uh, and so I think that, that, that's, that's one probably like big differentiator. Another thing which, um, I don't know if it's as clear, and I think this is true, but I, you know, call me out if you don't think it's true. I think we, um, I think we actually place a lot of emphasis on like the human-in-the-loop kind of like interactions in a few different ways. Like one, I think we've spent a lot of time on like visualization of data. Like, I think that's like a human-in-the-loop thing. I think we've, like, we've added an annotation queue. I don't actually think that that's common of a thing for people to have. We've added, on testing, we've added a bunch of things around— we've added a way to really nicely visualize comparison of things. And I think that's a human in the loop to like look at two things side by side and easily see which ones improve and don't improve. And then one of the things that we're working on that I'm really excited about is a way of like aligning evaluators with human preferences. So Hamel Husain kind of like wrote about his kind of like cycle that he did of like, yeah, checking the alignment and then manually improving the prompt. Did we want to make the evaluator prompt? And so then eventually you get from like 65 to like 95 or something. Alignment. We want to try, and I think Shreya Shankar wrote a great paper on doing that with like few-shot example prompting, and we want to try to actually make that automated as part of the product. That's not out yet. But I think like, yeah, like we're really interested in this like human-in-the-loop kind of like component. And then maybe like a small like third thing, which I'll call it just because I literally just tweeted about it after reading a I don't know if it was both Charles and Hamel Husain and I don't know, or, and Dan, I don't know how many of you were on this O'Reilly blog post thing. But there was one point around evaluations and the importance of kind of like pairwise evaluation. And so one of the things we added last week was like a way to do that kind of like natively. And so this isn't like run model A, score model A, run model B, score model B. It's run model A, run model B, score model A, score model B. Score them against each other. Um, and I think, I think it was called out there that that was pretty good at kind of like, um, making the results more stable. And we've seen that as well. And so that's like a small thing.

That one is really cool. I saw that just came out. It's really cool. Yeah, I just want to add, I was going to add a couple, like some more things actually that you probably are too embarrassed to say. But like, um, the thing that differentiates like LangSmith for me is like, okay, it's like, as you know, I'm a consultant. I'm in like all these different companies, working with them, and people have all these tools. And then like the places where there's Langsmith, like the support level is insane. Like anything goes wrong, anything, it's like, oh, it's like it gets fixed yesterday. It gets fixed immediately. I'm not trying to— maybe you don't want to have that ISOA like broadcast, but basically like that's what I experienced and it kind of blows my mind as like Harrison, like how in the hell do you do that? Uh, it's interesting. Another thing is like the tools are very like open in the sense that they have APIs and things allow you to hook into them. You don't even— you don't have to use LangChain to use LangSmith. You can like get the data out of LangSmith. The thing that is like really cool is like if you build tools around it, even if they like collide a little bit, like they— like Harrison and his team will like support you as a developer. They even will promote it. It might be counterintuitive, but that is really cool. If you want to build stuff, that team will help you do it. So I thought that was really interesting. I built some stuff that's not popular or anything, but I built some stuff.

LangFree. LangFree. Best logo in the game.

Yeah, LangFree. It was like a little bit of cheekiness, but yeah, that's why I like it because has this like openness to it that like not that many people have.

Yeah, no, I mean, I appreciate that. I think like just echoing that, like for all of this, we think it's so early for all this stuff. And so I love working with everyone who has ideas on how they want to do things, because I want to learn how you want to do things and then, you know, hopefully make our product better for you and a lot of other people as well. So.

A follow-up.

Cool.

One that's also about LangSmith. What's LangSmith support for human annotation and annotator cues? And is there, what about like concrete action items for fixing issues or user complaints?

Okay, I don't entirely understand the second part of the concrete action items, but around annotation queues, yeah, so you can send data to annotation queues either programmatically or manually. So you can manually, if you're an engineer looking through it and you want to basically say like, hey, subject matter expert, come look at this and provide some feedback and I'll talk about what you can do, but you can send that manually to a queue or you can set up programmatic rules. Actually, this is another thing that I think actually distinguishes Langsmith as well that I don't I don't think many/any other have is this kind of like, I think we've invested a bunch in these like automations to like easily move data from your logs to an annotation queue or to a dataset or do some online evaluation. Actually, that's another thing that I think also distinguishes is I think we've tried to make it really easy to do online evaluation with inside the app. Anyways, one of these automations is move it to an annotation queue. And so this is useful if you want to do something like every time a user leaves a thumbs down, send it to an annotation queue. And then you can kind of just like keep on top of this annotation queue rather than having to kind of like run a query through your log every day or something like that. In the annotation queue, you can leave feedback on things. We've added some hotkeys to like easily cycle through them. You can add them to datasets. And you can, you can edit the outputs and then add them to datasets, which is usually the most common thing we see there. Yeah, usually the most common thing is like correct the output, add it to a dataset type thing. And the reason you add it to a dataset is so that you can test against it in the future. Or, and this is maybe, this maybe gets to how can you kind of like fix errors, but like actually I think this is another thing that we probably do different than a lot of the kind of like other platforms. One of the things that we want to also do is start being a place for people to gather like few-shot example datasets, basically. I'm personally kind of bullish on this as an approach in the future, especially for personalization of apps towards users or companies. I think it's a bit more pragmatic than fine-tuning per user or something like that. And so if you want to like fix an issue, one of the things that you can do is fix it manually in annotation queue, add it to— by like correcting the output, add it to a dataset, and then basically you can pull that dataset down into your application during runtime, and we'll be adding some support for like pulling down like the 5 most similar examples pretty soon as well in case you build up big datasets. And then depending on the prompting strategy and the application, it's not like a silver bullet guaranteed to fix it, but it can be used to address those types of issues. Got it.

Yeah, I think that's my perspective on what is cool about Langsmith. It feels like there's a temptation with observability to make it this immutable record of what occurred in the application. And while that's very useful for traditional software, I feel like with LLM applications and with machine learning in general, what you really want to do is kind of like build up this rich knowledge base of what you have learned about how to deliver that application effectively to users. And that includes, and like, annotations, that includes like, like live notes. And it feels like they're— like LangSmith is kind of trying to be this like, like interactive database that collects that information together. And like, spoilers for future iterations of that O'Reilly blog post, but that's what I think a lot of us are convinced is where a lot of the value accumulates for these products. So that's— I like Langsmith for that orientation.

Yeah, I mean, everyone talks about data flywheels, right? And I think, like, this gets towards that.

Someone's asking, what's the difference between LangChain, Langsmith, LangGraph, Langflow, and LangServe? Is there any— how far are you going to take this? Take that namespace.

That's a fair question. Probably, probably not that much further. I think there's already a lot of things out there. Yeah, so I mean, maybe just going through these. LangChain was kind of like the base for a lot of these in some sense. It's an open source package. Started— it's the oldest one. Started around building LLM apps. There's 3 main things in there. There's an underlying kind of like runtime, um, so this would be ways to chain components together. Um, there's a lot of different abstractions and integrations, um, so for models, vector stores, document loaders, things like that. And then there's bigger kind of like off-the-shelf chains or agents that you can get started with quickly. Um, and, and so that I think that's a helpful starting point for a few of these. So maybe just jumping around a little bit, Langflow is actually not associated with LangChain, the company. Neither is LangFuse. Yeah, we do not own the entire Lang namespace, which also does not make this naming scheme super great. But Langflow is a separate company that added basically a low-code, no-code UI on top of LangChain. Which is great with us. We're very— we're more focused on kind of like the developer kind of like audience. LangServe is a way to basically easily add LangChain applications. It's honestly a pretty thin wrapper around FastAPI, and it kind of just like exposes LangChain applications with some specific endpoints on top of a FastAPI server. So lets you kind of like get up and running with that pretty easily. Basically deployment of LangChain applications. LangGraph is an extension of LangChain really aimed at agents. So we've had an agent executor in LangChain from the beginning. That has always been one class. It's been very hard to modify. It's been very hard to customize as we see more and more objective workflows coming online, specifically ones that want a lot of controllability because agents are still hard to get to work pretty reliably, um, we built LangGraph, which is a highly controllable, um, agent framework. Um, and then LangSmith is separate to the side. You can use it with or without LangChain. It's the observability testing, um, for LLM applications in general.

Yeah, I really liked LangServe because it made it very straightforward to put together like a Gradio app and then deploy on Modal, the, like, uh, some, like, little LangChain things I've been playing with, like, that, like, translation layer into FastAPI. It was like, I get streaming internal events, like, as an agent is executing Lang, uh, if you're using LangGraph with LangServe, you get these, like, kind of event logs, and it's in this nice, like, structured format, and then you can grab that and decide how to render it in your frontend UI however you like, like, showing little animations or states to users while your agent is like chugging away. And like, while it is a thin wrapper over FastAPI, it's like nice to have that just like, it's like a standard for expressing the state of the application and exposing it as an API.

So it's funny that you say that actually, because I actually don't think we have great support deploying LangGraph within LangServe because I actually think for like longer running agentic things, there's actually a lot of different things you might wanna do. You might wanna actually have background runs instead of streaming runs. You might wanna have kind of like built-in persistence for things. And so we're actually working on something more focused at this and I'm very, very excited by that. So basically, yeah, better, better LangGraph, LangServe. I, I don't know what we'll call it. As you pointed out, our naming team has gotten a lot of things in it, but I actually hope that that experience will be a lot better shortly. I can, I can maybe take the first one because that has 2 upvotes as one and was asked early on. That's top of my screen. During the course, we learned about a lot of tools. When would we want to reach for LangChain, or rather, what would be the classic use case? And I think this relates to the last question as well. And so I think the— I think LangChain we view as a great place for getting started to build kind of like LLM apps. You can easily get started with integrations off the shelf. You can easily chain different components together. I think where it really shines is actually more probably where LangGraph is now. So I think like if you're building and maybe like an application that has one call to an LLM, I think LangChain will benefit you in that you can get up and started and running with it. Super quickly. But that's a lot of them, and actually it has great built-in observability with Langsmith. Funnily enough, we built Langsmith after, but we've heard Langsmith be a reason that people stay on LangChain. But that's an aside. I think like, the, you know, for those types of simple apps, besides getting started quickly, LangChain may not provide a ton of value. I think the reason that we're really excited about LangGraph is that we see it providing a good amount of value for pretty advanced teams as they're building like agentic applications. And I think in particular, it has a nice developer experience for creating cyclical kind of like agents. And then it also has a bunch of nice functionality around built-in persistence, some different streaming, and we'll add some like background modes to it. And so there's other components of the ecosystem as well, but in particular for the orchestration layer bit, the LangChain versus LangSmith, or sorry, LangChain and LangGraph component, I would say LangChain is really good for getting started with simple applications quickly. LangGraph is really good when you have a more complex agentic type workflow that you want to build.

Um, I'm really curious about, uh, what you're seeing with the tools, like all these suites of tools. Um, a lot of them you have Python APIs and TypeScript APIs. Um, are the TypeScript ones becoming very popular? How are they comparing to the Python ones in terms of usage? I'm just curious, like, what do you see?

I think, yeah, I mean, looking at the numbers, we see Python's still more popular. I think they are popular for different things. So one of the things that we did last week with the JavaScript library is basically around this idea of generative UI, where you kind of pass the UI components from the backend as part of the agent. And render them on the front end, and that's not as easy to do in Python. And so for things like that, we see the JavaScript library being ahead and seeing more developers there. But I also think we see, yeah, I mean, we see more people on the Python libraries. I have like two hypotheses for why this is kind of why. One, like a lot of people are using LangChain and LangGraph, to build a bunch of experimental, exciting things. And I think there's maybe— I think there's maybe more people with that mindset in Python, I would say, of prototyping things quickly, trying to get something to work that way. And then the other thing that I'd say also is that a lot of these LLM applications really involve moving data around. And a lot of, like, data engineering tools are in Python. And, and, and so I think that maybe makes it a little bit easier to do some of, like, the retrieval bits, for example.

Makes sense. That's, that's pretty interesting.

I, I do think that, I do think that there's, like, there, maybe another way of saying this also is there's, there's definitely more prototyping in Python. And for some of the data-heavy retrieval bit, maybe that's happening in Python, but a lot of the, but, but, but a lot of the actual applications we do see in the JavaScript ecosystem. Like I know, I know WeChat is in TypeScript, for example. And so I think it's slightly different audiences. I do think, yeah, I don't know, so early it's tough to predict what will, what will become. I think if, if you think like, things like generative UI are really important, then I think that speaks really well to the importance of, of, of kind of, uh, uh, the JavaScript.

You talk about that a little bit. What is the, the generative UI? Like, what is that exactly? What's the kind of ideas behind that?

I'm going to butcher a lot of things because I am not a front-end expert by any stretch of the imagination, but basically the idea is that more or less the— if you have a chatbot and it's calling tools, it's not just passing like a message back to the UI, it's passing like a message plus some component for how to render that message and/or other information around it. So you could have a chat where you say like, show me the weather, show me the weather over the past 10 days. And then it doesn't just return like list of the last 10 data points, it returns, like, some nice graph where you can, like, zoom in and highlight things and click on data points or something. And, and that's a combination of the data that might get fetched by the tool as well as some component that is specified. I think Vercel has done a lot around this. They have their AI SDK. We did a bunch of work last week to hook in LangChain. So this is actually a big ask from the community is to make LangChain compatible with the Vercel AI SDK. And so you can basically use the AI SDK to render things nicely on the front end, and there's some backend helpers as well, but now you can use the LangChain components to stream things in the format that it expects. Let me, let me find a link. It's where we, we actually, yeah, we updated one of our repos around this, and we'll be doing actually some more stuff here as well.

Oh, interesting.

Yeah, it's an interesting idea. There's— I— okay, my hot take here also is that it's not actually like generative UI. Like, I think like some of like the WorldSim or WebSim type things are like more generative UI where they generate the full web page. This is kind of like you, you have to specify the components ahead of time and then you kind of just map how they interact with, with data. But maybe that's a hot take.

Yeah, I think, I think there's like kind of a graduated move from LMs putting out structured output can then be processed by a traditional software system into a richer interface than just a chat. And then maybe you like, you know, you start to give the LLM more control over how that, how it gets rendered. And it's not just about the like raw data it outputs, like which, which class or which JSON schema it chooses, but it can also like adjust the UI. And then the like full version of this, yeah, it does look something like WebSim, where the language model outputs like, you know, an entire React component, like, and or eventually like, you know, an entire web page or an entire application.

Side note on here, since we're talking, uh, real quick side note, because we're talking about UX basically, um, and, and transition from chat. One of the things that I'm extremely interested in is like, what is the right UX for like human kind of like agent collaboration? So like longer running thing, um, and, and maybe generative UI is part of this, um, maybe there's other things. But like, I think this is a super open, pretty unexplored kind of like topic, um, that I'm very interested in. So if people have ideas here, or want to explore crazy stuff, we would love to explore that with you. So drop a note in the Discord.

Yeah, I feel like I was using Devin for a bit and it did feel like a bit of a missive from the future. I was like getting on my phone to like check my chat with Devin to be like, oh no, you just need to like reconfigure that library and then go back and then later come in and like spy on it in the terminal and like watch it type stuff. And be like, and then message would be like, no, no, no, you're on the wrong track, buddy. And then like, you know, and then interacting with it through the other services it would talk to through the browser or through like APIs talking to GitHub and then go and review its PR. Like, and it was, there was a chat interface and the chat interface was used the way I chat with humans for like super like synchronous short communication. And then every other interface was open for other kinds of communication. And that felt really, really great.

Exactly. Yeah, I think Devin's a great example.

I was like, hopefully I can get in off the waitlist and use it.

I see maybe quicker question. What about for LangSmith? What are the GDPR and privacy like considerations? Are there servers in Europe? What are the plans?

Yeah, we are GDPR compliant. We, and we also do self-hosted deployments, but that's usually an enterprise contract. But yeah, we recently did get GDPR compliant. Nice.

One question that keeps coming up in the course and among students, and you're probably really in a good place to answer, is like, okay, people can ask me, what does Agentic mean? And I'm like, look, in my simplified, like my most simple answer that I have is like, it's an LLM that makes function calls. But then people remind me that, and then we're like, well, why is it called agentic? Like, what's in that word? Like, is it doing a lot more work? There's like a lot of debate. Can you tell, like, I think like there's a lot of, yeah, so maybe you can answer that because I think you have like talked to all these agentic things, like whatever.

Yeah, my, my take on this is that an agent in agentic system is a system where the LLM is in control of what actions to take and in control of the control flow of, in, in some way. And I think that's pretty tightly coupled to, or that's, that can be related to kind of like function calling, um, because function calls usually take some action, get some observation, pass it back. But yeah, but my definition would be like, yeah, an agent in an agentic system is a system where the LLM is in control of the control flow, where the LLM has agency over the control flow of the application. Maybe comparing that to like a chain where you always do step 1, then you always do step 2, then you always do step 3. Then you kind of start to get into like a gray area of like, okay, what if you use an LLM to like route between two things? Then it has some agency over which direction. I still wouldn't consider that an agent. And so then I think like when you start to add, when you add like that decision-making plus maybe looping is where like, I think like that really means you have control over the, the control flow of the application. But it's definitely a gray area without a super concise definition. But I have, if I had to give one sentence, it would be the LLM controlling the steps that are taken in the application.

Okay, basically I'm trying to figure out, like, is it a— is it like a term that's worth paying attention to, or is it just like a term? Like, you know, like big data is a term, and then like, you know, whatever. Like, you know, is there like meat in there of like things that you should learn, or is it just like, you know, it's a packaging?

No, I, I, I think there is because I think like I mean, again, I'm just going off of my definition, but like, I think like, if you think about like a chain versus an agent, there are like, there are pretty real differences in terms of like just the amount of non-determinism that that application could have, right? So like, you probably care a lot more about even just like knowing like what steps are being taken. Like, 'cause you don't know what those are ahead of time. So that's like, so then that maybe causes some UX issues 'cause you need to, how do you communicate that to the user? That maybe causes some observability issues 'cause how do you display that? Maybe some testing issues. How do you test for that? And so I do think there is a series of like new problems that come up if with, with kind of like the, the way that I'm phrasing it. At least. And I do think that, like, I do think we'll start to see more and more applications that are like this and yield more and more control to the, to the LLM. Yeah.

Um, so there was actually also somebody asking this same question in the Q&A, and their follow-up on top of Hamel Husain's question about what are agents was, what are the differentiating features of LangChain or LangSmith that help most with these like agentic workflows?

So for the agentic workflows, I would definitely recommend using LangGraph over LangChain. I think that maybe the two differentiator things there compared to other, um, agent frameworks out there, one is controllability. Like LangGraph is very low level. Um, we have one or two like high-level entry points, but it's very, very low level, which means that you have a high degree of controllability, which we've seen is pretty needed because planning is still not amazing for LLMs. And I feel pretty confident that it's more controllable than a lot of the other agent frameworks. And then also we've invested a lot in, I should maybe come up with a single word for this, but I'll say like persistent slash human in the loop. And so basically like LangGraph comes with like built-in persistence, or you can pass in a checkpointer, which will automatically save all inputs and outputs at each step. And then it also makes those steps very readily available to, to you as an end user. So you can interact with them and then resume from there or something like that. So I actually saw a comment down below, which is like, for agent collaboration, I believe a good UX is when we can go back to a particular state of the agent workflow and restart from that checkpoint, similar to what I think Devin does. 100% agree with that. And so like all the check pointers we've added are like towards that in, in that spirit. And I think that's a differentiator of LangGraph as well. And then for LangSmith, I think, you know, I wouldn't say LangSmith is like observability for agents, largely because I think we're still collectively figuring out what, how to make agents work and like what, what real agents look like. It is more generally like observability for like LLM applications. Um, but like as I mentioned before, it does, it does, it does shine for these more complex applications, whether they're complicated RAG or agents. And the line starts to blur a little bit, but due to, due to some of the things I mentioned earlier, it kind of shines there.

I occasionally hear people talk about, um, workflows where you have multiple, um, language models and maybe they have different strengths, and so they in some sense collaborate, or one of them might farm tasks out to the others. I've never seen actual code that did that. Are you seeing people doing that, and do you have an opinion about whether in the future that's like a real promising workflow or something that's just hypothetical?

Um, I, I have seen that. I think the most common place I've seen that is when you maybe— and there's different ways of framing this, right? Like But, but I'll tell you like where I've seen it and where I think it is pretty reasonable is basically where you have like the smartest model, GPT-4, deciding what actions to take, but then one action might be like draft an email and you use like Haiku or something like that for that because it's a lot cheaper. And so I think the main— yeah, maybe generalizing that a bit, like there's some planning steps that are done by the best model and then there's almost like tools or like there's— it's like functions you expose, and inside those functions you use another language model just because it's doing a much simpler task and a much more directed thing. Um, and, and so that, that I think is a pretty common workflow, um, that, that I've seen. Um, yeah.

Uh, one more highly upvoted question from Captivus. Uh, what is the best, most productive way to use LangSmith to rapidly assemble evaluation sets?

So my workflow for this is the first however many examples I think you kind of got to come up with manually. You can maybe try generating them, but in practice, you know, 5, 10 examples, come up with them manually. And then I think the feedback loop between the production logs and your dataset is pretty important. So whether that's you messing around with it, whether that's end users messing— if you're having end users messing around with it, then another key part is like collecting feedback. Like the more feedback you can get, the better. And so like basically 5 to 10 examples, put them manually in a dataset, maybe mess around with it yourself, You know where it messes up, so you can go to the log, edit it, add it to a dataset, ship it to some people, make sure to collect feedback, look at the ones with bad feedback. If they're relevant, add them to a dataset. That's kind of like the process that we see. Does that— I mean, you guys are more of the expert on this than I. Does that kind of like square with what you see people doing? Or how you do it?

Yeah, I mean, uh, usually start bootstrap with some examples that are either generated by like domain experts internally, or like because like you, you know, because you shipped a thing at least to like some small subset of users, and you can bootstrap off of something, and then you can just like expand off of that basically. Like either like generate synthetic data based on that, but Usually, like, generating synthetic data without trying to do any human interaction is a bad idea because it just, it just frankly, like, lazy. Like, you just interact with the system a bit yourself.

Like, have you seen, have you seen good luck with generating synthetic data, period?

Yeah, I have. Yeah. So, like, uh, Yeah, like if I don't have enough fine-tuning data, then that's like one example where, okay, like I can synthetically expand it in some cases. Like, yeah, it depends. But yeah, I can like synthetically expand it to like teach it, you know, teach the model like the right syntax of something or, you know, the right kind of rules to follow.

One of the things that— the reason I ask is one of the things that we've been thinking about is a little button on our datasets page that's like generate 10 more examples or something.

Oh, yeah.

It's on our backlog. Maybe after this conversation, we'll prioritize it more.

Yeah, usually I have some kind of synthetic data generation in some kind with all my clients. Whether that's like generating the data itself or generating synthetic inputs into the system, like asking, like the, in the LLM invocation, like the, the first step, you know, just like perturbations, as many perturbations as possible as into the input of that.

So yeah, something that, that I've done once that, um, we generated synthetic data but then have a, had a human label for two different pieces of raw from text of a given question. Here's the response. Which did you prefer? And then we ran DPO off that. And for a human to go through and read those and compare is relatively straightforward, much less time-consuming than actually writing them. Then we got good results because you do have a human saying like, what's good or what's bad? And as a result, the raw data doesn't need to be that perfect.

Yeah, this last one's kind of an open general one. Um, but maybe you'd be interested in this one, Harrison. So, um, what's a good stack for like an ML engineer who's now really excited about shipping applications? Uh, like how— like they're Python native. Um, it's like a lot to build, to like learn React or Next.js and all these things. It's a large surface area. So what do you— what do you suggest people do if they want to, like, build full-stack applications themselves as Python people?

Yeah, Streamlit and Gradio are pretty great. Use Streamlit for a bunch. We have a bunch of examples with them.

This depends obviously on maybe the polish and scale of the application you're trying to build, but like, there's a lot of LangChain stuff out there that's like, it's Python, but then it's like TypeScript, HTML, whatever frontend. And I always like fork those and like hack it. Of course, I'm not the best TypeScript person, but I just like, I'm like, okay, like, for whatever reason, there's a lot of LangChain stuff like that, like templates and stuff like this out there.

And with like LangServe and the LangServe plus LangGraph thing, we definitely want to go more in that direction. Write your graph in Python, 'cause you're doing a bunch of data stuff where it's more familiar, but you've got now this— we're all thinking about a React components library as well for some more frontend components as well, but at the very least, a JavaScript SDK to call it in a standardized manner. But yeah, I was taking this as you didn't wanna write any JavaScript at all and only Python.

No, I mean, I don't really write it except for the JavaScript part, whatever, I just use like ChatGPT like very heavily, be like, please help me.

Yeah, and to your point, like we do, we do have a few, yeah, we do have a few repos that are pretty good starter points for that. So we have ChatlingChain, which is Python backend, JavaScript frontend. We also have OpenGPT, which is, yeah, Python backend, create your own GPT thing with a nicely complicated JavaScript frontend, but if you just want to modify the backend, switch out the retriever, switch out the prompt, we have instructions for deploying it as well, so that should be pretty easy to do.

Yeah, I think one thing I would say is that it is pretty hard to deliver a full-stack application in only a single language. Like, JavaScript can kind of do this thanks to Node, but you really actually do have pretty different worlds in the, like, browser JavaScript versus Node JavaScript world. And then on top of that, you frequently are using a database that's written in another language and the interface to it is like fundamentally SQL. So it like, I think it's a, it should kind of be a long-term goal if you want to work on full-stack applications. Sure, use Gradio to get small things off the ground, fork existing repositories and use LLMs. So like, so that if you aren't doing anything too crazy, but your long-term goal should be to assemble like 2 to 3 languages that allow you to deliver full stack value from like from the ML up to the user interface.

I will say though, like what, just to foreshadow something that is coming, you should come to Jeremy Howard's talk and see what he says about this. He will unveil something new.

Exciting.

Something brand new? Wow.

I don't, yeah.

You guys are doing launches as part of this conference as well. I didn't know that. I would have prepared something.

Oh, it's actually, it's not a conference either. It's an accelerator now.

Oh, okay.

Any last questions for Harrison before we go? Hamel Husain or Dan?

No, I think that was really good, Harrison. Really great having you again.

Yeah, this is, this is just a fun conversation. Thank you guys for having me. It was a lot of fun.

Awesome. Thanks for joining.

Take care, everyone.

See you guys.