Replicate Office Hours

Okay, the question is, what would some things you'd love to see show up on Replicate in terms of like students pushing things? Like, what kind of things would make y'all excited to see? I think that's the question.

We would love for people to push fine-tunes that do useful or interesting things. Replicate for a long time now has been home to a lot of fun and interesting apps or processes related to image generation. People do fine-tunes that do something fun or cool or interesting, and we really like that kind of community experience. And we are working with a lot of people to try to figure out how to make— create a similar community experience for language models. And in part, that's been challenging because often what you do with a language model is solve like a business problem, and it's not quite as exciting as looking at, I don't know, emoji generation or something. And so I think there's an interest gap that, that we're still trying to figure out how to solve. And so if you, you're doing something that's interesting to you with a language model, we'd love for it to be on platform, we'd love to help promote it and show people what you're doing, how it works. Um, another thing I think a lot about is that much of the, uh, innovation that's happening with language models is with application layers on top of language models. So chaining prompts together, doing various operations across different language models, and those are things you can do on Replicate too. You can make queries in a Replicate model, um, so you, you can build sophisticated chains of operations. And personally, I'm really excited to see people start building things like that on Replicate. So not just pushing a language model or a fine-tuned model for inference, but maybe pushing an application layer that makes calls to language models. So those are the two things that come to mind for me. Really, we're just excited about what everyone's doing though, and we would love to help give you a place to run it and share it.

See, great. We have more questions. Okay, another question is, how enterprise ready are you? We have use cases for fairly sensitive data. I want to make sure I can get it past legal.

We are increasingly enterprise ready. We're working on SOC 2 right now. That's a hard requirement for you, you'll have to wait a little while, though it's escalated and we're making really quick progress. I think we have Type 1 in, not Type 2 yet. We have worked with people who do have data concerns, and we can have flexible control over retention. So I would encourage you to reach out, and we could have a conversation about your specific concerns, and we can see whether we could work with you to solve them.

It's great. Yeah, there's also a Replicate, uh, Discord channel which, uh, Joe will be in, and there's some other Replicate people in. So I highly recommend— like, that's a good, like, uh, conversation to also— because it might take some time to kind of go back and forth on the details. Um, okay, Wade is asking— hey Wade, um, is— I know Wade from Fast.ai. He's asking, here's something I'm confused by, and I'm not sure if it's even purely a Replicate question, but what is required to push an open-source function tool calling model that would work with the OpenAI API specification? Is there particularly a way I need to formulate the prompt or the template? Is there a particular way I need to serve the model on inference? And also, if it's not relevant, feel free to ignore the question.

It is relevant. I would like to tell a simple story, but I can't tell a simple story right now. We have an alternative API that is OpenAI compatible. That's just available for a select set of models. Language models that we maintain, and we are thinking about how to roll that functionality out to users. In theory, you could design a, a predictor that is OpenAI compatible, except there are some input types that we don't support, specifically lists or lists of dicts. And so if you wanted to make a predictor that looked exactly like an OpenAI API, I don't think you could right now. However, if you just wanted to implement function calling, there are ways that you could work around that limitation with, with COG's current supported input types. And I'd be happy to, to talk about that in Discord or, or elsewhere. So that's something people care a lot about and we're working on some examples. We also do want to support list input types. That's, that's in our backlog. Um, I would like you to be able to build an API that looks exactly like OpenAI, um, even if perhaps we, we don't support it in the same way that we support our, our OpenAI-compatible API.

Um, someone's asking— I think this is a good question, actually. Um, what are some examples of application layer— of the application layer of LLMs that you were talking about that you would like to see Has there been anything like that that you've seen on Replicate that you kind of like? And a related question is, we'd love to see some demos or examples of what cool use cases that bubble to the top. If you have it, if you have something in mind you want to show on Replicate that you don't, no pressure, but like if you want to, uh, you know, just like certain repos or something that's like cool, uh, feel free.

I will— I'll ponder that. I might have to get back to you. No, I could come up with a good list. Yeah, I'm not exactly sure what would come to mind right now, but I've also been just in the depths of bug squashing, and so I don't have the most up-to-date information at this moment. Examples of application layers. We've seen some people do some cool things with Reg. Uh, we have some internal projects where people use Replicate models for operations like web scraping. I think that we, we don't have a lot of these kinds of application layers yet. One reason for that is I think we haven't done a great job of showing people how to do it and, and reminding them that Replicate is just a place that runs models. You can put arbitrary code in a predictor that encodes complex operations. So that's something we have been thinking a lot about. And we're going to be introducing some updates to COG and how you can formulate predictors or applications in COG that'll make this a lot more apparent and a lot easier. The other thing is that we recently only recently introduced secrets. And one of the problems for like chaining operations across other models— so if you want to make calls to models from a Replicate model, couldn't really do that in a way that was, uh, that would allow you to charge other users for that usage. Now we can let you have an input for a user's key and they could specify that as a secret in a prediction and then it would use their account to make those requests to the other models. And so there's now a path to making chains of calls to other Replicate models, um, which is something I cared a lot about. And when I get the time, I'm gonna build some examples of that. That said, there's no reason you can't make calls to other platforms too. So you could have a Replicate model that makes API queries to OpenAI and then perhaps processes outputs with Replicate models, does other things. So really, your imagination is the limit with the types of operations you could specify within a Replicate model.

There's a related question to what you just said about secrets is, what is your approach to logging and evals? At least the logging part. Maybe evals, you have an opinion too. You might wanna, I put a, you can show if you would like the secrets, like how to do it. It's fairly new, I think came out in the last couple of days. I put a link to help you out in the panel, in the panel chat.

Thank you. How do I do secrets? I've only done it once and I think that was pre-release. Let's look. I know there's a tweet. Oh, we have documentation. Yeah, here's how you do secrets. Um, I should share my screen. We'll learn this together. We'll see how good Replicates documentation is.

Yeah, I haven't gotten a chance to use it either, actually. Zeke told me about it. Zeke's currently in the Discord as well.

Awesome.

Oh, he's, he's here. Okay, let me see if I can promote him.

Oh yeah, it'd be better for Zeke to talk about this, probably.

Let me get Zeke in here.

Oh gosh, one second.

There we go. Okay, I think he has to push a button or something.

There we go.

Oh, you're on mute.

Hey, hey y'all. I actually have a meeting in 1 minute, so I can't hang out, but awesome. Yeah, this is the closest thing we have to a real story around secrets, so. Changelog is your best answer. I think one thing to note, which is kind of a limitation right now with secrets, is that you can't access them at setup time in your model. So if your model is doing something where it's pulling down weights, you're gonna have to do a little bit of tomfoolery to move some of that into the predict method and cache it. So we still got some work to do around this, but at least unlocks some capabilities for doing some of these types of things in userland. And with that, I'm out. Thanks for inviting me momentarily. Bye, y'all.

I think everyone's gonna wanna know how Zeke does the background, which he has a blog post for. Could try to dig it up.

Yeah. All right.

Um, is my screen sharing?

Yeah, it is.

Okay, so in Cog there's a notion of a predictor which contains a predictor class and specifies a predict function, and that's what's used to build your, your inputs. And we now have a secret input type. If you create a predict method and set the type to secret, that will generate an API signature that contains an argument called API token, and the type will be secret. And we show that in web, it's redacted. You can view it if you want. And if you share that prediction, the, the secret will be redacted there. And this information is not logged, then you can— we can use it knowing that you will not be leaking your credentials. And so what I was describing before is a predictor that maybe allows a user to pass their, their Replicate token in as an argument, and then that could be used within the predictor. So you could have a predict method that makes calls to other Replicate models. Using the user's API token. Regarding logs, that's a big question. We wish, wish I could ask for context. Can people follow up?

Yes. It's an anonymous attendee, so I can't even— don't know who the name is. Uh, but they can.

Well, I would like— it would be helpful to know what's— what sort of approach you're referring to in that question. Can say we, we log inputs and we log outputs.

Um, so can you, um, I guess like, um, Yeah, so okay, like that's an interesting one. So like that's actually pretty useful, I think. Like when you make a prediction and replicate, like you get a permalink to that prediction and you can see all the logs in all the, like, yeah, the input and the output. And you can, whatever is being printed to the logs inside the predict function, you want to show something? Yeah, pretty useful actually.

So this is Replicate. You have a dashboard which gives you access to models, predictions, trainings, deployments, webhooks, et cetera. And let's see, I could go, go to predictions. And so this is under my user, all of the predictions I've made against Replicate. And we can see the models that these predictions have been made against. We can see the source replicate. You can make a prediction from web or via API. And we get some information about the prediction runtime, when it was made, and whether it succeeded or failed or is processing. So we could go look here. So here, It looks like I made an empty prediction. Oh no, I set the prompt to hi. There we go. So this is, this is stored data, and we store the input parameters including defaults, the output.

And I find this really useful when debugging, like when I'm working with other people. And we're kind of like playing around with the model, you know, a lot of times people will like have some input or something and say, hey, it's not working when I do this. I'm like, and I can't reproduce it because I have like, you know, doing something a little bit different than they are. So it's like really nice to have these permalinks. They can like share the permalink to the prediction. I can just look and say, oh, okay, like these are the different inputs that you, you gave. So I think it's like super helpful. Is the show logs going to show anything in this here?

Okay.

Yeah.

So that's like, this is really useful too. Cause like, I think in the predict function for this model, you're like actually printing out the whole prompt, the assembled version of it. Like really useful to see if like anything janky happened.

Yes. In, in cog, in your predict method, you have arbitrary control over what gets printed. So this is, this is code that we wrote for this particular model. And yeah, we, we care about the random seed, and we care about the formatted prompt because we have experienced misformatted prompts many times. It's really nice to see exactly what string is getting tokenized and passed to the model. Yeah, you can download these, you can share the prediction, I use this method a lot. You can tweak a prediction. So this copied all of the inputs to a new— to the input form for the model. And if you're trying to perturb your predictor, you can vary things by hand. So this is stored. I think we delete prediction data. I forget what the interval is exactly. And I think we do have variable retention. So if you're a user who's very security-minded and you need to guarantee a retention policy, we can help out with that.

Wade is asking a really interesting question that I haven't thought about, saying, is there an API to get at the log, all these prediction logs to curate a dataset potentially? Like, let's say you're using a Replicate model Like, and, um, you know, like in production or semi-production, you want to like use the logs to curate a dataset for like more fine-tuning or something. Yes, workflow that would make sense.

So when you create a prediction on Replicate, you, you get a prediction identifier that you can use to query the predictions data, and that will include both the input and the output and anything returned to the logs. And so there's not like, um, a bulk endpoint for getting logs across like predictions within a time range, though that's something that we're talking a lot about, more or less on the backlog. Uh, that's something that would be really valuable for us internally, and I think a lot of users would get value out of that. Um, But despite not having a kind of bulk dump method, as long as you keep track of your predictions, um, you can also query predictions. So you can get prediction IDs for predictions you've made, then you can iterate through and get logs for each of those predictions and any other metadata that you care about. So yes, there is API support for getting logs.

Um, there's another question. I have $501 of credits. And I really don't know what to do with it yet. So I got this idea from the website, fine-tune SDXL with your own images. And I'm thinking of training a model on some images of my kids and playing with the model. There's something I should worry about in terms of privacy or legal or anything else?

I mean, let's see. I wouldn't— there, I guess, always things to worry about. I don't know the date on our— the time range on our data retention. If you don't want images of your kids living in a database at any amount of time, then you probably wouldn't want to use this. So data will be stored for a little while, but it's encrypted and we don't share it with anyone. So I've uploaded images of my daughter to Replicate to fine-tune models. I don't know if that's a helpful answer. Legal, I'm not necessarily sure what legal implications you'd be concerned about. We follow US law and regulation. So any violations of US law regulations, I wouldn't suggest. Aside from that, I don't think I would have any concerns. I can say that it's really fun to fine-tune your own SDXL, so you should try that.

OK, let's see. OK, so we had a follow-up about evals. So the anonymous person said, by evals, I meant evaluating how a model is doing over time and seeing how users have been using it and what that looks like. Someone's asking if you have some approach that you like. We've definitely covered this subject a lot in the course.

I, I've done that at other organizations. We have not done that extensively at Replicate. Our relevant concerns within recent time have been more performance and reliability. And we have not really spent time doing any sort of investigation of what people are using language models for on Replicate. That said, that's something we're really interested in. One, it has great bearing on performance. We do see a lot of heterogeneity in how people are using Replicate. Some people will come dump a lot of predictions, and it seems like they're doing some kind of bulk processing or maybe running eval, and then other users, organizations have much different demand characteristics. We see an enormous amount of variation in input and output length, and I think learning a bit more about user intent and the characteristics of their requests and their outputs could help us build a better system. Um, thus far Replicate's been kind of agnostic to, to what users are doing. Um, So we haven't really prioritized a deep dive into that sort of intent evaluation. If I was going to do that, I would probably start with EDA. I would use Replicate's ability to dump logs, and I would look at a bunch of inputs and outputs. And probably start by doing some sort of clustering and just look at the data.

There's a related question I actually think is a really good idea. It's like, um, so Wade is asking, it would be nice if users can thumbs up, thumbs down the prediction and have it have that logged as well, I guess. Okay, so there's like two questions. One is, have you all considered that as replicate as some kind of affordance to make that easier to collect feedback? And the second is, OK, in the absence of that feature, what's the right design pattern? Is it to take the prediction ID and then log it to a database, basically, alongside the feedback?

That's a really good question. I was just thinking about that recently. We have some web apps, for instance, that implement chat with Llama 3. And if I remember correctly, I think there is some sort of feedback mechanism there. That said, I was just last night thinking about, I don't know, maybe the ability to add annotations to replicate metadata specifically so that you could associate perhaps some kind of feedback information with a prediction. So I could imagine thumbs up, thumbs down in web. It would still kind of have to be an annotation, I think, because the prediction already exists. And then really you would be wanting to add additional information to the metadata associated with that request. And I think that would be pretty useful. That's, I don't think, something we've really talked about. But I like that idea. And it would be useful for workflows that I care about. Short of that sort of like annotation mechanism where you make a prediction, in some way it's evaluated by a user or a system, and then you associate some kind of feedback information with that, that specific prediction. You could of course do that on, on your side, on the client side. So if you're logging, you could log just prediction IDs and feedback, you could log more of the information in your own database. We— one of the kind of questions and concerns that we consistently think about at Replicate is where to circumscribe the functionality of the platform. And there are many kind of especially language model oriented observability platforms that might have that sort of functionality. And we can't do— it's not practical to do everything or try to do everything on Replicate. And so we're constantly thinking like, what's a really high-value functionality feature that our users would actually use potentially if they can't use elsewhere? So where to draw that line is kind of a moving target. And this idea of modifying prediction data after predictions have been made would be really valuable. It would also sort of be oriented towards workflows that we have not completely thought through on replicate. So I think that's probably something we'll care a lot about in the future, just hasn't really been prioritized thus far.

Okay, we have time for one more question. This is actually by Andrew K, and I don't completely understand his question, something about secrets and something else. I actually allowed him to talk. Andrew, if you feel comfortable talking.

So I was thinking if you could create an embedding model and have it all hosted, and then you could query that embedding or the, the data, the RAG database to create additional datasets. You know, and, and the secret, you know, because it wouldn't be used all the time, you know, so it's kind of like as needed, you know, because, you know, I was wondering if that would be a possibility on your platform, the secrets aspect of it, because, well, the— see, I, I thought he was saying that the secrets you could you could, um, get payments or credits through it.

Oh, you want to host a paid product through Replicate or something like that?

Oh yeah, some— yeah, something like that.

So you mean you host some sort of process for which you've set up billing?

Peter Levels, like, maybe good to like kind of— like Peter Levels hosts his applications on top of Replicate, or whatnot, and like he has paid. It might be good to like talk about the design pattern of like, okay, how can you build like paid applications on top of Replicate, like as an indie hacker or something?

Yeah. Um, so we, we don't really have pattern to my knowledge for like billing users in a, in a Replicate model. I guess you could if you implemented it. So just, just to make sure I understand, Andrew, you mean you have a model that in some way implements billing? So user comes along and they pass some information, and then they run the model and you use this information to bill them. Is that the idea?

And more or less, but, but like, I don't know if, you know, even if you got— if somehow if they, if they had credits, you know, and they run your model, those credits go to me so I can run another model or, or use it for other services.

Yeah. What you just said sounds a lot like what, what we're thinking of internally as revenue sharing, and that's, that's something we're talking a lot about recently. I don't have anything I can share specifically, but we are really interested in that as a, a model for incentivizing collaboration on Replicate. And so the idea there would be you build a model and, you know, in, in whatever way we ship this, there would be revenue sharing, and so you would, you would get some sort of benefit, credits, money when people use your model. Yep. Yeah, that's something we're really interested in. All right, thanks.

All right, Joe. Yeah, that was really helpful. Um, we have to wrap up this office hours, but yeah, it was good questions. And yeah, thank you.

Thank you.

It's always great for you to pop in the Discord if, you know, there's always people hanging out there.

Yeah, I will. This is a really wonderful course. I was glad to be a part of it. Anybody has Replicate questions, please drop in the Replicate Discord as well. We can answer them. Thanks, everyone. Thanks, Hamel Husain.