Live CEOing Ep 716: Language Design in Wolfram Language [HumanEvaluate and more]

Okay. Hi, everyone. Welcome to another Wolfram language design review. Looking at a variety of different functions. Part of our incremental language development series. All right, first up, Human Evaluate. It's a function we thought about, like, 15 years ago and. Right. Okay, that. I. I bet I have notes on this from, like, at least a decade ago. See if I can find them. Yeah, I tried scanning through the design notebooks I have, and it probably predates those. I wasn't sure if Human Evaluate would be, like, a mechanical turk sort of situation or if it. You know, this is a subset of reinforced learning from human feedback. Well, this is a. Yeah, I mean, it's. It's a Somehow interesting. Maybe I called it Human Compute before. Oh, yeah. 2015. There we go. Decade wasn't too far off. Okay, let me see what I've got here. See what I have mentioned with Carlo Ricardo and others about the roadmap 2015. Okay, let me just see. Oh, boy. File system has a lot of these, And they're all in. How very confusing. Oh, here we go. Project accumulator post. Okay, there have been some summer school projects related to this. I'm looking through. Sorry, there's mostly things that are. Probably should have figured this out beforehand. Boy, something I should. In my. There's a lot of things in my file system which are currently only showing up as dead cache hits. Just a minute. Sorry. All right. Okay, we will be. Let's just see. I'm sure I can find. I mean, it's clear that there was text about this previously. Oh, gosh. What the heck? I mean, hold on one second. I think I'll be able to find it. I hope I'll be able to find it here. Just endless things called old files. But there was a one here. Okay, let's see if I can find this. Oh, see? Background material, slash, Human Compute. Okay, this was from the 2018 Summer School List of. Look at this. Oh, boy. That one could be done. Well, better now. Responder bot for spam phone calls. Mm. Let's see. Background material, Human Compute. So let me just go find that. I think that one was in Dropbox. That was the one thing I saw about, like, human stuff in your Dropbox. Okay, hold on. Let me try and find this 2018. Well, all right. Maybe we're starting with this background material. Oh, here we go. Okay, well, this is something here. I don't know where this came from exactly. I don't know who myself is. So. We had had a previous effort to connect a service connect to Mechanical Turk. Really long time ago. Anybody know about that? No, sorry. That was the thing we had built when Mechanical Turk was young. Wow. 2005. Service Connect didn't exist in 2005. Okay, here I've got something. 2007. Let's see. It's a whole discussion for 2007. Oh, there's somebody who has an alternative to Mechanical Turk that they were trying to sell me on. Looks like. Okay, looks like 2009, which is when Wolf Malpha came out. We had a whole discussion about this. Yeah, I mean there's hundreds of emails here. Okay. But I'm not sure they're relevant. I mean, I think this is. I, I don't know. I, I think maybe we're better off starting from scratch here. Yeah, we can do that. 2019, there's all kinds of discussion about this. Okay. Now, I mean we also have a lot of technology in our curation pipeline. I don't know whether this is, I mean, do we know we don't have anybody here representing that, like Alan for example? No. Could we maybe ping Allen, see if he's around? Yep, I can do that. Okay. All right, so just so we're clear, I mean the basic thing here would be some kind of human evaluate that, you know, defines an expression that now presumably, I mean, what do we expect will be the typical flow here for a human? It's like here's a question. Well, it's basically the question framework, isn't it?

Really, really depends on how you feed it back into the system. Or the reinforcement learning loop. You are using the human intervention to train an agent that then is used in a reinforcement learning loop. So it's a two step procedure. Then you have something like active learning. While you have a model, you're trying to estimate the weak point of the model using the model itself. And then you are asking a human to, to provide more labels and then you are feeding in those labels to train the model again and continue the loop. So in that case, the human is in the loop continuously. In the first case, the human is in the loop only at the beginning, unless you want to train a new agent.

Right, but. So let's look at our frameworks for doing this stuff. So we have the whole question object framework. Right. So in a sense, and this is a new thing that we didn't have when we were talking about this before, we have exactly the mechanism, because question object, we have usually imagined that its objective is we are assessing the human, so to speak. Right, right. As opposed to, I mean, by the way, we also have the ask framework. Right. That's what first came to mind to me, which is very cloud centric, but seems like, I think that's what we want here because I think the edge of this has to be, you know, maybe it's wrapped inside Mechanical Turk, maybe it's our own system. It's something where, you know, humans are being presented with things. So I mean, what, what's the basic idea? The basic idea is somebody would log in. I mean, just like our former kind of volunteer data curator program for Wolfram Alpha, which by the way we also might be able to use because we had some technology for that as well. Yeah, we had some technology there that I think is interesting. I think Alan would be the one who would know about that as well. Okay. Because that was a thing where you can take a set of tasks and mean that's, that's more in the offline way of doing this of humans rate things and so on. But. Okay, but the basic point is what we are trying to build up is machine generated this, you know, and human generated and, and then a human gives a response here. So I mean, okay, the most naive level, it's kind of like forms. Right, but the, the issue here is there's sort of an outer loop in which we say these are the things we want, want to get resolved by humans. Right. I mean, are you looking for a.

General, like I need human input moment in a computation?

Yes, we're looking for, you know, assume that there's a pool of humans that you have access to. Okay, which, which and somebody else will take care of, you know, pay the humans, et cetera, et cetera, et cetera.

Okay, but if you say pool and, and, and pay, it means like some of a low, low effort thing. But could it also be something like this method of ndisol that you, that has been selected like is not really working. So do you want to keep it or you want to change it? Like something like kind of expert knowledge. Like you are looking for something like this as well or just like low level labeling, like is this a cat or a dog? Just pick one.

Well, I'm thinking about that. I mean, I'm thinking about what. I mean, I think this is more, you know, this is the thing that, you know, in a sense the meta point in chat, GPT and so on is there's a lot of information that humans have just put out there on the web that you can forage. But then there's some very high value information about kind of, you know, this is how the AI is, is going to do something, you know, teach IT type thing. And I think what we're looking for here is, you know, an AI has surfaced a bunch of things where it's kind of wondering, you know, humans, please rate this. Am I making sense? Rather than. I mean, and, you know, you can imagine the loop where the, where the AI is saying, is this better? Is this better type thing? But I think most of the time this is not going to be an online. There's going to be an offline process. So, I mean, you know, from our point of view, you know, what is the. It's really going to be something which comes back, you know, it's not going to come back instantly. It's not like a computer where you say, you know, here it is and it's back in milliseconds. Right. This could take two hours. So, I mean, I think it's more like you give some computation and I mean, how does it work if we have, for example, cloud submit. Okay, well, let's look at how cloud submit does its thing.

It's going to be like session submit or local submit, like offloading a computation to a cloud machine instead of your local machine.

Right. I mean, in this case, you're offloading to a human to do it. Right. And. And then you. Where do you get the result back? I mean, if I say cloud submit two plus two, what's going to happen here?

You get four. Oh, no, sorry. This is submit. Sorry. You get the task object. So you get to. Yeah, you have to take. Assign it to a variable and then get the. Get the result out.

Well, so okay, so then we say result received. Okay, okay, so then we say evaluation result. What the heck?

Oh, I think the task was cleared out when it was completed. So I think the flow here is that you have to define something to do with the results before actually the whole task is completed. This should be some example of what.

A possible key is specified by Handler function keys.

Yeah, so you can say, for example, with evaluation result, you say X equal evaluation result or something like that. So in this case, for example, when the task is finished, you get a dialogue and dialogue gets the evaluation result.

Right. I mean, if I say for example here.

So the simple one that you were looking for is like cloud evaluate.

No, I know, but that's a synchronous thing.

Exactly. I don't know. I don't think this can work because you're in a different kernel. So this.

Where should this message dialog work?

It sends it direct. I mean, it's a different link. Maybe if they do print. Maybe it will print in the message window or something like that.

I have no idea why ECHO didn't work.

Because this is echoing my guess. I can be correct here, but my guess is that he's echoing on the notebook connected to the remote kernel, which is possibly no notebook, but it has no way to communicate with your notebook.

Okay, but this is coming back to this kernel.

Yeah, that's a good point.

You know, I had this experience trying to get ChatGPT to. I mean, I was connecting it are kind of amusing. I had a function which echoes from a custom plugin from Chat GPT and it was sending those echo outputs in totally random places. Like, I found one in the messages window. I found one in the notebook I was using. It just sort of sprayed them in different places. I would expect ECHO to put it in the message window in this case. Well, maybe it did.

Well, if you do print, it goes to the message window, so I assume echo as well.

All right, so let's try this then.

And now you open message.

How do I do that?

Window, the one before and messages. Yeah, right.

Yeah, that's hopelessly. Well, who knows?

But it was there. It opened on your other screen.

Okay.

If you drag it here, we can see it. So, yeah, it is doing something there.

Okay, fine. Okay, hold on. I can go fetch it. Just a second here. Well, maybe I can. It's more difficult than you think. Oh, yes. There we go. See if I can get this to work. There we go. All right. Here's my messages window. Fantastic. Oh, so it.

Okay, so you also get the echo here.

Yeah, it looks like I did for. For evaluation, result, arrow, whatever it is. Okay. In any case, be that as it may, I mean, asynchronous stuff is hard to deal with here, but, you know. So in the case, by the way, did we manage to find Alan or not? No, not yet. I didn't get a response. Kristen's working on tracking them down. Okay. So, I mean, It's. It's just a little bit unclear how this would work, because, as I say, your, you know, best case, it will generate a task object where the task environment is human. Do you see what I'm saying?

Yeah.

And then eventually it will be finished. And I mean, so then what would happen is this would at least conceptually, it'll pop up on somebody's machine and it'll say, do this stuff. And then. So I wonder whether we can trivially do this right now. In other words, the task. Yeah, this task, Somebody would have to check out the Task. Right. So in other words, there will be a queue of tasks. Okay, so how does it work? Generates a queue of tasks. You know, person picks a task that they want to do. You know, when they finish, this goes back through the. Through the handler, through the event handler. Does that make sense? That makes sense.

Yeah. I can assume we might have like different, like events, for example, the, like the stuff started. But we also have maybe like when the task was submitted, when the person accepted the task.

Yes.

You have like task time spent into different stages. So you can do something if it takes too long.

Task accepted by human, basically. Task underway by human. Task rejected by human. I'm sure these things exist in the.

We should look at what exactly? We should look at some API that is doing this now.

Right. But from our point of view, the plumbing, okay, and then task completed, and then we have. And then there's all kinds of things about running checks and so on. But that's less relevant for the AI stuff, I have to say, you know, it's just like the more you have of these, the merrier. And this like a task, it doesn't block the kernel. So we're saying. Correct.

Yeah, this is a submit thing.

So, so you don't. So it's not. You don't use this like in the middle of a code that then's depending on the results.

But if, if you want, you can do a task wait on it. It's like a SQL wait in JavaScript. You cannot, you can always, I mean you can always make an async code, not async by. By waiting.

Right.

But in the more general case, it's asynchronous.

Right. I mean, I think this is, I mean, the very beautiful remote batch submit. This allows you. So this is a case where, I mean, this, it feels very similar because, you know, some of these things, your submitting a batch job that might take a day to run.

Yep.

Now, an interesting question is whether, for Amazon in particular, whether the Mechanical Turk credentialing system, you know, authentication system is the same as this one here. Do you see what I'm saying?

Well, I believe this. With Amazon, you have one account, but you can have access to different. I mean, one account could not have any access to particular service. But I have to check this.

Right? But I mean, this here looks very much like, you know, remote batch submit. I mean, an interesting idea would be could we just use remote batch submit for human tasks where the back end is not a computer, but a human. See what I'm saying?

Well, I'll have to check exactly what's the difference between the session submit and the remote batch submit. If this just due to some technicality of how.

It's a huge piece of infrastructure to do with how this remote batch submit works, it allows you to access the job scheduling queues and all that kind of stuff. You see what I'm saying? Yeah.

So you're saying that this is strictly more sophisticated than special submits.

Yes, but it's more sophisticated in a way. I mean, look at all this stuff here. Oh, God, we're going to have issues with child labor laws if we have all these child jobs. Think of the same thing. It's different. Okay.

Okay. So do you see this as an interface to some kind of external provider like that we wrap it in something which makes it convenient to use?

Probably. I mean, it depends because I can also imagine. So, Alan, we're talking about this human evaluate, function, compute. And we're remembering previous, you know, things like the volunteer data curator program. Right. For which we built a certain amount of infrastructure. Right. I mean, there's nothing that I think would be useful in the present day. I mean, we, we sent people spreadsheets and asked them to send them back in and we imported things. Okay, fine. Okay. But so mostly okay. And the other question is our curation pipeline stuff, which again may not be relevant to this because it may be that what we're doing here is at a different level. That is that. I mean, I don't necessarily.

I mean, that was the first thing.

I was thinking of when I'm trying.

To digest what I'm seeing on your screen because I don't know much more about this. But that's.

That was the first thing I was.

Wondering if that's going to be useful for this.

Right. Well, I think the main thing here is for reinforcement learning.

Yeah.

Stuff. And the issue is. Yeah. I mean, I don't know to what extent this thing is kind of a disembodied thing, as in you just say, here's this thing. Is it good or bad? And to what extent it's important to have kind of real time feedback saying you're doing a chat, thumbs up, thumbs down at each stage because those require somewhat different infrastructure. From a machine learning perspective, is it.

Typically from a perspective you want another. You want a data set and you let it go for you have an amount of answers that you want, and when you have it, you take the data set and you start training.

Yeah.

Okay, so you're saying it is very much offline using.

Right. It's not like the robot is trying to feel its way around the room with reinforcement learning somehow. Or learning to balance a pendulum or something. Or learning in a game.

Exactly. This was the point of the human in the loop thing. But what it was done, it was they trained the reward model using this, and then they put the reward model into reinforcement learning. So you are using this human feedback to simulate human judgment. A human, A human evaluation.

But not in real time.

No, no, no. What is used is in real time is that when you have the simulation, you have a model that simulates a human motivator. Then you do the training in real time, the reinforcement learning in real time. But at that point, you are dealing with two machines, not a machine and a human.

Right. But so for us in the reinforcement learning stage, so let's imagine that we've got. I mean, we've built a certain amount for this. And I always get confused by what's actually useful or not.

In this case, we built a little bit, not too much, honestly. And this is one point. Like, if you want to use this for enforcement learning, I think we should first design reinforcement learning from ground up.

Because what we had before was kind of crazy. I mean, in my view, because it was very.

With the device framework.

Yes, it's very real time, which I don't think is really the point here. Right. I mean, in other words, what, what one wants is there is a, you know, reward function type thing which, which basically just says, you know, which is some kind of function that could be, that could be obtained from the human feedback system. And then. What you're trying to do is training against that function as opposed to training. I mean, we, we've had that in the, in the whole active classification thing and so on. Active prediction thing. Right. That's, that's surely what that is, is something which is trying to figure out how do you poke at the function. See what I'm saying? Yeah.

I mean, the idea of this is basically that you have a long list of unlabeled data and you have a noracon function. So here, this hash +2 is the function that knows the answer. Then you have a long list of data which you don't have labels for. And the point of active prediction is like, I should try to call, like, imagine that the, the labeling function is not like something like this. It's more like. Exactly. A human or like an API call. So you want to use it as few times as. As possible. And, and this is.

You don't really care if you've trained Something based on a bunch of human feedback stuff. You've now trained it and it's now cheap to call that thing many times.

Exactly. At least why you do that.

Yeah, right. But, but so, so then in this case, what's the difference between. So, so what you would typically want to do is to take an existing model, you know, a NET model of some kind, and you want to basically tune it up. Right. With. With. Is that, is that a fair, you know, reinforcement train of the model with the training function, with the reward function?

Yeah, but this is, this is generic reinforcement learning. Yeah, right.

But the question is, what we've done in the past was a very messy thing which involved, you know, worrying about devices and real time feedback and this and that and the other. I think what's emerged is that there is a form of tuning a model, so to speak, that. I mean, you know, actually, let's ask a couple of questions.

No, no. Okay, but, but, but the reinforcement learning step is still in real time. Like that. This is what, this is my point from, from before. Like, if we want to use this thing in reinforcement, this is a separate step that basically creates the environment for reinforcement learning. So if you want to discuss if.

This is something where this thing would run n iterations and give us back a new NET model. Yeah, it ran. You know, everything is real time. It's a computer, it's computing. You know, it's not.

Yeah, but, but, but this has nothing to do with the, with a human part except that the reward function was created that way.

I understand, I understand.

So I don't have a good design for enforcement learning in general. We don't know.

Right, but so to me, okay, what would fine tuning look like in our framework? Because what we don't have, I mean, we certainly have. Here's a model. Now go train this model. Right? We have. I mean, you know, if I were to take, you know, NET model of, you know, mnist, of Lynette training Stata, right. I could take that model. Happy, happy model, and I could say, let me, let me make. Just for fun, let me train it on. I don't know what I can do. I can. Pictures of cats and dogs or something, or pictures of. Let me do this. Let me, let me just make, Let me rasterize. Let me make a. Goes to rasterize. A comma, style. Sorry. Of rasterized style. Hash, comma, 30. Say something like this. And let's say that mapped over the Alphabet. I've never tried this. What's going to happen? You know what I'm going to try.

Yeah.

Okay. There we go. Net train.

You need to revert this.

Oh, it's the other way around. Right. Thank you. Okay. And then net train of this NET model here. Boy, this is really nice design. Wow. I never tried it this way. Well, let's see if it does anything.

It should. Oh, no, because you didn't modify the network. Now you have to basically replace the output with more classes. You have 10 classes and a class decoder that is going from zero to 10.

All right, so let's just do this.

Exactly. So if you do that and then mod.mod 10. Exactly.

Come on. There we go.

Okay, now this should. Should work.

Yep. Cool. Soros. Okay. Is it surprising to you that it's going in jumps? It seems to have learned.

Well, the error rate is not related. It's not one to one to the loss. Yeah. Because it's a piecewise thing. If it moves to one letter or the other.

Okay, great. Let's take this and. And as our test, let's feed it an S here. Let's see what it does.

Well, I mean, hopefully it has learned the. The training set. You should try like a different S, but.

Yeah, I know, I know, I know. But let's try. Okay, let's just try something like rasterize, style S, comma, 30, comma, font, family, arrow, you know, act. I don't know.

Let's try this, whatever that is, like Times New Roman. Oh, you have raster instead of rasterize.

Okay, let's try that.

Well, you gave one example of everything and is also squished because it's resized to a square. So. But yeah, I mean, in principle, this is going to do something. Yeah.

Right. So what we're talking about here is the.

I think. What if I can interpret your mind? What you're talking about is a design meeting or enforcement learning.

Yes, but I mean, I'm just trying to understand the basics here. Okay. So if I'm understanding correctly, like active classify here what we're doing, the difference is that whereas here we are trying to make a prediction function from nothing. In the reinforcement learning case, we are taking an existing model and we're saying, I mean, in the case. In this human feedback case, what we're really going to want to do is basically just this, Right. You've done a bunch of human feedback. The only difference is the selection of the questions that are, look, the loop is something like this.

But there's no difference. It's just how we wrote the function. You can start reinforcement learning loop with an untrained model. And you could have active prediction with an already trained predictor. That's just the initial condition.

Okay then, so let's be very practical. Okay, We've got some LLM for example, we're trying to do. You know, we've got, we're trying to do RLHF for the thing. Okay, so it's going to surface certain things where it wants a human to opine. Right. Model surfaces things it wants to ask about.

No, this is again, we need to keep this separated. The way this was done was that it was already all clear from the beginning. You had the language model generating a bunch of continuation and you ask the human to rate those continuations. This is step one. Then you take these votes and you create the reward model.

I understand.

And so it's not a model.

Okay, step one. You just said there are a bunch of continuations. Rate them. Right. What continuations? What are you doing? The continuations of.

Well, you have a prompt repository.

Well, right.

You as a engineer, you define that maybe it's a prompt repository that you did for the initial find with supervised learning.

Yeah, I understand, but. Okay, so all right, fine. So it's not surfacing things it wants to ask about for itself, so to speak.

No, this is more what active learning is doing.

Yes.

Kind of judging the performance like, or estimating what is like the boundary region. And there's a lot of theory about. Right, exactly.

That's what I'm saying. This is at the point where. Okay, so you know, okay, look, the, the thing about training a reward model, okay, we kind of understand that's, you know, that is a supervised learning story.

Here.

Right. And for that which is, you know, a fixed set of questions and you want a human to opine on the results. That's where humans submit.

In this case, for example, the question, I mean in the case that you are thinking about, the question is like you have this initial string, these 10 way to continue the string, sort them by your preference.

Fine.

This is the human evaluate task. And then you get the data set and you do something, I don't know, like eliorating to, to. To take this training set and make it into a scholar evaluation.

Yeah, fine, that's fine. But then what's the difference between what you do next? And you're saying one thing I could do is take that training set. I could, you know, one thing is take the, the human generated, you know, input output training set. Input output set and train with that. But the, but the other thing to do is and Train.

Sorry, sorry.

The original. Okay. Train the original model.

And this is what happens. This is what this is like step, step one, because you do an initial fine tuning, right?

So that, that's, that's initial fine tuning. Okay. Then the step two is train a model from scratch. Train a reward model from scratch from the input output pairs.

No, no, no, no, no. This is the point. You train the whole model from scratch from the data that you collected with the human submit.

I know that's the input. Output. Yes, that's what I mean. Take the human generated input to output set.

Okay. Or you do it twice. Okay. This is not for the world. Okay, let me clarify the whole pipeline because otherwise we look at it in different pieces and it's semi clear. So for the specific case of language models, what you do is you take a language model, you have a repository of prompt, and then you ask humans to continue those prompts. Like tell me these, follow this instruction answer more nicely. Don't say this thing. And then you have the human continuing doing the job of a chatbot. So this is a first human interaction. So this is the first human submit. Just continue the prompt as a human. And then you do some supervised training that's normal training of a language model like learn to predict, to predict the continuation. But now is a human continuation of a specific nature. And this is the first bit. Then you have a second human interaction which is. Now you have a model which is better at following instructions because it's seeing examples with a first human interaction and you have it generating different possible continuations. Like you ask for a question, you have four answers and then the second human interaction is the rate, you know, sort according to your preference these, these four answers. And on that sorting you train the reward model that is in turn put into the last step, which is reinforcement learning. So we have two calls of human evaluate. Let's say in this, in this.

Yeah, right, fine. But we, we. The reinforcement learning part is a part that is like our active prediction, but slightly different packet, differently packaged.

Yeah, because active prediction is, is looking at specific inputs where the, some external model is estimating that you need to improve.

I know, I know where you were given these things here, right?

Yeah, exactly. So you have this unlabeled thing and you need to know when to label them. In this case, for reinforcement learning, it's typically not expensive to call the environment. It's called like the oracle function here is called typically environment because you have the agent that interacts. The nomenclature is from robots interacting with the outside world. So the Reward model here is the environment and the robot is a language model. But I think we are looking at this too closely to this specific problem.

I understand what I'm trying to understand. There's a final reinforcement train or something. Reinforcement learn, whatever it is, which takes a model. It takes two models, the model to train and the reward model. And what's going to happen is it's going to start selecting different possible cases and it's going to then, you know, see what this model, this model goes input to output and then the reward model runs on that and you know, then, then it has to adjust this model and the output here should be a, you know, improved model. Right? Correct.

Yeah. I mean, like, it depends. It depends on what? I mean, there are several techniques for enforcement learning. I think that the one used. Okay, right.

But I mean, and the issue is, you know, one of the questions is which samples do you take? You know, in this case, this thing should be figuring out which samples are worth taking.

No, no, no. This is not typically what's happening. What's happening is that you still have your prompt repository, but in this case, differently from the typical training, you're not getting an answer token by token, like in the first supervised learning or in the unsupervised thing, because you want it to finish. So, so you take your repository, you let your model go and generate a bunch of continuations from 10 prompts, 10 continuation. And these are all hyperparameters. This is the reason why it's so finicky. Reinforcement learning. And then you unleash your reward model and say, I think you did well, I think you did not well. And then you feed the score into this and then the loss function of the model takes into account the score.

Right. But the whole point is you're doing it on the whole output in the LLM case, perhaps different from some other cases, like I don't know what you do in the, in the robot case.

Right, that can also be. Yeah, because it depends like, you know, if you, if your robot needs to like put a ball into a basket, it doesn't matter where it is in between. Yeah, you only, you only can give, you know, like a well done when the ball is into the basket. Or you could have a different regime where you wanted to take the shortest possible route so you have a penalization.

Or it's playing some video game and it tries to see whether it can do the whole game and win.

Exactly, exactly. Of course, I mean, if it is too long, you have stability problems. Because if I just tell You. Well done. When you win a chess game, it's very difficult to understand what was correct and wrong in the middle.

But the fundamental point here is this model to train has certain. I mean, it's. You know, it's not a single token at a time. It's the folded version of that, the nested version of that. That's going to generate the whole output.

Exactly. Run it. And then I. Let. Let me read it. Okay. I like it. I don't like it.

Well, the question is here. For this reinforcement learn function, we have to figure out, you know, how. What are the levers that it has on the model to train that it can change. Right. And that's where all these different techniques for reinforcement learning come in.

Exactly. Like PPO or like, you have, like, the cubal base. I mean, there's an off policy on policy. There's big families.

What is the typical. For our case? Okay, let's say we were using this model, some very simple model here, what levers would be. And let's imagine that our. Oh, I don't know, our full thing is reading a whole word or something, or reading a whole number. And I don't know, we have some loss function that's based on. I don't. I don't know exactly what it would be, but where. You know, where it's this. Like. Like, for example, let's imagine that our whole function was just something like map this over, you know, a list, a.

List of letters, and then you have the letters, you join them into a word, and then you have to say, is this the right word or not? Something like that. Yeah.

Yeah, that's a good example. Yeah. So let's imagine that this was letters. And then we say string, join of that. And then we say dictionary lookup. You know, dictionary word Q of that.

Okay. This is. This is a discrete reward. So it's. It's difficult to take the gradient of this.

Okay, okay, fine, fine, fine, fine, fine. So. So it's like, I don't know how we do that. You know, how close to a. You know, how close is this? Okay, so now what do we do?

You, for example, you can. I mean, typically, what you want is that you want to give an estimation.

What are the levers that we. Given that we've got this thing. We've got this loss function, what are the levers we. Then how do we know what levers we can pull here?

Like, what are the other. The other parameters of a specific reinforcement learning model that you're using?

I understand, but in the End you want to modify this. NET model that the end goal is to modify that. NET model. Right. So I just, I'm just asking how would we get inside the. NET model to be able to modify it given that we have, you know, essentially what we've got here more accurately is, you know, function of the underlying model. Right. And what we're trying to do is to train the underlying model based on the reward model applied to the function of the underlying model. Is that right?

Well, I mean, the, the reward model is used in computing the loss. So basically what you have is that you apply the underlying model to your thingy. This is generating, in this case, probabilities of letters for each. And then let's say very broadly speaking, you try to assign a value.

Look, you've got some function of the underlying model that's applied to. This function of the underlying model is being applied to data at every step. Right. In, in the case of an LLM, it's, you know, keep running until you get to a stop token.

Yes. Okay.

Okay. And then the whole point is that what's different here is that you are applying this reward model to the whole thing that was generated.

Yes.

Not to the individual steps of the underlying model, right?

Yes. In this case, yes.

I'm going to claim it's kind of almost the general case. I mean, like you're playing a game, there's a thing that, you know, takes a step, an individual, you know, turn in the game and then there's the whole picture which is how you're going to win or lose. And then.

Yeah, but like, for example, stuff like Alphazero, like the reward model is more like that you are trying to estimate your probability of winning at every move.

Okay, but I think in the end this is. But the way you're doing that, in the end you're going to win or you're going to lose, but that's the result of a whole series of things. Whole series of steps.

Yeah, I mean, to be more precise than that, I will need to like refresh my knowledge on this. So I don't want to get too detailed and possibly say one thing.

I strongly suspect this is right because I've been quite confused about, you know, what the minimal way to state all this is. I suspect this is it. And then the thing which is not obvious is, you know, how do you dig inside that function to go and make modifications to this thing that's inside, given that all you're actually seeing, you know, in other words. So. Yeah, but this is the standard problem you know, how do you, how do you update. It's like you've got a big neural net with many layers. How do you update individual parts of that? I suspect this is correct. But coming back to. And, and this will be a very useful thing for us to clarify how this works, right, because, because I suspect this outside function, unlike the functions that we already have built into, into layers in a neural net, right where we know how to differentiate them and all that kind of thing. You know, the confusing thing is that outer main loop function, outer loop function is not as. Obviously, you know, we don't have that in that same framework. So I'm saying. So that's why it looks different and more challenging to do. Learning against it, so to speak. I am going to need to go in a second here but, but the, You know, back to the human compute thing for a second. I think what's becoming clear is that the fundamental operation is you submit a bunch of expressions that could be questions and you basically want people to provide a mapping from expressions to results. And that mapping could be framed in terms of the expressions being questions and the results being what they would do in question answers that would then be fed to an assessment function. Or it could be anything. I mean it could be like, like what's the first word that comes to mind given this particular thing. It's just some mapping from inputs to outputs defined by humans. And then the workflow we have to figure out is just how do we, you know, how do we present those as jobs in Mechanical Turk or some other system. Does that make sense?

Roughly speaking, yes. It's more like, you know, it's very similar to some like controls we have. You know, it's like a picker, like a text field. Depending on what you are asking, if you're asking. Right, but that's type something in to select something.

The question framework exactly deals with that issue as to some extent does our form framework. I'm saying, yeah. Other words the issue here is more the question of how do you present it to whom, who, who picks the jobs, you know, out of the queue, you know, etc, etc, etc. How does that all work? That's which is where the whole framework for you know. And I think if, if we can do that we now have, you know, we need this additional kind of thinking for this. I really do think this line is, is kind of the, the summary of what's, what's going on the. But anyway, so you know, I think what we should do for the human submit thing, if we think it's worthwhile, which I think it is, is understand what these different platforms are and whether it's easy to make a service connect. Whether for example, in the case of the Amazon one, the remote batch submit thing, which already works well on aws, could almost directly be used for that. And I was not expecting that this discussion would take the whole of our time here, but it has. So I think we have to wrap up here and should I schedule a follow up meeting specifically for this? Like a shoot off type meeting? Probably. I mean, I don't know what the priority of this is right now. I think it's worth. I mean we've got a lot of other things going on. Right. We may need this for some of the machine learning projects we're doing, in which case we better have built it, you know, if we need it. So I think it will be worth. Yes, it will be worth a little bit of assessment of how hard is this to build.

Yeah. Okay.

And maybe Jesse who worked on remote batch submit might know the answer to that from that side.

And for human submit, most of the work is. Is like non machine learning related because it's like interfacing with something.

I think so, yeah, I think that's correct. I don't think it's for your team. I think it's for the service connection team probably. Okay. And. And they may already have stuff that was done because I know we did do stuff years ago on this, so let's try to find that. And yeah, the part for the machine learning team is more this reinforcement learn type functionality which I claim is not as scary and undefined as it looked before. I mean what do we have? I. I need to go. But. But we have some stuff. Right. For. This is such a horrifyingly not the point thing, I think. What is this? Where's the. Okay, these are the states. But where is the actual learning?

No, there's no learning here. This is just an environment.

Okay, but how do we use this environment?

Well, basically this is to generate the. Is the oracle function.

Yeah, I understand. How do we use that oracle function? What function are we using? We're using it inside Active. Active.

No, no, inside Netrail. Inside Netray. Let me find the. If you search reinforcement learning, you should probably. Okay. OpenAI gym has. Should hopefully send some example. Okay, now this is just a link, not a second.

Great. There are the video games.

That might have been an example. I have to check because this was done years ago. That might have been just an example in marketing pages when this was released about how to use this, but. Yeah, let me check that and.

Okay, all right, fine, fine. But then I'm claiming that this is the kind of thing we eventually need for various functions here.

Yeah, I mean, we have several design proposals. So, like, if we have a follow up meeting, I'll bring up what Jerome did about this stuff like last year and. Yeah, the stuff we have to look at. To look at in that case.

Okay. But I think it's going to be for some particular functions here. We can make this work and. Okay, sounds good. All right, see you guys soon. Bye.