Live CEOing Ep 780: Design Review of LLMFunctions features

Hello everyone. Welcome to another Wolfram language design review, part of our version 14.1 cycle. And we are looking at LLM functions. So what is the first issue we're looking at here?

First issue is about the raw sort of service execute that users can run. So I have the text there. Basically there isn't a standardized form between all the services, meaning like Dropbox, Google, et cetera. So we would like to have some sort of standardization at least for these LLM based services, since there's quite, you know, a few service providers popping up.

Yep. So what's the. So what's the issue here? I mean, in terms of.

Well, I guess the question is, is it fine to return the whole association that we get from the service or should we have some processing on it where you only get certain bits of information? So far we've been doing it where we do some processing. So I've, you know, the differences highlighted there, one of them is what we call the RAW request and the second one is what we call the cooked request. And so we'd like.

Sorry, I'm totally confused here. Okay, so sorry. There is a service execute with a raw chat and this is with a non raw chat.

Exactly. So I'm trying to minimize my speaking here today. Sorry. Because my voice. But this is something we inherited from the framework. So the framework right now has a bunch of, let's say, internal requests that are basically, you know, you send the API parameters as they are documented. Now the requests that we document, they are like so called cooked or processed. The advantage is that we support stuff like image object association. So we support all our internal language expressions in a nice way.

Yes.

So now I asked around and I couldn't find any consensus about this in historical service connections. So my issue is we all agree that the input of a process request is massaged and is nice. But what about the output? Should we stick with the RAW output from the service or should we do some processing? An extreme case is the anthropic service connection. The anthropic service connection, when you do chat, it gives you just the text.

Okay.

But the service will give you an association like whatever you put in that message with some id, some information about the model, you might have some information about the.

Well, I understand there should be a request you can make that gives the RAW version as from the service, but the default should probably be something that is packaged by us.

Okay, but the thing is, RAW requests were never documented, so we cannot. And I mean this is historically. I mean this is the thing I found When I started doing this stuff. So I wanted to bring up this case just to decide maybe what do we want to do with these LLM services? Because in many cases you will be dropping away information that is important.

Like, okay, listen, listen, listen, look, it's a fast moving area. We're not, it's, you know, having a. I mean, many of these other things are fast moving as well because people have changed. I don't even know whether it's still called Twitter for these purposes. Yeah, that's something. Somebody should look at the URL.

Twitter still works, but I know for how long.

Yeah, I'm sure it will keep working for a while, but I mean, it's kind of weird that we have that as our top service here. I think almost everything in the API has been closed down, so we should. Right. So can somebody who is responsible for this external services page?

Well, but yeah, we can figure it out in the team who should be doing that.

So rearrange this to not put something which basically has had its API killed first.

Yep.

I mean, I believe Reddit still has a, you know, alive and well API. But, you know, if you pay for it.

Yes. Yeah.

Oh really? Okay. I don't know how it works, but.

We can bump up chart model like, or like refresh this a bit.

Yeah, yeah, refresh this page to, to put things. Okay, now my next comment is I think it is useful to people to be able to get the RAW output, but I think that that should not be the default. Okay, so there should be a request you can make in the service execute that says give me the RAW form. The default should give you the cooked form. Does that make sense?

Okay, yeah. I mean, for the way the system works, I think. Okay. Okay, then I have a question for you. The row form is for just the output or the input and the output, because the input of the RAW request is whatever the API is documenting. It can be an arbitrary nested things. It can have like very complicated stuff and is very system dependent. When we do the cook request, we clean it up.

Okay, so look, look, what you've got here is a RAW chat request, right?

Yep.

I think that's a very sensible thing to do. That. That's the RAW version that is based on their API. It may change every week. We're not going to, you know, it will have standardized conversions and it, you know, if you, if you write code that involves raw, the RAW API call. Well, guess what? If the service changes, it won't work anymore. You know, if, if you go through the cooked Call, then there's some chance that we will have adjusted it and made sure that it actually works. Does that make sense? So I think what you have here with a RAW chat request and a chat request makes is.

Okay, so we can document those RAW requests.

Yeah, I think so. Just don't document the output. Just document the fact that it is a RAW request and this is the format that are all requests. But don't document this detail here, because it might change. Yeah, just document it is a RAW request and the structure of the RAW request. Presumably we have central documentation for what this means in terms of API parameters and so on.

Well, no, because this is. This is. This is taken from the service. This stuff was never documented. This is why I'm remembering this.

Listen to what I'm saying. Okay. This turns into. In the end, what is this API? This is a. You know, it's going to turn into some post request or something. Is that right?

Yep.

Okay, so there's a standardized way that this turns into a post request, right?

Yeah, I see what you mean. Yeah.

Okay, so all we document is it's a post request. You give the Wolfram language side of a post request the same as you would give for URL Execute, right? It's the same format as urlexicute, is it not?

Yeah, it's a similar format.

It's not similar. It's the same. The conversion we're doing on whatever is happening here is presumably the same. That we're. We're just calling the same code, are we not?

We are calling the same code, but the way the framework currently works is that these row requests, they list the parameter names explicitly. So you cannot really put something arbitrary and let it fail and let it through. So I think this is why we doing that.

Why are we doing that? Why not just keep our lives simple and just have the RAW request do the same thing that URL Execute does?

This we can do, but requires framework redesign. I was hoping to have something require redesign. Well, because, I mean, I don't write service execute code. I only write parameter.

Okay. Okay, wait a minute. I assume that in this meeting we have whoever actually knows about service execute.

Do we? I know, I know.

Go ahead.

I know about service because I'm. I'm kind of pushing for the redesign, so I'm in the loop. Okay, well, there's a developer working on this.

The RAW request should just do whatever conversions we do and things like URL execute and then send that onto the API. Am I making sense? So we don't document what this Is we say if you want to know how this works, go read the external service API documentation.

Okay, so we, we create a way to have something open, open ended where you can just try and fail.

Yes. Which is the same. It's basically URL execute, but it will have authentication that's specific to the server.

Exactly, exactly. I mean this is, this is nice and we can, we can add this to, to, to the, to the list of like to DOS for the redesign. But then what I really care about is what I put in the output of the, of the process request. Like how much? I mean, because there's some, some minimal sort of post processing that is, listen.

If you have the RAW request, you bring back the RAW results, period, end of story. Don't process the results there because just going to get it wrong because you don't know what's going to come back. In the case of the cooked request. Absolutely, you process the output. What's the question?

Okay, no, because I was acting on the assumption that this new thing that you asked for, I don't know if it can be ready for 14.1. You asked for the RAW request to let everything through and not do any validation and just let it go. Okay, so this, this. I have to see how, how we can.

Whatever what comes back, just don't process it. Yeah, okay, the RAW request, just bring it back in the standard format that we will bring back whatever a JSON result is, you know, the imported JSON result. That's it. That's it. Because we have no control over what comes back. And somebody adds, you know, on Wednesday, somebody adds some special new field to the output from the LLM that tells you whether the LLM is having a good day or not. Let's say we're not going to chase.

Yeah, no, of course. That is never processed. That is never processed. Yes.

Okay, fine, fine. Okay, so now onto this. Is there anything you want to ask about this? This is, this is processed and documented by us, Correct?

Exactly. So for example, this one here is very evident in the input because you could say content could be an image, for example. And we do all the job under the hood to create the.

So I don't understand why has these lowercase names and things like that. Is that because that's their names, right, Exactly.

So the point of this one here is like some services, like anthropic, all the services that we did after OpenAI, they do full processing. So if you do chat, you only get out the text.

Okay. As opposed to what?

Like something like this.

Okay, so I do think we should try to standardize these if we can.

Okay, so we. Okay, so we attempt like some heavy processes.

But you didn't attempt it here. This is pretty raw.

No, this is just basically extracting that from the one above. But I'm.

Okay, but I have total kind of naming conventions.

Exactly.

Okay, so I think that's wrong.

Okay, that's good. I have code to do that for our interpretation layer. So we can expose requests to do that.

Look, I think the default for a thing you call chat is to return something that we would understand where what we get back from Mistral is not that different from what we get back from llama or whatever. Am I making sense?

Yeah. And if you open the relation to LLM services that every put there, this is the middle layer that we created to abstract away all this stuff which is used by Chatbook by LLM functions. And you see this is an internal function. But you see that this is a pretty standard response format.

Why is it an internal function? Why aren't we documenting it?

Well, it's called Chat Object and Chat Evaluate. Okay, so this is just a layer that basically is like a hourglass. All the service connection go here and this one goes to all the system functions like Chatbook, LLM synthetize.

That's exactly how it should be set up.

Okay, so what's the question? No, well, I wanted to show you the output if that is something that you would consider acceptable. Which at the bottom of the page. Yes.

Well, I understand why it says LLM.

Services chat because that's an internal function, but I'm saying like just look at the output. Is that output something in term of massaging that is acceptable for documented stuff?

Documentation of what? Of a chat?

No, no, no. Service Execute chat. You were asking for a conformed output of all the services. So this is a proposal basically which is easy to implement because we already have something like it. Okay. I mean I'm going to use something.

Like this, you know, RAW response, wxf.

I don't know those I'm going to drop because that is what I need internally. But the other things is basically following our format.

Okay. Yeah, I mean it looks okay to me.

Okay, then this is gonna be.

Have we gone through these names? If we're gonna standardize on these names, we need our naming conventions here. Am I making sense?

We can go, we can do this now if you want.

Okay, give me the names. What is it just these or is it a whole bunch of others as well?

I mean, so far it's Just these. That you can add more if you need more fields. But these are the fields that we need. And if you have two requests or two responses, you could have two more that are LLM. Like two requests or two responses.

I'm sorry, I don't understand what you meant.

So if the interaction with the LLM has a two request or a.

Okay, well, so show me what that format is. Show me what we can get. Because.

Well, you just have one.

What's that?

You have one extra field. You have one extra field which is called two responses. And you have a list of LLM two response.

Okay, look, I think we should say here. I think this should be called LLM Role. Okay, what are its possible values?

Well, typically assistant, system or user in our convention.

Okay, okay, fine. What's content? Why isn't that called message?

Because it could be an image or. Exactly.

Well, messages can be. Images can be. Messages can be images, too. Hello, MMs.

Well, that mimics the. You see that the input is role, user, Content. Hello, world. That is also what you would put in Chat Object or LLM Synthesize.

Where did we get those names from? Role and Content. Did we make those up before? Have we already put those in? Okay, what function has those?

LLM synthetic, like chat, Chat Object or LLM synthesize.

Are they in the LLM evaluator?

No, no, no. This is a payload. This is the input that you put.

Okay, so LLM synthesize, you could have a string prompt. I'm sorry, why did that documentation not show up with its. Oh, this is a prototype build.

Yeah, I think this is. This is. No, that's a property. No, I think this is. Yeah, I just recently pushed on CVS so it doesn't have all the updates this can take.

You're proposing that in the LLM synthesize, the prompt here could be this association. Is that correct?

Yeah, more so. If you want. If you want to discuss the prompt, you should open the multimodal thing that we have in this agenda.

Okay, but I mean, that's where this is going. That's how this interacts.

Exactly, exactly, exactly. So this is, what you see here is an expansion of content. So this is the only content. Because LLM Synthesize, it doesn't have a role. The role and content is for chat. I think I made a mistake before. Sorry.

Fine, fine. But if you fed it in here, it would just ignore it, right?

No, no. What do you mean? This is. This is. This is working. Now, if you use a. You see, this is a real output.

I. I know that. I'm saying. Okay, okay, I'm saying you are feeding certain input to the LLM. Okay. Yes, that input has tagged pieces.

Yes.

Those tagged pieces include the. I mean, I have no idea why you have to say type image here. Because it's a freaking image. You can see it's in it.

Yeah, but you see, look down at the short end syntax that also works. You can just provide a list that's just the fully specified version of it. And is our proposal to be future proof? Because we could have multiple types eventually.

Yes. Okay, all right, fine. Okay. What's the difference in content and data?

No, all that you had there, typed image data and type text, data, text. All of that is the value of content. So in the case above, content is just hello world. In this case, content is a list of association with multimodal input.

Okay. So I could put this full association as the first argument of LLM synthesize of CHAT Evaluate.

Sorry, that was my mistake. It was my mistake.

LLM synthesizer. And it would then ignore the role because it doesn't care. It just says, here's the content. This is what I'm synthesizing from. But if I put it into chat evaluate, then it will pay attention to the role.

We could. Right now, we, we. Right now we only accept this type of input that you see here. But we could also accept the full role content input and ignore role.

Okay, but so for, For a chat. Okay, so then this has an implicit content arrow, whatever, wrapped around all of this.

Exactly. Around the whole list.

Yes, I get it. Okay. Seems okay to me.

Okay.

Okay.

Just. Just cute examples. Those next two.

This is the nice loop with the image. Synthesize the one below.

Why don't we just do a nest list? Can we do that?

Yes, we can.

I'd like to see what happens.

I can prepare one for you. It's going to take a while to run. Yes, run. Because it needs to synthesize the image every time.

But.

No, I understand that. That's really cool.

Yeah.

Let me just look at this for a second. Okay. Yeah, looks great. Okay, I'm happy with this.

Okay, so this is using all sorts of things from the From Baba stack that you have to put together so far. You know, like erlm, prompt evaluator.

Very nice.

Okay.

Very nice.

Okay.

Okay, so we need to discuss on this.

Well, no, if you like all of this.

No, I do. I think this is all fine.

Okay, now. Yeah, these are some little changes because now what we do, we want basically like to report a bit Better on tool processing. Because tool processing can take a long amount of time. If you can. If you continuously do back and forth. Yeah, so just saying waiting for the answer is not super informative.

I understand, I understand. Waiting for.

We put a couple of proposals down there.

I understand. I'm thinking for a second.

Okay, yeah, sorry.

I mean, the service would be something like OpenAI. Is that right? Or what would it say?

Yeah, yes.

Is that the whole name of the service or is it like OpenAI Turbo 7.2?

No, no, it's the name of the service connection. So anthropic, Google, Gemini, OpenAI, whatever you use, whatever you have in your element.

Okay, all right. Okay. I don't think counting the number of tool responses is all that useful. I mean, in other words, Do we know the name of the tool?

Yes, I was thinking the same thing. Right now we could report on the name of the tool.

I think that's what we should do. Calling blah, blah blah tool. Now, is that elapsed time, the total elapsed time, or is that elapsed time of that particular tool call?

No, it's a total one.

Okay, so put it on the line below.

Okay.

And we should just say waiting for blah, blah, blah answer, calling with a capital C. Blah, blah tool.

Okay.

Or calling tool colon, blah, blah blah. Because I'm not. I'm concerned that the tool names may be a little bit weird.

Yeah. Okay, so we just sanitize it by putting it at the end in a separate space. Yes, Fine.

Right. Calling tool, colon, whatever it is. And then so elapsed time, blah, blah, blah. And then you'd see various tool calls flash up as they need to be. As they need to be as they're being run.

Exactly. Which can be fast or slow, depending on like if you are asking for some like entity property query or something.

Don'T even bother to put it up. If it's. If it's less than 500 milliseconds, just don't put it up there.

Yeah. And we, we added a stop button. So if this is taking too long, you can. Instead of like aborting, you can stop and see the conversation. So far.

That seems very good to me.

Which is especially useful.

The conversation or not?

Well, if you have chat object. Yes. Or in LLM synthesizer, you can take the string and put it back and continue from there. But in chat object, you can directly take the whole chat and continue.

Well, I think that will be less and less true that you can put it back and continue. Because I think there's going to be more and more state information.

So far you can. Yeah, but still, I think it's useful to. To kind of loop out without losing what you have so far.

Absolutely. Looks good to me.

Okay. LM tool. Okay, so we, we always had our custom tool calling, which is what is called textual tool calling. Textual tool calling means that we do prompt engineering and we explain verbally to the model that it can call tools if you write special tokens that we interpret. Now, models are starting to have tool support, so we added in a new parameter to a learn configuration, which is this tool method that can be textual or service.

Where is this being listed?

If you scroll up. No, stay up. The first example. LM evaluator. Tool method service or tool method textual. If you want to decide on the names, now is a good time.

What does service mean?

Service means that we use a service parameter for tool calling. We format the tool requests as they want, and we get the tool response from them without, you know, doing any prompting.

I'm confused. How does it know. How does the LLM know that this tool even exists?

Well, you see that in the LLM configuration, I have tools goes to Tools. That's if you open the closed section at the beginning. Tools. That's the definition of the tools variable. You see, I define three tools. Thermometer, altimeter, and date.

Okay, fine, fine.

Okay, so this is what the LLM know.

Okay, and you're saying that the LLM intrinsically knows how to call these without us having to give something in the beginning of the prompt for that specific LLM query.

Exactly. Some of them do have specific API parameters that you can use to pass this information along.

So when it says tool, method, arrow, textual, I don't think, I mean, doesn't that mean that in every. I'm a little confused. That means that you have to add to the prompt, to the. To whatever prompt you say, like, hello, tell me about phishing. You add tool calling information to that initial prompt that you're giving. Is that correct?

We do it automatically. Of course, you don't have to do anything. And we also do the parsing automatically. So if the LLM answers with tag to request to name, we parse that. So in the end, for example, in the chat object, you will always see, and this is a new in the conversation LLM2 request, LLM2 response. Then you can ask for the full text and see the nitty gritty interaction, but we do all the parsing for you. So now the thing is, the textual method will work with any model that is Able to understand it, but let's say potentially any model. The service method only works for services that have this enabled. But it's interesting because it allows you to save prompt space because you use a separate channel.

Sure, sure, sure. No, I understand. I want calling it tool method or tool calling method or something tools.

I mean, I'm open to suggestion for the name. This is just something that is there to make it work.

Well, it seems okay. I think it's okay. Anybody else got comments on this? No, not for me. Is it really service or is it built in? What, what is. What should it be?

I'm just afraid the built in seems that it's our built in, which is not.

All right. Well, I think this is okay. I mean, this is a moving target, so this may yet have to be changed. But not. I mean, we can continue.

Well, I'm happy to say that a bunch of design that we did internally, for example, that, you know, that content thing with data type and these tool stuff with the LEM2 response LEM2 requests. It's basically more or less what APIs out there are using. So it wasn't so stupid after all.

Well, right. You know, conceivably they looked at the design. We already did. But you know, I don't think this is an independent system.

No, no, but I'm saying that it could also be that is it like multiple people converge to the same smart solution.

So I'm saying that convergence evolution always can happen.

Exactly, exactly. So I'm saying that it should be a moving target.

We're happy with what's going on there?

Yes. Okay. Okay, let's keep scrolling then. Okay, so this is the new type setting for tool calling that you already look at in the design meeting.

Okay.

Okay, so let's, let's go down. Like I pasted this on top because this was an old build, so this didn't have it. So yeah, don't look at this. This is a working example, but with the old design. Let's keep going.

Okay. Why do I need to look at this?

No, no, no, no, this is fine. I mean, everybody will show you something interesting that you ask for the current date and we have a date tool, but the LLM just doesn't call it and makes up the date. You know, this is what can happen.

I love it. Fact and fiction have finally merged.

Yep. Well, one good point is that you can, with our system, you can actually easily check if the tool was made or not. Yeah, because we, we always report with the LLM tool requests.

Yes, it's nice if they, if you.

See no request and no request has been made.

Yeah, okay. Okay, so these are the services that we have currently.

Yeah, yeah, we are like before wrapping all them up, I wanted to clarify a bit that point about how to document the connections. But yeah, we can have all of these lined up for 14.1.

Okay, good. I mean I, this is a different discussion that has to do with partnerships and so on, not for this meeting exactly which ones we're supporting.

Okay.

I've never heard of some of these anyway. Who knows? I've heard of many of them, but not all of them. Okay, what's this?

We floated this idea. So basically the thing is LLM Evaluator and LLM Configuration are fine for like chat LLM synthesized chat book, but not for stuff like semantic search or image synthesize where you have different tasks. So AI configuration is a superset overlang configuration. We floated the design at one meeting with you. This is a bit, a more polished proposal for what we could do.

And the point of this is, I mean, because that the reason we're doing this is because many services have our Swiss army knife type things is that.

Yeah, well, I mean, let's say that you, you do semantic search, you could use one service or even a no service or like local version for text embedding and a different service for text synthesis to get the final answer, you know, like RAG style or maybe in a function with even more task, you could use yet another service for image generation.

So basically the point that you're making is we've got functions like LLM Synthesize or Chat Evaluate and we want those to under the hood, they're using some whole collection of different services. And the point is to be able to have one central place. So you know, I gotta say that the System Services, you know, that's sort of the idea of the System Services project is, you know, oh, you've got, you know, satellite image tiles, you've got, you know, URL shorteners, you've got all kinds of other, you know, systems that you're going to call. And in principle, you know, we will have defaults for many things, but in principle you might want to use a different satellite image provide or something. And the only difference here is that we think that this is still a, you know, we don't have enough of these. Completely standardized that we just have, oh, use our defaults most of the time. It's still a sufficiently moving target that you might want to, you know, change these around. Yes, but the Fact is that the correct solution to this, I think, is System Services, which is a general way of saying, okay, for, you know, for my function repository, I want to have this local function repository as well. For this, you know, satellite, you know, for this map provider, I want to have this blah, blah, blah. Am I making sense?

In full generality, we could also support functions there, you know, like audio embedding could be run. This function I have defined that runs something on my computer.

Okay, but. But you understand what the System Services process.

Yes, yes.

And I mean, to me that, that seems very similar to this, because by the time we're defining, you know, seven different things you might do here, you know, I'm sure for System Services, we have things which define, you know, we should have at least things that define kind of, you know, what default voice to use for this and that and the other. All those kinds of things.

So if you open the UX for reference while designing this, we thought a bit about how this could be interact with the pref. So this could be something like this that you have a. I understand it's.

Very similar to System Services.

Okay. And we could have, you know, but this is a close thing. But if, you know, if you pick OpenAI and OpenAI, it has models for everything. You will just stay with that.

Yeah.

Then you can open and begin. Okay.

This is exactly what we're talking about for System Services too.

Okay.

Okay, so where is this? Does anybody know? Does Avery. Were you involved with the System Services project? Is that Bradley's project?

I'm not involved, so I assume Bradley.

Okay, well, that's what I recommend having a discussion about that. I don't know what the state of that is right now. You know, if we can explore that. I, I think, I think that's what we should be doing because I think that the, the number of these services is going to just multiply. You know, it's going to be, oh, what do we do for the, you know, what, what are we doing for the ar? You know, you know, our AR published thing, it's got something too, and that's then going to be linked in to some, you know, neural radiance field version of the, of the. Whatever it is, you know, it just, it's just going to get more and more of these things and we have a framework for this.

Okay.

We don't want to build another one. What's that?

Good. Good to know. Good to know.

Right. So, I mean, but we need, we.

Need also something programmatic. Yeah, of course. To use in place of LLM Evaluator for these functions, we need an option.

I don't understand what are you talking about?

What if you scroll up? Yeah, like where I have AI Evaluator. So this is what we floated that you could say, AI Evaluator goes to AI Configuration and have an association with a bunch of things. So this is a generalization of LLM Evaluator and LLM Configuration.

I understand that. I understand, I understand. But I mean, the question is, I mean, this is exactly what we were thinking through for system services.

Great.

Because I mean, it's the same thing. Let me give you an example. Okay. If I use Geo Image, for example, or geographics, there is a thing, there is an option there. Geoserver. Okay.

Yes, I remember.

Yeah, right. And all of this stuff is, you know, deeply related to that. I don't think we figured out whether there are separate options and different functions to call into the system services, how that exactly works. Okay, but that's what we want to do in general.

Okay, so you're saying let's hold on this general design instead of introducing a new thing and directly hooking into system services.

That's what I think we should do. But I think that, you know, if we can get a design meeting about system services soon, because this is now a driver for really getting them, that would be a good thing.

Okay.

All right, Anything else we cover here?

Can you scroll up? I just, I'm afraid you might have missed something just by going up and down, but maybe not. No. Okay, perfect. Everything done.

I think this is good. I, I, I'm, you know, this is exciting stuff and. Okay, I hope that we will be able to start, you know, we, we plan to start publishing our benchmark results based on all of this for, for these various different services. So hopefully that will be a thing that will be coming soon. Yeah, the. All right, thanks all very much and thank you. See you soon. Thanks, bye.