Live CEOing Ep 741: Language Design Review of Machine Learning Features for 14.0

Sam?

Yep.

Okay. Hello, everyone. Welcome to our second design review of the day. This one is about machine learning. Okay, let's take a little look here. So we. We did a review fairly recently of text summarized, didn't we?

Yes. So most of what we discussed was implemented like this week.

So I have to look at the.

If you want to look at the documentation page, I think it's going to be. Yeah, it's fine if you look at the page because I modified that with what we discussed last time, Which seems you don't have updated documentation.

Oh, gosh, not again. Okay, can somebody trace this perhaps before the third design review of the day? Anybody here who can chase that, please?

I can ask to get your docs updated.

Yeah, yeah. I don't know why it's not working. Okay, so how are we going to look at this?

If you want, I can share or like whatever you prefer.

Well, you could just update the documentation like you.

Well, I know, but I mean this. Whatever.

Okay, whichever. Right away.

No, no, I'll do that. I'll go ahead and do that. I just thought it was done. Okay, still running. All right, let's get a sketch of what's going on here.

So, okay, so basically we are keeping this release externally is quite slim because we are doing a lot of work under the hood, especially around statistical functions and modeling function so that they can be modernized and work with the. The new tabular stuff. So that is ongoing. At a good point. Yes, yes, this is. Okay, so we can start here.

Okay.

Okay, so changes like there's normal property. The topic specification has been moved to the first argument here. We need to do at some point a final decision. I put synonyms in the like between abstract and description, summary and rewrite and then I implemented the more things that we discussed.

Okay.

Others are coming. So.

I'm just going to get some random piece of text. I get the text version of this.

Most of the stuff came in today, so I don't know how much you have, but if you want to try the older spec, they're still going to work. So like the summary and abstract, they are going to work. So the default one.

We have no summarization other than one done through an external LLM.

Right now. No, we discussed this at the beginning. We don't have like immediate alternative to do that.

Okay, but so we have the service connection stuff coming that's going to Support zillions of LLMs.

This is what I said that that's coming for the past, that's coming for 13.3. We have a clear release plan and we are also going to release.

How many LLMs are we supporting? Many?

Well, three with the new release, we are going to have OpenAI, we're going to have a cloud by Anthropic and we're going to have Palm by Google, which is kind of officially.

I thought we were also getting Llama and Vulcan and all these.

We have support for Hugging Face, so you can get those and we can start registering those. I mean, we are finalizing the registration framework this week. It seems to work. So Rick is switching to that for the chat notebook and I'm gonna switch for which thing we, we basically, we create an abstraction layer that's going to be used by Chat Notebook LM synthetize and so Ricky's already working on porting Chat Notebook to use that. So that thing is moving very fast. And at that point we can register things ourselves and push out packlet updates to add new LLMs. But we can also make very easy for users to, you know, pick their own, register it and have it in the, in the interface so they can select it in chat notebooks and use it.

That's great. Okay, let's talk about this. Okay. We went through a long discussion last time. Neither of these is the right term for this. You can't say no, I mean, we went through this at great length last time. What did we conclude on the notebook?

I had this abstract, but it's not that clear. It's better to use description and same for summary is better to use rewrite.

Well, wait a second.

So I put in the.

Somebody needs to go listen to the recording then. Because we, we discussed this at great length.

Okay, I go through the recording again. Okay, look at the transcript. Yeah, I look at the transcript. It's probably better.

I'm sorry, this is just not right. I mean, okay, you know, we went through this great, great length and that style.

Yeah, I think I, I missed the end of the discussion apparently.

So you better go because this isn't right, you know, this is. Yeah, no, I mean you can't have. One of the things that text summarize does is to produce a summary. I mean.

No, no, yeah, we, the new synonym is called rewrite. I'm gonna drop the first one. I just, I left the old one as a. I don't think it's right.

I'm sorry.

Okay. I had that in the Judah ticket. I, I, I, I Listen, I don't know.

Did somebody here remember better than I do what we concluded? Is that what we concluded really?

And Then we got other feedback from members in the audience from that live stream. We have yet to follow up with that, though. Yeah, I have more in line to be implemented, like the ballot list and stuff. But.

I mean, and I don't think this is particularly if. So come on, let's. Let's avoid the typos here. Summarizes. Generates a summary or, you know. But this says summaries, right? That isn't right.

Yeah, sorry, I should. Yeah, I have to reread this, but this has not been DQA so far.

No, I understand.

And my. My eyes and I don't understand.

I don't understand what the order of things listed here is just alphabetical order.

Yes, no, it is. It was alphabetical order. And. And as soon as I remove up certain summary, I will resort it. So. Okay, those are just as a comparison.

No, this is not a good idea. When it says paraphrase and then it says in the same style paraphrase, I would expect to be in the same style.

Oh, well, I can just fix the description. Like give a short.

Yeah, I don't know what this should be called. Right. I mean, this is. And we've talked about again, the text discusses versus just what it says. You see what I'm saying?

Yep.

How is that reflected here?

Well, description is a text discusses and rewrite is one that keeps the same tone, which in your version is still called summary. So if you want to try it and check the difference, you can say text summarize of the same string with summary and see what it would say. You didn't put the second argument. Yeah.

Which also doesn't have autocomplete.

No, I have a PR in another repo for that.

Okay, Which one is this?

This is.

This will be called rewrite. And he's done the shortening with the same tone. And the one above is what you call description. And it's doing in this text, blah, blah, blah. Or this text does this and that.

Well, that's not terrible. But then for God's sake, write this correctly. Description of, you know.

Yeah.

Description of what the text says. Rewrite of the text in the same style.

Okay.

Okay. And I. I would not put a verbs here. I would put nouns here.

Okay. So. Yeah, key sentences from text. Yeah, keywords. That makes sense. Yeah.

Okay. I'm. All right. I. This is probably. Okay. I mean, we should just check if that was what we concluded last time, then I think we should consider it. Okay.

I can verify that as well later.

Okay.

Okay. The other big thing that is coming out Is what we just discussed the LLM update to custom and arbitrary LLMs for all our LLMTech.

Just talked about that. Yeah, that's great. We just talked about it. Okay, let's move on here to next external object.

Yeah, so this thing here, we are deploying right now, cuda. There was a bit of work on ARRI side because we decoupled the Cuda library from MXnet so you don't have to download it twice for MXnet and for this one here. And we are just testing basically everything is working on the packlet downloads before pushing everything. But CUDA works. And now Matteo is working hard on finishing the core ML so we can have the support for Mac. But yeah, I think. I think we'll be ready.

Okay, so what this means in practice is you can. In the neural net repository, we can have things which are alien creatures, alien Onyxes going around on the. On the Savannah or wherever onyxes live. Is that right?

So we can already. We can already have those in the. In the. NET repository if you want.

So what is this now?

They're faster.

The advantage is that we. You will be able to use target device GPU with NET external object, and that will work on a Mac as well.

I see. That's very interesting. Right.

This is super, super important also for our internal use of networks. Yeah. Because we can use this to have GPU acceleration to like.

So wait a minute, so does this mean the core ML support, does that extend. Yes, to our own. You know, any neural net that we currently have can now have target device, you know, arrow GPU on a Mac and it will work. Is that correct?

If we, if you export it to. I mean, the only caveat is we need to be able to export it to rnx, which we. I think almost all of them. We can.

For example, all of our, I don't know, image identifier and all those kinds of things.

Exactly. We could. We could redeploy them as ORNX files and then. And then import those on the Mac and run them on gpu.

Okay, but I mean, so the plan is to do that for all of our internal models, Is that correct? By internal models, I mean models that are in the product deployed for a variety of purposes.

Yes, yes, exactly, exactly. As soon as I have confirmation that this works, I'll start.

Is every model that we use in the neural net repository?

I think so. And most of them are documented. We have an ongoing effort to document all of them.

Okay.

So something tricky, but that would be immensely helpful given that we have this as a primary deployment route is to add the encoders and decoder support to. Net external object. Because right now this is very naked. It's just direct access to the method. So you need to do the image decoding yourself or whatever it is. So I mean, you can still use the encoders and decoders. We have examples in documentation. You just use them as a function instead of attaching them. But it would be easier if you could on import just specify this port named blah blah blah in this network is an image.

Right.

Well, this will reduce errors and stuff. So yeah, we are also actively working on this and I have a branch open. It's just a bit tricky with the way the framework currently is, but I would like to release this 4.14 as well.

Sounds good to me. Okay, so just so I understand, I mean, if you've built a network in our system and then you export it as onnx.

Yes.

Then this is saying that I could run it natively on. Can I run it on iOS natively through Core ML?

Well, I mean we do have networks exported with CoreML running. We are trying to. For our math detection. We are helping the people doing that app to do formula detection using Core ML. But that is using CoreML directly though we exported the network to Core ML directly. So honestly the answer is that I don't know. I'll have to check because this relies on the ONNX runtime. So the ONX runtime of course works on Mac. I don't know if it works on iOS iOS.

Okay, well, okay, but I mean, so what is the tooling around getting a. An independently running CoreML inference system? I mean, do you just output a file?

We need to. We need to define an export. I haven't looked too much into that yet. If it is priority.

Well, I don't know. I don't know. I mean, I don't know how useful it is. I mean, you know, is the bottom line that there's a dot ONNX file that has. That unfurls itself into a giant thing that can be used to run inference on some, you know, device or how does it work?

Well, I mean, we targeted Windows Linux Max for now. So I have to investigate how this works on mobile devices.

Well, how does it work on. How does it work on a Mac?

For example, we have a On Next runtime right now. It's built with CPU support and the plan here is to build with GPU support. So basically the On X runtime will run it on the neural engine instead of.

Yeah, I got that. I got that. But what I'm asking is if I want to make this standalone. Okay. And I just say I've got this network. Look, you know, let's be concrete. Okay? So if I say NET model, you know, Lynette, for example, okay, here's a net. Here's a model. If ever it comes up. Come on, what's it doing? Okay, now I want to. I want to save that and run it natively on my machine. How do I do that? Do I say export?

You say export.

Or that to O N X onnx.

Exactly. And then. Oh, what is this?

We need a file, right?

Oh, yeah, sorry. Silly. Always forget about the file.

Well, let's just do an export string, just so I see what it is.

Oh, dear.

Well, hopefully it's not too big. It is too big.

And we just crashed.

I think it's ready to put the wrong string.

What's that?

Oh, it crashed. Nice.

It crashed. Yeah. So that's something.

That's going to be a very long string. I would try with export if you don't. If you really want that. But anyway, this will generate in our next file and then you can use your next runtime directly on your. On your computer if you want, and try to send. What is your end game here? What do you want to do?

How do I do that? How do I create a temporary file?

Create file.

Okay.

And you can just put create file in there if you want or whatever. Yeah.

Well, I have to give it a dot onx.

No, no, it is. If you say the comma onnx it will take care of it.

But won't it get the MIME type masked up?

I have no idea. Let's try.

I just crashed.

Did it? Yep. Interesting.

Okay, so this wasn't a successful test.

No, this wasn't. But I mean, I know it works because I spent the week helping Markus straightening out his pipeline with the segmentation models. Yeah, this creates definitely. Most definitely in master creates our next file. And then if you want to use it from our stack, you just import that as next object. And I'm adding direct import using the head. So you can just do NET external object or file. And that will create the NET external object for you.

And why isn't it just NET model a file?

Because NET model is used to download stuff. This is what this is. A file on your computer is not a resource.

I understand that, but it would know. I mean, I don't think there's anything ambiguous about NET model of a file. I mean, what's the Difference between NET External Object and NET model?

Well, NET External object is an alias for the resource system. So it's basically doing resource data on something.

No, you mean net specialized form. Yeah, you asked netmodel, you said NET External Object. Net External object.

Oh, sorry, yeah, sorry. Netmodel is an alias for resource data. NET External is a wrapper for a representation of ONNX model in. In our memory.

Right.

So it doesn't get converted. It doesn't. It doesn't.

I understand, I understand.

And, and the idea is to support other other types of network frames.

So it's some random binary gunk.

Yes.

What is a binary array that has.

A, you know, a runtime binary gunk. Runtime.

It's a managed library expression. Right, to be precise.

Okay, yeah, okay, fine. Okay, but then, but if we saw a file which was a ONNX file inside netmodel, it would not be ambiguous.

No, no, no, no. We can distribute ONNX files through. We already tried that. You can even distribute NET External object directly as a serialized expression. We also tried that. So you can basically do netmodel foo and you get out a NET External object once foo is probably uploaded to our cloud.

Yeah. Okay, wait, what's foo? Foo? Foo is a Onyx file, right?

Yeah. I mean, for example, image identify is called wolfram image identify. Net v2. So if we do, I don't know, Wolfram segmentation net v1, and that wraps, you know, is an alias for a NET External object instead of NET chain or NET graph.

I understand.

It will still work the same.

Okay, next question. On these different devices, random, you know, iOS device, Linux device, whatever, you know, if I feed in a dot ONNX file, like for example, with standard MIME type openers and things, if I, you know, put a GIF file on. On a random device, it's going to open with a GIF viewer, right? Yeah, but now with an ONNX file, I want it to open. I want it to become an inference engine that takes certain input and processes it. But how does it know where to get the input from? Other words, what is the standalone driver on? You know, I've got this file. It is a neural net. How do I put a harness around it so that it works on a random.

So like within a notebook? Within the Wolf?

No, no, no, no, no. Not within our technology stack. What exists?

Okay, well, you could. You use the ONNX runtime directly. You install a Onyx runtime and then you say whatever they use. I actually don't know the details about how to use it. I.

But that, that's great. But it doesn't put the harness around it particularly.

No, no. You will have to do some console command, like, I don't know, like load this file and pass this input. Right.

So. So, okay, so. But that's not something we. I mean, like for Arduino, for example, we provide a harness, do we not? When you do a microcontroller programming thing, we. You know, I could have an Arduino that actually controls my thermostat or something. Yes, because it has enough harness around it to do that. So I'm just trying to understand if I want.

You kind of need to say on which pin or whatever, as it's called there. Right?

The. The.

Yeah. In the Arduino case. Yes. Right. So the question is, where's the data coming in to this thing? It's running. You know, there's a. There's a disembodied neural brain running on the iOS device and something is feeding data into it and then it's running its inference and it's blurting out the output to somewhere.

We haven't. I think that, that's really useful to think about. Sort of. That to me is like a deployment model. Yeah, Like, I think we need lots of them for, for stuff that we develop here.

Well, I, I know but I mean, we have standalone look, for example, you know, we have this whole framework which is almost arriving for standalone deployment of Autumn Engines. Okay. I'm just trying to understand, you know, like, like for example.

You know, the.

Wolfram script wrapper is an example of something which provides a command line wrapper for our technology stack.

Right.

So one could imagine a similar command line wrapper for, you know, an Onyx inference engine. Does that make sense?

Well, I don't know. Command line? Well, maybe.

Well, I mean it's just one, one, one mode or. Or it's a library that plugs. I mean, some other thing.

Yeah. So I don't have a super good idea for how one would do it on iOS, what that would be.

Yeah, it has to be some ide. I mean it has to be some. Something that you write code for from an ide.

I assume Swift seems to be a good thing that it should hook up to.

Yeah, right. But I have no idea. But go ahead.

But I think that there, there are other such similar modes. Like how can you export it to be an API on AWS or a service on aws.

Yeah, I know. Right. I mean, just like we, we've obviously we did a great job many, many years ago with you Know with our API mechanisms and so on. Right.

But I mean these are sort of, you know, this is mobile and cloud, which are two sort of nearby modalities in a way or environments.

So are you saying that we should offer like some way to wrap this procedure?

Basically, let's provide a workflow that describes how to do it even if we do nothing else.

Right.

Okay.

We did do for iOS many years ago. We wrote a blog but I think how to do it on iOS but I think things have improved and we could probably do it better.

Fair enough.

What did he do? He did some Mushroom classification.

That blog.

Post and then ran it on his iOS device developed in Mathematica.

Massive documentation updates. What did you change?

Open Predict New and compare it with Predict. I also have a Classify new, but.

I like Classify better.

Well, Predict New has more examples, but the notes are the same, they are both updated.

So.

Why is it going to two lines on your machine? I tweak the space all the time. Okay. I have to make it even smaller. Of course. That should be on one line. Input output.

I don't understand what line for what.

This you see, but you see in. No, you see in feature. Feature, feature. And then no, just below you have class. Class, class. You know, those should be on the same line. So we basically we rewrote all the notes, created images. Created images for every method and started documenting all basically type of input that you can give in all the examples. So predict is all pushed Classify. I'm reviewing some, but there was. Yeah, no, we started spending a lot of time on documentation because we noticed we were some like seriously lacking in terms of everything we could do and.

Well, it's important.

Well yeah, because people didn't know that we could do some stuff to turn. Start reading bugs and support tickets. So.

Right, good.

That's a minor thing. We added progress support. We add in progress support and basically every version we are taking some function, adding, progress reporting.

Where do I look for Dendrogram Tree?

You already look at that. We skipped one release because I'm waiting for some updates in the tree framework to come in because I did custom stuff and we decided it would be better not to have custom stuff here. So you see all these new edge functions. I just got an update today from Ian that he is actively working on that and he's going to update me again in a couple of days.

Okay, fine.

So this is a big thing that we start. We just started touching the surface of all our statistical modeling functions that have been sitting there for a long time. So what we did is, I mean I rewrote the whole property framework so that we don't have crazy dependency messes and we can actually extending and make it possible to add new properties on the fly. Also we know this could be done with the function repository or whatever and we have started to bring them in the modern world like different input types, support for uncertainty in the arguments, support for quantities.

That's very important.

There's a lot of stuff that they don't know about right now.

Yeah.

And this is going to be ongoing. You know, the biggest change is going to come.

What about the output? Making kind of nice blobs for the output rather than some of the crazy stuff we had before.

I can, I can definitely change. I have, I have a prototype for that. I didn't put it in for 14, but I can definitely change that to a summary box. The. You mean the fitted model?

Yeah, yeah, yeah.

No, no, no problem, no problem. Also, one thing that I wanted to add and I have, and I have a problem on my machine, but just that you're talking about summary boxes. We have this expression called discrete indicator which you know, if we. Is basically a way to represent Boolean values is very useful for.

Never would have guessed from its name.

No, it's, it's. Well, you were.

I do know that name. I know what indicator functions are.

Yeah, yeah, yeah. So it's, it's. It's. It's what machine learning people call a one hot encoding or dummy variables like statistics people call it. So if you have a categorical variables with five values that's going to result in five variables when you fit it. And then the formula will have these discrete indicator functions five times for the different values.

Yes. Which is very similar to our. I mean it's closely related to our bool functions and things like this, isn't it?

Well, that's basically the discrete indicator X comma A is basically bool of X equal. Equal A. Yes, but with the difference that that also holds all the other possible values that X can take.

Yes, it holds.

And when it has a list of possible values.

Exactly.

Thinks can be a symbol, you know, it. It. It knows what the possible values are.

Okay.

And if something is symbolic, it's neither of those values.

So wait a minute. I mean.

Okay, do we have open, open the page for it. It's called discrete indicator. This has been around for a long, long time. Yeah, I'm, I'm not.

It was introduced in seven.

Really?

Yes.

I've never used it.

You don't really use it. This is created for you for models.

Yes. It's when you want to see the formula. If you fit. If you've only used fitting and regression for numerical variables, you've never seen it. If you've had categorical variables, you would have seen it.

Okay, well. Okay, so what's the point?

So my question is, given that we are making these models massively more useful in terms of data types and processing and stuff, this thing will pop out more. Can I make type settings similar to root object for this?

I hate the root object typesetting nowadays.

The new one as well.

Well, what's the new one? Let me look.

I thought it was greatly approved.

We need to lock you in in a room with ITAI so you can. Because you guys.

Oh, God, not this again. Not this again. Come on, guys. I want to get the freaking polynomial out. Get me the polynomial. I click here, I get nothing. That's. No.

You have a meeting with it and he says he's going to do whatever it is you guys are arguing about. And you meet every week in other meetings, and yet you do this.

Make sure that this gets it. Whoever's doing project management, make sure this gets in the next meeting. Make sure this doesn't escape. This needs to be in 14.0. This is wrong. Please fix it. Anybody who able to communicate that. Yes. No. Who's project managing here? What Project manager is here.

I will. Yeah.

Okay. Can you. Did you understand what to do here?

No, I don't.

Yes. No, no, you didn't understand what to do, so.

No, no, no. I will pass it to itai.

Right. We must.

I. I got it.

And. And it should definitely not print the polynomial. For sure. That should be easy to get, but it should definitely not print the polynomial. That's exactly what we're trying to avoid.

I don't mind that.

Yeah, but. But it should be. I agree with you. Absolutely, completely agree. No argument at all.

Right. This is nonsense. I mean, this is.

We have agreed a million times, Stephen.

Okay. I don't know why this isn't done. I just don't know.

I don't know either. You can stay in the same meeting with it. I says it's going to do it and then it's not done. I don't know.

Okay. So sorry for opening this. Pandora's pass. And it will be in 14.

What's that?

We will make sure to pass it to Etai so that it's in 14. Okay, so my question was more about the visual appearance of this, like something which is smaller than that. Has a little frame around it, so it reads, you know, if you see that inside of a formula. Yeah, Sorry, please go.

Yeah. Giulio, I really, the way to have those discussion is to have constructive suggestions instead of trying to get a free mandate to just do something, because I think one needs to iterate. It's like every time we've designed this, we've had to iterate quite a few times before it gets sort of good. Do you know what I mean?

I mean, like here, I don't know whether you want to display the abc. I. You know, there's a. I don't understand this. A.

Well, that's not the. Maybe. Well, so, you know, the first thing is this formula. It's like X. Okay. So when X equals A, it's true. You know, so you can put in.

Yeah, I mean, so I'm going to put an X here and it'll not evaluate.

Right, exactly.

Okay. So now if I say table of this, you know, X abcd.

Yeah, sorry.

Yeah. Okay. Well, I mean, the obvious thing to do here would be, if you think this list is going to be fairly short, would be to have something where you, for example, have this list. I mean, this isn't going to look quite right, but. This. I mean, that looks terrible. But. But, yeah, actually we need to. I mean, this isn't right, but you get what I'm saying.

I think we should come up with some. I don't think that that's going to work, Stephen, because A is going to be many of them. Yeah. And the other thing is that I think that the essential information you need to sort of know is that. So these things are usually called factors in a model.

Yeah.

Okay. So what you want to know is that if this factor is like, male or female, and that's what you want to stand out, you know, this is the male factor. This is the female factor.

I see what you're saying. So you just basically want.

I forgot to put the proposal, actually. I had it ready. I send you an email with an image, which is the render. The render thing.

Okay.

You know, or if you have, you know, like the typical quality scale, like a five, you know, strongly disagree.

Agree.

Those.

Right. But you have the variable here as well, so.

Yeah, because the variable might. It might be called, you know, a city. Equal. Equal. Rome and then the ab. Is that correct? Exactly. Yeah, that's a tooltip. So if you over it, you see, it can be Rome, Chicago or New York.

This looks reasonable to me. I mean, as a first approximation, this looks reasonable, Roger.

Yeah, looks Pretty reasonable. But I still want to iterate on it a few times. It just.

Yeah.

Want to see some more variants?

Okay, I, I would. Also in the tool tip, I don't see a good reason to not give the, you know, the thing as explicitly discrete indicator, blah, blah, blah.

Oh, I see.

I don't see the reason to hide that. I think, I think you might as well because otherwise it's like, what the heck is this?

Okay, I'll do one more iteration with Roger and, and then we can, we can submit for approvals.

Yeah.

Okay.

So then it's going to be exciting when all these statistics functions really are modernized.

Yeah, I think, I think it's going to be. I'm, I'm positive. Next version we put a lot of work with Summer is just that there's so much to do that we think it would be stronger if we have a, you know, a release fully focused on those.

All right, fine.

I think we can innovate on every part of the workflow so the data coming in, how you control it, how we describe it, first of all, so you know what the heck it does and secondly, what comes out. So I think all aspects of the workflow will improve and will fill in.

What about missings?

Yeah, no, it will deal with missings. That's another thing it needs to do that's important.

Yeah. All right. The net repo, we have a lot.

Of network coming out still. So this is the least updated to last month of the new networks. There's some big features here like stable diffusion clip and bleep, which is for image captioning, the image classification using sentences instead of classes. And which of these are we using.

In our internal functions?

So the one that I think, Let me see, what is the unit? I don't think we use many of these honestly. The one that we use that we are writing right now, the documentation because they are uploaded but without being public yet, is the new segmentation models. Yes, Sam, exactly.

For what?

Did you know that, Roger? No, it's a very cool model that basically.

Sam, is there. Well, in the next review that you have in an hour or something, we'll talk. Sam, all along.

Oh yeah, okay. I'm not gonna say I'm gonna spoil it. It's gonna come out in, in the image processing review.

Okay. All right, so I have a question. I mean in, in, you know, something like pose estimation, is there a good reason that we don't try and just make a function for that?

You know, we could just add that to facial features or, or like no, no, sorry, we don't have one for. We don't have one for full body.

But I think why don't we have a function for this? I mean it seems.

What is, what is the output from POS Estimation?

I think exactly.

Key points for everybody. Part.

I mean at this, at our summer school and high school program and so on, several people have used this quite successfully. It was rather straightforward to use and it felt like a thing that was just a component that could be used. You know what I'm saying?

Yeah, no, I'm just, I'm just asking out of curiosity what comes out, you know, like. But maybe it is like facial features. It's a function like that.

It's a function very much like that. Yes. You know, it's. Let's look at it.

Yeah. Basically facial features we built as a collection of. Basically to wrap a collection of machine learning resources around face analysis.

Pose estimation. This one here.

Yeah, So this gives a bunch of things, gives bounding boxes, gives key points by type and also segmented by instance.

So those points that come out there are these kind of labeled like knee.

I assume that they are.

Yeah, yeah, they are sorted.

So.

So here we're just displaying the points as the naked input output. Sorry, the one below is showing you this like the segmented basically key points. So it's not just knee, but is this knee is of the same person as this other knee? Because. That's right, exactly. So this is doing both object detection and on object marked people is doing the segmentation.

Yeah, not just a huge collection of knees.

Exactly, exactly. This is, this is. This is quite difficult.

Yes.

And of course you have some parameters to decide, you know, how much you want to look around to decide if this needs part of the same person as this other knee.

I think we should build this into a system function. I think this is obviously, you know, there's just lots of random applications for this.

I can see this like, you know, even in video processing or something like that.

It would be.

Yeah, this could be cool. Yeah, it could be nice to have this.

I mean, you know, it looks to me like it needs some packaging and so on, but it looks like it's not far away.

Okay. And we. I mean, I think the plumbing is already all in this page. So it's mostly the packaging as you said.

Yeah. So question, Like speech to text at this point, how multilingual are our speech to text systems?

We. Okay, we added a bunch of networks. So at this point is just. It should be just English. But we've Added a bunch of multilingual networks and I know that Carlo was working on those, but I have 14.

Are we going to have multilingual transcription?

I cannot answer to that. Sorry.

I don't think so. But it's possible that we have. Let's bring that up in the next meeting, Stephen, with Carlo.

Okay. Okay. All right. Well, looking good. Anything else we should go over here?

Nope.

Okay.

Well, one thing I can maybe tell you about is that we have a tentative plan for being able to add new connections already in 13.3 that involves basically releasing two function repository functions. One that will make it like. While we are still waiting on the service framework redesign, we can just skip the wait and release function repository item. That will make immensely easier to create a new service connection. And then we would need a sibling function that will take that connection and say, oh, you see, you can use this connection as a valid chat provider and that will basically make it register it for Chat Notebook, Chat Object and lamp synthesized so it can glue itself.

Into the GUI basically. Is that correct?

Yeah, exactly. The idea is that the GUI will basically use our registration mechanism and basically query what has been registered. So if you manually register a new provider in your session, then it will be available through the gui.

You know, one thing that I wonder about all of this know in packlets we have the ability to register things into the system, right? That's what packlets do is they. Yeah, one of the things they do is they stuff pallets in in the right place and so on. Isn't this the same kind of thing?

Maybe, but I think releasing one packlet for each of the. We already have one packlet for each service connection is just a matter of telling the system to use. I mean the service connection is already available.

I'm saying just as a packlet can announce to the front end that here is a new palette, stick it in the palettes menu.

Right.

Why can't a packlet announced to the chat notebook system, here's a new, you know, LLM provider, stick it in the the appropriate menu.

It could. The reason why we have this intermediate layer is that not every service connection supports every parameter and there's a lot of renaming and a lot of things there around that. So basically that is a middle layer that tries to do all the translations necessary.

But I mean in the end the registration. You don't want the user to have to run a registration function. You want.

No, just say package, we are doing this, blah, blah, blah, okay.

Happen. I mean and possibly. Okay, well, everything is pre registered.

Exactly so stuff is. No, no stuff is pre registered. And you know, if we want to add a new system ourselves, we can always release a packlet update, let's say of LLM functions that will register new stuff. This system is useful for two reasons. People might want to experiment with their own. One is released today and you tomorrow you want to play. Or organizations might want to point their custom endpoint on some Azure like AWS, EC2, whatever. In that case we are not involved. But we should give them solutions also can create them. They ask. So, so you want a registration mechanism that is independent of us so that people can actually point to their own private stuff.

I agree, but I'm. I'm just saying somebody, in a perfect world, somebody would make a packlet but connecting to, you know, I don't know what, you know, turtle LLM or something. Right? Made by who knows what, you know, then. And, and that would be a packlet because there needs to be a packlet to specify the connection. Right? But as soon as you packlet install the packlet it would attach use the appropriate hook to attach itself to the GUI for the LLM functions.

Well, that's not a problem because if you, if you know what you know, if you have this registration mechanism, you can put the registration code in your packlet and as soon as the packet is installed the registration code runs. And that's.

Yeah, I understand, but what is the registration code? I'm just saying that we have a way of packaging registration code in the packlet repository system or in the packlet system. I mean there is hard code that you could use to install a palette in the palettes menu but you don't have to use it because that stuff is automatically done. I think I don't completely know how it works. Is automatically done by the packlet system.

The point is that the packlet can, Sorry, a palette can be anything and we are fine in displaying anything. But you know, if you create a service connection that is meant to be used, let's call it as an LLM provider, then you know, we have a format. It needs to understand LLM configuration. It needs to understand our standard message payload.

I don't think it's true that you can have anything for a pallet anyway. I think somebody should have a discussion probably with Connor about the packlet system to just see whether this is something that should be inserted as a thing that gets registered through the packlet system. I might be wrong. It might not be useful.

Okay, we have talked to Conor about this. He's on board with our plan, but maybe we didn't push enough on this thing. We can have another.

It's nice to just have a packlet. As soon as you type packet install, everything just works. But you say you can do that anyway because you can have hard code functions inside it that install it in the hook.

I can create a packlet that does it.

Yeah.

So.

All right.

Okay. This is why I want to create those function repository functions to basically test how that can be automated.

Right. I'm making a small point that might or might not be useful.

Okay.

All right. Sounds good. See you all soon. I'll probably see you in the next meeting.

Right. Okay.

Thanks for now.

Bye. Bye.

Okay, bye.