Live CEOing Ep 789: Design Review of Machine Learning Features

Hello everyone. Welcome to a second design review for today. This is looking at machine Learning for version 14.1. Okay, these changes to LLM synthesize, LLM function and so on. Is there a design change or is it just functionality change?

Well, there's a little bit of design change in a sense because you can also add. If you open the page, for example, for chat evaluate, you will see or align syntax. You'll see an example. It's a very obvious design extension. Instead of just putting text in, you can add images.

Yeah, you put an image.

Right, exactly. So it's already supporting a list. And scroll up. Yeah, there. So the list was already supported it. Now you can add this, I think.

This is quite straightforward into the stream of material sent to the LLM.

Yeah, I mean most models right now support multi part messages. So it's just adding a new element which is tagged as image and that's it.

Okay. Text summarize. Wait a minute.

Yeah, I'm going to add. I mean, the only change here is support for vector databases. I wanted also to look into supporting normal search index object. So one caveat here. And if you bring up the page for text summarize, it's going to be easier to explain. We support a source argument which is called text. And I say in the table, possible format text include. Yeah. So in general, this is the whole text. But when you give it an index, like semantic search index or search index object, I think we should only support this if you give a topic specification which is equivalent to, in this case, equivalent to searching those indices because.

I see. Yeah, okay.

Otherwise it could be huge. You don't really want to summarize the whole database. It would take forever. So I think that in general you typically want to summarize something in the database, which is the advantage of having the direct hit here.

Right. I mean, it would be useful perhaps to be able to say, what is this whole thing that I've got. I've got a search index object. Tell me about it. You know, it's a question of how one, how one handles that. But it seems like, oh, this is a search index that has roughly this and this and this. It's kind of like a linguistify problem for.

I was thinking about linguistify specifically. I discussed that with Shadi today. And if we have linguistify for files, it might work as well for the objects that have a direct representation of those files, like audio. We were thinking about audio and image. But this is also something that could be done here.

Right. I would think that that was the place to go. But initially having search, index, object, arrow topics seems like a reasonable place to start with text summarize.

Okay.

Semantic search is completely new and that's a vector database play.

Yes. If you scroll down, there's a whole section about this actually that thing that was wrong copy pasting on my side.

Okay. All right.

But yeah, so just to wrap up on this, so here for image synthesizers that I would like to do, but I'm a bit unsure about how to handle this because we had this potential idea of AI Evaluator.

Yeah, but then you brought up LLM Evaluator. But I think we can. I think we can call it LLM Evaluator. Even though, I mean, for goodness sake, you know, people are using Vision Transformers and so on in image generation. It's. I understand that it's not strictly an lm, so to speak, but I think it's become common usage.

Okay, okay. The only thing that I was a bit unsure about is like the idea that we floated about AI Evaluator was to like be a superset of LLM Evaluator and contain specific tasks. So that thing is also like needing design because like something like semantic search, we will need like two configurations, one for embedding and one for synthesis. So yeah, that's something that we.

Evaluator gets to have an association of lots of little pieces.

Exactly. But last time you suggested not to use the evaluator and to use something like system services. But that doesn't seem to be a thing.

What do you mean it doesn't seem to be a thing? I mean, we're not going to build some giant structure here that is about to get built somewhere else.

Okay, so I can just wait on that and then who can that when it's ready.

Okay. Alarm, synthesize, submit. Is that used by our.

Yeah, yeah. Those are where like ideas that like floated up for this version but didn't make it. I just want to make sure that these are like in the plan. This is not ready yet. I mean, this thing came from interacting with Theo is basically like a synchronous version overlap synthesize.

Right. I just don't know what back because.

I mean, like, what's that a task? This is the classic submit thing. And the kind of a challenging thing here is to support tools which are, you know, inner tasks. So this is a nested task thing and we have a prototype for this, but is a challenging design. We want to do it right. So we are pushing it to this. But this can be super useful.

What on Earth is this.

Which one?

Knowledge graph. I don't know what anybody means by that. They always mean different things and sometimes what they mean is incredibly thin.

Yeah, this is something that was brought to me by multiple people. And Joffrey has a prototype of this. It's basically kind of extracting the knowledge from an LLM and connecting things and building a graph out of that. But show me an example. Exactly. This is like for next version, not this version.

Okay, let's talk about neural net back end and then let's talk about vector database.

Okay, if you scroll down. So the main change if you open the page target device new that summarizes basically.

The change which page Target device new.

Target device new.

Yes, it's new. Is it in his new World Cup?

It's the same. The case is sensitive on Mac.

But it's capital N lowercase E. It doesn't matter.

It is like this, but on Mac it doesn't matter. Or on any case insensitive file system is going to be the same. Okay, so here we are proposing a completely new structure. We used to have only CPU and gpu. Now what we are proposing here is. Is to update GPU to have a broader meaning. So gpu. And Matteo was actually suggesting to actually rename gpu. I mean still support it, but rename it to something like accelerated, which I don't like. But I put here because I understand the reasoning behind. Which is something broader that basically looks at what accelerators you have available on your machine.

Meaning that's a neural engine.

Exactly. So depending on your system ID you would.

But it's just we haven't. I mean the broader name for all these things are accelerators GPUs, neural engines.

My question is this target device. Presumably the default is target devices are automatic. I can't see why you wouldn't want to use.

Because in general there's much less support automatically on those than on like you're not. You don't have. You're relying on some library supporting all the operations which is not always the case. Like CPU will always run those things.

I understand but. But I mean if they can run run them. If they cannot run. Notice that they cannot run and run them on a cpu.

I. The goal would be to have automatic. We are working very hard to. To. To make sure that that's not becoming slower for testing things than actually.

Zoo of what actually happens underneath.

Right now it's not too big. We are thinking of adding basically two things like CUDA is simply explicitly stating what we always supported as GPU and CoreML and DirectML are new and they enable to use basically Apple hardware, which was, I mean since Apple ditched Nvidia, we didn't support accelerators on Apple. And DirectML, which is based on DirectX. And that means that you can also use non CUDA compatible GPUs on Windows. There's in principle a fourth one which is OneDNN, which is the one from Intel. A bit unsure because from initial tests is not so performant. So if it doesn't go too better.

Than cpu, the people at intel about it, what they think we should do. Maybe you can initiate that through partnerships.

Okay, good point. So we added a first layer of automation so you don't have to say core, ML or directML. I mean you can if you want to target it, but you can say just GPU and it will figure out what to do depending on your platform and what you have installed. Like if you have, you know, Cuda Enable cards or if you don't, it will go to different paths. So in, in the current setting, target device GPU is a very portable setting that would work on any device.

Look, target device arrow accelerator doesn't make a lot of sense. Wrong part of speech.

Yeah, probably, yeah, it would be accelerator if we add that. But I sort of, I think that that's the future. I'm not sure if it's the right name yet.

I think GPU is a brand and people kind of say, am I going to use GPU or not?

Yeah, okay, I see a point. Like possibly right now, Steam. Yes, but you know, a neural engine is not a gpu. But I don't know how many people know the difference.

Yeah, right.

Because There are technically three Intel SoCs.

Intel SoCs are also getting the equivalent of a neural engine. Yeah, and there could be other things. I mean, the thing is that's probably going to be a little bit more complicated before it gets simpler. Like instead of cuda, it could be rock, which is radon. Radon, you know, language.

Yeah, but I think, I mean, look, the way to deal with it being very complicated is either a bunch of string names or some structured association of these things. I don't think we should try to clean it because as Roger points out, it isn't very clean. The distinctions are not clean. The, you know, the, the. Oh, I have this extra GPU and I also have, you know, onboard acceleration of some kind. How do I combine all that stuff, blah, blah, blah. Gets very complicated.

Well, I think that I like this proposal here because basically we educated People, that GPU is the accelerator and that's still going to work and it's simply going to do more. And those explicit names are just, you know, if you come here and you want to look exactly what. What's been used. But I imagine most people just use GPU and it will just work on more systems than before.

Well, the only issue is somebody who says, I've got a Mac, it doesn't. I didn't buy a GPU for it. And yet it's still going to work, which is why automatic is really important.

Yeah, yeah. I mean, we.

I'm with you. We want the automatic.

I think we should document automatic here, which is, I mean, okay, how do I find out if I have a GPU or what we consider a gpu? Is there a dollar system capabilities or something which gives me that information? Can I even find it in if I go to. I mean, here, look, this is kind of a scary thing that it probably isn't here. Right. If I go to system, Can I find out if. What kind of gpu ism I have?

I think you can. If you go to front end, it will list it under front end. Yeah, because we use it for graphics. That's what GPU stands for.

Yeah. Okay, well, where do I have. Did have.

I'm sure we had it. Maybe it's.

All right, so listen, listen. Let's figure out how we get the information about whether you have a GPU in System Information or elsewhere.

Well, I'm sure we have another way because Matteo has hooked us. I need to talk to Matteo tomorrow morning, and I'm going to figure out what he's using in neural networks.

But I'm going to make the point. It should be in system information. There should be a way to query it. And then we should use that to determine what automatic does in this case. And we should document automatic here.

Okay.

Okay.

And so let's say automatic by default. If you have some accelerator available, it's going to use it.

Yep.

Otherwise it's going to go back on cpu.

Yeah, exactly. Okay, so now another comment. In NET Train, I would suggest that the progress monitor gives some indication about whether it's running on GPU or not or what it's running on. See what I'm saying?

Okay, yes.

It doesn't right now because the whole point is that's kind of a feedback loop. If it's like, oh, my gosh, this is so slow. And you see, oh, it's running on cpu. Well, that's not right.

It says that. It says that right now.

Okay, fine.

It says CPU or GPU in the method. Okay. I don't know if it could be made more visible probably, but it says that.

Okay, well, we could unroll the GPU one and what it actually is of running on.

I think that will happen automatically, probably with the mapping, but I can make sure. Yes. Okay.

All right.

Okay, so basically, just, just to clarify, this new support is gonna happen for net external object, of course, because that's using directly ONNX models, but also for layer, net chain and graph evaluation.

Although you're saying that that won't happen instantly.

Actually, it kind of will.

Okay, Julia will explain.

Well, so right now, all our layers, they exist in our framework, so when we need to evaluate them, we instantiate an evaluator of the chosen backend. So instead of instantiating something with MXnet or other backends, we are going to instantiate ONNX evaluator. So this is happening already right now. And in general you have a cost. The cost is on the first evaluation of the network. The first evaluation. That means that if you train, of course you're going to evaluate many times. So you don't see that for inference. It's always been the case that the very first evaluation is creating the evaluator, so it's a bit slower. And then the subsequent evaluations, they don't need to recreate the evaluator, so they are very fast.

Do we give a progress monitor that indicates that?

I think that we should typically not where we can, because it was never kind of above like 2, 3 seconds. We should make sure that we should.

Do it even for two seconds, because people, okay, look, I've seen this and I'm like, oh my gosh, it's so slow, I can't even do it again.

Okay, I didn't know this point.

Even I didn't know this point. Now another question is, can we freeze dry the thing that it is creating? What is it exactly?

Creating something in memory, like it's a memory, like it's a blob.

Why can't we not just read it if it's going to be the same blob? For what class of systems or machines is it the same blob? So, for example, we had this issue with a compiler that, you know, we thought we were going to have to create different versions with different machines. Then we found out there was a slightly more portable way we could do it, blah, blah, blah. But my question is, can we cache those blobs? Particularly if it's coming from the, from the Neural net repository. Why not just cache those blobs for different kinds of machines and just read them in. Do you see what I'm saying?

Yeah, I'm thinking, I'm thinking that the size is going to be a different magnitude.

That's what I'm going to. That. Is the size much bigger than the specification of the neural net or is it kind of. This is the bitmap of the neural net. It's potentially smaller.

No, it seems to be the same.

Okay, well, okay, then, you know, if once we. I mean, the neural net repository is capable of having the plumbing to know you're running on this very specific kind of machine. Right. So it could download just the thing for that very specific kind of machine. See what I'm saying?

Sorry, I jumped. What did you say? You, you download the. The optimized binaries basically for the machine.

Yes, for that particular machine.

Okay. We can need to think about that. But we can do that before we can start with the more popular network and see.

Right. Then we can just completely get rid of that latency.

Yes, yes.

Okay.

I mean, networks backings, like system functions are the first network candidates for this, I think.

Yes. Okay. Gosh, there's quite a bit here. Okay, okay.

It's less than you think. Vector databases is the biggest part here.

Okay, so we already went through this design. Is there anything that doesn't work about this design?

I just want to make sure that you like all the names which we didn't finish. I think that those, those things for databases. We were pretty happy. Roger was suggesting to be a bit more defensive in the design and call vector database. Vector database object.

I think it's a good idea. I think it's a good idea.

So vector databases at this point becomes vector database objects, I assume.

Yes.

The one that gives the list to them.

Yes, yes, yes, yes, yes, yes, yes. This is good.

Oh, sorry, let's misspell.

What do you mean create a new vector database? That's not the right description of semantic search.

No, that's. Sorry, sorry. I got a notebook quickly to send it to you. This is probably not the right one. Semantic search is searching through a vector database. Semantic search index is the empty one. And then create semantic search index is the one that creates that. This is copied over from Create search index. I think this is what you suggested for the design. One thing that was floated up in one of the team meetings is that semantic search index could have a name that reflects the concept of embedding, like semantic embedding index, Embedding database Or something like that. Because, I mean, the real difference between, between this one and the vector database is that this is managing the whole embedding thing is having an extra layer that is doing things for you, like chunking.

I mean, my question is, if I want to know what's the embedding vector for this piece of text, how do I get to that? It's with feature extract, right?

Yes. We could also add a function. I have an internal version of that, which, which is text embed. If you want to do that. If you want to do like text in vector out.

Why isn't that feature extract?

That could be done in feature extractor. It should be feature extraction.

Yeah, yeah, it should be.

We could add a new property there.

Yes. Why is it a new property?

Because. Well, can you pull up the documentation for feature extraction? I think you already have something on text.

Yeah, we do.

So it should be like a different one, ideally.

Okay, so this works by taking a set of examples. Is that what you're talking about here as well? You have a standardized.

This is why I'm a bit uneasy about this because this thing here is built to create like an extractor, like to train a feature extractor. Sort of is like classify, you know, you can get the extracted vectors as a side effect.

Okay, the main purpose, mfcc, for example. Yes, I know. This was not what I thought it should be. I mean, you, you know, I was told it was always irrelevant. So here you are actually getting the vectors out. Exactly right.

Exactly. Feature extract. Yeah, sorry. Even myself, even I get confused about this. Feature extract is performing the extraction. Yes.

Okay, well, I thought that there were some named feature extractions that weren't based on examples where it was just.

No, this is something. Some design that I proposed at one point. And we need to circle back to that because we, we don't have a nice way to access naked extractors without examples.

Well, that's just silly.

I know. I mean, I agree.

Maybe we should have a function called embedding vector of. Or. Or. I don't like just the word embedding. On its own, it's something. Embedding, embedding. I mean, given that we. We bit the bullet and said vector database, maybe we should call it embedding vector. And then embedding vector. Takes as a first argument a named method for doing it. And as a second argument, the, the actual, you know, the data. See what I'm saying?

As long as we can have an operator form.

Yeah, right. Of the. The, you know, sort of embedder and Then of course, the data that you're trying to get that for. I think this makes perfect sense. I mean, you know that that's what we're effectively doing in feature space plot when we're. I don't even know how it works. I mean, in. In feature space plot has a feature extractor.

Yes, that's calling feature extraction followed by dimensionality reduction. You can, if you want to create it on your own, is feature extraction.

Right, but why are you using feature extraction when, I mean, are you giving it all the explicit examples or are you just using a fixed feature extractor?

No, the feature extractor depends on the examples. If you do space plot of images, it's going to use an image extractor. If you're doing feature space plot or like something else, like vectors, it's maybe just being standardized or something like that.

Okay.

It's basically the same as. The same thing as when you do classify, you know, you can throw anything at it and make it.

Don't. Don't we want the standard extractor? We want it to be compatible with other things? Don't we need the standard extractor? I mean, we're embedder, let's say, for sentences, don't we? You know, our vectors might be totally incompatible with the other vectors.

You need the method. Of course. This is why I was thinking about. I have the backend ready for something like text embed, which will work similar to LLM synthesize where you give an LLM configuration. We can be a local model or can be an API model. It will do the embedding. So that is a clear design. If you want to do something broader, we need to think carefully about this because it needs to be super reproducible and very well versioned and documented.

Sorry, sorry, sorry. We had a whole discussion about feature extraction and different versioning for it and so on, but. So now what are you saying? You're saying embedding vector of.

It's the same beast?

Yes.

And you might want to specify my internal function of text embedding works the same as LLM synthesize. It takes a bunch of text and an LLM evaluator. And the LLM evaluator of course can go to a local model if you have one, and something that it will be possible to call local LLMs. But the basic syntax is that. Because you need to be able to specify what exactly you are doing, because that needs to be mirrored in other applications.

Yeah.

And the point is that once you put the vectors in the database or wherever, then whenever the Vector is just a vector. It doesn't carry any information. So you need to use something like semantic search.

We discussed at great length. And, and when we design feature extraction.

Yes.

I don't completely remember how that came out. I, I want to look ahead on this agenda. I'm sorry, we, we, we need, we obviously need this embedding vector thing. I mean it really does suggest maybe that we should not be generating these naked vectors without any versioning information and that perhaps what we should be doing is having an object that has embedding and versioning information. And if you really want just the vector out of it, you can say normal of it.

That's semantic search index. Yeah, semantic search index is exactly that. Because it contains information about each option will control how you do the embedding, the chunking and the embedding.

Okay. Okay.

So for this application we are covered is more like. If you want like in full generality. And the other one was just want to make sure that it's the right name. Is like a LAMP prompt generator.

What the hell.

The last one on this list, we look at that for rag. This is like.

Oh, yes, yes, right.

I have documentation for this. If you split up this some examples. Yeah. So this wraps a function and you add it to prompts. So in this case it's just, you know, this is stupid eco. But just to show what it's doing. The one below, for example, is adding always current date time using date string. And it's prepending that to every prompt. And of course these are just like simple examples. But you can use this to hook up semantic search and do rag. You can hook up whatever you want basically here.

So there could be a list of prompts here. One of them is an LLM prompt generator.

Exactly, exactly. At the beginning, you remember that we named this like rag tool. But then we decided that this is really belongs to prompts.

Yeah, it's a good idea. Okay.

Okay. So if this is approved, I'll just speed this along. So now something like that came out of our internship project. Is this random classification data that it's kind of like. It's minor, but interesting. I think we have data generator to test pipelines in many fields. Like we added random color when we started to do color processing. This thing here is basically generating classification problems or clustering problems, depending on how you see that. So by default you can specify how many examples you want.

And how do I say I want pictures of cats? Everything wants to be a picture of a cat. How do I specify?

Well, this is. This Is like right now, this was the internship project of the first version is numerical. We can start expanding this to handle different data types at one point.

Why don't you put it in the function repository for right now?

Okay.

I'll talk about it again later.

Okay, so then for linear model fit we are working on, the big update is going to be like formula specification that's tied to tabular. But in the meantime, while that is being organized, we basically revamped all the functions. So I think I showed you that in one machine learning meeting. If you open the linear model fit page, for example, you will see that the list of properties is very different now, is much more compact. Basically this stuff was developed before dataset.

Sorry, we won't document, but we will support the old ones.

Yeah, yeah, everything is supported. If you still. If you use it, it will come back. So now you see, if you go to the property list you will see some of them are new. And basically each one of these is replacing like four or five properties of old.

Okay.

So it makes it a lot more manageable. So this is the list of properties that we. The final big one with excise properties is the one that we don't have anymore. And it's basically because every data set was split into data set headings, the columns, rows, content and simply because we didn't have a table like structure to put stuff in.

Okay. This is now using tabular. So.

Right now it's using data set also because sometimes no rectangular. Yeah, but it works. Yeah. And the other update that we did on linear model, nonlinear model is that now it fully supports around.

Yeah, it's a big deal, which is very nice. And it supports units as well, right?

Not yet, but we have it in line. We have data option and quantity. This is the list of things that you want to support around quantity and dates, because this was never added. And around in the end supporting that on the whys, let's say the responses was quite easy. For the independent variables. There's some stuff in the literature. We just picked one implementation which is an iterative method to propagate uncertainties. It seems to work quite well. It's quite fast. That much overhead and I mean in the end we want to make this function modern in the sense that we will have to eat everything that is sensible to eat. And for me this includes around dates and quantities.

Yes, I mean you could imagine other things like geopositions, things like that.

We can have a reach at least in the end when you kind of steer this towards like classify and predict.

You're just doing one dimensional objects right now.

Yes, but, but, but this stuff. I mean, the point is if I don't want to add a feature extractor option for. To this until we cycle back to feature extraction.

No, I agree.

And we stress because, yeah, it's just propagating something that we know we want to redesign. So these wrappers are kind of very obvious and they come about when you do real data. You will have uncertainties and things like that. So it's very nice that we don't fail anyone.

I would really like to get going if there isn't a lot more to talk about. Did we talk about Decision Tree previously?

No, no, but. Sorry. This is, this is stuff is for the new version. This, that XXX new version. This, this is just to. I left. I left the plan here for this. This is just to show you what.

All right. So this is what now.

Exactly. So.

All right. Okay.

We focus mainly on semantic search and LLM stuff for this release and the big neural network update.

Okay. Is there anything else we should make sure to look at?

I think we covered everything.

Sounds good. All right, good work. Okay.

Okay, thank you.

Bye.