Live CEOing Ep 807: Design Review for VectorDatabase, SemanticSearch & LLMPromptGenerator

Okay. Hi, everyone. Welcome to another Wolfram language design review. This one is about vector databases, semantic search and rags. So let's take a look. Create vector database. So what is that ID thing?

Well, this is something I'm still working on. The thing is, we want these things to be ideally, by default out of core. This stuff can grow really big. So I'm working with Carlo to. Basically, as we discussed in the previous tabular meeting, we talk a bit about generated asset allocation. So, like I'm learning fast how to use local objects to store files to store these indices.

Okay, but why aren't we doing this the same way we've done a lot of other out of core type things? Where in the notebook interface it says, it's a blob that says underneath, you know, not stored in notebook, blah, blah, blah. Do you see what I'm saying?

I see what you're saying, and I'm a bit conflicted about that. I mean, text search doesn't do that. Yeah, like search, index, object.

You always imagine that would always be out of core.

Yeah, that has a file backing it. Yeah, right. Like video. That doesn't do that because it's always out of core.

No, no, no. Video has a way of doing that. Video has a way.

Look, can you bring in core video?

Well, let's do audio. Hello, computer. That's in the, in the file. But now this there says store in notebook. Storing cloud public. Store in cloud private. See what I'm saying?

Yeah. We can have something like this. And I guess the store in auto will be disabled. Like when you have a very long array, if the size is too big.

Sure. I mean, you can do it if you want to. It's not a very good idea.

No, I think it's actually disabled if you create a very large. I mean, I've seen the UX element for that.

But anyway, yeah, okay, but I mean this storing cloud public is clearly useful. And presumably if you say cloud deploy of this thing, that's what will happen. Well, no, maybe not. Because you know what you're going to be able to do is if I say cloud deploy of this, and then I feed the resulting cloud object into a vector database object of cloud object, it should give the same result.

Yeah, I mean, this thing can have like one or multiple files, depending if you store just the vectors or the metadata. And I guess they can all be brought in memory if you want.

And what I'm saying is if I cloud deploy this thing, I don't know if I can do that, but can I Do that.

I mean, right now that thing is in memory, so you can cloud deploy it.

Yes, it is in memory.

Yeah, we have both implementations. I'm working on this training classification because I think the primary implementation will be out of core. But of course we mean for prototyping and also it's generally good to have any memory version, so it's not something that I cannot implement.

Okay, but if I say this, it's not going to work. Well, that failed for maybe different reasons, but no.

Yeah, because your build probably doesn't have it, but yeah, did that. Yeah, yeah, it failed by loading a library that is not there yet in the layout.

But yeah, okay, but what I'm saying is this thing should generate a cloud object and then I should be able to say vector database object of that cloud object, and that should be equivalent to this. See what I'm saying?

Okay, so would that imply that vector database object or the vector database object is like a nop?

It's not a nop, it is ingesting it. Setting up the plumbing to use a cloud object as its backing store.

No. Yeah, but you deployed an expression which is a vector database object. So you know, if we generalize what you. Did you say you do vector database object or that cloud object, that cloud object is equivalent to what you cloud deploy, which is a vector database object. Yeah, so I'm saying if you are nesting, well, then you have.

Then you have to copy it. Cloud deploy something which came from the cloud, then you've got to copy it.

No, no, I'm saying different things. I'm saying formula wise, if you do vector database object of that cloud object is equivalent of doing vector database object of the original vector database object. So you have. So I mean, the syntax you're suggesting is basically nest like that. You know, you can. You can give a database object as a database. Yeah, okay, you got it. Yeah.

Okay. But the main point here is that whatever you're going to default to and create vector database. How do you default. How do you specify the default of where this is going to be set up? That's a persistent location thing, isn't it?

Yeah. This is something that I want to discuss in this meeting, actually, because I looked around. Persistent location is something for symbols and we don't have any option about that because it's the second argument of persistence symbol.

And we have a use for persistence location.

That'S also not an option. Yeah, that gives you locations as object. Okay, so the closest thing I found to what we're doing here is video function have this generated asset location option and if you look at that documentation page that has. You can take several things like a directory, a file, a cloud object. No, I don't have it here because it's not a new one. It's called the generated asset location. Yeah. Okay, you can click that.

That's from video, right? That's, that's.

Yes.

Well, I guess I would reuse that until we have some better way to do it.

Can you click on that so that we maybe look at. What is it? That's a nice table with. Okay, here. This is file, local object, cloud, object, directory. And there's none to store in memory when possible. So this is covering. I mean, the reason I picked this is because it's basically covering the cases that we care about.

I think it's okay for now. We might, but go ahead.

Yeah, yeah. I kind of agree that it's probably not the best. But is the better? Is it the best so far? Let's say so. So I'm refactoring currently the object to basically be. I mean, taking the idea from like service connection or the index created by text search, where the object is basically, unless you have it in memory, the object is just a pointer. The ad is used to recover it from a list of local objects. So in that case it's going to be very lightweight.

Okay, so when I create the vector database, I could have done that with a bunch of things in it.

Yes, you can do. Yeah. And you can say. Yeah, exactly.

Now what happens if I just do this? Okay, well, insisting that it's in a list of lists.

We discussed this. I didn't add conveniences around it. For example, this could be a list of length one if we want.

I agree. I mean, what we want to do in general, there's a feature extractor that we need to be using here, isn't there?

Yes, we could have a feature extraction option. We have a problem here because we prototype like a couple of months ago this AI, I think it's called feature extractor, Extractor function is ahead.

Okay. Yes.

So we prototype this AI configuration that then we dropped. So one thing, I mean this could be in principle a function. F. Like in feature structure we have like a name structure, a function, but it could also be with a stretch, I mean, of notation, like an LLM configuration. I mean, how do you hook into. A very common use case would be.

I don't know what the right way. I don't know how do you get an LLM based feature extraction? I think you just say, I Mean it's using NET model effectively. I mean, you can just say feature extractor. Our own. NET model of BERT or something.

Yeah, if you use NET model, that would work because it resolves. It's basically the case with the function.

Right, but so here.

But I don't know. Do we want feature extractor here? I was imagining that this thing will go into semantic. Into the create semantic session index.

I don't see why it shouldn't. If this works here, we're going to need it here.

But the point is that, okay, so in my head, this thing here deals with vectors. Semantic search is the thing. And create semantic search index is the thing that goes from like text, for example, or other things.

Fair point.

Vectors.

Okay, so there's a vector database. How do I look something up in that database?

If you keep scrolling down, you see, you have like, for example, I was using an empty one, so I have add two vector database.

Yeah, I understand. And how do I extract something from the vector database?

Keep going. There's vector database search db you see, I put like 1, 2, 3, all properties. First five. And this is very similar. We copied the idea from nearest. You have an element, an index and a distance. Metadata is empty because you haven't used it yet.

This is not something nearest can do to make a data set like this.

No, we can return it as a disposition in case. But typically we already designed several functions where if you use all, you get the data set. If you give an explicit list of properties, you get dispositions. What are you trying to do there?

I'm trying to get a vector database search. Okay, so this is going to give me the closest thing to that. Oh, gosh.

Yeah, sorry. I have a bunch of normal to put in. Internally we are using optimized version, but yeah, okay. I think this is already fixed in master, but yeah, okay, that's nice.

So what happens if I want to get the three nearest things to that comma?

3.

Okay, Nice. I'm a little bit confused why it came out in that order. 013. Yeah, sorry.

Well, you, you, you randomize that. Yeah, it's not the same as before.

I understand, I understand, I understand.

Okay. Okay. Yeah, sorry. Oh, you're comparing with yours. Yeah, yeah.

Okay.

We have like prop is an optional second argument. So if you don't do property, you can use N and otherwise you can use both N and property. Property N N.

Optional third argument. It works like nearest. Yeah.

No, nearest doesn't have the property. That's something not when you apply has when you create it. You see, it has the things data element arrow property. So it's on creation. So if you create a nearest function, the nearest function has a fixed property that it returns.

Okay.

I don't love this design because it forces you to decide what to return at creation time.

Yeah.

Which is not so relevant for vector layer bases. It's relevant for nearest. Where the creation of the nearest object happens near to its use.

Yeah.

Okay. All right, so add to vector database. I get it. Vector database objects. Fine. And the distance function. So these are all. Are those. Those are not strings, are they? Yes, they are.

Yeah. I mean, these are properties that you can ask for. I mean, the distance function property will return what you put in the distance function option.

Yeah. Okay. Metadata associated with each vector.

Now there's the bit of design here for metadata. I'm not sure this is the way we want to give metadata. I'm playing with some syntaxes. So basically Here you have 100 vectors and you have an association with it could be multiple properties. Now I only give one, which is like one piece of metadata for each vector. This is useful because you can do joint search. Yeah. So if you scroll down, actually two examples of database search, one is a normal one.

We've got some examples of this. I mean, classify and many other functions have examples of where it's like all the places, all the values, type thing or.

Yeah, I mean, we support both syntaxes. You could say prop goes to a list of 100 elements. Or you could have a list of 100 association. Prop one goes to something. I mean, it's easy to disambiguate.

It's basically a transposed of what you might expect for that association there.

Sorry, I got distracted for a second. Could you say this again?

So that's saying for every vector, you. You basically transpose them with that two times range 100. And that's how they get associated there. As opposed to instead of having a list of a hundred associations with everyone having Prop 1.

Exactly. Right. I mean, but this happens in visualization functions. It happens in classify. There are lots of places where it's kind of like, do you have the whole list arrow, another whole list, or do you have pairs? And I don't know the correct conclusion here, but let's take a look here.

We typically would like to support both if possible, but the way of. Also this one here, you could try it in different ways because you could say random real 103 what I had there. And then you could say that arrows. A list of hundred associations.

Yes, that's What?

And then you have a third way, which is to merge them and have a. You know, you have a list of 100 elements, which is a vector, arrow and association.

That's also the thing you just said. I don't think this is really right. I think that that should be instead, You know, table. Blah, blah, blah. Two. I.

Well, in the practical case, the one above is more useful.

You say that. I don't really believe you.

It's more efficient, even though it's a little bit weirder.

I sort of. I don't. Yeah, maybe. I mean, it depends on where the data is coming from. It depends on, you know, your particular case there with it's a single range. Maybe fairly maybe. I just don't believe that's a very general case. And it's inconsistent with our other syntax, so it's just not going to work. Am I making sense? Packing. Yes.

Classify supports both orientations. Yeah. I mean, we can document one. No, no, no. If you think it's better.

Does Classify support the thing that you've written?

The classify doesn't have metadata, but you can, you can, you can do. If you open the classify page, you see that you can have a list of associations or an association of lists.

Yeah, I know that, but that's not the same thing as what we're doing here.

Is the first table, the syntax for input.

You know, there's too much of these here with very arbitrary, weird stuff.

You want a lighter image?

Yes. I just think this shouldn't be such a big thing relative to all of these things. It's just wrong. Okay. The fonts are too big. It's just wrong.

Okay. We can make it smaller.

Yeah, but I mean, that's something that needs to be done generally in documentation. You don't want these things where these numbers are meaningless. I mean, we go to a lot of trouble to make the things we say here be meaningful. We don't want a bunch of meaningless crud just sprayed into the whole thing. Am I making sense?

Yeah, it can become like more.

Can you communicate that? And. And I think we need to communicate that to the whole documentation team and so on, because I really don't want to see this. This is not right.

I mean, I can talk to. Talk to Roger and then we can have a pass with Tim and prove this.

Yep. It's. Had to get done without sending explicit message about it. Okay. What am I supposed to see? These things here.

Okay, so this is. Shows the threaded and untreated syntax. Yeah. In, out, in, out or in, in, out, out. Yep, the one. There's one though.

That's it. That's the syntax. And. And that's what you should do as well. And don't mess around with some crazy, crazy syntax here.

Okay?

It's either. It's either one way or it's the transpose with a big arrow in the middle. Not. Not some subsidiary case.

The transpose. Sorry, what is the transpose? When the transpose of the second one that you have.

No, I'm saying it's. It's this or it's lots of pairs. Right. Or it's the. The vector. I don't understand. You're telling me the vector database is just vectors, so have it be just vectors. So I'm saying, except this case, you can. I mean. Oh, no, I see what you're saying you could do. Yes. You could do things like vec. Arrow, meta.

Exactly. So I thought that you meant that by the transpose.

Yes, that's what I meant and I agree with that. Okay, okay, okay.

All right, let me take a look. So here is, how do you access those metadata in search? So given that we have a narrow syntax for creating it, we played with an arrow syntax for actually searching. So if you see the second example of back to database search.

I see the example.

Okay.

You know, what are the other cases like in optimization functions and things like that, where you have multiple criteria? Do you see what I'm saying? And by the way, vector database search, you're always searching for one element at a time. True or not true? Yes, it's true. Right. You don't have a list of things that you're searching for. Yes or no? Yes or no?

Well, you can, you can. But then I think it's not very different from mapping. There's some efficiency in vectorizing the search, but the syntax also gets more complicated and the output type gets more complicated because you're going to have a matrix of results like the first five result for three vectors.

Yeah. Okay, well, I mean, so this is what we're immediately doing. Okay, let's take a look here. All right, so I don't think this is obviously crazy.

To us, your previous questions. What we are doing is like use. Typically you have. You have a list and then. Or use. And.

Yeah, I know, but even if we don't typically have a function, we typically have an expression.

Exactly. But that was before association. Way before association.

Yeah, I know.

And we are designing something similar for tabula. You know, you just telling people to get used to think in terms of columns.

Yes. Okay. Semantic search index why is there a comma light there? That's the name of the search index.

But yeah, it's the same as Create one.

Okay. What's it doing and why isn't there a status Right.

Right now is using our semantic. He's probably setting up.

We don't want to see this. No, but we don't want to see it.

This is going to be replaced by a. NET model. This is semantic Bert. So sorry. Sentence. Sentence Bert extraction. It's something from the Wolfram function repository that I have it for this demo. This is going to be a. NET model. So it's going to download a NET model?

Yeah. Okay.

Yeah. No, it was not so slow on my side when I did the first time. I did like a month ago. You never use external evaluate.

Maybe not with this build particular set of dependencies.

I mean, it's interesting to see how it is from scratch.

Yeah. Not very pleasant, but interesting. Okay. When will this be done? The replacement of this?

Well, I'm checking the model these days. It's going to be a quick thing to. We don't even need it to be documented in the repository. We can have it up immediately.

What is all this junk? I don't know. Why Space down there. That's a bug.

This. I don't know. I have to ask Ricardo.

Deal with this extra space. That's just a bug. Am I making.

Yes.

Who's here?

Is it? I'll tell Ricardo.

My guess is the line wrap and it's leaving space for all the crud that's in the line wrap, but it isn't actually showing it. So it's either a line wrap or it's got to not have the space.

Yeah, no, I understand. It's probably because it's dynamic. Yeah, I'm going to check that. I'm surprised it took so much meat. Now this stuff is because it has a lot of dependencies. I'm more surprised by the stuff before. This is actually. I understand what he's doing now.

Okay, well, I mean we don't want any of. We're not going to have any of this stuff, right?

No, no, no, of course not. Yes. It's just because I don't have the. NET model up yet.

Okay. Who's working on that? Is it you or is it somebody else?

Me and the people from. Net Repo. But yeah, it's mainly me for this functionality.

Okay, well, I really want to see this demo, so I. Yeah, yeah.

I mean we can look at the.

Syntax question, by the way, from Ethan on our live stream. Asking for a function that gives text embeddings. I thought feature extract gave text embeddings.

We had the discussion last time, we need to cycle back to that. Then we didn't have time to do that for 14.1. But why? Because what I showed you last time is that the current design for feed restruction is not functions. Is the. Can you open the fetorestruct?

Yeah. What is this doing? What is this? What is this garbage? Well, this is a bunch of random Python crap.

No, I know. I mean I already told you, I don't know why. Why you are printing Python debug messages. This is from external evaluate.

I can't even imagine. I can't even imagine that's what we are saving people from. Okay, so took a while, but it worked. Maybe it didn't work. Okay, now how do I, how do I do something with this? So now I say semantic search.

Exactly.

Percent11 comma.

That was a light page of Wikipedia. So you can check stuff.

Yeah. Why is this so slow? Why is it so slow? What is it doing?

Yeah, because by default this is generating an answer. So if you want to get only the chunks, you can do comma chunks and that would be instantaneous.

Why is an LLM to do this? I thought this was a vector database. What are we doing here that don't go through a freaking LLM. This is a vector database, not an LLM.

No, this is a typical use case for this stuff. We can drop it and we can have a default property to be the actual chunks. So if you do that, the same thing syntax you have and then you say comma chunks. Which by the way, Bob suggested we rename to items. And I'm not really opposed because I don't like chunks. So if you agree, we could call it item. So these are.

What is it showing me?

Yeah, this is actually the result. So you are putting a piece of text there. And these are the most similar pieces of text in.

Why are they in these size of chunks? Why are they in these sizes? What chunkified them?

Because these are word chunks by section to make semantic sense with some padding around. So you see that they all start around a section break.

This looks like kind of a garbagey thing because I mean like.

This works very well. We use it also in text. Summarize it chunks by, you know, in this case you have double new lines and it chunks by that then that's fine.

But it didn't seem to get any useful result here. I'm certain that this page has the word rainbow on it.

Yeah, but this is trying to find. This is matching embeddings. It's not matching words. So if you say something which is semantically related to a sentence, it should be able to find it. Let me check in the page. I don't even know if the page has rainbow. Hopefully yes. Okay. It has one mention of rainbow and it says the observation and study of optical phenomena such as rainbows and the aurora borealis offer many clues as an issue of light. This is the only mention of rainbow in the page.

Great. So why didn't it bring it right up? Why is it giving me all this garbage?

Because this is. I mean, this is matching meanings. Not.

But it should work. Rainbow is far enough away from other meanings that embedding vector should be. Should be something reasonable. Okay, I'm not impressed at all here. I mean, if Rick finds this useful, great. But this doesn't seem like an end user thing.

Well, there's one difference that if you want something that you can run on your laptop, it has to be as smaller models. If we do embeddings using.

We already talked about this. We build the embeddings and then we're using them. We're doing the vector database lookup on. We're going to do the embeddings in the fanciest possible way. But the vector database lookup is local, right?

Yeah.

Okay. I mean, so you're saying that the default should be something using like LLM functions.

Yeah. Yeah. Well, I don't know the default. I don't know. For this case, you would still need.

To generate the vector that you're gonna.

I see.

Search using the same method.

Well then that maybe isn't a winner. I don't know. What are we gonna do for. Well, what you are doing for the LLM interaction functionality. Obviously there's an LLM in the picture because the purpose is to interact with an LLM. Correct?

Yeah, yeah, that's. That's.

I'm just confirming from. From Rick. That's right.

Yeah, yeah, yeah.

But this is not what you were talking. What you was talking about because this interaction is just a compilation synthesis.

Yeah, I think usually like the stuff that semantic search would give in its results isn't very useful to humans. It's typically something you would want to pass to an LLM. Like it's going to be something kind of ugly.

Why is this so ugly? Talking of ugly. Well, I guess the double new line doesn't help.

Yeah, I mean, I mean I. Yeah, that was put there manually.

Prompt Generator supposed to do anything here?

Yeah, I don't know why this was merged like two days ago. It should let me see my version. I mean if you undo, you see what it was doing. It's simply taking. Why it scrolled up so much because.

The undo went to a different place.

Okay. So basically this is a function that resolves to a prompt that then is added as system prompt before the rest.

Yeah, I got it.

Yeah. So in this case we are just simply prompting the model. Always with that. Yeah. So you can ask what day is today? And you get a decent answer instead of like, I don't know, I'm an lamb, stuff like that or a random thing. So with something that you know, just with this simple thing, you can do something interesting. This is something that every came up with. This is pairing up this day prompting with the WolframAlpha as a resource tool.

That's nice. Okay. Okay.

And the one below is combining semantic search with the prompt generator, which is a typical use case. So in this case here we are using semantic search to provide context. So this is the page of light of Wikipedia. And you see I say connection between light and Aphrodite. And the one above is with the generator and the one below is without. So the, the one below is what the LLM knows on its own is using the source.

Yeah, that's nice. Well, LM evaluator is nothing. What does that then do? It uses.

No, no, it's a default. He's. He's just due to emphasize that there's no prompts there. You could also drop the option entirely.

Well, this is great. Okay, so the points to make here. There's plumbing that you need to finish getting actually working. And let's look at some the design issues that remain. I hadn't really internalized how useless semantic search is. It has a name that makes it sound really cool.

Well, it really depends on your vectors. Like I want to expand that function to put in like stuff like images and other things, like everything that can turn into vectors.

Semantic search with images were useful.

Exactly. I mean, I think that the main issue here is how you generate the vectors. So like single word search, like for a small embedding model, single words is not going to be a winner because.

It'S how you chunk it to generate the vectors. Basically.

It'S two things that how many chunks you use, not just how small they are, but how many you put in the context. And also the quality of the vector embedding using different models is going to affect this dramatically. So you Ask why it is not picking rainbow. Because this is trying to make some sense out of sentences. And a small model probably doesn't have enough reach, like semantic, especially if you want smaller vectors, if you want super long vectors and you have a super advanced model, it might encode finer details. So this is why it's really depending on that.

What's the difference between semantic search index.

And.

Vector database really?

Well, semantic search index it basically you can do that with only vector database, but you need to create a lot of extra glue code because you need to chunk the text, you need to embed the text and then you can put it in the vector database together with the chunks because you also need to store how you chunk the text.

So this is something I just linked. It's like a more complete example. So this has like 350,000 vectors, something like that, and it's pointing to pieces of documentation. So if you wanted to get a sense of what this can do, like little tiny examples I think is kind of hard to really show how strong it is.

Why is it so slow?

It. Because it's loading all the data from disk each time. So once you had it in session, it's actually extremely fast. It's just because I'm doing it through a form page.

Okay.

It's interesting. How do I make a list of primes together of numbers together with the factorization? Not bad.

And the link should actually take you to the actual snippet once the page loads too.

That's a little weird. Okay, okay. All right.

Yeah.

Harry on that live stream is commenting that the semantic search might seem pointless without an LLM, but there are applications like paper analysis where there's a direct link to the search. And this is another example here. So why is this? I mean, we're generating all of these crazy things. What should we have been generating? What, what, what. What things should we put here that gives us like the position or something like that rather than the actual text? What's the analog of this here? How do we get something where we have named chunks there?

Well, the problem is that those are anonymous. I could give you. We store the source. In this case, the source is a string. If you put a file, we save the file name or a link, we save the URL. But in this case you put a naked string. So we just have to say that it's a naked string and we can give you the equivalent of that if you put in how it's called string. Take.

Yeah.

Hold on.

The fundamental thing you put in a vector database is A bunch of vectors. Right. So you're making vectors from chunks of some kind, from items of some kind. Right?

Yes, sometimes it. Hold on. So what Rick has done, like the image case.

Right. So what Rick did over there is he made chunks of these sections of documentation pages. Right. Okay, so in terms of this, it will be create semantic search index of a list, an association with the name of the chunk and what's in the chunk. Is that right? Because that's not what any of this junk is. Right? That's not what any of that is. Well, this doesn't seem right to me. I mean, what, what. How did you create this, Rick? Because I think what you should be doing here.

I just linked a notebook that I think could show kind of what. How it works. So this is the. That underlying function, but I'm asking for snippets. These are the chunks that correspond to each of the vectors. So I generated these from documentation, you know, in markdown, and then generated embeddings from those. And basically I have my vector database, which is just the vectors, and from that I get the indices.

Yeah, I know. I don't think it's the right way to do it. I think what we should be doing is the following. I think what we should be doing is here, we should be saying there. It should be a list of the form, you know, text, stuff, you know, stuff to index. Arrow, tag. See what I'm saying? Arrow tag.

Yeah.

And in fact, so this, I guess I need to clarify a bit more. These chunks of text, those are some. Those correspond to some of the vectors in my data set. Most of them are actually generated from text that I synthesized with an LLM using these chunks as input and some additional prompting. Okay, so there's. In the end, need a one relationship there. I'm sorry, so for. There can be many vectors that.

For a single tag.

Yeah, for a single tag, yeah.

Okay. But I think the default way of setting this up should be create semantic search index from something like this. And same with vector Database. Create Vector database. I mean, okay, this is a possible alternative syntax, but the generic syntax should surely be, you know, just.

Well, vector goes to tag is similar to what we have. It's just. I think we even support it. Instead of an association of properties, a single property is a naked property.

I think one of the common use cases of this would be, you know, those tags would be files like file of some string or whatever, and you're indexing a bunch of stuff on disk. But the stuff to index is something that you might have asked an LLM to talk about your file. I think tag is just some arbitrary expression that you.

Yeah, sure.

Right.

But that's the thing that you'll get returned when you do semantic search, right?

Yeah.

The thing you should get by default is that right hand side tag, unless there were no right hand side tags, in which case presumably you should get the stuff to index as by. By default.

Yeah, that makes sense.

Yeah.

Okay. And if what you were given in the Create semantic search index was a pure. Was this pure thing? Well, I think that then this is what it should do. You shouldn't need comma chunks here. This is what it should return by default.

Yeah, I think we said. That's on me. I think we already said the other time that chunks should be the default property.

Okay, fine, okay, but then. Okay, all right, so then to create something like this, you would be. I mean, a very obvious thing would be to create something that is a bunch of URLs. If you feed it in, if you feed create semanticsearch, index the URL, I assume it will do some kind of import.

Yeah, it's importing the page as plain text, not as HTML. Then to comment on that, I'm chunking it.

What we want it to do when it generates chunks from that page is to generate them with tags that correspond to, you know, a search string on the page or however you go to. To scroll into the HTML page. Do you see what I'm saying?

Yeah, yeah. Different depending on the file format.

But seems, yeah, it's not super obvious. This is something that they want to iterate on because like, we are definitely tagging them with a page address or the file name. There's a bunch of tagging that is done automatically where it makes sense. But for example, if it's a book, you might want to tag the. Do you have a title of a chapter or something like that? That should go in. But I would say it's like the version two of that.

Oh yeah. Really what we probably want is a chunking function has nothing to do with any of these functions. That's a completely separate function. Because being able to chunk a piece of text is already useful, right? We already have, you know, snippet, for example, that gives you, you know, a little snippet from the front of the text type thing. But in the end, I mean, you know, we already have text sentences, for example. Am I making sense? So there's an argument that we should have something called like text chunks. I don't know exactly which should be in which Some property setting should give tagged text chunks which could, for example, be tagged by their string positions.

Well, then we need to expose more things because we ideally, you know, how long is the chunk dependent.

I know, I know, I know, I know.

On a tokenizer, for example. So we would like to expose tokenizer function.

What like?

Well, these chunks are not decided by string length, they are decided by token length. So one thing we could do is also expose some way like a text tokenize of such, which could be using both like a tokenizer, like GPT tokenizer or do text words mean that's also a form of tokenization?

I think those are both reasonable things. We're not going to get them in for this version. I think for this version we should concentrate on the things that we know what to do with them. Like lists of things with tags.

Yeah, externally.

And we can maybe make some WFR functions that munge text to put it in the right form.

Yeah. So one thing before we have to wrap. So here Rick has snippets. Bob was suggesting to call these items. This also plays well with the max items option because you say give me a rainbow spectrum of colors, max item 3, 4, 10, automatic.

That's fine. We can call them items, but they're going to be the default. So we're not going to name them most of the time.

Okay, no, no, fine. Yeah, but I need them for other. You could call the. You have properties, like give me the item embeddings extractive actor. So I mean, I need the name to get related properties.

Okay, let's come back to Ethan's question about text embed. What is the story with that? If you're going to give embeddings of these chunks, then we've got embeddings somewhere. So how do we get. Go ahead.

I mean right now we use a. NET model or the things you can do. The documented way to do this in the language right now is to use service execute. If you want use an API. But we can have. No, we discussed this last week. We could have an embed text function. We have text summarize. We could have text embedding.

Fair enough. I think we should. But I mean, I don't understand why it isn't just feature extract, but it happens to call an LLM.

Okay, let's cycle back to that. Can you open the feature extraction page? Let me see if it's a feature extraction or feature extract.

Feature extract. Well, feature extraction is.

Yeah, you see, this is annoying because this is. Feature extraction generates a function, but feature Extraction depends on some input. So this is something that I complained about before I joined ML and I still have a beef with this design. This cannot give you extractors by name. This needs data to create the extractor.

Yeah, that's wrong. That's wrong. I mean, we should have. Look.

Yeah, you could. For example, you could give a bunch of vectors and you say that you want to have standardized vector as an extractor.

I thought we had.

Yeah, this is the same as the other one, is just applying the function directly.

But how do you get. We had a thing where you give it this kind of stuff and you just apply it to something.

Only NET encoder, for example, the neural network framework has it. Net encoders are doing what you think.

Well, look, it says it's experimental.

I know, and I discussed with Etienne that we both liked that it was experimental, and I suspect that you left experimental for this reason.

Yeah, well, there was also the question of when you change. I mean, it's like random numbers. Okay. You know, I have just unbelievable amounts of code that depend on the fact that random integer is going to give the same sequence for a given seed forever. See what I'm saying? And extraction, you know, as we've Learned from the LLMs doing their thing, there's a better feature extraction and a worse feature extraction. You see what I'm saying?

Yep. I mean, this stuff in the ML framework is all versioned. So if you create a classifier in version 14 and you import it in version 15, even if, I don't know the audio extractor has been updated, it will still understand that. You know that I remember working on.

That design at length.

Right.

But I think that, you know, this is just wrong. I mean, this. This should just have a way. And it probably should be called something other than, you know, there's probably. You know, I'm sure we can come up with a different name so that we can keep compatibility and so on. I mean, it's more like feature vector of whatever. So that's a separate.

Well, this is not just vector. Yeah, this is pipeline. So, for example, you could have a list of colors and you can call conform colors to make them go to the same color space. That's an extractor. And then you have. I mean, I like more the name that we had internally, which was the ML processor, because this is a processing pipeline. You have confirmation.

We have to wrap because we've got a good meeting. But.

Yeah, I know this is a big design.

The main thing that we needed to go over here. Okay, so you've got plumbing to finish here. Then, you know, then. Rick, maybe you can comment on this. I assume we're going to make. Where are we going to put the vector databases? We're going to have a separate vector database repository. And if so, has that started to be built?

It hasn't started yet. We've just briefly talked about the possibility of doing that. Whether or not that's a new repository or if it can be integrated into the LM tool repository, we're not sure yet.

Oh, I see. To put it in the LLM tool repository, but it's not really necessary. I mean, like your. Okay, in this system here, are you using an LLM in the middle of this? You are, because you have to generate the feature vector that's coming into the.

Yeah, so just, just for getting the embedding vector, when I use this in other places, I also use a FAST LLM to rank and filter the list as well.

Okay. So that's to make an LLM tool repository thing, which. But I mean, my question is. So for example, I really want one of these that is going to. You know, when I'm writing things and I have some kind of deja vu that I've written something conceptually like this before, you know, I want to be able to do that search easily. You see what I'm saying?

Yeah.

So I would like to just have a. An LLM tool. It could be, but someplace which is ultimately going to turn into just a little form function deployed thing that can do that. Lookup.

Yeah. So I've started work on something like this for arbitrary files. I haven't done much with a lot of files yet, but the few examples that I've been trying so far is very promising. So I think something like this could be applied to what you're talking about.

But. But I mean, basically what you're saying is the thing you need is not just a raw vector database. You need a thing with a certain amount of dressing around it or it's not going to work. Yeah, I mean, it will be closer to say it's a semantic search index, which presumably contains. I mean, what does this return when it says. When you say create semantic search index, does that thing there contain the embedding information?

It contains both. It contained the text information and the embedding information.

No, no, no, I know that, but does it. Can I extract from this the function that I need to use to generate the feature vector for an incoming vector, you know, an incoming piece of.

Yes, yes. You can extract the vector database and the Vector database has. I mean we can short the property vector database do vector database and that's going to give you one of the above. Do just database without that tool.

Wow, that's pretty AI ish. Should probably add that in other places as well.

Yeah, I talked to Roger about it. It's very old actually is a message generating function that we had since a long time and now this one here has. Let me see how is the property is called.

But by the way that's a useful thing. It should be done much more broadly with the entity system and so on the. Did you mean this and that?

Yeah, why not? Can you do properties? This should have.

The embedding.

No, it is. It is saved inside. I probably forgot to expose it.

Should be able to expose it so that we have a way to.

Absolutely right.

And I think it should be in the semantic search index. And I think even you know the fact that you do a semantic search by not just subvaluing the semantic search index seems to me it'll. Well, I don't know. You. You want properties of the semantic search index I guess to be the sub values.

We have this in fitted model. I. I think it's very confusing design if we use it both as a object with properties and the function.

No, it's hopeless. But it's not obvious to me that we shouldn't just have information of semantic search index be the thing that gives you these properties and then semanticsearch index sub value be the lookup.

And then all the option of semantic search goes like in. In there.

Like I mean I think but I'm not even sure then we need semantic search. I hadn't really realized how raw what comes out of semantic search actually is. You see what I'm saying?

Yeah, I mean I guess people mostly when they talk about semantic search people mostly. I mean it really depends if it is about images or other things you want the objects but in that case the chunk. The chunking function is identity. So it doesn't really matter. For text a lot of people think about the second. The second step when you do the LLM synthesis on top of it because it's basically a cleanup phase and LMS are very good at digesting prop text and making it a decent sentence. So I think this is the main difference that it's a different DUT type.

Okay, but look, The question is what functions are we going to put in here right now?

Can you scroll up to the top so we have a full list?

Yeah.

And we decide if we want to delete some okay, so are we fine with the vector database?

I think we're fine with the vector.

Function and prompt generator. Okay.

What is semantic search doing? That vector database object isn't doing.

Is managing chunks, the chunking and the metadata. Basically it's automatically managing the metadata compared to the naked database, which is vectors. And you have to do everything manually.

For what Rick is doing right now, he just needs the vector stuff, right?

No, because you're not putting a vector in the query, you're putting some text that needs to be converted to a vector.

Yeah, I don't really believe in this chunking thing because I think that needs to be.

I don't really need it because I've already done the work of the chunking. But if I was starting from scratch, that would be super convenient.

But you think the chunking is right.

As like a basic automatic approach? It seems pretty good to me. I think anyone that wants to make it really good and dive in and understands the data could make it a lot better if they did something manually. But I think having something done automatically is useful.

If you have create semantic search index and it's given a list of files, does it do the right thing?

Yeah, I mean, it chunks each file annotates the chunks with what file is from and then, you know, it's a flattened search. Then you have chunks from everywhere and you want to search, which is just.

Like text search does.

I think so, yeah.

All right. I guess these functions are okay, but look, we really need to get something where, where, you know, there are some WFR functions and so on that have well controlled chunking systems. See what I'm saying? Which probably Rick has been building anyway.

Yeah, like, I have some really good stuff for notebooks that I could publish in wfi.

Is it published there?

No, not yet, but I can, I can do that.

Okay, but I mean then, then for example, if you take a notebook object and you do create semantic search index on that notebook object, what's it going to do?

No, good question. I didn't think about notebook object, but if we, if I had to prevent that, I would assume something very similar to what Nick? I mean, if Rick has the code already, I would just use that because it's basically importing the notebook.

No, I agree.

Create the sections and chunk the sections.

All right, but the important thing here is that with the LLM prompt generator and the stuff, we can do a much better job of tool calling, which is what we really cared about here. And Rick, how close are you? Are there any more impediments in getting all the plumbing working with, you know, out of core vector databases and so on?

Not that I'm aware of. Like, I've. I've tried Julia's version of this with the. The out of core stuff, and it all worked flawlessly from what I gave, the exact same results as my implementations.

Great.

Does the. Does the vector database work in the cloud?

No, but for a very specific reason. The cloud team has whitelist the dynamic library, but. Except for that. Yes. No, no, we met already last week. We just needed the RE jobs running, and they are running this week. So all the code works. They just need to add the library to the whitelist.

So it will work soon is what you're saying?

Yeah. Yeah.

Okay. All right, sounds good. Okay, see y' all soon, I think. And unless there's anything else you need to raise, we're really late. Next meeting. Okay, I gotta go. Live stream. We're about to do another live stream, so talk to you all soon.