Live CEOing Ep 765: Language Design in the Wolfram Language [LLMs, Graph Functions, and More]

Okay. Hi everyone. Welcome to a Wolfram language design review session, part of our incremental language development series. And we seem to have several interesting topics today. So let's start off with LLM configuration.

So we have probably. Yeah, should we take this? So here the idea is that we have successfully packaged both API parameters and top level parameters that we made use of into LLM configuration. So you can see here like for example, specifying the model temperature prompts. Tools, for example, is something that could be either. We have a custom way to call tools, but some APIs have the functions or tools that provide them. So this is quite flexible and functions can ignore the keys. They don't know what to do about complain if it is wrong. So it's kind of working well. Now how do we extend this to functions that are not doing text generation? I would like to use this in image sensitize to specify different model names, different services. So if you go open the extensions here, the. So I put here, this is a low level service provider basically that I can hook ImageNote using this function. And this is implemented with LM configuration. So it does the job. You can immediately use Dall E3 and so this was worked in a branch. I can add this to prototype. You can play with this. It works exactly the same way. There are a few issues here. If you open the design issues, which.

Is actually the topic, this is kind of a mess because this is sort of the parameters being passed to the model that you would like to default in some way from the configuration, but then you want to be able to give them explicitly and image synthesize. Right?

Yeah. I mean if you scroll down to these, basically maybe we identify more here live. But the three main things I have here is like. First one is a slightly annoying naming issue, which is the most minor. Like this is not really an LLM and stuff for Whisper for audio is not an LLM. Not terrible. But yeah, the worst is defaults might not make sense. So if you set $LLM evaluator to something that might not make sense for different tasks. Yeah, because you might want to have a model for audio, a model for images.

So maybe we need a more sophisticated thing. Maybe we need some kind of AI evaluator. I mean, you know, in what sense? Maybe we need an AI evaluator that is a superset of LLM evaluator and there's more like the system services thing where it's an association.

Yeah, I was thinking about an association. Yeah. And then we need to decide if it is by function if it is by.

No, I think it's by function. I think it's going to get complicated because I think that even in, you know, for example, LLM Synthesize can perfectly well should be able to take an image as a prompt, Right? So I mean, I don't know if that works right now. Does it?

Sorry, my connection jumped. Can, can you, can you save the question again?

LLM synthesize take an image.

No, that, that, that doesn't work in top level. We are hooking that now we have added it. I mean, we have support at the service connection level. We need to propagate it to. To top level.

Okay, so basically what we've got is, you know, LLM Synthesize is generating things. I mean, the concept that you're just generating an image, like image synthesize. My guess is that everything in the future is multimodal, Right? So you might generate an image with a caption. So the only question is given. Well, that's the question is what's the target? You know, because it could be the case that, that in the future, LLM Synthesize, you know, we'd feed in a picture of a duck, generate a picture of a duck, and out would come the image of a duck. See what I'm saying? I mean, that's what's going to happen with multimodal LLM synthesized, is it not?

Anyway, with multimodal you can have both in input and output. I mean, so far multimodal means that in input you have images and text.

I know, but that's not going to stay the case.

The thing is, who's gonna. Yeah, that can be that. Or that can also be that. The API is using different models for different tasks. Like they could try to offer that they do manually what we do. That's why they do automatically what we do manually. Now, like, you do one call to an image generation model and then you feedback the result into the multimodal text generator to do captioning or comments about the image.

So the issue right now is this. We have a function image synthesize. Presumably we're going to have a function audio synthesize before too long, right?

Yep, yep.

Why is it LLM synthesize and not text synthesize?

Thanks for the question. It was the original name. It was that. And then we decided to hold on using that name because it might have been something more general in the future. That was the original discussion. So it's a scoped name in a sense.

I see.

And we don't want LLM we don't really want text function, we want LLM function because who knows what that's doing.

Yeah, I mean in hindsight it was good from this point of view because we didn't think about multimodal stuff at the time.

Right. So what functions do we imagine we're going to have? We have image synthesize, We might have video synthesizer. We have right now speech synthesize, Which hopefully will get better. But I think also speech synthesize may be able to have some metadata, like saying, make this be, you know, an angry Italian male, age 35 or something, you know what I'm saying? Whereas right now we have voice styles. But I have to believe that coming soon will be a description of what we want. Am I making sense?

Yeah, yeah. This is also the trend, like for images as well.

Yeah.

Like instead of having like discrete classes, you have like both captions generated and text to find the image like as a fuzzy string. So the trend is definitely to move away from discrete things and just have an LLM part that parts in the string.

I understand. I'm not sure how well that really works because as we make the bridge to the symbolic world, image identify, going to a definite entity that we can then compute with is worthwhile. Am I making sense?

Yeah, it's application based. If you want retrieval, then it's useful to be able to say if you're looking in your gigantic list of pictures and you want to find that picture where you were young with blue trousers on the beach, that's not a class. It's well then good that you have a fancy description. But what you say is a different case is when you want to go on with the computation and do something with the, with specific class.

So you're suggesting that there is an image search, for example, which is the thing you just described, that's yet a different use case.

Yeah, I believe that Google is doing that in Google Photos. I, I don't know because I, I, I deactivate that because I'm afraid of it. But I, I've seen that in news. I don't know if anybody here has an Android phone, but yes.

Okay.

And yeah, is it similar to what is now being discussed with vector databases? I do believe they generate some features.

Okay, so look, what we've got is, okay, the main issue we have is that $LLM evaluator is set to an LLM configuration.

Right, right now, yes.

Okay. So the default, So I think the kind of, the short term solution here is to make LLM evaluator be able to be an association.

Yeah, it's slightly annoying because it is already. You can already. That's a valid syntax. And the thing is that that thing gets wrapped in LLM configuration and this is, this has been added to be a shorthand. And so you have satisfy all this.

Well.

I mean, yeah, we could disambiguate. That's. That's always a possibility.

That's a bad. It's.

Yeah, exactly. All these experimental and work in progress. So if you think that this is really a much better use case, we could just break the design just bringing this.

I don't think so. I don't think it's a better use case. I think it's a hack. I mean, Let's see. I mean we could do something like this. See what I'm saying? And then the thing, we could still have LLM evaluator which I think is useful right now. I mean, you know this, this story has not ended, right. We. We don't know whether people are going to call it. I mean right now LLM is a brand. It's just like at times in the past, you know, AI back in the 1980s, 70s, whatever. Everything we do in Wolm language would be AI today, you know, AI has started to mean stuff done with neural nets. You know, if you had a rule based system, it's just not called AI, not making sense. But that's a, you know, it's an evolution of the language and I don't really know where that's going to land. So.

Also if you look at, at what you propose, that's also tricky because for example, the text embedding, text completion and messaging are all text based things. Do they all map to the same configuration?

Well, I think we could have this, you know, we could have something where it goes into more detail. You know, we can have an association here where it just says text. You know, generic text is blah blah blah and text completion is this and so on.

I think we need to experiment a bit with this. But yeah, this is something that was brought up today and I think this kind of works as a stopgap for sure. And our LLM Services layer gives already a good hint about what these keys should be. Yeah, because each one of those function in LLM Services is collecting different APIs from different service connections. So I mean each one of those, it symbolizes API calls of you.

So we could start with that AI evaluator rules.

I'm not super fond of the name, but I don't have any method right now.

I'm not either. But I mean, I think. I don't know what brand other than quotes AI to use. So. Okay, so then the way this would work is these various functions which can use this could get their AI evaluator rules, which could either be just a pure LLM configuration, which means use this for the default action of this function, or it could be this more elaborate thing here and then there's a $AI evaluator rules.

Yes, exactly. I think that as a structure it makes sense. We add one layer of complexity, maybe we can strip it off before the next release. We start working with this, we see.

How it plays, we leave the LLM evaluator and you know, trumps the evaluate AI evaluator rules if it is provided.

What about AI configuration object like LLM configuration?

Oh gosh.

I just put like a. A sketch in ch.

Which chat?

The Twitch chat.

Yeah.

Okay.

I use synthesis instead. Synthesized.

Not.

Not tied to a specific function.

Yeah, no, no, I think that's right.

And then the automatic at the end would be the fallback if you're not. If you don't specify something, it would just.

Well, but what do you suggest that we have LLM evaluator goes to AI configuration.

No, I'm saying AI evaluator rules goes to AI configuration. This would just be a generalization of what we have in for LLM evaluator and LLM configuration. But we could use the same.

Right, but this is then a second level.

Yes.

What we're proposing here is a configuration object which has another layer. Right. I don't think that extra layer will be terrible. Even in. Even if it was just, you know, a single configuration, knowing that this is a text synthesis. Configuration seems useful. I don't, I don't disagree. This is reasonable. Yeah.

Right.

And this would work for multimodal. You know, you could pull things out that you need for the service. If one service for a multimodal request once some settings for audio and some settings for text, you would have it.

I think it's a reasonable thing to try. Let's talk about the size thing and so on. Right. Size is something generically set. N I don't think should be generically set. That's up to things like I mentioned the size to decide what that value should be.

Oh yeah, yeah. I mean, this is because the LLM services uses LLM configuration as a second parameter. But this is internal stuff. The N is an argument in image synthesize, size is image size option and N is an argument.

Okay, but fine, but I mean, we might want a Default size for a given model. And I don't see any reason why we shouldn't set an image size in quotes here.

Yeah, yeah. In that case. Yeah, in that case it would be image size. Yeah, of course. So it's clear that it's image size because it's confusing otherwise. This also solves a problem that was my last. Issue. Parameters with different meanings in different designs.

But I think you should still say quotes image size. I don't think you should use the image size symbol in the association.

No, I agree. Because not our option. This is. Yeah, yeah. This will be on the same synthetica level as the other things. It shouldn't be.

It shouldn't look different. This seems reasonable to me. I mean, I'm a little bit concerned. I don't know where this bouncing ball is going to land. And I'm also concerned that. And it might be worth looking at some of the stuff in the tutoring project where, you know, Teo has been working on this. This vote of 3 LLMs type thing and how we do that here. See what I'm saying?

I was thinking we want something like a LLM graph or LLM chain, kind of analogous to what we have for the neural nets where you can basically implement like a voting system where the graph is like a tree or it starts out as a tree with three nodes and then you have another node where those connect into that makes the decision and each one, it takes the inputs, builds using a template. Each node would have a template. And then does an LLM synthesize on that?

Right. Because each one is going to have to have its own prompting in general. Yep.

And potentially its own configuration, like different models and things like that. And it should all run asynchronously and, you know, some. There should be a mechanism for cancelling certain nodes under certain conditions.

Well, so it's basically code. It's not clear that we need to. Okay, so what you're saying is the AI configuration could be a function. I mean, I think the design of those pieces that say. You get what I mean? I, I agree. I think it'd be interesting to have some of those functions that says, I mean, what are we doing? It's. Instead of LLM synthesize, it's really a parallel LLM synthesize followed by. In other words, you take three things and then you have to do something to them. You've got a list of three things. I mean, it's. It's code, basically.

Yeah, yeah. You can define it as any arbitrary graph where the output of One can feed into one or more of others and you can build up more complex. Complex.

Look at what we're going to run into here. We're going to run into directed asynclic evaluate, which is basically doing that. I mean, for graphs in general. Let's look at that in connection with this. Okay, Because I think this function may be the function whose functions become LLMs. Am I making sense?

Yes, that does make sense.

So let's look at that in a couple of minutes. Okay?

I have a short anecdote to share that may or may not be relevant, which was that last week at jmm, somebody came to the booth and asked how do I create the graph of iterate? I mean, I'm translating a little bit, but how do I create the graph of iterates of 10x + 2 on z12 without writing any Mathematica code? So I said, we turned it to the chat notebook and we said, create the graph of iterates of 10x2 mod 12. And you know, and it just did it. It was very impressive.

Nice. Well, it's probably because we have some good examples somewhere we need to that it has a chance to read. Okay, What is. Who's. Who's speaking for vertex Stratify? So we have Brad here who authored these, and we are you.

Yeah, I think you've asked for this vertex stratify to be brought back to ild because we keep seeing it in. In all the different projects that we do with directed acyclic graphs.

Okay, so remind me what it is.

What does it do?

It's basically a directed acyclic graph is going to have a minimum path length from its initial vertices to any vertex in the graph. And this vertex stratify just finds. Basically finds those minimum path lengths and writes the vertices down in a list of lists where the first list is vertices on the first level, second vertices on the second.

Wait a minute. The general case of this is the whole foliation thing, right? We've got. This is making foliation. This is making a particular foliation. We've got these other functions. Nick has this function to make, or maybe it's your function at some point to make all possible. Sort of. This is the cosmological rest frame that might be useful to hitai.

The. It's the. The main single output of vertex Stratify is the earliest arrival foliation, I guess you would say. It lists the vertices it. As early as it possibly can.

Yeah, right. But these are all space like hypersurfaces in the sense that these are all simultaneous. Everything in a sub list can be simultaneous. Is that correct?

Yeah, I think so.

It's basically level zero, level one, level two.

And then.

And how does it relate to breadth first search?

Well, the breadth first search tends to go level per level. It goes to the first level searches everything. Second level, searches everything.

So it's ordering. It's doing default ordering is. So I'm now forgetting we've got all these functions which are in wfr, which we did in the connection with combinators for leftmost, outermost. In fact, we've done that also for tree scanning. Right. The leftmost, outermost, et cetera, et cetera, et cetera thing, haven't we?

Yeah, that's not the design we used. But we have tree traversal order that has depth first, breadth first, and leaves first.

Okay, so this is. So what will be the analog. Okay, so we got a tree traversal order.

That.

So. So can I get out of a tree, if I use tree traversal order, can I get out of the tree the list of nodes in the tree that I will visit and the order that I will visit them?

Yeah.

How do I do that?

Do tree level.

Okay, so if I say random tree of like 40 or something, and now I say tree level, that comma, tree traversal order.

Comma, tree traversal order, breadth first.

I really understand that. What am I seeing here? Am I seeing these layers?

Each subtree in breadth first order. If you just want the data, you can say all goes to data.

Quotes, data.

Yep.

I don't understand that. So this isn't.

This looks very similar to vertex stratify. The thing that it would. That it's possibly missing is the extra list structure that divides them into levels.

Yeah. So this is the list of all the vertices at level zero, then all of them at level one, then all of them at level two.

2.

But it's in a flat list.

So just so I understand. So. So I mean, the point is that in a directed acyclic graph, we can have merges, which we cannot have in a tree, Right?

Yes. Which do the same example, but use leaves first. So this is going by negative levels. And negative level is determined by your children. The corresponding thing in a DAG is that your positive level is determined by your parents.

Right.

You could, if you wanted to use vertex stratify to get a similar output, you could just do vertex stratify of the reverse graph and we could even.

Let's just say Vertex stratify.

Yep.

Okay. So let's just do vertex stratify here of the tree graph of that first graph. Right.

Well, make sure that you get directed edges, though.

Well, do we get directed edges from tree graph?

I don't remember.

It should have a directed edge option, just like.

Yeah, I would.

Yeah, it's directed by default.

Okay. Okay, so I'm officially confused here.

Well, it's because the.

Oh, I see. Because the node names, you might.

If you.

If you do vertex replace on the tree graph, you can get. Get it simplified. I think you just want the first of whatever.

Yeah, right, right, right, right, right.

Well, you'd have to map.

Okay.

So now all that we see comparing this vertex stratify output to the other earlier tree level thing is that there's a random disordering per level, which makes makes sense because there's no criteria for vertex stratify being applied to try and straighten out the vertices in any particular sub list. Any. So far, any order is.

Is it.

Okay.

Whereas with a tree, there is a definite ordering.

Brad, does it just put the vertex names in canonical order?

It.

It might. I don't know. I haven't. I haven't given that much thought to it. Okay, so, I mean, that's something we could discuss now if there's some particular order.

Well, I mean, for a tree there is, but not for a graph, right?

No, for a graph there could be too, because the. The graph object itself in its vertex list as a canonical order. So we can try to test that if we just do the. The vertex list of tree graph and see if they're appearing in order. Vertex list is kind of the closest thing to vertex stratify.

Yeah, I get it. Oops.

373. 373.

Oh, yeah. Okay.

You have to map first at that.

I think, first onto this. Yeah.

So now we can check. Okay. It looks like 11 is first and then. Oh, so there 11 is before.

Yeah, so I'm confused with this.

This is probably in like depth first order.

Well, in other words, you've put a con. You've put an ordering on the vertices when you create the tree graph. And the question is, what ordering did you put? And you might have put a depth first ordering, which might or may not be right.

No, I don't think it would be depth first, but it could be discovery first. Like this algorithm is going to go. I'd have to look.

Whose algorithm? I'm not talking about your algorithm. I'm talking about. This is Ian's algorithm here. Yeah, for.

I'd have to check, but I think I put them in Depth, first order, so that the branch turns out correct. I think that whatever order it is, it was just so that the tree visualization has the branches in the correct order.

I see.

It's sensitive to the vertex and edge ordering.

Okay, all right. Okay, so. Okay, so what we've got here is, you know, we've got this directed acyclic graph and you know, we're gonna, we're going to. Presumably we are not going to have a separate DAG construct which hopefully we don't need, and that our DAGs are still basically just graphs. Right. We're not going to have a special DAG object. Right, yeah. Okay, so now I could see a.

Use for a DAG object that has ordered branches, but it's not necessary for this functionality, I don't think.

Look, we need graphs with ordered. Ordered, you know, with ports. Right. Same deal, basically.

Yeah, we, we were talking about that last time we talked about directed acyclic evaluate too.

Okay, so I mean, that's the thing we need. And do we have Charles here or anybody from the graphs group? Charles was invited. He declined to come, but we can poke him if we need to. Okay, well, I think we should time to poke.

I think since we already have DAGs as our current unordered graph objects, I don't think that the grass with ports or ordered branches should be a prerequisite for this functionality.

No, it's not. It's not, it's not. There's still a definite order. It's just that. Yeah, okay, so to me, I don't know if, if we can poke Nick, see if Nick is around, that would also be helpful because I think, I think, you know, we. We've got this function that generates all possible foliations, right?

Yeah, so does mine. And just to zoom back to when we were arguing about this last time, there was a discrepanc between what Nick did and what I did, and I can only prove what I've done.

Okay, well, okay, so for example, here I think I have. Okay, so that's a depth first ordering, right? That's a depth first scan. And somewhere here I've got many different possible foliations. Right. Make sense, guys. Yep, that's a dag. And that's its many foliations. And you know, to me, where's foliations? Where does it set it? Okay, graph foliations. Okay, so this is.

Basically all, all partitions that respect the order from the.

Exactly. Directives.

Yes, but, but I've investigated that closely and I don't know if that's what's being produced. Because vertex stratify. If we go look at the WFR page, we can see there's another usage that's not just the default usage. And it's going to give you all, all of the things. And we did. Nick and I did not get the same results.

Okay, where's the. Where's the non.

Scroll up there. Returns a comprehensive set of stratifications with at most n additional levels per stratification. Because the. Okay. The other thing you can do is like, look, this stratification here has four levels to it. But if I say that I'm willing to add one additional level, then I'm going to get. Let's just try that and see what happens. So first, this, this example that we're looking at there. Yeah, just copy that. If we copy that whole thing.

Yeah. Okay, but I mean, there. So there's a graph here. Okay.

And then just extract the vertex stratify G0 and. And replace that whole statement there, the whole labeled statement so that we can just see what's the output of the function is. Right. Okay, so add. Add the extra argument comma zero and see if that changes anything. I don't expect that it says that.

There'S only got an extra layer.

Right.

Because there's only one. Now, like, change that. That's a rigid graph there. But now if we change this to one, then all of a sudden there's a bunch of different options.

Right.

Although now that I'm looking at it, I don't know if it should be an option. Should it be an option to put a blank on the first level?

Probably.

Or to leave a blank anywhere? Probably not.

I think it might need to be. I mean, Nick is the expert on, on what we've done with. I mean, well, do I get a concept?

Do I get to be an expert too? Or is it just Nick?

You can be an expert too, Brad.

Thank you. Because I've written three or four of these blog posts now.

I know, I know, I know, I know. But I mean, we've, we've done, You know, we've had different.

Oh, when you say Nick, you mean Nick with no C?

That is correct.

Okay, that I was confused for a.

Little while there, but now I understand. I. I am not an expert in this. Now the. Hold on one second, guys. Hi again. Okay, so. All right, so treat your traversal order. You know the definition of which of these foliations you get which reference frame is. Well, let's take a look here at directed ACYCLIC evaluate. Because I think it's also related to this. I mean, basically what we're doing here.

Well, actually there's one for vertex stratify, if I remember correctly. There's one interesting neat example that's worth looking at. It's on, on the page there. Yeah. Saying like certain, if you have a graph like this, this graph that we just looked at when we did G0, it, it only had one result. Why is that? Because the entire graph is, is what I called a rigid subgraph. And there are other graphs like, I guess this one here is an example where if you try to do vertex stratify zero, then you get a bunch of different options because there's a vertices that are floating around there and they can be perturbed up or down a level.

Yes, I understand. I mean, it's like it's simultaneity surfaces. It's like you can sometimes have only one possible choice of simultaneity surface or you can have multiple different reference frames with different choices of which events are simultaneous.

Yeah. And I think the use case for having these, for extending the time access with regard to direct data cyclic evaluate is like maybe we're doing a parallel calculation where we want to evaluate vertex functions all at the same time, but we're constrained by the number of cores that we have available.

Yeah, yeah. Right. Let's zoom out here. Okay. So the function vertex stratify, I don't know if it's the right name. Not a terrible name. Is, you know, it has a default potentially which is this breadth first traversal or this cosmological rest frame or this, you know, everything as quickly as you can kind of thing. Right, yeah. There is also a depth first version of this presumably.

Is depth first defoliation, though.

No, not necessarily, but you can start it. Yeah.

Illustrations on tree traversal order leaves first are obviously foliations. But depth first has the level sets like inside each other.

Right. So the question is, can we label the foliations? And this is kind of like, you know, one thing we've got a canonical foliation which is this everything as quickly as you can. Foliation. Right.

Yeah.

We might have another one that's canonical, which is everything as slowly as you can.

Yeah, yeah, that's true. I think Danny made that point back when we were talking about this.

Okay, so then, then we've got all of the other reference threads. Yeah, maybe.

Brad, you know, the answer is the everything as slowly as you can. Is that what you would get from, from vertex stratify of the reverse graph?

Yeah, I think so, probably. Okay, that'd be my guess.

Which is your leaves first, I think.

Yes.

Okay.

And are those the only two foliations for trees since they have a unique parent at each node?

Well, it depends on the rigidity thing. So some like if there's a short branch and a long branch there for the foliation, the short branch can get permuted forward.

I see.

Yeah. Okay. All right, so let's then to me look, I think vertex stratospy, its default seems clear and it seems like a good function to have. Now the question is, how do you get the other foliations out of it? Okay. And this reminds me a lot of find path and find path by default gives. It still gives a list, right? Find path still gives. Am I. Am I right if I say find path of this thing or this thing here? Right. If I say find path of that comma. I don't know, 3 comma 8, for example. I see. It'll give a sub list there. I'm just concerned about. I mean, if I say find shortest path here.

Because you can also do. All right.

Yes, you can. So there. If I say all comma all here. There's only one of them.

No, that does not seem right to me.

3 comma 6.

You should just specify how long your path should be.

Yeah, yeah. I think you have to write infinity.

Or double O. Yeah, okay, fine, Fine. So in the case of these foliations, the question is what are the selectors that we want for these foliations? Because they very quickly get very long. And Nick has this. Include permutations and not. What is the relationship between graph. Graph foliations is kind of. So if I take this and I say graph foliations of this. Let's say include permutations to false. That's only one foliation, right? That correct. I think that's correct. Yeah, that Right.

Maybe we should try the other graph example where that's non rigid from the WFR page because that will.

That one will probably.

That one right there.

This one here?

Yeah, I think seed rand. If you get the G0 there, it should be fine.

Just this.

Yeah.

Okay, so now we take that. Not the right thing.

Okay, can we compare that to vertex stratify with the extra zero argument?

You've got. You've got the analog of include permutations to true, I think.

Oh, well, let's test that and just look at the length of those two lists and see if they're the same.

There's Also another property for whether you can allow to put a vertex in it in one of the stratifications. So it's called bundle. I called it bundlefoliations. It also make counter to how many.

Of them there, how many did we get over here? It may be the same.

Actually, I think it's the same.

It's pretty close.

Yeah.

Okay, so that's good.

Yep. Okay, we need.

So if we're really serious about this, we need to do a detailed double check, making sure that Nick and I have the same definitions, that we get the same numbers every time.

Yeah, yeah, yeah, I agree. I agree. So I think what we need to do. I don't understand Nick's bundle thing, but this permutations thing is useful. That's a useful. And probably by default we don't want to generate all the permutations. We just want to generate the canonical order. Because most of the time the big problem is going to be, you know, because any time you can permute any of these in any way you want, can't you? Each one of these space like surfaces can be. Can have arbitrary ordering, right?

Well, if. I guess if they get off in different disconnected sub graphs, maybe. I don't know.

Nick.

Do you mean within each of the foliation you can permute vertices in any order?

Yes.

Yeah.

Yeah. Right. So, so. So.

But I don't think that's what mine is doing though.

Well, I don't know what it's doing, but it. Yours was nice and fast. That was.

Well, they're not permuted. They. They are always sorted with initialiation, stratification as you.

So why. How, how. What are all of these include permutations. Just means that we also get 8, 6, 9. Is that correct in here?

No, I don't think so. No.

What does include permutations mean?

It's a. Well, I can't really remember. It's a algorithmic thing. What to actually do when you consider when you put your new vertex in affiliation. Because you always have a choice. And you can do it like in all possible ways or in the first.

Can I call. Okay, so I don't know if there's a trivial transformation from this to the all permutations, then we probably just want by default to return this thing. And if all you do is to say, you know, permutations of each of these things. I. I'm not sure. But. But okay, it's not.

It's not that trivial.

Okay, fine, fine. All right, well. Okay, but clearly we need both the canonical foliation and we need a way to generate either a selected set of. Of. Of foliations. For example, we could do the same thing that we do with find path of, say, give us a thing which is stratifying into some number of. Of. Of strata, if you understand what I'm saying. That will be a way of. Because clearly there is a. There is presumably a stratum where everything is being done separately.

Yeah. If the number of strata is the same as the number of vertices or something.

Yeah, right, but. Right, but I'm saying that as a way to select between. Because we're going to get very large numbers, very. Even a very small graph. We're going to get very large numbers. And the useful thing is going to be the canonical stratification and other ones that are. I mean, I don't know how we can say what other ones are going to be useful. Get what I'm saying?

Yeah, I think what you're saying is that we need to put some time into thinking about how to thin the results out because there are going to be too many results.

I think that a possible approach would be to model what we do with find path, where you give an argument that says what the number of stratifications, the number of slices in the foliation should be.

Well, that is already the second argument to vertex stratify. But it's a different something.

Rel. Yeah, it's different. Same idea. Right.

But that, that. I don't think that's going to solve the thin. I don't see how that solves the thinning problem. For the thinning problem, we will like have to specify maybe properties that each level needs to satisfy or something.

Well, which is similar to the depth first, breadth first type story.

I think that depth first and breadth first will play into a resolution of this problem. Yeah, because there's extra.

Let me, let me just point out this is the problem that I've been yackerating about for like three years, which is this problem of what are reasonable reference frames. See what I'm saying? And we've talked about this a bunch of times. You know, there are ones that are deeply fragmented. There are ones that look like, you know, that look like this, which is a weird reference frame. Right. That's a. Well, that, that's just an ordering. But you know, we have something where we have. Well, down here we have something where some of these are very weird reference frames. See what I'm saying? Of what's simultaneous. Am I making sense? And there's some that are more reasonable. Okay, so all right, so we want to have vertex stratify. The name is not terrible.

Can I suggest graph levels? Because everything as fast as possible and everything as slow as possible exactly correspond to positive and negative levels in an expression or a tree.

We'll consider that Graph levels is a possibility.

Graph strata, maybe.

Maybe. I mean, there's also the foliation term.

All right.

But I don't know which is the right thing. Okay, let's go on to directed acyclic evaluate, which I think is a very interesting function. We have Charleston as well now. Okay. All right, so the basic point of directed acyclic evaluate is there are values coming in from sort of the leaf nodes and at each non leaf node there is a function that combines those leaf nodes. Is that correct?

Yeah, I might say it backwards that the values are coming from the root nodes.

Yeah.

Okay. But this is essentially the same thing as an expression tree.

This example, of course, this works for dags, Right.

So the point is that in an expression tree, this is evaluation of an expression. The point is that there can be shared common sub expressions, which makes it a dag. Right.

In the general case.

Yeah. Okay.

Right.

I'm not quite sure what all of this is. I mean, these.

These are just the highlighting is the values coming in and the frames are the functions at the nodes.

I think that's bugging usage. Basically. The latest thing we've used directed acyclic evaluate for is to. We had these game graphs that Andrea and I made and we figured out a way to take the entire game graph and prune it down into a minimal version using directed acyclic evaluate. So that we got from like having a proof that the second player is going to win this game maximally. It takes 10,000 vertices, but then you could prune it down to only 40.

Okay.

And that the. The restriction of directed acyclic evaluate is that the vertex function is the same on every vertex. And another one is that it does not.

The.

The order of inputs is not. Not critical or not paid attention to by. When evaluating these vertex functions.

So, so this is essentially. I mean, to me the obvious name is something like graph evaluate.

Or.

Ian and I have been talking about this last night. Maybe propagate vertex weights. Because one of the main design changes that needs to be made from what's at the WFR right now is that we'll probably just return the vertex weights, whether that's as. Maybe as an association.

So.

To me propagate vertex weight sounds okay. Because really what's happening it's not.

It's not. That's incomprehensible. Nobody will understand that use.

But I mean, kind of like vertex weight fold.

Right. But graph evaluate may be misleading because we cannot in this case do undirected graphs. And we also can't even do cyclic graphs because the cy. We don't have a. Any idea of what to do.

No, I know. Welcome to time like curves. I mean, you know, that's the, you know, it's the fodder for science fiction of what happens in a universe with close time like curves. That's, that's what we have when we have cycles on the graph. We don't have a notion of time. We don't have a way to stratify time. That makes sense. Yeah.

But may be related to paths more than just like evaluating or folding something. I had this physics example when you essentially use it to evaluate path integral version on a graph and it just the same is basically this. You can use it to compute all the paths and sum them over.

Well, so you're saying. Okay, hold on, let me just understand what's happening here. So the most obvious thing would be a, you know, a function F applied at the function F applied to the weights. Oops, yeah, you just had that above vertex weight fold of F comma. Now I don't understand at the nodes that. So what would happen is the nodes, the intermediate nodes wouldn't have weights on them initially.

No, the. The way that Directic is directed acyclic evaluates usage is set up. It needs a graph initial values and the vertex function. And the vertex function has a. What, what this basic example is showing here is the vertex function. It actually gets a little bit of extra information in addition to just the preceding vertex weights.

Okay, but what I'm suggesting is this. I'm suggesting the following vertex weight fold of this comma graph assumes that the graph has already had vertex weights added on its leaves. Yeah, I don't know what's that.

And edge weights too. Basically fully awaited graph with the following edge weights.

The second argument for the vertex function in directed acyclic evaluate is the edge weights and the third argument is the edge.

We need edge weights for probability calculations.

Yeah.

So the function that we're putting there first, if we make it too simple, then this vertex weight fold, if that's what you're going to call, won't be able to calculate everything we need it to.

You're saying that that function is going to fill in at every node. Okay. So it's taking Input to F is. I don't quite understand why the input to F isn't just a list of edges.

It's the parent vertex weights, the parent edge weights and the parent edges.

But if it was just the edges, they have the edge weights in them, don't they? I'm trying to understand what's that?

Just, they just can contain tags. You have to query for the weights separately.

Yes. Where are the weights stored?

The weights have to be input.

They're stored globally for the graph. Is that correct?

Yeah, the same.

Brad, correct me if I'm wrong, but I don't think directed acyclic evaluate actually uses the vertex weights and edge weights from the graph. It's.

It's not. It's not, but it could. It's not currently, but it could.

How do you put them into the graph? Remind me how that works.

Well, can you kind of like any other option, like vertex labels, edge labels, you just say.

Okay, so okay, if I just say. If I just say graph A goes to B, B goes to C and then I say vertex weights goes to 1, 2, 3. So that vex weight, singular. Okay. How do I find those vertex weights?

Is it annotation value or something like that?

Yeah, yeah. Charles, can you say please, annotation value of a graph.

Then second argument, vertex weight.

Is Charles here?

Yes, I'm here.

Okay, but so the vertex weights are stored as a metadata to the graph. They're not stored sort of microscopically granularly inside the graph. Correct.

Right. I mean all the property, anything which is not topological structure, annotation associated to the graph, not granularly.

What is the input form of the graph on 395.

Well, the input form doesn't represent anything. Doesn't represent how we store. Right. We build an inform. But because you ask for.

Okay, but so the basic point is that in your folder here, what you're doing is you're taking at every node down here, you are taking the weights of the edges that come in.

But more importantly the weight of the vertices.

That came in from here that were evaluated so far. Yep. What about the names of the vertices?

That. That is handled through a third argument. And that can also be important because if you have the name, especially if you have the name of the vertex that you're on, then you can start. It's a little cumbersome, but you can start switching the function that you're evaluating. And we have to do that for some of the games. Proofs.

Yeah. Okay. Okay.

Evaluate a different function for player one.

And okay, so the, the function F needs to take.

Can I just say the third argument is the edges and from that you can get the parent vertices and the current vertex.

Yes, I understand. Okay, so the F is taking the incoming. The vertex weight, The vertex. The edge weights, which should be equal in number to the vertex weights and the edges themselves. Right?

Yes.

Well, that's not crazy. So, for example, if we want to do probability calculations, we would be that function for probabilities, we would simply be adding probabilities. Right. So it would be a total of #2. Is that correct?

No, no, because the, because you have, you're going to have the, the edge weights are. For the probability usage is just going to be dividing by the out degree of a particular vertex, but you're still going to have some value vertex. So you've got to multiply the value that's on the vertex by the edge weights. And so I think that that's probably going to be a dot product of number one and number two.

Yeah.

And I'm.

Okay, so is that right, that it's the vertex? It's literally that.

Yeah, probably. As long as you have the edge weights set correctly and the vertex weights too.

Okay, starting to favor graph weight fold. Since we have vertex and edge weights.

Yes. It shouldn't be vertex weight. Right.

Well, but okay, here's, here's the question then. Are the, are the edge weights going to get rewritten? Because the vertex weights are being written lively. But like with this probability usage, you have to put in all of the edge weights and they never get edited, they never get changed. So for that reason I'd say keep.

It vertex weight fold, because that's the only thing that's getting rewritten. Yeah, well, so but then this will return a graph. Is that correct? In which the, in which the vertex weights have been rewritten?

I'd rather that it didn't do that because then we've got all sorts of extra overhead time for the users inputting the annotations and then when they get the graph back, extracting the annotations. From a practical point of view, I'd rather just be able to input the initial values as an argument like you would with Nest or like you would with the extra.

Okay, but that. We can do that. We can do both cases. Right. So this here would return the graph and I need to do something here. So hold on a second.

But I don't think we can have two output forms, can we?

Well, we can if we have a third argument. I mean, so if we say the Graph and then we have INIT here. Where init. How do we know what order INIT is in? How do we know what the leaf nodes are that we're going to give or the root nodes that we're going to give the INIT for?

The root nodes have no in component.

I understand that, but what order are they in? It's a big mess to figure out which ones they are.

The order doesn't matter at all. The order. The first thing that vertex weight fold would do is calculate a foliation or the stratification. Once it does, then it knows how to proceed through and calculate all the values.

Here's the problem that I have in this graph. We don't know for this init, we don't know what the, you know, it has, you know, five inputs here, right? Yeah, I have no idea in what order those inputs.

Yeah, so just put it in as a list of rules or as an association.

Okay, so you're saying that that init could be B1 goes to weight 1, et cetera, et cetera, et cetera.

I'm not saying that it could be that. I'm saying that the version that's published on WFR already is that.

Okay, all right. Okay, so that's, that's giving.

There's a question of if it should be a list of rules or, or a function.

And I think maybe we could have some flexibility. Function is good. Association is good too. List of rules, good.

I think that the list of rules is sensible because the graph is of finite size. You can always generate this function by just saying. By mapping it over vertex list of graph. Yep. But in the case where you give that initial, then to me, the output, given that you've had that init, then the output could be a corresponding list, could it not?

Yeah, yeah, I agree with that.

Yeah.

And if you don't give the init, the output is the graph.

If that's good for everybody else, it works for me.

I think it's sensible.

So you'd have. In the two argument case, it would need to be awaited a vertex and edge weighted graph.

Exactly. And then it would overwrite the vertex weights. Or maybe by doing this operation, I.

Think right now, directed acyclic evaluate uses the vertex names as the default vertex weight.

If no weight. If no weight is satis is specified.

Yeah, that's fine. I mean, it can do something like that. That's fine. I think that's a little weird because. Because the vertex names are unique, which is a kind of a weird thing to do.

I think it would be sensible to just initialize all vertex weights to 1.

For probability calculations because that seems sensible. That seems more sensible to me that the default is vertex weight to 1. If no weights are specified.

Well, why not zero?

Charles, what does vertex weight go to automatic do? Is it different and different functions? I think like the find path functions has default weight. Is one.

No, technically all the weight always one.

Okay.

That's what we should.

Same with edge weights.

Charles.

Edge weights actually is the other way around. In general, the edge weight are always the edge weight are 1. The vertex weight. We don't require them to have a specific value. So we don't have any case where we actually need a specific value or set up a default value for the vertex weight.

Okay, so we can pick one here. I think it's a reasonable thing to do.

The dot product will just return path.

Count by default, which is great. So F being the dot product. If F is dot, I don't even know. We might have to do 1/7 2 there. All right, I think we got a plan here. I think I need to run along here, but is there anything else we can cover right now? This looks like a good plan and I think we should start off. Vertex weightfold is not my favorite name, but let's start by with that name seem reasonable.

Yep.

Thank you.

And I think it's a great function and very useful. Particular people can understand it. And then I think we've also got a plan for whatever vertex stratify or vertex levels or graph levels, whatever it should be called.

Danny linked to the Wikipedia article for graded postset.

We're not going to call it that.

Might be related terminology there.

It's absolutely related terminology. I mean, as is foliation, as is reference frame, as is, you know, traversal order. Yeah, there.

There were a bunch of terms in there that. That. That's what I was getting at. They call them ranks, among other things. Each level I also notice.

Lieutenant colonel. Yes, thanks.

This is also not a huge extension as far as I can tell from topological sort.

I never remember what topological sort does.

Well, it gives a sort a sorting that's compatible with the grading that.

A.

Sorting of the nodes compatible with the grading given by vertex stratify, for example.

I see. Well, that's a good connection to make.

So it's possibly just a catenation of vertex stratify.

That's what it looks like to me.

Yeah, the default vertex stratify.

I mean, it's a concatenation of a vertex stratification. Whether it's the one in that particular function? I don't know.

Yeah, because there's a possibility to rearrange per level.

Well, why is the.

Yeah, why is two so far up there?

I don't know. This looks wrong. I mean, something is different. We have to understand this. There's not.

There's not always a unique topological sorting. Right.

So this may be a particular foliage.

Correct.

And not the one that we're picking by default, which is probably wrong. And we probably should rethink topological sort for this case. So whatever traversal order we use for stratify, we should also probably be able to use for topological sort.

I. I would be careful on that. I mean, Charles probably can comment, but I'm guessing it was written for efficiency and we shouldn't force it to do something different. If that's the case, fine.

But this is a similar operation to this stratify operation.

Yes, very much so.

Well, I mean, the fact that there are 17 different things that all lead to the same end result of wanting the stratify operation means we need the stratify operation. And it's no surprise then that it's showed up in a bunch of stuff Nick and Brad and I have done.

But topological thought probably also appear in more things. So it's a matter of point of view.

Whatever. I mean, my point is that this general operation of, you know, forming an ordering of vertices by some kind of foliation type thing seems like a generic operation, and we should have that function. Which is what we were concluding anyway.

Because graph users will be gratified that their dags can be stratified. Say that a few times fast.

All right, well, if I can make one more suggestion. If we're willing to explore the different vertex stratifications, maybe there's a chance to put an extra argument on a topological sort and do something similar there.

Yes, I agree. I agree. That's what I was suggesting, that it should agree with what we do here. All right, I think we got a path forward, so to speak. Talk to you all soon. Bye bye.