Live CEOing Ep 680: Language Design in Wolfram Language [ExpressionTransform and WFR Review]

Sa. Hi, everyone. Welcome to another Wolfram Language design review meeting, part of our Incremental Language Development series. And we seem to have a few topics here today. Extension to Nest Tree and Tree Fold. I have a feeling there's just a lot more to do with these, but. Okay. To allow Nest Tree to take a list or association of operators. Gosh.

Right. So in this first example, you're using regions to calculate the children at the country level, but you're using subdivisions to calculate the children at division level.

So you're saying this is saying at different. With different part numbers. Use different. I mean, we could imagine the same thing for map, couldn't we?

Yes. Like a version of mapple that takes a list of functions. Yep.

That says at different levels do different things.

Exactly. It's the same concept.

And it's not a supplied typewrapper. I mean, it's not a wrapper of some kind where we're saying, just like we were talking about, you know, things like, you know, map immediate evaluate, blah, blah, blah, you know, map.

We can already take a list of exactly two operators and this would, this suggestion would allow a list of any length.

I mean, one could imagine something like this where you say, you know, level selected, you know, one or two goes to F3 or anything or something goes to G. Is that crazy? Might not be useful. I don't know.

So you're saying that in these functions in which we map something or we do something at given levels and we have two separate arguments, this will be like a combination of those two arguments.

Yes. I mean, but this is something where you can select. I mean, here, if I'm understanding correctly, this means zero means the root. Is that right?

Yes.

So I'm. Yeah, I'm not entirely clear on what problem we're trying to solve. So what does a list of two things mean here?

What does a list of two mean right now?

So the first three, it's meaningless. Like you would just apply the, the list to. To the function.

Okay.

Oh, so what is the. What is the thing that takes two. You said something takes a list of. Exactly.

Tree Fold does have an option for a list of two elements. And that. Does that mean it repeats the first one for all of the non. Leaves and it uses the second one for the leaves.

Oh, boy.

And so then what would an arbitrary length thing mean? Use that guy at that level one.

Level two and so on.

Right. Well, that, I mean, that's a complete disaster, right? Because then you have a two level tree. Well, I guess maybe that would.

No, no, you just have to send it at the end. I mean, what you got to do is. Okay, there's tree fold. Yeah, look there.

For what it's worth, the thing with NEST Tree is equivalent if you do through compose the list.

Right. But my point is that. Right, so. So you have some big tree with say, four levels, and you give it a list of four things and it applies one to each level, and you give it a list of three things and it applies things to the first three levels and I presume ignores the fourth level. I'm not entirely sure.

No, I think what it would have to do, given the existing syntax, is it has to pin 1, 2, 3, 4, and then the last one has to be rest, so to speak.

So the last one is always applied to the leaves.

Exactly.

And. And if the tree is deeper than the. Like, let's say you had this 1, 2, 3, 4, but you had something of depth 5, it would repeat the fourth operator, which is consistent with what it's doing now for the. The case where you have a length two, because you give an operator for the root and it repeats it for everything except for the leaves.

I see. So if you had a list of lengths 3, it would apply the first one to the root, the second thing to all the intermediate levels that are not the root, and the third thing to the leaves.

Yes.

Okay, so why do we want this in the first place, this thing about having, you know, different levels? Because you're saying we're using a tree to represent something where the levels on the tree. And we're going to have the same issue that we have with parts, where in the end it's going to be like we really want named parts. That's why we have associations. We don't have trees with symbolic levels, do we?

No, I don't think so.

It's more that you might have different kinds of entities at different levels and hence different notions of children. Currently, Nest Tree handles the case well, where all of the nodes have the same type and the child operator is always the same.

Right. But the thing I can imagine is that you want to label the levels in a Tree with Country, ADM1, Administrative Division, ADM2, ADM3, etc. Do. We don't have a good way to reference level three, that is named County. So, see what I'm saying?

Yes, it's a separate thing, but yes.

Assuming some sort of rectangularity.

Yeah.

Right. Well, I mean, we don't have a named levels mechanism for expressions, for general expressions either.

Right.

Yeah. I mean, I would like to claim that that would be a useful thing to think about and something that may be relevant to our new sort of data containers.

If we had such a thing, they could be used in the association as left as keys of that association there.

Yeah, exactly. Right. So there's the question of named levels for expressions, etc. I have no idea. I mean, you know, let's say we have, you know, something like this. How would we even think about a named level play for this? I mean, we would have to have a wrapper around this that says, you know, named levels. I mean, I don't know if this. Any of this makes any sense. And then lots of things would have to support that. Does that make any sense? I mean, again, maybe. Maybe it's coming with data container, something like this. I think so. Sorry, Jose, what you said. What did you say?

I think this needs a more powerful container, like data container, not a dedicated object.

Right.

Because I think you know that. Then you're hopelessly lost of like. Well, what happens if you have a thing, right? Does like. Does the named level, if that's embedded in something bigger, does the named level wrapper magically disappear? And there's no way to get that head out of the.

All right, what is the relationship between data container and trees?

Well, at the moment, what's being worked on with data container is, you know, rectangular things, which is not trees.

Boy, is this accurate. I definitely do not know my British royal family genealogy lineage based on this. Okay, never mind.

I think it's a hierarchical character that is common to both. It is that trees are more general.

I'm sorry, a common. Yes, okay.

It's only the hierarchical character that is common between trees and deeper arrays.

Right, but. Okay, so the thing in SU here is, does it make sense to have nest tree support an association syntax with numbered by level number, Right?

Yes. Or the list. I think actually the list is like more fundamental. And then the association is that weird.

Thing about the leaf anchoring is for.

Tree fold only, not necessary.

Yes, but I think the fact that in trees it happens relatively frequently, that we want to map different functions at different levels.

I agree.

And this is the sort of construction which would allow us to prepare something clever to call entity value efficiently.

Yes, I agree. I mean, this seems reasonable. Okay, do other people have comments about this?

So do we agree with the.

I'm not entirely clear what this is.

Okay, so this, the proposal is that the first argument of which functions tree fold, nest tree. Which other ones?

Just those two.

What about tree map?

Well, treemap Takes level specs. Right. And including sort of named level specs.

Right.

Not, not just so I'm not quite sure how those things would interact.

Right. So this is the discussion we were having before about merging the two arguments, the first and the third.

Yeah.

So it's less natural here, but the idea of mapping at different levels like map all is interesting.

Treemap. Okay. Treemap defaults to mapple basically. And this is what levels you're mapping on. And the thing on, on offer, so to speak, would be a tree map of, you know, one arrow F, two arrow G. On the tree and then the level span would be the third argument.

Perhaps it's a matter of having respective lists. So for example, in this example you were showing when we have 1 and 2, we could have a list of.

F and G. Sure. What's your point?

I'm trying to maintain the current structure of three arguments and still generalize it with one and two in the third argument.

I don't see anything wrong with that. I mean, that's just a saying that. Right? I mean, but if this argument here.

I think actually the level spec is independent of the function. Like I think it makes sense to have a list or an association with any level spec. I guess if, if you use the association you say levels one and two, but then the level spec is level three. You're just applying the identity because you didn't specify an operator for that level.

Right. Well, my, my contention will be this looks reasonable. Anybody else want to comment on this? But by the this I mean the ability in these functions applied to trees functions to segregate what function gets applied based on what level it's on. Objections?

I'm only thinking of the case in which mapping a list might have had a meaning. Is it possible that user might that be?

I mean, mapping an association does have a meaning and that's more scary. That could cause trouble.

Right. Mapping an association actually does seem like a real problem because that's pretty, pretty common thing to do.

That's a problem, guys.

I mean, it's a little ugly, but it could do a list of rules.

I mean, mapping a list I don't think is worth worrying about.

The only time I can think of that is if you're preparing for some crazy thing with through. But I think that's not very common.

Yeah.

It, you know, we could wrap an association in another layer of list like have that as a, as a workaround. If you actually want to map the association.

Nobody's going to know that. They're just Going to get the wrong answer. How useful would it be to do the non association form, just the list, not support the association.

So the list form you need to list the operators at every level starting from the root.

How do you do it with the association? Do you have a blank on the left hand side?

Can you use a list of rules.

Instead of an association?

Well, that's what Crystal was saying.

Well, I mean like here. So you say one goes to F, two goes to G. But you didn't have to say zero goes to anything. Right.

And how do you do the zero in the case of the list?

In the case of a list you'd have to put something there. It'd have to be like the equivalent.

Where's the there?

Like identity comma f comma g. I.

Understand that, but the first element is the 0th element.

Really it is the level 0.

Well with 3 or with map that would depend on the third argument.

Right?

If the third argument says from 0 to something.

Yeah, well that's an interesting claim. I mean that is shifting where you are referring to with the first element.

And that might be why we need a third element.

Sure.

For math. And I had thought that there wouldn't be a shift that the levels would just refer in both cases to the levels in the original tree.

I think with tree is clearer because you typically map on everything. But if this was ever to make it into map, then we would have to be more careful because map defaults to one.

Yes, okay, hold on. Good point. You know, I really don't like the fact that the association thing is not going to work.

Yeah, I tend to agree with that now.

So do we do a list of rules then?

I think we should start with a list. I think that's the.

You know, Nick is commenting. What about supplied?

Not sure how that would help.

Well, so I mean there's. It's a little bit complicated. Right, because you would be putting like.

List in the second argument of supplied.

It will be this basically.

But wouldn't supply be associated with each function rather than with the list of functions?

Oh yes, but there's a pure function around the whole thing in this case, isn't there?

I don't understand this pure function.

Yeah, this pure function is.

Yeah, this doesn't work.

Oh, I super don't understand it actually.

Yeah, no, I mean the correct version of this would be. That's very weird.

No, I think this is doing something very deeply.

No, I agree. Supplied should be on the right hand side of each rule. It should be one arrow supplied around The F. Right.

Why?

No, because this case is totally different because this is a pure function. This isn't even the usage where you pass an association.

No, I know, I know, I know. This is an alternative to passing an association. This is.

I don't think it is. Like, what's the hash level even doing?

Like what? I mean, that's saying that this has treemap.

This, this isn't an association of operators. This is just a pure function that, like, returns an association.

Right?

Oh, I see.

This is structurally completely different.

Yeah, right, because it didn't put the expression in there.

Well, just try for your own sanity. Just write a version of it below where it is what we're talking about.

Okay, what is it?

So you tree map and then write an association, and then I guess one arrow, you know, f, two arrow g. And maybe, you know, f and G can be pure functions. And you'd put supplied around the F and around the G if you wanted to pass them supplied arguments.

Why does that help? I mean, what we're trying to do is select from the association using the supplied level number.

Supplied what?

Okay, look, remember, supplied in treemap, right? Supplied, Supplied. Just to be clear here, we can have F of hash expression, hash level. Okay, what's that?

I think I finally see where this is going.

Let me have to say supplied here.

Yes.

Right. Okay, so now what. What I think Nick was saying is why not use this to do the selection?

To do the selection.

No, no, look, look, look, look, look. On level one, this will bind to. That says one here on level one.

Yeah, I see that.

It'll bind to one, and then on level two, it'll bind to two.

Okay, yes. And then these expressions will bind to. Because this tree map is going to visit every node of the tree and every node of the tree expression is going to bind to the payload of the node and level is going to bind to the level number. So I think it would work. I mean, it's kind of stupid and inefficient because it's going to construct everything on the left here.

Maybe you could use rule delayed.

Yes, yes, perhaps. But I mean, I think this would work.

This is basically using supplied to have one function that behaves differently on different levels, rather than trying specification for different.

Yes.

Right. Although you couldn't. Right. If you wanted to distinguish between tree leaves and not leaves, you couldn't do that. Unless we manage to supply negative levels or something.

Yeah, which we have the option to do in the supplied specification. You could also say this could be leaf or not type thing as a value. I think we even had something like that in the supplied mechanism. I admit this is rather complicated, but I'm trying to think. I mean, I just don't think it's going to work to have an association there.

I like the list of operators because it extends what we're already doing with treefold with the pair of operators.

Okay. Is it useful enough?

Another motivation is what Ian has in the next section, which is a generalization that includes both directions of going down and then going up. It's a promising idea. There is too much material here, but I explained very quickly and then Ian can continue. So the idea is to manipulate a tree or an array going down in analysis steps and then going up a bit. What dataset does already in query language. Yes, it's just that rather than having an unpredictable way of knowing whether an operator will be treated going down or going up, this has two lists. The list of going down operators, the list of going up operators. So it's much cleaner and you can have at the same level something happening when going down and when going up.

When you go back up, has it evaluated?

Yes, yes.

Right. So that's the big difference. So in terms of the kind of the execution graph, you see the recursion going down, it reaches the bottom, then it does the evaluation, and then you see the next part of the execution graph, which could be of a completely different expression coming back up. That basically the picture.

Yeah, Basically you get a result from each branch that you took and you apply results.

I think this is evaluating at each step. So at each level that you cross, it will evaluate the function that you have there so it can potentially modify what you have and continue going down. Same going up.

Well meant.

Yeah.

These are all starting with the seed value E to the E to the E to the E, and then using different operators to build different trees.

One of those should be, you know, have an echo in it so we actually see what's being seen at each step.

What can you tell me, what does this mean? This is. This is going down and going back up again. Is that right?

So this the second argument. Can you highlight the second argument expression transform the if. So that is the going down operator and then the third argument is a pair of operators.

Oops, did I get that right or am I confused here?

After the ampersand.

And now what's this?

So here we have two ascending operators.

Don't you need. Won't you need an echo inside of each operator or.

This, what Steven's doing is going to just print the list of operators. I don't know if that's what he says.

That's not what I want.

Well, in that case, I think you need the bracket inside the ampersand.

That's what I thought. Was confused by being told otherwise.

Yeah, I thought you were just trying to print the list of operators. I'm sorry.

Okay, this goes there. This is a good case actually of where you don't actually know where the bracket is going to go. Actually, we should, we should make sure this case gets fed to the folks who are trying to do bracket completion because this is not an obvious case. Okay, let's take a little look here. Okay, so we drilled until we got to the none. We logged until we got to the none. Why is there an up to there and what happens with up to up to? We put down an X because.

Because you hit the none. So when you hit a, when you hit the leaf.

Can we see some other examples of this? Can I see something where we have actual non trivial operations happening?

Yeah, there's a neat example section and there's a comparison with query section.

So how does it choose between up 2 and up 1.

Up to? Because it's the descending operator returned none. So that's the last level. So it applies the ascending operator for the last level.

Okay, so this is. So this is similar in structure that the first thing is everything except the last level and the second is the last level.

It's basically nest tree composed of tree fold, same arguments.

So this type of idea is much used in data set with group by and so as you go down you do group by and then the data set is being modified so adding more levels and then when you go up you do aggregation stack type of operations like total, etc. So, so this is an idea of allowing you full flexibility of what to do going down and what to do going up.

Is it obvious that you just go down and then come back up again? That you don't bounce around halfway and things.

It could be generalized. Yes. I guess this is just the assumption of just going one down, one up.

One of your functions could be another expression transform, right?

Yeah. Wow. Okay, so that's just bigger picture here. Okay, so map goes down an expression tree spraying things into it. Then okay, then those things might evaluate. Then one thing you could always do is redo the map. You wait for everything to evaluate. You bubble up to the top again, then you go back down in again and evaluate stuff and map again. This is an equivalent because why.

The levels will be different you'll have new expressions.

Yeah, think of group by again, a new level appears in the middle and shifts all the levels below.

Okay, but.

Which means if you were to chain this as a separate operator after the fact, you would then need to know to apply your operators at one deeper level.

Oh, gosh. Okay, let me try again.

So.

Let'S see the reason. I mean, the F could evaluate to a 10 level object. Every F could make a 10 level object. Right? And so you're saying that knowing where to apply G, know that. How does it know? Does it apply G to wherever the. I mean, let's say that it made a 10 level object. Does it apply G10 times as many times? Or does it apply G only at the places where it had made that complete 10 level thing?

The second one that you said, basically you're gonna have a call stack. Like there's a function that called the function that produced the children. And once it gets its results, you apply G to that.

Okay, to the results. Yeah. Okay. And this is useful, right? It's. It's essentially like a map and reduce operation, isn't it? You come back up. It feels somehow ungeneral.

It's for divide and conquer algorithms.

Okay. Where otherwise known as essentially map reduce. Okay, so, okay, so how does the Sierpinski divide and conquer?

So you start out with the point, the three outer points, and the descending operator generates a list of three sets of three points for the three smaller triangles.

Yep.

So that's your map or your okay, operation.

And then on the way back up, okay, so that at the end, when it's gotten to the bottom, it's made a bunch of points, basically. So for example, if I said, just to understand what this does, if I said map. All right, why did that do that? Oh, because the arguments are in a different order.

Okay. What is the structure? Okay, I'm sorry, I'm playing with my own examples here. What are the arguments to? What are these functions supposed to return?

What is this? What is this? What is this? Which three?

The third argument does. There's no third argument to mapall.

Right.

Matall doesn't need a level spec. It has an implicit level spec.

Mapall has an expression already. This is building an implicit expression that never gets evaluated except for the explicit ascending operators that you give it.

Okay, so if I replace this. With an F to prevent this from blowing up. Okay, so what will that be? That will be not quite resolved because it didn't quite. But in the other case, the reason it blew up was because I had I'm confused. I had. Why wouldn't it be? Why didn't I have at the end of that other thing? Oh, because I couldn't. By the time it was in the structure of the list, it was not going to evaluate properly. I'm a little confused. Can you explain to me what was going on before?

I. I'm not sure. Like. So circle points of three is a matrix.

Yeah.

I don't.

Question what exactly you're mapping onto. Okay, I understand what this is doing, threading.

I'm not sure how this ends up getting so deeply nested.

Because. I'm not sure. I think it's level specs are wrong. All right, look, I think this idea though, okay, you go down, you spray functions onto on things, you come back up. I know, but on every place you come back up to. Okay, let me try and explain it this way. There's an anchor point which is the sub expression whose head is the thing you applied with map. As you come back up, you get up through a bunch of those anchor points because the Fs that you put there are still there. They may have innards that, you know, you don't get back to the level where the F is until you have recursed back to that level.

Yeah.

And to clarify one difference between expression transform and query. Well, so query like compiles things to maps and things and basically sprinkles operators. Expression transform is in the nest in the fold family where the operators immediately evaluate. So it's not like building an expression and then evaluating applies and evaluates the operator.

Yeah, I understand. Isn't that an implementation issue, not something else?

I don't think so. Right. I mean, whether.

New structure or just mapping over structure that you have. Sorry to cut you off, itai.

Yeah, but.

Right, right. Well, for one thing, when you write, if you do things iteratively, you have some control over evaluation. Whereas if you sort of replace everything and then just let things evaluate, you're at the mercy of the evaluator.

Right.

Those are going to get you to two very different states.

Yes.

For example, I use this internally in expression tree to prevent evaluation.

Okay. But basically, okay, it was going to go down to a certain specified level, then it's going to come back up again as it comes up. It is applying a different function as it come back comes back up. The normal map would apply identity as it comes back up, Right?

Yes.

I'd say it's more the other way that the, the descending operator's identity and on it and the operator that it's mapping is all the way back up. But I guess I don't know the actual.

Why. Why is it that way? I don't understand that.

I mean you usually bottom up depth first traversals.

Ok, let's look here.

I think it doesn't matter in this case, given that all the they are all going to be inserted before evaluation in map.

Okay, so that's sprayed Fs everywhere. So now let's imagine that our F is actually something that doubles the depth here. So let's say that the F is instead U of V of hash.

Right? And so the depth is coming from the point when you let this evaluate. But say if you had hold of like if the second argument of map all was had hold, that wouldn't happen. Whereas expression transform can eat things with head hold just fine.

But the basic point is that every U of V of something is an anchored object. We know where it is. You can pick up that sub expression.

Yes, but in expression transform the new operators would act on the results. So they would act on the U hats, the V hats, rather than the original.

No, I know, but. No, but they're acting on the whole U of V of D. They don't act individually. The thing gulps the ascending function gulps these whole objects, doesn't it? I think it has to, yes.

The ascending. Yes, yes. I was thinking on the descending.

The descending. We just did here. So I think ascending takes whatever was written by the descending thing. It takes whatever sub expressions were written by the descender and applies the ascender to them after their innards have been evaluated.

Right?

Yes, To the results of wrapping the descending function.

Not just wrapping, but evaluating the descending function.

Yeah, right, wrapping. Well, wrapping the descending function and. Yes, and evaluating it because. But the whole point is it hasn't, you know, each one of these is a sub expression where it can pick it up. It knows what the sub expression is. That's the surprising thing, because normally when you do evaluation, everything's going to change. But because this thing spawned an evaluation, basically it's applied to the ascending function is applied. Another way to put it is. Is applied to the results of evaluation of the. Expression that the descending function spawned. Does that make sense?

Yes.

And because it always knows what it spawned, it's never confused about where it is when it spawned that thing, even though the structure kind of changed completely. The ascending function knows where it is because the its corresponding descender. So, so another way to put it is, okay, there are pairs, Descend, ascend that are applied to particular sub expressions. One on the way down, one on the way up. Right. In other words, you can pick the sub expressions. Like for example, in this case here, we pick the sub expressions. Boom. That's been applied to those. Now we can think of stage two, which is for each of those Fs, First replace with the descender. Okay, so here's a way we could do this. If we said this F of x blank goes to I don't know what X +1.

Do you mean?

Arrow. I certainly do. I see it doesn't sub do it. Okay, but the thing I was trying to do was I replace it with something. So as it comes up from the bottom and then whatever I replace it with, I then. Oh boy, this is confusing.

So I just pasted something into Zoom Chat, which is helpful to me. So. Right. I basically took the example, the first example only. I replaced E to the E to the E with hold of one plus one to the one plus one. Right. So the first thing applied to the whole expression.

Right.

Then it applied at the first level, which evaluated to one and produced a log of four. And then at the next step, it didn't see any of the ones or twos that were originally there. It applied it to the result of the previous step. So I found this. Right. Helpful and understanding how the evaluation is happening.

Right. Okay, let's go through this again. As you're saying, there is a one thing that it does is spray stuff on the way down, then it aggregates on the way, or it can aggregate on the way up. I'm not really understanding more that there.

Is no down until you apply a descending operator. The descending operator defines the expression that you're going to traverse down. And it need not be the one.

That you started with, as this example here shows.

You know, the expression you start with could be the number one. And the descending operator returns a pair of numbers.

Oh, boy. I think I more or less understand this. I don't think I quite have it in a clear way.

Right. What's not? What's not? I guess it decides to stop when it encounters a none. It's not entirely clear to me when it stops.

Right.

When it returns an atomic expression or a zero length expression.

Yeah. Which is the same way that map all. I mean, it could also be given a level spec that it would bounce from that level.

Specifically it does. Here it's saying infinity. You could say three instead of infinity.

So let's do a slightly more realistic example. Okay.

So I claim and three really means three steps because. Or three down steps.

Right.

Because you. Yeah, those levels may have nothing to do with the levels of the input expression.

Okay, where's that? Where's the data? Oops. The data, somewhat peculiarly, comes first. Okay, so let's do. Okay, now my idea was on the way down, I want to add one, add to 2x plus one. Okay. On the way up.

I would suggest to add a head. Wrap the 2/7 +1 in H or something like that so that we see where it's placed.

Hold on, I've got a different idea. Oh, boy. Okay, who understands this? The down is showing me the output, right?

Yes, yes.

So down actually returns an array of the same depth as it's what it's given.

Okay. Maybe we should be echoing the thing that was given. Actually, we should probably echo both. Can we do echo evaluation or will that confuse things totally?

I would just put echo around the slot.

All right, Echo label. Well, let's just do an echo because we don't know where it's going to be.

Well, those are going to be actually the same thing.

Yeah. Okay, so this is a down. It got to the bottom, started recursing back up.

No, where your cursor is.

Or do you mean top? You mean. Yeah. You're saying this is at the top?

Yeah, this is one.

Right. And it evaluated. Right. It did two of that plus one, as we can clearly see. Or just.

Yeah, okay. Yep.

So that was zero.

Right.

And then it. So then it took that, used that. Or I guess it went to. Where is that next input?

Come on. Oh, no.

It goes one level deeper and it finds the first matrix, the 3, 5, 7, 9. And then it applies there, the down operator.

Oh, okay. There's. There's an extra set of braces that I didn't see. Okay. Yeah.

Okay. Oh, my God.

And there.

So you see 23. When you applied down to 23, it gave 47. So that's how it knew to stop. Because 47 is a raw expression. So that means 23 is the last result.

Yes. And now it comes back up in your up.

Can you change slot to slot 2? Which echo at slot should be slot 2 here. Yes.

What the hell is slot 2?

The first argument is the original sub expression. The second is after it's transformed by down, I think.

Right?

Yes. The first argument is the payload.

Maybe we want to print echo. Both of them.

Right. So look at the 23. Down gives 47. So 23 is a Leaf.

Just. Let me just understand the whole result for a second. Just understand what the hell.

The whole result and does down have both a pound one and pound two or only up?

Only up.

Okay.

And only for the non leaves. The non leaves is just a single argument. The leaves is a single argument.

Okay. So it doesn't do the most obvious thing.

Well, because it's applying the operator multiple times as if. As it goes down.

Right, so. Right. So. And then when you say original, does that mean like referring to parts in the original input expression or like what.

Can you do comma H after infinity?

What does that do?

So H is applied to the same expression that the down is applied to. And it gives a payload. So the down. And so the down is applied to an expression and gives the children. The H is applied to an expression and gives its payload. And both of those things.

So we're seeing those. Right, right. So the. So because you added the H. Right, that. And we're. We're seeing the H in the up labels there because we're echoing £2 rather than £1. Yes, yes. Right. But I, I still don't understand what Exactly. Where exactly £2 comes from. Is it like the last thing before this thing, the last down was applied to it?

Well, so you apply a down and you get some expression and then you replace every, every sub expression at level one of that result with the result of recursively doing expression transform to it. That result expression is. Is £2. It's the result of down after you have recursively applied expression transform to each of its subparts.

Is Christopher following this?

This seems pretty confusing.

Yeah, yeah.

It's still not entirely clear to me what £1 and £2 are. So £1 is. Right. And like when I go to. Right. So like if I'm at the next to last line. Well, actually, Stephen, can you change that echo at pound two to echo at list pound one two comma pound two. Just.

So that's going to give completely different result. It's not. You don't want the.

Oh, right.

What do you want it to return? We can, we can just echo something and then return something. We use pound one as the label for the echo.

Like the two argument form of echo.

Oh, yeah.

So pound.

What do you want to do Thing.

You want to return and then use.

£1 as the label second argument. Like that. Yeah. God.

Right. 7. So 7 11. You see the h of 711 that corresponds to the 711 that you see above. Right.

Okay.

To which down had given 15 comma 23. The result of recursively transforming 15 and 23 is given by the list h of 15 comma h of 23.

Okay, so let me rephrase this in my own words. So when we hit an atom or I guess a lengths of list or an expression of length zero, we just get the result and we're done. Is the up even applied to it? Because I don't see any 47s here.

Only if you do a list. In the third argument of expression transformation, by default, identity is applied to the the atoms.

Right.

And then, although they need not be atomic expressions, they're just the. The last thing.

Right.

And then, so for, then for each, I guess, output, we from down, we. We pair it up with the input that produced it and we wrap those in H and then we apply our function.

Yes.

Okay.

And that's what's going to ultimately be. And the result of that function is going to be what ultimately ends up in the result.

Yes.

Okay, I'm fairly confused. I mean, the big picture, we still think it's map produced. Basically it goes down, it sprays functions in, it comes back up, it is progressively aggregating as it goes down, it's progressively applying functions to smaller and smaller subparts. Right. I mean, let me just understand something here. Right.

And. And I guess it really is important to talk about these as steps rather than levels because.

Right.

There are no lists in that final result because it sort of, I guess, bubbled up and gathered them together with H and R Total. Oh, there's a total there. So total will flatten it no matter what. Okay, so I don't.

Sorry.

Yeah, so I'm not sure if there will be levels or not. I guess I was confused by the total.

So I would say the one difference with, between this and both MapReduce and Query is that in like MapReduce and Query, the descending operators are like filtering and sorting an expression that you have here. The descending operators, you know, it's the difference between structural recursion and generative recursion. If the descending operator can return an expression with new sub expressions and you recurse through that, Which I don't think MapReduce does and query does not do.

Okay, let me just understand. Map all goes first down to the leaves and then it comes back up again. Right? That's what we're seeing here.

Well, I mean, no, what we're seeing is, right, Map all sticks everything in at all levels and then normal evaluation occurs. So since there's nothing with a held attribute in here, then leaves get evaluated prior to the express, right. It's heads, then leaves, then the down values is the standard order of evaluation.

I see. So you're claiming the echo doesn't fire because it didn't have. It didn't fire on the way down inside map, because MAP was just constructing a structural thing.

Correct. And in particular, if you, you know, put a hold at. At in front of say the three, four, then you. You won't get any prints from any echoes from. From there you'll see the literal echo inside of.

Oh my God, what's that? What the hell is that?

Well, it went inside everywhere, at every node, it inserted the function. But once you're inside the hold, it's not allowed to evaluate. So you get ever more compound heads.

Oh, gosh. Okay, okay. So the way I should think about it, even though I do think a map was going up from the bottom, that's only because of the evaluation thing. But it is going down from the top.

But again, the evaluation, it's structurally recursing through an expression.

I mean, since I happen to know what the C code is like. Right. It basically goes down each level and sticks things in until there are no more levels to stick things into.

But.

Right, that's irrelevant. The evaluate. Right. It never allows the evaluate evaluator to fire, of course, unless there is an association in there, as we discussed earlier today.

Yeah, yeah.

Once we have immediate evaluated, then it'll be more interesting to see what examples like this do.

I think the way to think of this is that MAP polish is insert all and then evaluate all while expression transform inserts and evaluates at each level. And each level means the levels we have right now, which might be different from the other ones because the previous evaluation changed the levels.

What previous evaluation?

So you evaluate, you insert an evaluator at the first level. That changes the whole expression below. But the new expression has levels too. So now you go to the second level of the new expression and insert and evaluate.

Really? No. How can, how can that work?

I mean, so I think it's the difference between structure, like structural and generating. So it's not that you start with an expression and that you go through the levels of that expression, it's that you start with a seed value and you're creating an expression with levels variables. But that's just a seed value. You don't. You care nothing about the structure of that expression. The structure that you care about is purely the structure defined by whatever these descending operators.

So then is the claim that. What this expression transform is doing is that at step N it applies the function to. Only to the expressions at level N of whatever the last generated thing was.

No, it's always at level one of the previous generated thing.

Okay, which is why I go back to saying it's not useful to talk about levels. We should be talking about steps.

Right. The comparison with levels only makes sense with trees because it's not the levels of the expressions we're not going to solve.

Let's understand what we're trying to do here. You know, for many years we've had the ascending descending operator concept in Dataset. I have never been happy with the. The idea, with its generality or not. This is an attempt to modularize and generalize that functionality. Fair statement?

Yes, I think so. Here you know, whether a given operator is ascending or descending, as opposed to.

Reading the novel at the top of the data set ref page and trying to define what will happen.

Right.

And query can only do filtering and sorting operators, not generating operators.

What's an example of a generating operator.

Do? So do expression transform.

Yeah.

First argument one, Second argument do list of two slot comma two slot plus one ampersand.

This is so horrifying. Comma G comma 3 comma what?

Comma G comma 3. Yep.

Can't even imagine what this is going to.

The whole point is it's. It's generated a tree and, and folded.

It.

If you like. The thing that you're just. You're traversing is the one.

If we do tree of percent will we see a. Nicely, a nice G.

Do tree instead of G. Literally do expression transform with tree instead of G. This is basically how nest tree is implemented.

Ah, okay. Yeah, this makes sense.

This is the thing I'm claiming query cannot do. This is not sorting or filtering the one. It's generating new sub terms on the.

Way down or on the way up.

On the way down. The way down is defined by that. That pure function.

Okay. And on the way up. And we're wrapping these in tree. So eventually it wraps it in the root and tree and we get a tree out.

Yep.

That's a nice example. Let's go ahead.

As this goes down, it's finding levels that were not there at the beginning because we didn't have three levels in the one. But as this goes down and generates new levels, things get applied to the new level.

Yeah. All right. This is beginning to make a bit more sense. It goes down, it makes an expression.

Or changes the existing one.

Iteratively in this case, I mean, it just. Every subpart is being, you know, split here.

And that's why the ascending operator takes two arguments. The original expression that way that you have a payload and a children.

All right, we're getting there, but we ran out of time. So to be continued another time. And you know, the more clear examples you can get, the better. With some echoes and some trees and so on.

Yep.

But I think this is, you know, look, I have to say this is one of these, these things. I'm remembering all the different functional programming constructs that we came up with. I mean, there was a time when even fold was hard to understand, which it isn't anymore. And we'll get this one, this one will become easy. Just isn't yet, at least for me. And then, you know, we already know from data set that this is a useful kind of operation. All right? So for another time. All right, thanks a lot, people here and people on live stream. See you all soon. Bye Bye.