Live CEOing Ep 720: Language Design in the Wolfram Language [ReapAssociation and more]

Having a couple of these examples.

Ah, okay.

I don't think keep cases is absolutely deaf. Devastatingly bad.

It's. It's one of the name options we have is a subsection there with a couple of other ones we looked at. Yeah. We don't like. Yeah, Sift was another one. I kind of thought of it might be a little esoteric and filter would work, but we have lots of filters.

Oh, no, no, yeah. Let me see. Sieve. That's a bit obscure.

Absolutely not Sieve. Number theory connotations.

I said sift, but yes.

Yeah, yeah. Sift is. I'm okay with.

Yeah. My only concern with keep cases is that as soon as we call it cases, we start thinking about patterns.

Which doesn't. Delete cases support the list structure too. I mean, Okay, if I simply say here, Cases.

Cases takes a pattern. Right. So you'd have to do alternatives. At. At. Yeah. Yep. So in this case, the same result. The same result. It would not be. If you want to preserve the positional cases that didn't match, which was where the idea originally came from was, you know, this is in some ways a filtering way to preserve the structure of the data. Whereas, you know, cases is inherently going to give you back a list, a flat list.

Right. The problem with prune is it's the wrong way around.

Right.

Okay, what is this doing? What is the.

Why not keep elements? Because you have delete elements that also takes a list of. But it doesn't accept a level.

So delete.

Maybe makes sense to extend delete elements with the level and also keep elements kind of would work the same way. So they would be.

Well, this is. Removes all elements, But one could give it a thing to put in their place. I suppose. I don't know why one would do that because one could just use replace for that. But.

But is the first argument being a.

List more restrictive than the first argument.

Being a general expression there?

Maybe so. But look, I agree, keep elements is reasonable in the sense that.

Matches syntactically what we were talking about.

Well, it doesn't seem bad actually. I mean. This funky syntax with N instances, one could even support here, Keep only up to that number of instances of each of these things. See what I'm saying?

Yeah, yeah. I can't think of an immediate use case for it, but we could certainly support it.

I can imagine. Once. But. I think this is a reasonable name. Now, in terms of the level spec, What does it do at the multi level case?

I mean, the multi level case is tricky because, you know, if you if you take the, the toy implementation I have here and you put in level two, it's just going to flatten it down to just two elements in, in this particular input, I guess three elements.

Sorry. I'm sorry. If you do what here?

So the, the first line I have there is a toy implementation of prune and if you change it from. In that matrix form example, if you remove the braces around the two, it'll be a multi level spec and it'll just flatten the whole thing down to three missings.

I'm totally confused. It's because it, because it looks at. Because it's like map. It looks at the higher levels.

Right. It looks at the higher levels first and says, ah, these aren't in the list of expected question what the level.

Spec is supposed to be.

Right. So if you were going to do this with a level spec, you know.

Is there a point in doing it with a level spec or can you just map this thing over the thing and get the same result?

I mean, you should be able to map it. Yeah, you could map out minimally, but yeah.

Well, I mean, why do you have to map out? Why isn't there just a map? Whatever you can do with level spec here, you can do with a map.

Yeah, that's true. Okay.

I don't think we should have a level spec. I think it's confusing, as you say, because it could, it could smoosh the. It could blow the thing up at a higher level and then it. Right, this looks good. Let's do it.

Yeah. One spinoff question then, is delete elements currently is set up for lists. I believe, I don't believe it works on arbitrary expressions. We could potentially extend it to. If we wanted to do that with keep elements. Or we could just say keep elements and delete elements only work on lists. Oh, actually delete elements does work on arbitrary expressions. It just wasn't.

Yeah, I kind of thought it did.

Yeah. I mean the list case is certainly the simplest case.

The object can have any head, not necessarily list.

Yep.

Okay. Okay, Easy enough then.

Right, let's go on merge and keys.

So this was a suggestion that Brad had and I can have him talk about it a little more, but looking to get some more information about how about the keys during messaging and whether this is something that could be generalized.

Yeah. And you know, I was thinking about it a little more today. If the keys themselves contain some important information, then when you're doing the merge, you could also be switching on keys exactly how you do the merge.

Yes. But what does merge normally take?

It's just the values.

Using the function to combine values. Well, why isn't this a supplied play?

Yeah, that would be kind of a natural place to have it there.

Because, I mean, there are all kinds of weird things like the position and the association index.

Yeah, right.

There are all kinds of things you might want.

I. I think the positions in the association, it would be a little bit lower down the priority list.

Yeah, I understand, but we. What are we going to do? We can't change merge to have that function take more than one argument. So what are we going to do? We have to have a thing which basically specifies some kind of handler keys or something.

Yeah, I guess I hadn't thought about the intersection of supplied and messaging, but it would be a nice way to effectively have that meta information available.

Yep.

Okay. Yeah, I will pass that along to Etai for the next discussion on supply, just as a use case.

Fine. You know, the fact that we keep on calling this thing supplied, even though it's had many other names is a sign.

Yeah. I think it's pretty intuitive what it means when we call it supplied.

Right. What were the other names we had for it, which we don't remember anymore?

We had including. There's some variation of includes.

Okay, well, maybe some others, but we should get back to talking.

Sorry for my ignorance here, but how does. How is supplied a workaround for this thing about missing keys?

Because supplied is a way of specifying. Look, the idea is that you say map supplied F comma and then this can tell you, you know, the index, the value. And this is then supplying to F both index and value. Whereas normally without the supplied F just gets the value. Get what I'm saying?

And I guess key could be in there too.

Yeah, exactly. So I mean, for example, or keys plural or whatever we can have named. The whole point is that in a given situation, like Map, for example, that you can put a supplied there, which, which has a certain known collection of possible values that it can fishes out of the map environment. Get what I'm saying? So here in this case with merge, we would say this, blah, blah, blah, comma supplied. And then the comparator, I'm saying. So you could just say supplied of this.

Right, Okay, I think what I'm getting here is that it puts more power into the user's hands about having extra arguments and then how to order them.

Yeah, yeah, yeah.

You add like meta information to the.

From the environment. It's information from the environment. In this case, information from the merge in the case of map information from the map, like, you know, what was the depth, for example, of this at which this, this F, you know, is, is occurring? Let's say this map had been a map all, but it would have different depth information, et cetera, et cetera, et cetera. It's like supplying environmental matter information into a function application.

And as to your question, Stephen, I think supplied is next up on the list for lcc. There have just been some scheduling conflicts that have kept us from getting all the important parties together.

Okay.

So we had started talking about this last time also the idea of modernizing reap or having a variant of reap that allows us to create an association. The main sticking points being, you know, basically we'd have a result key and then we would have the different elements that have been sewn with their associated tags. The single argument form of reap will grab all of the sewn keys, but you don't necessarily know what order they're going to be in. But the sticking point we ran into is what would we call the unlabeled or the untagged sewn elements? Anything can be a tag that is.

Actually documented as none.

I, I understand that, but the problem is if they're in an association and they're all called none, they will crush, they'll merge. I'm saying. Well, actually, No, I see, I see. What you're saying is this is the list of all things sewn with the tag none, correct?

Yeah. So the single argument version of so is equivalent to so of whatever, with none. I had forgotten about that last time we were looking at this.

Okay, fine. Well, then that's an answer.

So then the question is, you know, I don't know that there's a way to really integrate this into REAP itself.

Well, let's look at it. Let's look at reap. Let's see whether reap has a property option. Okay, let's look at these other cases. Reap reaps, only ones that match this. Okay, so that's like a key cases. What's this?

So I believe that one basically applies F to the tag and elements. I haven't really used that.

If you made F rule, it would almost be the same as this, except it'd be a list of rules instead of an association.

I think this could fit with a.

Fourth argument being association. And this would be a pair fg, like we use in distribution and things like that. Sorry, distribute.

Right. It's kind of. Well, if you had association as a.

Fourth argument, is there anything that would actually work for F.

Or would it have to be rule?

Right. It's almost like you want a magic string. Is the third argument that does this.

Yeah.

Otherwise I think it would have to be pure function slot 2, right?

Yes, something like that.

Well, that's an argument for reap association. Is that the right way around? Is that what will be association reap? No, it's reap association. It returns an association, right?

Yeah.

Or it returns a list that. The second argument is association. Depends if you want this result zone level or not.

Well, maybe reap association. I mean, it probably needs that pattern argument too. It doesn't need the F argument, but it does need the pattern argument, correct?

Yeah.

Yes.

But for the F, it could have a property argument.

On the the resource function. I did support that F argument, although it was a long time ago. I don't remember exactly the behavior.

Well, okay, so one possibility is there's a single association that has both the tag, the sewn tags and the results in it. Because I think this two level association is pretty ugly. The question is what's the name of the magic? I mean, because the tags can be anything, right?

Yeah.

And you do need the output of the entire evaluation.

Sure.

So you do need a. Something's wrapping the association.

Okay, so the question is, I mean, we do not want to have something where you have a list of the outer level and then it's an association that seems really ugly. I mean, the third argument of reap association could be a property thing that picks what form you want this output in.

Why does reap association doesn't return rule Deloitte for the tagged one for the result is it. It's a delayed rule, but not for the tag.

This one here or those. That one's there.

The none one. Maybe I like. So unrelated result something that I want to be unavolated. There is no easy.

Yes, I agree that's a reasonable point. But. Okay, so the way you use this is.

Yeah.

You'd say, you know, R equals reap association of blah blah, blah things sub result things sub. So, and I think it's okay. This looks fairly reasonable to me. I wouldn't put in a. Go ahead.

I like having the top level association like that because your evaluation might return something like nothing or a sequence or something like that. Which would be hard to figure out with a list.

Yes. Okay. All right, so I think we got another winner there. Association key deflatin. That doesn't sound promising. In terms of writing.

Yes.

It may be descriptive, but it's a little Wordy.

But there's an association word association needed.

Well, hold on, let's see what it does first.

I. I made it as an inverse for another resource function called association keyflatten. Or maybe I made the other one, I don't remember which one I made.

What does it do?

So it takes like a. It basically just creates a nest of associations from a flat association where your keys are just the. A list of the positions.

Yeah. So each, each element in the list in the key is another level of association that's been basically squashed in this case.

Yeah, I understand.

This is a reconstruct association from.

It's kind of a transpose operation. Except that it.

Yeah.

What's that?

Yeah, yeah, it's very much like a transpose association.

So we have an association transpose, don't we?

Yes.

What does it do?

I don't think we do.

Oh no, don't we? Okay.

I think if it's. I think if it's square transpose might.

Work on an association and general utilities. We have one, a rectangular.

I mean.

What do we have in general association transpose? Well, maybe this is that. I mean this should be. This should be a version of that where there's a level spec.

That that is more for like if you have a list of associations, it will give you an association where each item is a list and vice versa. So it's kind of a different thing.

And what is this doing then? I mean this is just exchanging list keys.

Yeah, this is more of. I guess you could call it like a key transpose.

This is kind of. I mean it's interesting. I just don't quite understand. I mean it reminds me of, you know, join across, which some people think is the most magnificent thing in the world. And most people don't understand what it does.

Yeah. So I guess a use case that I thought was interesting with this is we already have file system map which works on basically a tree like structure for a file association and creates these sort of nested levels of associations. And oftentimes you get things like file names that will be, you know, if you break a file name up into its elements, you'll get lots of matching lists that will be like this where each element numerically in order will correspond to a level of basically a flattened association. So if you file name split a bunch of files, you'll get a file structure or you'll get an association structure that looks a lot like this. Being able to merge it back into a single large or deeply nested association can be useful.

I Agree.

You know, if that's strong enough of a use case to have a system function, I don't entirely know, but I've seen data like this occasionally and wanted to basically squash it into a single case.

Here's the deal. Here's the deal. There are a zillion array and association, you know, array like list arrays, association arrays. There are a zillion of these operations which we kind of need this general formalism to talk about. Whether this is, you know, from string diagrams to this to that. There's just a lot of array gymnastics. And I feel like we don't have a good principled way of talking about them at this time. You know, we, we've had in the past a certain number of things like flatten and transpose and things of this kind of.

Okay, we have the option allowed heads and that is able to tell some of these functions you are talking about, that associations are allowed.

Yeah, that's pretty obscure. I mean, what I'm asking is, you're trying to rearrange. I don't have a good sense. And perhaps it would help if there were graphical representations of this. Of what, you know, this various array gymnastics, what they mean. And how do I, how do I, how would I ask for some, something like this? You get what I'm saying? I mean, in, in. Like in the modern world of LLM based programming, you might just give an example of what you want and it might deduce this is the gymnastics you're trying to do. I mean, in other words, there's got to be a way to do this that either you say, I understand I have a way to represent this complicated nested, you know, list, association, combo thing in a way that's some kind of diagrammatic thing. And I can say I want to turn this into that or some kind of symbolic, you know, almost type level representation of these things. I want to turn this into that. Or I literally just say I've got an example of this. I, I'm gonna, you know, I want it to turn into that by example. Do you get what I'm saying?

Yeah, like sort of coercing something into another pattern, essentially.

Right?

Yes, yes. Yeah. I mean, and. Right. But the question is how to specify that. Do you get what I'm saying?

Yeah, I know what you mean. It sounds difficult.

Well, right, but I mean, I don't know. Yeah, I mean that, that's what, that's what I'm, That's what I'm trying to.

You have a flat list of integers, for example, and you Specify a pattern that's like a list of lists with, you know, triple underscore or whatever. So you're just defining a pattern where it's a list of lists. Could we somehow infer that that should be partition or something?

Yeah, well, right. I mean, so this would be. You just give. Let's imagine that the thing that you gave was this. That's the target pattern.

Yeah.

Yeah. Well, that's an interesting idea. I'm not sure how much ambiguity there is in what flows where. In other words, if it is the case that you say I have a bunch of things and their flattened form is always going to have all elements in the same order and now I'm simply going to sort of cluster the elements. In.

I think in a lot of cases this would be something like find instance where there isn't like a unique solution to this. And we would just kind of have heuristics to figure out a sensible thing to do. Like if you didn't have underscore three times, for example, in that first list and you just had, you know, blank sequence or whatever.

Yeah.

Maybe we could just automatically try to make a squarish shaped.

Yes, yes, yes, yes. Yeah. But basically what you're saying is I'm going to make a picture of what I want now. Try and put it in that form. Yeah, I think it's interesting and I think that that has a much higher chance of success than to give another 18 functions that, you know. I mean, I'm not sure how complete and whether it's necessary to even make it that complete. Because as you say, I mean, in this particular case here, what we'd be saying is the pattern we want is blank arrow. Oops, blank arrow, association blank. Maybe we don't even need to, you know, whatever. Blank, triple blank or something pretty much. And then it's this thing here or dot, dot. I think this is an interesting idea.

Yeah.

I mean, one could imagine generalizing this also to strings and things. Well, go ahead. What you can say.

Yeah. I mean, it's a very fuzzy concept and, you know, I don't think we could get things exactly right all the time, but maybe we could make something that's useful enough. Most of the time.

I think this is the direction to go. And we should talk about this in connection also with this whole string diagram type stuff. I think, because that may be. There may be an actual type formalism. Okay. There are three. Three related things. Type test specifications. This idea before related things. Okay, so what do we got? We've got type specifications We've got LLM examples and we've got well, these patterns, pattern based structures and then we've got sort of the string diagram type stuff. Graphical representations that make sense.

Yeah, yeah, makes sense to me.

So our goal will be to bring those together with including. Yeah, no, that's good. All right, let's quickly look at some of these others. Oh my gosh. Yeah, it's sort of interesting.

Is this also related to. I forget what we were going to call it. Comap.

Yeah, it's somewhat similar to comap where you're basically applying a list of functions to an expression. In this case we're. We're getting. The keys are maintaining the, the head that was given or the decoration as the rule form goes.

But we didn't actually get what happened to comap.

It's in the queue. It's in Danny's queue. It just wasn't.

Do we agree we're actually going to call it.

Lowercase M?

I'm a little bit suspicious about that name. I think it's one of the.

We've gone round and round on it a few times.

All right. Okay, let's not do it here. Well, so if we talk about that again, let's talk about association through at the same time. Okay.

The last two are probably going to be slightly more complicated discussions but select positions we started talking about and we. Jose brought up this relationship that position is to cases as count is to cases.

No, I said it.

Oh, Danny. Okay, I'm sorry.

I don't know why I'm grabbing credit but.

Well, I'm misattributing credit but the idea was that having a, a test based version of positions is potentially nice, but if we're going to go into a test based version of that, you know, do we want a select count also do we want a bunch of select related functions? You know, I tried to do kind of a deeper dive on the pattern based functions and you know, for the cases that we're looking at, I think select position and count are the, the primary ones. You know, we're not really going to have a select replace where we're using a test based replacement because replacements inherently are going to.

Yeah, I mean look, we've had the code count by family, right. And this is now a thing a little bit like count by story.

It counts by.

Right, yeah, I understand.

Yeah.

But this is another thing, a little bit like by. It's another adverb type thing.

Right.

And I feel like if we, if we could come up with the right adverb. I mean, it's almost like positions by or position by count by. Is it. I mean, I guess most.

Yeah, the buy functions would have a test function. Right, Right.

It's a little weird. By isn't quite the right. But is there another adverb that we can use there? It's not adverb. I mean, another preposition we can use there.

Well, that would, that would also handle like giving. Select a level specification. We wouldn't actually need that because you could have cases by or whatever we would call it that would essentially do that, right?

Yes. Let's think about what that preposition could be. Okay, I want to make one general point and then we should wrap. You know, as you all know, I've been working hard wrapping up this LLM function, all these LLM related functions for 13.3. I do think that, you know, now that the aliens have landed type thing, we're, you know, there's just an awful lot of LLM related functions. And I kind of think that we should pivot a little bit in terms of these ILD discussions to look at, you know, LLM spin off functions, because I think that they are, you know, they have high potential in the short term. And I guess we can talk about this. You know, these things that I'm writing about these things and what's going into 13.3 is happening fairly rapidly. Probably by next week even we'll have some more information about this. But I, I kind of feel like I, you know, like I've. There's a, you know, the LLM function function is happening. You know, we know how that's happening. And I think there's another idea of an LLM block function which we should be talking about. So I hate to blow things up, but, you know, it's what we get when the aliens have landed time.

We would need an LLM blockchain. Of course.

It also might be worth looking through all the functions we have and see which ones are. Would be useful as utilities in that ecosystem. Things that have to do with templating.

And what templating or NLP stuff.

Right. Because there might be some of those in the ILD list already. That would be helpful.

And seriously, it might be a good idea to consider at least temporarily merging this with another group since you have people working on this that are not in this group.

Sorry, sorry, sorry. I spaced for a second. Excuse me, Looking at something else. Could you say that again, please?

It.

It would make sense to temporarily merge ILD with whatever group you already have. Not everyone is in this group.

No, no, I know, I know some are. But a lot of the thinking that you guys are doing here is going to apply, you know, to these other kinds of functions. Except that I think that they have more bang for the buck right now because.

No argument, but, but some of the expertise at the implementation level lies outside this group.

And I agree, I agree. But I mean, one of the things that's happening is because of LLM function and the related functions, a lot of the structure that has to be built now does not depend on detailed LLM knowledge. That's. That's. Yes, I agree with you in terms of these meetings, but I'm also saying that I think the good news is that, you know, sort of the generic ILD function building operation can be applied now to build out a bunch more LLM functions.

All the more reason to merge expertise.

Yeah, no, no, it is, it is. All right, sounds good. Okay, I think we should wrap up here and talk to you soon.