Live CEOing Ep 834: Language Design in Wolfram Language [CasesWindow, Unthread & More]

Okay. Hello, everyone. Welcome to another Wolfram Language design review, part of our incremental language development series. Okay. Cases window.

Right. So sometime in the past, we had talked about functions called takebefore and take after, which were somewhat complicated by the idea that you had to use a relative part spec. And so we had generalized the idea to this notion of a cases window function, where you basically say, you know, I have this expression, I'm looking for this pattern, and I want a window around where it occurs.

I mean, my gosh, you know, we could have map, windowed map, and things like that too, if we, I mean, you know, there's all kinds of windowing stuff.

Yep. No, I, well, I mean, windowed map would be something quite different. Right. I mean, of. Right. Because map applies to everything in, in the list. Right. Whereas cases only take certain things from.

Are you saying basically block map is what I'm saying. Window map is.

Possibly. Wait a minute.

Why does that happen? Oh, I see that. Does it do this? What do you know? That's what I would expect. Okay. Okay. So now I don't understand the cases. I mean, does it reject the case that doesn't have enough padding around it?

Well, so that, that's one of the design questions. Kind of COVID a couple of things I've found just in toying around with putting together what we had previously proposed. You know, there is the, there's a trivial case or string version of this.

I'm just not very convinced by this whole thing. I mean, I don't understand why you're not using sequence cases if this is what you want to do.

Because, because sequence cases is right. It's matching against the whole sequence. And here you're, you're looking for context around the thing you map, which I guess you could do, but like that.

Yeah, I think that's much better defined because that way, you know, if I wanted this to potentially be, I mean, if I wanted. How would I do that? 01 repetitions?

Well, you'd use repeated. Right. Repeated blank lists. List 01 in the second argument.

Right. What did I just get there? Oh, I see. Because it matched. It didn't refer. Yeah, It preferentially matched the zero case. So if I did this, wow, Can I do this? 100.

That's a range. No, it's a range. Right. That's min and, and max lengths.

I'm surprised we don't allow alternatives there. And it's arguably something we should do. I'm not feeling this function. I'm sorry, I, I, it just seems really hard to define, you know, with the end cases and all that kind of thing. And, you know, if we have this one, there are 25 others, or at least half a dozen others that we would also think we wanted to have. And I agree that being able to, you know, in a supplied world, being able to pick up the flanking things, even in a map, would be very useful. But I think that's sort of a secondary thing. And I think if you want to do it as primary, why don't you use sequence cases? I'm not buying any argument from me.

That we need this function, so.

All right, okay, let's move on.

Sorry, just one comment. I think what you were looking for before is overlaps goes to true. Because in sequence cases, overlaps by default is false.

For this one here.

Yes.

The second one. Yeah, the one below.

Oh, I see. But even. Well, that one.

Right.

Oh, I see. I didn't even get everything for that one. Right. This is a good example to put in documentation.

Yep.

Okay. All right, let's continue unthread.

Nick, you are muted.

Yep. Yeah, sorry. So, yeah, again, as it said, this is effectively a partial inverse of thread. You know, we. Not everything can be unthreaded if it doesn't have a thread like structure. There's a bit of a question about, you know, how we handle the fact that if you provide thread a single argument, we basically repeatedly insert it. We might have an option or something like that in unthread to condense or preserve repeated elements, if that makes sense. So that final example, where.

Is this actually useful, this function?

I found.

I'm not sure I've needed it. What's that?

Yeah, I found myself having, you know, I'll have data in some form where I've got a head that is effectively threaded through the data and I want to. I go through some hoop of, you know, transposing it and removing the head to get back the. The structure that I was expecting previously.

I mean, if, if you're, if you're avoiding evaluation leaks, then a pretty common operation is to have a list of held expressions that you replace with a held list of expressions. Obviously you can do that with replace, et cetera. It's certainly not an insurmountable problem. This is potentially useful, but I'm not sure we have to have.

It looks to me like a WFR thing. Or did it come from wfr?

It came from wfr.

Yeah. There's a version on wfr.

Yep.

So I think this operation is interesting, but I would have expected it to come from transpose itself. And in fact, we had added an option called allowed heads, which when it's fully extended at the moment, it only works with associations and lists. Then it will be able to do this sort of operation.

Yeah, it would be able to do this, basically. In other words, this will be a transpose with a loud heads, including.

List, list command. Exactly. So at the moment it doesn't work.

I think this is the right way to do this. That's exotic. Yeah, yeah.

Now it's because I didn't think to make sure that the second argument is not a option before typesetting it. That's a binder typesetting buglet that I will. That I will endeavor to fix.

But the operation is interesting. So there are these cases in which people want to do lists of geopositions or vice versa. So one direction we do threat, the other direction it's multiple.

I see what you're saying.

Is there a technical issue with why arbitrary heads don't work, or is it just something that hasn't been done yet?

In transpose, you mean? Yeah, in transpose, the main priority here was lists and associations, and that case works. I think we just have to work on extending it to general heads.

Right. And that's. Right. It's not a small. Right.

It's.

It's right. Even Rob couldn't do this in an afternoon. Right. This would be real work to handle general expressions.

Yes. And general permutations.

Yes.

Okay.

So the case of your position is handled by your position itself. Your position of a list of your positions will do the right thing. So something like this is not urgent from that point of view, but it's an operation that users want to do.

Right. But it's. It's seems difficult for many things that listify automatically. I mean, there are either things that work just fine with listed inputs, you know, like. Like, for example, if I do, you know, graphics of disk of list. Actually, disk doesn't work well for that. But if I say point. Well, no, any of those. Those things where I'm, you know, if I do dis of, you know, I.

Think this has been extended.

Yeah. I mean, that works fine.

Yep.

But I mean, it is in fact more efficient if I do. If I threaded this, I don't know what I'd get. I might get a total mess here. Now I get that. Okay, so this is less efficient than this. But I. I think that's a detailed implementation issue which we should fix separately. Am I making sense?

Right, but I get your point. If the users had A million of these. How do they go to the. To the single disk?

Yeah, but I mean, that's only the fact that they need to. Seems like a silly implementation issue. We should either do it automatically if we recognize.

I mean, that. That that won't. Right. I mean, like I said with the held expression case. No. Right. I mean, you want to be in control exactly when things move around.

Yeah, but that's a more sophisticated case and I mean, honestly, we need to come back to our held expression manipulation discussion of several years ago. Okay. All right, let's keep going.

Yeah. So we have this internal function already called effective precision. And it differs from precision in that it is specifically interested in numeric functions. So when we look at, you know, precision is going to dig into the expression and look for numeric elements, and we'll get the precision that it's operating at. The effective precision will give us.

I don't understand. What's the difference between what.

What does the word effective do here? I noticed that effective is used in the definition of precision.

Danny, are you.

Yeah, I'm here.

What do you mean effective? Which. Which effective are you asking about?

This one here is going to say effective precision of this. Right. Which is a bad name for it, in my view.

Yeah, it should be something like numeric precision.

Well, well, that's not any better, right?

No, no, no. The issue is this.

Well, it is better because it's the precision of the numeric parts. No, hold on.

It's absolutely not the precision of the numerous.

Guys, it is because you.

You don't go inside the non numeric Q symbol heads.

That's.

Guys, guys, guys. Look, look. The point here is F shields the imprecision of its innards. Right. F of point one is still a symbolic thing, right?

Yes.

And that's not at all related to what I just said.

Okay.

That was dripping sarcasm.

Oh, okay. I thought it was the same thing as what you just said. Yeah, it was. Okay, fine. Okay, good, good, good. Right, so. So the issue is there are non numeric Q functions that shield things from. That sort of shield things from being a pure number for precision. And the fact that, I mean, does it even document. Does precision even document what it does in the case? I mean, precision for a list presumably gives the lowest precision of everything in the list. Is that correct?

That's correct.

Precision gives the lowest precision of any number in the expression. That's certainly how it's always behaved. Whether that's explicitly documented anywhere, I don't know.

Okay, so. So what?

I caveat that Machine precision no dollar sign is lower than anything.

Right.

Okay. Okay. So my suggestion would be to add a second argument to precision that determines what's going on. In this case, for example, like heads. Yeah, like heads or like, like a list of like an allowed heads. Some kind of allowed heads type option that tells it these heads.

Yeah, why not? Why not? Exactly an allowed heads option which would. Which, you know.

Well, you might want to include a whole class. So it could be something.

Right? Yeah, so. Right, so we, we could have. Right. The default would be allowed heads goes to all and we could have an allowed heads goes to automatic. That is whatever effective.

That's just numeric. In the automatic case it would be numeric functions only. Perhaps.

Yeah, that'd be fine.

Although a numeric function. I don't understand why a numeric function doesn't already. Hasn't already evaluated by the time precision.

Sees it because not all of its arguments are necessarily numeric.

I see what you're saying. Right, I get it. Right, so allowed head. Go ahead.

A sign of 3 point times x.

Okay, but in this model with an allowed heads option. I see with. Okay, so what you would be saying is in sine 3 point times X the default allowed.

That has an effect of precision of machine precision.

I get it. Okay. So the, the thing here that would be the default would be allowed heads arrow all because that's its default. But allowed heads are automatic. Would in this case give the same machine precision result.

I think I slightly disagree with those choices. So we have always chosen automatic as what the function always did.

That's not true. I mean that is that, that is the common case, but that is certainly not true.

So what's an exception to that?

Propagate. So propagate boards option would be a bit of an exception.

So allowed heads has named families like list like. So we could have a family called numeric which might be useful in other contexts.

Yes.

Yeah.

Anyway, I think where we should go with this effective precision thing is an option to precision that makes sense to people. Yes.

Yep, yep.

So details to be worked out. All right, let's go expression to function.

So this is basically a utility function for taking a non pure function and turning it into a pure function. It was mentioned that this had come up, I believe in a previous tabular meeting.

Yeah, yeah, yeah. I've certainly encountered this a bunch of times. I don't know whether it's semantics is even vaguely clean.

Yeah, I was trying to get a handle on, you know, there are going to be edge cases that are going to be thorny, if you have a function which itself has a pure function in it, how, you know, do we handle?

Well, we then expression to function and function to expression.

Since you have to specify the variables anyway, how is this different from just wrapping it in the head function and evaluating the expression?

Because function doesn't evaluate.

Well, I mean, you wrap it and evaluate the second argument.

No, you can't do that because function is hold all complete. But you could use a width.

It's a width and a function.

Yes.

That's horrifying. I mean that's a good reason to have this function. If the. Okay, so let's say.

Right. I mean, if. If. Since you're going to want to evaluate the argument anyway, the way I would do this is apply function to the list var comma expression.

Yeah.

It'S hold all. It I not hold all complete.

Yeah, I've used evaluate and function many times.

I haven't done that, but I obviously should have done. Well, maybe I have. I have no idea. Huh.

It is hold all. Ok, my mistake.

Okay, so that gives that. But if I say this. Yeah, but by the way, things like this, we, we really, we're trying to have a better tip of the day type thing. I mean this should obviously be on the documentation page, but this should be one of, you know, 10,000 tips that are out there because that's obviously, you know. Okay, so that's how you go. So I don't quite understand why function to expression isn't just apply the function. Or even. Yeah, I mean, I understand. What. This doesn't seem useful. Maybe I just was being silly here. Yeah, I don't understand the, the. Yeah, I don't understand why this isn't. As you said.

Does function expression just return the body?

Say that again.

Does function expression just return the body?

Yeah, yeah. I mean the question is, is that accessor. I need that so rarely. I, I just don't think. And usually I would apply the function to something to get, you know, if I'm hacking, I would just say last of the function. It's not lost, but. But part two of the function.

Right.

I don't think this is a thing.

Right. Which. Well, which I mean it, it. It's up to of minus two. Right. Because it could be a single argument function. But.

Well, what would you do in a function expression of know if you say function expression of hash plus one, what do you return?

Well, I would return hash plus one.

Yeah. Which is not very particularly useful.

It. It can be if you're building your.

Functions sure, but by the time you're doing that, you know how to extract those parts, right?

Yeah.

Right.

Yeah, no, I def. I definitely do it by just applying it to variables because then I know what, you know what the variables are to further manipulate the expression. All right.

I don't think this is one association through.

So this is basically a version of through which produces an association and has a nice utility of being able to label the keys. By default, the key is going to be the function being applied. You know, I often will have a data set and I do something like mean, median, standard deviation and just positionally have those as lists and have to keep track of them sometimes create associations using something like association map instead.

So through the ordinary through would apply to this case here, right?

Yep.

How the heck could association through. I mean, if it's going to have a name like that.

Shouldn'T this be association comap at this point?

Yeah, that's what I would think.

Yeah, I guess that would be a better name for the argument structure. I mean, not a lot of things have through like argument structure. It'd be slightly.

This isn't just a through because you have a list.

No, it's a comment.

Well, through through is a comap. But association through I don't think is. Because it's giving the list min, max, standard deviation and comap. You'd have to do something like the association min goes to min, max goes to max, standard deviation goes to standard.

Deviation and then comap that it extends the analogy. Like we have association map for map. Why not have association comap for comap.

Where what does association map do? It does this.

It gives the. The list A goes to F of A or the association A goes to F of A, etc.

Yeah.

Why wouldn't association comap do exactly the right thing here?

Yeah, I mean that would pretty much give us.

Sounds like you're suggesting the same thing. Maybe it's good.

So right now fgh there on X gives that. Yeah, I think it's the. I think it's exactly right. It's association comap, the direct analogy map.

But. Oh, right. Yes. And then if you gave it the list abc, then it would comap would give you. Can. Can you put. Replace X with the list ABC just so we can convince ourselves that.

Yes, but this is going to be G of X etc etc. Etc. But here in that comap, if it'll just put ABC in the arguments of F like that.

Yep.

Yep. Association comap makes sense.

Do we?

Like.

So under the. Under the examples section, the the second one, there's this. Or, sorry, a little farther up, there's this. F goes to F as a string labeling mechanism that I found kind of useful playing around with it.

We really have an analogy there, but I think, I think it's nice. I don't see any problem with that. I think it's a little bit of an abuse because that use case would.

Be more appropriately handled with actual comap.

Should it be reversed.

Yeah, you would. You would reverse it and instead of being a list of rules, you'd be an association. And this was be literally comap.

I see. So what you're saying is it will be comap of the association. F goes to F, G goes to G onto X and just does the right thing. Yeah, that looks right. It should be an example. Okay, so then do we even need the label ability in association command? No, we do not, because it's going to confuse them.

Yeah, yeah. It would make functions that are rules getting applied a little trickier too. So I guess not functions that are rules, but heads that are.

I mean, we could have something where you have an outer rule outside of the list fgh, but I don't see the point of that.

Okay, yep, association comap seems straightforward then. Right. These are some utilities that Danny put together for converting to and from.

Hex.

String, converting reels to and from hex strings using specific exponents and bit totals. We don't really, I mean, we have various conversions of integer string and integer name. We have real digits. That doesn't quite do this. You know, it doesn't really have the same argument structure for representing.

Um, Sorry, go ahead.

I mean, unless you're teaching a course on IEEE arithmetic, why would you want this?

Yeah, that's what I was wondering.

I think it was more of a. You know, this is not entirely uncommon notation for reals. Right. I mean, we, most of the time that we're storing them as data, we're storing them as doubles anyway or as floats.

Okay, question, question, question, question. That in, in the raw data formats, if you're writing an importer for random, you know, hcf7 or something doesn't exist. But, you know, is it, is this going to be relevant for that or not? Is that where we think this is relevant? Because I, I, you know, the actual bit pattern, I mean, we can. How do you get. You can get the bit pattern by saying real digits, comma 2, can't you? I mean, what, what? Okay, let me understand something here. If I say real digits 1.1 common. Okay, that's the base 16 version, is it not? That is the hex string. And if I say from digits. Bluff, well, it's probably not wrong.

Right. Well, I think, I think the, what you're not getting is you're not getting the exponent in the bits. I mean, I presume that what, what Danny's thing is doing is it's taking, you know, the 64 bits of the float and representing them as hex patterns. I mean, Danny is welcome to speak up and, and consider.

Yeah, that's what I'm doing.

Yes.

Right. Are you sure this shouldn't be like an import string, you know, case?

I, I was just going to say I've needed something like this for byte arrays before. So if it was import string import byte array, it would work there as well.

Yeah. Where import byte array has a bunch of hex numbers in it.

Yeah.

Presumably you'd want the hex to string to, to real for that case, not the real to hex string.

But yeah, yeah. But if you want to export a byte array.

Right, right, right.

Well, okay, so.

Right, I'll accept. Well, except possibly in the compiler, right, we don't have the cast double to, you know, byte array of 8 bytes function.

Right. So I mean, the possibility would be a hex string import format where we say import, you know. Well, you have to say what we're importing it as. I suppose we want import this, You know, and the import element. Then this would be hex string and then real.

This is import string, right?

Yes.

I, well, right. Well, this is assuming you're starting, you have the string. Right. I mean.

Or byte array, I mean the other one.

Right. But this is not what Danny's function is doing. Right.

What is it doing?

Right. It's taking a real and converting it to a string.

Yeah, I understand. So that's.

There is the opposite.

There's both versions.

Yeah, there's both.

That would be export string. Right. Of 1.1.

We don't need the real there, we just say hex string.

Okay, so what, what does hex string do with. I know, a list of things. I, I, I don't. We have to define all a well defined format.

I agree. It's not necessarily a well defined format. But the useful case of this would be there's an array that was dumped out in a, you know, literally dumped out from memory and the array has a bunch of real 64s in it. I mean, I think this is kind of complicated and I'm not quite sure where one would use this. I mean, this seems Like a, I assume this is a WFR function. Yes.

Yeah, yeah.

Jamie has put it in the WFR also.

Yeah, I'm not, I'm not seeing this as a big. I mean, I, I can see cases. Look, if this is a building block for an importer, that's one thing. I don't know if that's what it is. What is it? I mean, why, why are we looking at this? I guess.

Well, I don't. I mean, the question is which, which people raised at the very beginning is, is it just to niche a function to want to promote?

And. Right, and. Right. You could do export string of 1.1 real 64 right now, which will give you the bytes. Right? A string of the bytes.

Right.

I, I was approaching it more from, you know, the angle of. We have various integer string functions, integer string and integer name, that use comma and integer representations. Is there a case to be made for having different real string representations? I mean, I don't know that we're going to have real name.

Yes, one, one particularity of this, which I think Etai brought up earlier, is that this is an attempt to match IEEE arithmetic as best I was able.

And what about endianness and things like that?

It's not mucking within it. It's, you know, It's a straight 4 byte or whatever. I, I don't do anything with byte reversals.

But, but, oh well, no, but you're not, you're not giving what's actually stored in memory. You're giving. Right. The, your most significant bits are appearing on the left, which is different from how this is stored in memory. So in fact you are, you are in fact doing something with the byte reversal.

But I don't understand what's useful about this. I mean, if it's. There's one thing which is I'm going to use this for an export format. I'm going to put it out in a certain form. Then I need to worry about endianness and God knows what else. Well, maybe I don't need to worry about endianness because I'm just doing it myself by hand, but maybe I do because I'm trying to feed it into an array to an existing format that takes account of endianess in a particular fashion.

No, I, I think this is doing the right thing in the sense that, I mean, the string that you get is the correct string that's not in end independent as far as I'm aware.

Well, it depends on your, on what it is you're interested in. Are you Interested in. In the sim, Stephen, I put a.

Yeah, I see that.

Yeah. Right, so you see that he. His. His output is reversed relative to the actual bytes in memory because we're on a little Indian machine.

I don't even know what. Don't, don't. Aren't all machines now the same endianess?

I love machines.

What's that?

Most machines.

Most, but not all.

I. I don't even. I mean. Okay, fine.

ARM. ARM CPUs can operate in either little Indian or big Indian.

Oh, brilliant. Okay.

And. And power machines, they. For a long time they were big Indian. I think maybe now they. They can be either. I'm not sure about power.

What's power? I'm sorry, I don't know what that is.

IBMs.

Oh, Power PC, then I'll call.

Well, it's no longer called Power PC. It's now called Power.

How confusing. Okay, and.

And in all caps.

Okay, okay, okay. Packing more power by having a capitalization of its name. Okay, yes. All right, look. Okay, what is this useful for? I mean, if it's useful as part of a collection of utilities for importing and exporting raw data, then let's make sure we understand what that full collection is. The truth is that we've never done much with byte arrays. We've. A few times people have, and I don't know what there is in the function repository for byte arrays. Is there something that will take, you know, we'll get the strings out of a byte array. I know people have built that functionality.

What do you mean we'll get the strings out of a byte array? It's called byte array to string. It's been in the system since at least 12, if not 11.

What do you mean? So if I say, how do I get a byte? How do I get. If I got a core dump, for example, how do I get. Is that.

That.

That's the case. Byte array to string. So that. That is the equivalent of the strings function in unix, is that right?

No, that is the thing that will convert the byte array to a string.

Okay. What I was asking for was the UNIX strings function, which will try and find reasonable length strings in a byte array, for instance, kind of forensic byte array stuff.

Okay, I guess we don't know what you're talking about. Yeah, I know what you're talking about. I'm familiar with the strings function. I hardly ever use it.

No, and one uses it when one's trying to. Okay, so here we have. I'm super surprised we don't have this. Words from Executable is similar to what you're describing.

Yeah, that. That. That sounds like. By. Right. A string. Like the UNIX strings function.

Quite. Why this word cloud has overlapping fonts, I have no idea. Somebody should report that bug. How could this possibly work? Does it. Well, I don't know. There's a dot.

Right. Well, it probably uses read to byte array to.

No, no, no, I know that, but I'm just. The exe file.

Because it was run on Windows where that would actually work and return a. An executable.

Oh, this is a wfr. I shouldn't have. My standard shouldn't be that high. That Bitcoin blockchain1 is very similar to.

Yes, I have to go to another meeting.

Okay. All right. Is there anything else we're covering here? And Nick, perhaps you could. Can you send me something which is the. I. I just feel like there are important issues that have been sent to the. Is the list which I'm not seeing showing up here?

Yeah, I mean, some. Some of that is delegation between ILD and lcc. But yeah, I will. I'll send you the. The parent tracking.

Can you mention some things that you think are more significant that we're supposed to be showing up here?

Yeah, I mean, there. There's some. There's stuff that's pending in graphs with ports. There's stuff that's pending.

Yeah. So graphs with ports. Hold on. Graphs with ports. Arguably should be its own custom meeting. That's going to take a whole meeting.

It's. It's kind of been spun off. Ian and Nick Mertzen are meeting about it regularly and kind of road mapping out and also talking to Charles. Things like supplied have largely been spun off into tabular and kind of LCC adjacent.

Okay.

That's where the graphics summit also is handling. We have traditionally taken care of the error handling project and.

Well, where is it? What's happened to the error handling project?

It's somewhat in limbo because of personnel changes. It's on my list of things to ping people about.

Okay.

Yeah, I can. I'll send you the summary I have and then maybe we can touch base again on what we want to move up the queue.

All right. Because I think this meeting seemed a bit weak. I mean.

Yeah, I mean this. We're recovering from a couple of different application areas taking up time and some other things being outshined by other pressing projects.

All right.

Some of it's been delegating to other meetings and some of it is just. Some things are in the back burner because people are busy, but. Yeah, we will.

Fair enough. Okay. I just think we need some, you know, let's make sure we hit the harder hitting stuff, so to speak.

Right. We want to have some media discussions. Some of ILD is looking at small one off functions.

I know, but there's also just. But by the time there are lots that really aren't going to make it, I think. All right, good stuff. Thanks a lot.