Live CEOing Ep 828: Language Design in Wolfram Language [Color From Expression and More]

Live. Hello, everyone. Welcome to another Wolfram language design review, part of our incremental Language Development series. Okay, let's take a look at what we've got here. First, color from expression.

Yeah, so I think this came up in another meeting, I think, related to the Tabular project, talking about ways to create colors from arbitrary expressions. So Bob has this function that's in the function repository called Hash Hue, and there have been a number of similar functions created by people over the years that all.

Well, I mean, how does this relate to our whole color palettes project of trying to get, you know, these families of colors that work and so on?

Yeah, I mean, so there, there's a subsection farther down.

I don't think it's at all related to that.

Right.

Well, I mean, the colors we want out of this need to be like that one. There isn't a winning color.

Right. There's a section farther down using themes to try and curate this a little better. The main trade off is the better coordinated the colors are, the less guarantee there is that we're going to have uniqueness for expert colors.

That's basically the things we're doing for plot curves and so on. You know, that's the kind of thing we should be doing here. This is the same problem as the. I've got 10 curves on a plot. What colors should they be? And we should solve it in the same way. Now, one other thing that is a problem.

I mean, there's an important distinction in that for the plot curves, you know that the inputs are just like an index, like, just like the number of the curve, whereas this one, it's an arbitrary expression, kind of like a half a point, right?

Yeah, that's a fair point.

I mean, I mean, that certainly is one very common application of a color from expression function. Look, the hashu is a different application, possibly.

Okay, look, I think that, you know, one place where this is really relevant is, you know, Let's say, Come on, what the heck? Oh, this is the prototype build, so.

Who knows, Might be a little funky.

I'm going to show you something. I type percent look at that ugly thing where it says temporal data. Yeah, that's. That's a bug that we should just fix. Okay, that's a mapping. But my point about this in general is this kind of like iconic, you know, let's say I make a plot, you know, You know, this, this idea where we like an iconic hash of what's there. This is something we built, remember, for the cloud. We tried to do this for Notebooks know, what's an icon that would represent a thing? I mean, to me that is more useful probably than a hash color. I mean, a hash color is amusing, but I think that genuinely useful would be an iconic representation of the thing. Do you see what I'm saying?

Right. An important question is if the two things are just slightly different, do you want the icon to be slightly different or radically different?

Right.

For my use case, when I created the resource function, I wanted them to be radically different.

What was the exact.

Right. So I wanted, I, I, you know, I wanted to look at a table of data and if there was some value that wasn't identical, I wanted it to be glaringly different. And a lot of times it was numeric values.

Right, well, how does that work with machine epsilon and so on?

Well, I mean, you know, a thousandth off, not a millionth or billionth.

Well, which, which one? I mean, you know.

Right, right. So certainly what does hash do with, with, you know, machine epsilon? Yeah, it's, it's. If it has a different representation of memory, it's going to have a radically different value.

I mean, I don't know what hash does. That's a good question. Let's see. I mean if I say four point plus dollar machine epsilon and I, then I say hash of that. Oh God, why am I getting an internal self test error for undo atom? Who knows? Interesting, right?

Because I mean, right. That you're adding it, you're adding it to too big a reel.

Right?

Oh, I see, I see, I see. If I say one. Yeah.

You need to add it to like one or, or something less than one.

Okay. That's pretty wild actually. What it's doing there.

Interesting that 3 and 4 were the same value.

Well, I know, but look at that. It's every, every pair are the same.

Oh, huh.

Possible issue for hash. I'm more interested in the. Can we make. I understand your point that. You just want to sort of color code. Wait a minute. These are all alternate versions of the same thing, is that right?

Yeah, yeah. There are different versions that sort of fiddle with those parameters of how far apart you want things based on minor differences and how visually distinct they should be.

Right.

And remember, Stephen, that one of the ideas where we discussed this is things like, you know, asymmetric key pair or.

Sure or no, that's been successful.

Or even. Or, you know, I mean, imagine a rank five sparse array.

Right.

You're not going to have an icon that represents the content.

But what we've done with the. What am I saying? What we've done with like the, the cryptography stuff is it's not just a color. Right, right. We can tell those are different and if I do. Yeah, I mean, you know, we can tell they're different and that's certainly a way to do things. I mean we could do it with pure color, we could do it with this type of thing. I mean, I feel like this isn't our highest priority thing to solve. I mean, we've got solutions in wfr. Why do we think there's a system function in this? It seems like there are just too many different cases. You know, I would like a way to thumbnail an expression that would be helpful to me, I think. Maybe not, I'm not sure but, but you know, what you're imagining is that it's a style. I mean, you know, really it will be something more like know hash style of EXPA would display Expo in a style that depends on the content of Expo. Is that correct?

Yeah, I'm thinking, I guess the, the, the way that this had been presented to me specifically with color was related to things like it highlighting differences in, in data frames or something.

I understand, I understand, but I mean, but, yeah, right, but, but that. How are you going to highlight the difference? I mean, if you, if you wrap hash style around, You know, then that would do it for you. I mean, in other words, in other words, then you'd get everything that's here. You just, you'd say for that column. I assume there's a way of specifying here how to format that column. There is, I think. And then you just say hash style is the way you're formatting it.

Yeah, like in, I guess a slightly different version on the link where it says wfr. Bob has an example that uses prime data where it's basically so it's in the subsection header for qhash. There's a link to WFR and I think the first example has a.

Mel. You send me a link called for it. Why doesn't this allow me to copy the link? That seems to me I was having.

Some issue with copy and paste in.

No, no, no, look, look, that's a bug that should allow copy link H. Okay, yeah.

Can you note that. Carla?

Yeah, yeah. What?

Sorry, I was. You right. Clicked on a link and it didn't allow copy. Is that what happened?

Yeah.

Okay, that sounds vaguely familiar, but maybe not.

I, I just don't think this is a good idea. I mean, I just don't think this is a. I think it's amusing but this is not a good met. You know, this is not a good idiom.

Yeah, or like the, the first example under scope was the one I was thinking of. Sorry, not the. So we have this version where we have a bunch of crimes and you know, you could apply, instead of replacing it with the color, you could basically do this, you know, crime description of apply hashed style. And then you get a visual. You know, you may not be interested on, in exactly what the crime description is, but you know, okay, these are, these are grouped together.

I think this is a good WFR thing. I, I'm okay. From a standing start, I'm not seeing this as a, as a convincing.

System function.

Yeah, okay.

Yeah, I think it, it had gotten kicked to us and, and I wanted to look at it, but I'm right.

I mean it's, it's a fine thing to talk about at some point, but I, I just, I don't think it's. I'm not seeing. I think we should talk about it in the context of iconizing expressions, which I think is worth having. That's something we've wanted multiple times.

So we had started to talk about this in a previous meeting, but we didn't really have much time to dive into it. Right now we have create UUID which uses a UUID version 4 spec, but there are seven different UUID specs. There's technically an eighth one, but it's basically whatever schema you want. And the main hitch would be, you know, that we need to add in. We already have a prefix for the first argument, so we'd basically need to be adding in a second argument that would be a version or we need an option that is the hashing method. There are advantages.

Last thing. Base 36 string. That's the way it's being presented, right?

Yeah. So you might want it, you might want it being presented as a base 36 string or a hex string, et cetera. That'd basically be a format as a secondary thing if we're doing extensions.

So it's quite useless. What's quite useless?

This was a comment from Ricardo when he was looking at, I mean the.

The existing.

But I don't know the existing prefix. He didn't really like the first argument.

As trivial to just add your own prefix with string join.

No, it's not. Well, I see. It's just purely.

Yeah, it's just concatenated strings.

That's crazy. What happens if I put in.

Ah, it only works for strings. Right.

So why don't we just have the Version number there. Let me say create UUID 7 has V7A V7UUID.

Oh, we certainly could. The. So the, the minor hitch. So under the, the closed subsection there are all the different UUID forms or schemas kind of fall into different groups. 1 and 6 basically do the same sort of thing of concatenating a Mac address and date time hash. Their arguments are reversed basically. So the timestamp in the UUID 6 comes first so that it's okay.

Who cares? I mean, if the format. If Create UUID of N creates a UUID of version N uuid, what's wrong with that?

There shouldn't be a problem with it, except for I think it's version four. And what's the problem? I mean, there are a couple of UUID forms that require parameters.

So version parameters.

So 3 and 5, they require a namespace and a name. So they're used, typically used for things like DNS, but they're basically deterministic hashes. So if you want to create a.

Uid, why don't we put that in our hash function?

I think it would be fine to put them in hash. It would basically just mean that we would say Create UUID doesn't work for version three and five.

If we say three arrow prefix, we could handle that, couldn't we?

We need two. You need a namespace and a name. So I mean you could say three arrow list of whatever or three arrow association namespace goes to blah.

Or it could be just list three, you know, parameters, which I think would be more standard.

Yeah, I just like having the, the, the naked three there.

Are we.

The name namespace aren't just concatenated in the uuid, they're not. Okay, well, I would propose something like this.

Are we confident that UID schemes will forever be known by integer version numbers?

Well, if they have been stable, we.

Could deprecate the design that says we could deprecate the function this. We could de. Document this functionality, but giving us room in the future to, you know, in, in n years to change it. Okay.

Yeah, I hadn't thought about using integer indexes for it. I was thinking, you know, if we're adding a second one, but since they're just currently indexed off of integer values that make sense.

And what about the format? This solves the.

Yeah, this solves the version problem. The second question is, you know, do we want to have byte array encoded UUIDs or hex string UUIDs or do we just say they're, they're coming back as hex strings by default and you can encode them from there.

No, I think it'd be nice to have a second argument like we do in hash. I mean, hash has that second argument, right?

Third argument.

Third argument.

Yeah. Right. I think we should support it. I don't see why we wouldn't.

Okay.

And in terms of supporting a UUID as a hash, I think we should do that. I think it'd be useful for people if they see it in the list of potential hashes. I think that will be a helpful thing.

Okay.

They were called, you know, UUIDV3 or something like that. Okay. All right.

Yeah, we will move forward with that and if there's some issue, we'll circle back. But that makes sense to me. Right. Some time ago we had talked about it would be nice to have a function to generate passwords in Wolfram language. I have a prototype version in the prototype builds that does kind of the basic case. There's. There's a bunch of parameters. I've kind of based this on what generatesymmetrickey does, but there's a few questions. So, you know, by default there's certain assumptions. We're including letters, numbers, some common characters, which we'll probably need to think of a better name for. But there's a question.

No way on that second thing. No way. Just generate a freaking password, you know, like any other system does.

My main point with that is that as we get particularly into passphrase or this word based password section, there are a lot of parameters you can potentially tweak and it might be nice to have a dialogue somewhere.

I don't understand why generate password has a number here. I don't think that's useful. I wouldn't put that in there. I would. I think we can use that argument for something much more important. Okay. Like, I mean, what I would do is to have generate password be have, you know, its default. That's not a great password. It's too short, isn't it?

It's defaulting to eight.

We can't. What do most password managers now default to because they're longer than this?

I'm pretty sure 12 is pretty common. 8 is usually the minimum length that's allowed on various.

So let's not go for minim because there's no point. By the time it's a password like this, nobody's trying to remember it. You see what I'm saying?

Yeah, yeah.

We'll bump it up and make sure that we show the tooling to go, you know, generate passwords Set system credentials to that, etc, etc. Etc. You know what I'm saying? I would say that what we want to do is to say something like this and here we would say, you know, words, whatever else.

Yeah, I mean there's again a parallel with generate symmetric key kind of has the same notion of there are methods and then there are parameters that can be passed in for methods. So you might say words and it generates a word based password. Yeah, I guess we could say random for the machine version.

Characters.

Characters, yeah, yeah, yeah. Characters and words seem like obvious, obvious choices. And I mean those should just be default and we will change them over time or whatever else.

Yep. And I think having an association, you know, words goes to association with whatever fine. Controls you want. That's good.

Or first argument. That's just the association.

Yep, yep.

That has all those things, you know, password type, arrow words and you know, I don't know what the password length means when it's dealing with words.

So if you look on the, the word base section down there, generally a passphrase or pass word based system will have a number of words that you're using. Like the implementation here is kind of basic but you know, this version generates three word passwords. It includes some extra things like separators and padding and there's a bunch of different standards. Like Apple has their own standard that they Recommend For Apple IDs. There are algorithms for generating web 16 or 32 bit that are word based but guaranteed to be 16 characters or.

I don't understand why this junk is in here. I mean I thought it's word basically to introduce like word word word with dashes between or something.

That's one.

Yeah, but then the website complains that you need special characters or that you can't use a space.

It also increases entropy. It's still human, memorable generally.

No, these are not. That one is not. I'm sorry, that one is not. It's just not. It's. It's. I mean, okay look, if we can name these different password schemes then. You know, if we can come up with good names then we can have a bunch of presets. I mean, you know, maybe there are things like Apple ID who just uses names. I mean if there's a standard for that. See what I'm saying? Wpa. There's. If there's a standard for it, why don't we just use that as a name and you're generating a password suitable for that type of thing.

Yeah, we can certainly do that.

I'm sorry I missed something. What are we claiming that Apple ID represents?

There's a recommended password scheme that uses passphrases that Apple has on their Apple ID setup.

Right. So, I mean, somebody needs to go research this a bit and come back with. I think, I think we just have to come up with names for, you know, and I don't know, for non us keyboards, for example, there are all kinds of additional things. Right.

Yep. There's extra characters, so there's different dictionaries.

Right. And it's like language. We probably just want a language option.

Yep. There's, you know, in keyboard layout, alt.gr can create problems with some special characters and passwords.

Yeah.

So, yeah, I think, I think the idea I generally have is that, you know, we have flexibility with a lot of parameters and I like the idea of having presets that just say.

Right. I think we should have presets that are suitable for, you know, where. I mean, conceivably we could have a default way to do. But I don't think it's necessary. I think people can just remember, you know, I'm going to use generate password with words and that's how I usually do it. Okay. But I also want to make sure that we have a workflow where those passwords can get stored in the keychain system, you know, credential thing and so on.

Yeah.

And also if you are generating a password for a. A cloud object, for example, how that all works. Makes sense.

Yep. Yeah, those would be.

And we might want some WFRs that basically will generate it and send email about it and so on.

The dialogue you suggested would be a good function repository thing too.

Okay. Yeah, that's a good point.

All right, what's this here?

This is like.

As opposed to a flipped phone.

Yeah. So this is a variation on reverse applied that just switches the. The order of two arguments. So something I regularly run into with like date conversions is that some, some date functions expect the first date to be earlier than the second date and some data sources will return things in the reversed order and I can reverse them. But if I want to have. If I want to get date difference in years, then I need to reconstruct and map things across.

You want flip applied, which flips the first two arguments. Is that correct?

Right. Or we could look at some argument structure to do it in reverse applied. My only qualm there is. Is that too far of a departure from reverse applied as a function name.

Reverse is the first N arguments. Why the hell isn't this reverse applied?

F2 because we can also specify which arguments to flip. This is reverse applied of 2f comma 2. But we can also say I want to switch the first and last argument, which would be something we could extend in reverse applied.

Well, it would be more like permut applied. I don't think it's flip applied unless it's just two.

Yeah. So I guess the question is if we want to extend the second argument of reverse applied.

I wouldn't.

Does an operator applied do this with the torque uniform?

Not quite. I mean the.

So I mean this would be reorder applied where I could say 3, 1, 2 here, for example. I mean, I don't know how useful this would be, but it's certainly. It's somewhat useful actually because you have to do a bunch of applied and lists and all this kind of stuff. I'd be tentatively in favor of reorder applied. Can we think it through?

Is.

Is that a permutation list or is that a cycles? In which case maybe it should be cycles.

I was thinking of it as a permutation.

Could it be the same spec as operator applied?

What is operator?

The example you have on your screen right now would work with operator applied. That's exactly what it does.

The thing with operator applied is it accepts the arguments in a curried form. And this is not doing that.

This reorder applied.

But the.

What do you mean accept or does it only have its. Does it only do curried form? No, for operator applied I think operator.

Applied allows it to be curried.

However.

What do you know, it already works. So what does that to do?

Yeah, so why do we need a new function?

So if you. If you add a d to the sub values there it becomes interesting.

Right?

Operator replied, doesn't care how you query the arguments.

Subtle. All right. I mean, so reorder applied could be the same as operator applied but not do this, but only reorder the ones that it's asked to reorder and leave the other ones in there in their sequence. Basically.

Yeah.

That seems somewhat useful. Go ahead, what were you going to say?

I was going to suggest. Well, maybe we add support in operator applied for cycles and then you only specify the right the things you want to move and in that form it knows.

I think it's obscure. I think knowing that operator applied because people are going to think of operator applied in a different way. I certainly do.

Right. I'd honestly forgotten about this usage form for operator applied also.

All right, so tentatively, I think reorder applied makes sense. The cycles thing I Think is very weird.

Why?

I mean maybe we should. Well, we could support it, but I.

Mean if you have, you know, five arguments and you want to just flip two of them, cycles of, you know, two comma four is much easier than, you know.

Fair point. Fair point. Fair point.

Yeah.

Fair point. It's reasonable. So reorder applies.

And we could also. And we could also support two way rule for, for the just flip two case.

Oh, I like that. That's a good idea.

Yeah. Would it be safe to extend that into operator applied also?

I would think so. I'm not sure. I'm not sure because I don't know what's happening with the, the sub valuing.

Yeah, that's a, that's a question to look into at least.

All right, let's have a quick look at the last thing on the agenda here. Trim.

Yeah. So this is actually one of our pretty old WFR functions. I think Tally originally wrote it. It's basically a way of, of specifying a rectangular form for ragged arrays. So I have a ragged array I want to operate on. Map thread something else. It doesn't work because it's not consistent dimensions.

Well, I mean you might want to put zeros in it or something like this. I mean you wanted padding, you might.

Want to pad it or you might want to trim it. And this is kind of the. There's a trim left and a trim right version. I think it's maybe a little weird to.

I don't think that makes sense because you can have an eight dimensional array and I don't know what the heck left and right are then. Look, I think this general question, I mean there are a bunch of image processing functions that have padding options and so on. I mean I would, I mean I would think that something like. Or array. I mean because it is an array. So you can't say it's a. You know. You get what I'm saying?

Yeah.

Rectangularize an array and the default would be chop off all the proboscis, you know, chop off all the tentacles, so to speak.

Basically have the, the shortest row dictate where you chop things off.

Right. And failing that I think it would be dims here. What dimensions do you want? And now you've got a whole kinds of stuff about is at the beginning, is at the end. I mean here again there, there are all kinds of issues about do you chop the. The beginnings of rows, the ends of the rows, etc. Now here also there's, you know, now what we have an analogy here like the center Array, for example, I think we have, you know, a b here, comma 10, right, comma x would be that. Or should this be rectangularized array? I think rectangularize is, is okay, pretty obvious what it means. And then. So that's the, the padding argument. If you need a padding argument. I think by default it's just, you know, continue with that index, if you understand what I'm saying. That's the either cut down, you know, have that index. Dims is, has very obvious functionality. Right. It's like have that index. You've already got elements 1, 2, 3, 4, 5, but you only need four of them. Cut off the fifth one, you are given elements 1, 2, 3, four, you're going to add a fifth one. Okay. Add the fifth one at the end. Other words, you've got existing part numbers. Use them.

Yep.

Now, okay, so the rule is any part number that's already there stays the same in the rectangularized form. Does that make sense? And then I think it's completely well defined what, what this should do and seems useful to me.

Yeah. Then, then the only ambiguity is whether we, we pad and trim from the right hand side the end of the row or beginning, which we can parameterize if we need.

No, it has to be from the. It has to be from the end if the part numbers don't change.

Oh yeah. If the part numbers don't change. That's right.

I think it's the only consistent thing to do is don't change the part numbers. Okay.

Yeah.

I think, I think anything else is going to get very funky with different left. Right, whatever.

Right. I guess you could always reverse it.

If you want to that level if you wanted to. Yeah, if you really, really wanted to. Okay. But I think this is, I agree that this is useful. This is part of sort of repairing data type thing. It is unclear to me what the default when you don't put. I mean a lot of other things default to zero when you don't put. I think that's what we should do. Center array defaults to zero. Think this should default to zero as well.

Okay. It'll just pad out to zero. Pad out with zeros.

Yep.

Okay.

All right.

But it won't trim already existing bleeding zeros.

No, no, because it's got to keep the part numbers the same.

I think it really should keep the part numbers the same. I think this is useful, but we just don't want to be messing with, you know. Oh, I, I mean, you know, just to be clear, This is, I mean, its relationship to Pad. Right. If I say A, B, C like this and I say pad, That's what it does. I mean, it's not clear why rectangularize is any different from pad. Right.

It shrinks rather than grows.

I see what you're saying.

Yeah, we have the ability to shrink it.

Right. But rectangularize, the general version of this. Pad write. If I said pad write this comma, 4 4. Right. It does that. Which is what I'm suggesting here as well.

Yep. But if you said if you did one one, it would one, one, you know.

Right.

I think it would trim. Yeah.

So this is rectangularized to different defaults of padrite.

In which case, why is it a different default?

Because this defaults to padding.

Rectangularize has to compute the minimal dimension for each level.

And so what is Pad. Right. Doing in that case?

The maximal dimension?

Well, maybe it should be.

You could just have. You could just have a magic token in the second argument to pad.

Right. Yeah, yeah. But nobody's really going to find it. Yeah.

So. Okay. Oh, so rectangular. If we did rectangularize on that input, we would end up with the matrix containing a.

No, I think we'd end up with a. We'd end up with. We'd end up with that. That's rectangular.

I see.

Could just be the row ab, though. It's ambiguous.

Right.

It's always a column first thing.

That's an interesting question. And that's another issue. Yeah. I think you're trying to preserve. What are you trying to do? I mean, I think this case is pretty ambiguous because you preserve. The number of elements is the same both ways. I think it's like a. You know, I'm thinking of it as some kind of not convex hull. But, you know, imagine that you have these points in. Yeah. How do you do this? Do you understand what I'm saying? I mean, can somebody think about this? I agree that.

So I don't. I mean. Right. Saying it's column first, I think is not useful. If we say it's innermost level. Right. We minimize the innermost level.

Right.

Just the name rectangularized puts me into linear algebra mode and I want to think about it that way.

Well, what would it mean in linear algebra terms?

I don't think it would mean anything. Right.

Okay, fine. I need to get going. Anything else we should cover here? I mean, I think that this is a function, I think rectangularize and. Or trim, but I think rectangularize is a better term because I think it makes it clearer that you can go both ways.

Yeah. We'll prototype it and see where we end up on the.

Yep, sounds good. Okay, thanks a lot, guys.

Thanks all.

Bye.

See you later.