Live CEOing Ep 767: Language Design in the Wolfram Language [BitFlip, KeepElements, and More]

Hello, everyone. Welcome to another Wolfram Language Design review, part of our Incremental Language Development series. And we have a few topics. I don't know what the first one is. Gaussian predicates. I'm about to find out.

Oh, yeah. So we have Adam here also to opine, but I think the idea for Gaussian Integer Q originally came in through the wfr, and Danny and I talked about it a little bit and think it's useful. And there's also potentially a case for Gaussian rational queue. But we wanted to explore a little bit the relationship with the Gaussian integers option. I don't think we want to have something like an integer queue that includes Gaussian integers as an option, but we also don't want to balloon the space if we're going to have other numeric tests that account for whether something is complex or not.

Why is this so unbelievably slow? Oh, it's. Maybe it opened on another screen. Yeah, that is a stupid, stupid thing. I mean, this is some Mac user interface guideline that things open on the screen that you were somehow lost using or something. We. We just have to fix that. I mean, there's just, like. It is so confusing to me. Everybody finds it confusing. It's like, why are we doing this? Okay, wait a minute. Why is that? Maybe this isn't even the same version. Gaussian integers. I mean, I don't think this is the world's most important issue, but this. I mean, why do we need. What do other people think about this? Integer Q with a. Gaussian integers are true versus Gaussian integer Q.

My general preference is I tend not to love predicate functions with options.

I think this has the same typical issue that we had with Real Q. Do we mean a number that's real or. I mean, is. Is one a Gaussian integer? Because this is a bigger set than the integers.

Yes, I think it should be.

Yeah, I do, too.

So this is why we ended up with Real valued.

Right.

Real Valued numeric Q and Real Valued Number Q.

Right.

Okay. But implicit Gaussian integerism is not really a thing. This is like integer Q. There's no implicit integer Q. Yeah, right.

I mean. Right. I guess, you know. You know, I guess I'm 30 years too late, but. Or some such. But, you know, it is weird that, you know, integers and rationals and reals are all domain specifications, but Gaussian Integers is an option, right?

Yes. Isn't there a field specifier that is the equivalent of that what. What's the analog of reals? The for reals, you know, were for, let's say integers with scrota2 appended. Adam, isn't that a field specific? Isn't there a way of specifying that as a field?

It's an extension. So, so algebraic number fields we. We treat as extensions. So if, if you go extension goes to I, this will be the same thing.

Okay, but there isn't a. A single named object.

Nope.

Well, I mean, I consider this somewhat a corner case, but I think Gaussian integer Cube probably makes sense. And the fact that there isn't a Gaussian integers domain is unfortunate. But I think this extension field mechanism is reasonable. That anders it I says we cooked this goose a long time ago. Yeah, I'm not a big enthusiast of Gaussian rational Q.

One thing is that there is an exact number Q which tests exactly for Gaussian rational. Because all exact numbers are Gaussian rationals right now.

Okay, so I think we have Gaussian integer Q. We don't have Gaussian rational Q. Make sense.

Okay, related question. Should we have rational Q? That is useful.

We don't have rational Q. Nope. We should have that by the time we have Gaussian integer Q, we should have rational Q.

But again, I make the same question. Is one a rational Q?

Yes, yes, yes. I mean, just like we have rationals, we should have rational Q, I think.

Right, but the point Jose is making is that the right.

I mean, we didn't want real Q because it was confusing.

One does not have head rational, even though it's. So it would be a bit odd that it's rational Q.

I understand the issue. Adam, what do you think about that issue?

So I have a function rational number Q that I used everywhere, which exactly tests for rational number which is integer or head rational.

Maybe that's better. Okay, integer Q obviously tests for something with head integer. Right.

You guys have real valued number Q and real valued numeric Q now, right?

Yeah. Rational number Q would be somewhat consistent with that naming convention. Probably don't need a rational numeric Q.

I don't think so. Well, it doesn't seem bad. Rational number Q Other people's thoughts, that's fine. Yeah, seems useful to me. Okay.

Yeah, we'll proceed there. So there's a ref page we can look at for this bit. Flip is a function we proposed a little while ago and I implemented time ago. Yeah, implemented a initial version of it. There are two design questions here. I talked with Mark a little bit about what we should do, because one of the proposals we had was to allow negative coefficient specifications to basically say, count from the left instead of the right. And when we do that, there's the potential to overrun the initial set bit. And so I thought the two obvious cases of what we can do is either error and say, oh, you're past the zeroth bit, we aren't going to do anything. Or Mark's proposal was we basically just return the input unevaluated, as if you're toggling some fictional fractional bit.

The issue is, okay, so the coefficient of 2 to the k is the bit you're flipping. So n 0 means you flip the ones column.

Correct. So if we're counting from the right, we can basically set whatever bit at infinitum, more or less that if we're counting from the left.

Well, we can set that in a. Is this runnable in this version?

Yes, it should be runnable in the prototype build.

So that's a power of two.

Yep.

Right.

Now, what does it mean that. Where are you counting from the left from? I mean, there's an unambiguous starting point if you're counting from the right.

Right.

So what we currently have implemented is that if we're counting from the left, we're counting from the final set bit.

Yeah. Right. So if I say bit flip of this comma minus 200, it should divide by two. Or is it minus 199?

No, no, no, it added one, right?

Yeah. If you do minus one, then it'll divide by two.

Oh, I see, I see, I see, I see, I see. What?

Right? You, you, you added one by. By. Or, Or I guess added two by.

Okay. Bit flip of percent 3 comma minus 1.

Yeah. Sets it to zero because only the very, very leading bit is set. If you look at integer digits of percent three, it'll be 100000.

Oh, I see, I see, I see, I see, I see. So that, so that just turns it into zero.

Correct.

So in fact, my divide by two is not even doable.

Yeah, not with that input. Yeah, but the question is, you know, so what does bit flip of 2 minus 3 do? We're counting past.

I understand. We're counting past the end of the ones column.

Right. And so what I've done for now is basically just we return the input unchanged, as if you're flipping some fictional bit that's past the start of time.

That's what we have to do. I don't see anything else to do.

Yeah, the only other Thing I could think of is we give some sort of error.

But that's maybe what happens with negative numbers. What do negative numbers do? Huh?

Yeah, I think it's just doing that to the. You know, to the. I think it's ignoring the sign and adding it back in.

I don't even understand what this did. That changed 111 to 11 0, didn't it? Why didn't that change 111 to 11 0?

Oh, is it doing it in the two's compl.

I'm totally confused. What bit flip of 7 comma 0 should give 6.

Yes, you're right.

Yeah.

So are you doing this in one's complex? In two's complement?

What do we do for bit set? Don't we have some two's complement story for one of those? Or bit. Bit compliment? Don't we have a bit complement? And doesn't that. What is it? Bit not. Okay, So I don't understand this. What does bit not do? And. And is this doing something consistent with that?

Not sure what bit not.

Well, what the center did. Integer digits of -7 assumed to be.

Represented in 2's complement form with an unlimited number of digits. So I think bit. Bit flip is probably doing the same thing, which is probably right.

Yeah.

So it just needs to be documented.

Yeah, we'll add documentation for that.

Okay. Okay.

Yeah. If that looks good, then we'll make those changes and add it in. Keep elements is another thing we talked about a while ago. This is kind of a sister function to delete elements, but instead of removing whatever elements you list, we keep them. There's a ref page too. There's also a substitution argument so that you can fill in elements which aren't missing or elements which don't match with something like missing or none or zero or whatever you want to. But there's a couple of design questions I had in the process of.

It's a bad sign that I didn't remember this function existed.

It's experimental, which is probably why. No.

I don't understand why. This isn't like delete cases.

It's like delete cases within implicit alternatives.

I was not a fan of this function.

I think this was introduced together with unique elements, which is a function I must admit I use a lot now. Yeah, but not this one.

So what's the difference between unique elements and union?

Unique elements is a way of doing multiple complements at the same time. So if you have several lists and you want to understand the differences, it's super useful. So it's A bit like symmetric difference, but it keeps the structure so you know where the differences are.

Right.

I use this a lot.

Yeah, I use it a lot too. Like if I'm generating a new unit table, I want to know what things disappeared when from when I last generated when things were added. It's nice to have that.

So I see. So it's kind of a more symmetrical version. Okay, I get.

Was called all complements and function repository.

Yes.

Okay, so keep elements. What is.

So I guess my point was that we introduced both together and delete elements is a lot less useful, I think. And keep elements is probably as useful as.

Not useful exactly as delete elements.

So maybe the story should be we get rid of delete elements, not add keep elements.

It's possible, yes.

Yeah. We had at one point discussed keepelements as a function called prune where basically you say I have these elements I'm looking for inside of a list and I either want to just keep those ones, in which case it's basically just cases or if you scroll down and look at some of the examples, there are instances where you can replace any instance that doesn't match with a second argument or I guess a third argument. So that second input where we are keeping all the B's and D's and anything that doesn't match that we are inserting a missing instead.

Not something I would guess from the name of the function. I don't know. This function is not completely doing it for me.

Yeah, I mean, as. As I was putting it together, I think there's a number of questions and you know, maybe as Itay said, that's more of a. A reason to look at whether we want to have delete elements in the first place rather than extending is this is starting to turn into a whole elements family of functions. We have unique elements and keep elements and delete elements and they are kind.

Of like Delete elements is a questionable function. Okay, when did we introduce it? Eleven?

Maybe not that far back.

Thirteen. One.

Okay.

Okay.

We will put keep elements on ice and circle back and talk about maybe de promoting delete elements.

All right, let's keep going here. All same by. Oh, this does seem somewhat useful. Right.

I found the operator form to be particularly useful too. Right.

Yeah. All same by last or something.

Yeah. And this is somewhat similar to. We have the. We also added the same as function which is basically an operator form of sameq.

Seems good to me. Any complaints here?

No, I think it's pretty straightforward. I don't think there were any real edge Case issues.

But maybe if you think the operator form is so useful, you should put that in the. In the basic examples.

Yeah, I can do that.

String applied.

String applied is kind of a super function. The idea is that we can provide it an arbitrary function that works on lists and it will implicitly call characters on a string and apply that function. So we looked at one point of having something like string union or string sort and rather than having a bunch of different string x XY functions, basically have this general function that lets us characterify and operate with an arbitrary function.

So how is this related to like mapping on an image? Like we've got image data and things like this. Okay, so you're saying if I say string applied of union of I am a fish. Okay.

Yep. But reverse in there, it's string reverse or sort. It's string sort.

Of. So.

Oh, was this based on string function? Yeah, string function was the function repository I need. Does it only work with built in.

Functions or you can put. It's got to return a list of strings or you know, it's got to return a string for a string.

Well, that's obviously a bug though.

The error handling should be fixed.

But yeah, the error handling. See it fixed there.

There's not.

There's nothing good we can return from this.

I see. Because uppercase Q is not returns billions.

Right. And you can't stream join billions. You can do tostring composed with uppercase q and it creates some ugly true, true, true, true, false, false, false thing.

Yeah. I think there's an example on the ref page of composing a string function farther down.

So is it then string applied or characters applied?

It is, yeah.

That was what I was kind of wondering too. Kind of a character map. Right, except that it reconstitutes it afterwards.

Yeah.

Honestly, character applied might make more sense because we've got, you know, reverse applied, for example.

Means.

Right, that's reverse applied. I mean it's nice to have the string brand in there because it's a very string related thing.

Is, isn't this more string map?

No, because it recons. Well, I see, I see.

Yes, exactly. A map takes. Eats a list and returns a list. And this takes a string and returns a string. Well, it, it applies because your function.

Is given the list of characters.

It's not applied to each of the characters.

Yeah, sorties are very good example there.

So why do we not provide this and then have an operator form that is our. I mean, we're forcing people into an operator form.

Mm. Yeah, I suppose I could see that.

This seems less obscure. I mean the operative form requires kind of higher order function thinking, so to speak.

And then string apply of sort would.

Would also work as an operator.

Yep.

Yeah, this would work just like map. I mean, just like apply works.

What's that ellipsis sneaking out on the doc page behind you. Is this supposed to work on varx?

Yeah. So you know, if you. If you scroll down, you can see some examples under I think scope where you know it. We basically need a function that you know, like with delete, you can delete.

I think. I think this is a mess. I think if you want to do this you should say string apply of delete hash comma 3. I think this is cutesy, but hard to understand what you have here with string applied with multi ARCs or delete.

Of three, which is, you know, another operator form. Right. You could do string apply of delete of three.

Yeah. Drop doesn't have an operator form, does it? Because of our args.

Yeah, it does not.

I think this should be string apply and I think it's a good function. Okay. I wonder if there are any other.

So I'm thinking that this is a bit like subsetmap in which the contract is the. The function has to take lists and return lists.

Well, specifically it has to take lists of strings and return a list of strings.

Yes, but there is some sort of restriction on the collection of functions that are admitted.

Yes, Which is of course different from ply.

Exactly.

Right.

We could have something like string templates insertion function option. But the restriction. I agree.

I think string apply is reasonable. I don't think it's most important function in the world, but I think it's reasonable.

Okay, we'll signpost pretty clearly in the docs that there are restrictions on what functions work and we'll fix the error handling there too. That uppercase Q example you had. This next one is something we had talked about at one point or another. It's become a little more prescient with working on some of the future of Arrays project with the ideas. We have a collection of states and we want to conform them to some common standard, much like image conform images does. There are a couple of questions, mostly about the one argument case. You know, the two argument case. You can basically provide a reference state, or you can provide a collection of options or a combination thereof. But what do we do in the automatic case? Do we want to basically tally through and say, oh, this is the most commonly calendar.

No, no, no. We either want to do like the SI unit thing you know, canonicalizing it to a particular calendar system. Okay, look, what are the possibilities? There is a current calendar system for the user.

Yep.

Canonicalized to that. That's one possibility. Which seems like not the worst possibility.

For example, in conform images, there's basically a ranking of things like color space. If there's a bunch of grayscale images and one RGB image, it canonicalizes to rgb. If there's hsb, it has to. Yeah.

What does it canonicalize between HSB and an rgb? What does it canonicalize to?

To rgb, I believe.

Yeah. So that's like the SI Units game. Yeah.

There's basically a set ranking of, you know, and some things like alpha channels. If an image has an alpha channel, then all of them will have alpha channels. Not to lose extra information.

Yeah.

So.

Well, I mean, there's no information loss and dates, is there?

The only information loss would be doing things like going to a finer or a coarser granularity. I have a bunch of years and a bunch of days. If I go to year, then the days kind of lose information.

But how do we go. And we can't change. How do we conform them. How do we conform them with different granularities.

Right. I was. I was going to say if you go. Go from years to days, you've also lost information.

We go to the first day. Right. Which is what we do in other. In many other functions.

Yeah.

Well, so wait a minute. So in other words, if it's an instant, the year will turn into the instant at the very beginning of the year, is that right?

Right. That's kind of our central dogma with dates of. If it is a extended date, implicitly through granularity, anything that needs to resolve it to a finer granularity, we'll use the first instance of that granularity. If you go from year to month, it's the first month. If you go from.

So in that scheme, we're saying it is the calendar conventions of the current location together with the finest granularity of everything in the list, Isn't that right?

Yep.

That would be the canonicalization.

Yeah. And again, like, if you want to do something else, the second argument is there and you can specify what you do.

All right. I think what we just said is probably the right thing.

Okay, Sorry, I have one question here. I'm thinking of. Imagine that all dates have the Julian calendar. Do we keep it or do we still change to Gregorian? Because can conform always change to Gregorian?

I think we should always change to the current calendar to wherever you are as calendar system. If somebody sends you something and they're in a different calendar zone and it has to recompute. That's a bit silly.

Exactly. Because we want to use this in a situation in which speed is important. So if we realize that we can avoid conversions, I think we should go for that. Or at least having a second argument that says automatic or inherited or.

Yeah, all right. Okay, so. So what you're saying is, okay, case number one, you want to canonicalize to your current lock.

Perhaps we can write commonest in the second argument or something like that. And that indicates he look for the most common.

Yeah, that seems obscure. And then there's a tie and then blah, blah, blah. Well, your point is, if there is. If everything's a year, but there's one thing that's an instant, what do we do?

It's a bit like common units. Common units. If it realizes that everything is in feet, we don't attempt to change to meters.

Right, So I understand that, but what happens if. If there's 10,000 dates that are all years and a single solitary instant.

You.

Know, we're going to lose information either way. If we move everything to instance or if we move everything to years?

Well, I think we move everything to instance.

I think we need a flexible second argument that allows you to specify what you want. I mean, of course, if you write a date with an instant in the second argument, it will go to instants.

You mean date granularity, arrow instant in the second argument of conform dates.

Or it also takes a second argument that's a reference date. So that version of conform dates, comma now means pull the parameters from now and apply them to all of the dates.

I see, comma today would give it to a day, comma now would give it to an instant.

Yep, yep. Or if you give a date on a different calendar or a different time system, it would pull from there.

Okay.

An easier way than specifying all of the options.

Yeah. Okay, so what are the questions here? So the question is the single argument form. What should it do? And I kind of agree that if everything is already conforming, it should leave it the same. Otherwise I think it just needs a promotion schedule. That's what you're proposing. If there's a single instant it, you know, a single thing which is an instant it would all promote to instance. Correct. So that's like the RGB case where RGB is preferred. Presumably SI units are preferred in the units case.

Right.

So I think we just need a promotion table and it's Instant, you know, Gregorian instance. What else do we have to figure out? Type, isn't it?

Yeah, granularity calendar type, time zone, time system, and theoretically date format also.

Okay, time zone, that's tricky. Time zone. What do we do with time zone? If they're in incoherent time zones, what the heck do we do?

We could either make an arbitrary hierarchy, which seems a little weird. You know, we could just say, oh, GMT is preferred.

But I don't think we should go to gmt. I think we should go to. I think we should go to current. Your current now time zone over gmt. I think GMT is just going to be confusing for everybody.

Okay, yeah, that's a good point. We have dollar time zones. We'll use that.

Okay. What were the other things?

Time system is pretty much always going to be our default. But if you have a bunch of dates that are all in Thai, for example, we can preserve that and date format, basically same thing. We have a dollar date string format that we reference. So we can.

Which is. Which is your current one.

Yep.

And so. So the basic point is. Which is a little confusing. Okay. If you want to always get it to your current thing, you just say conform dates, dates, comma, now or something, and the now will be in your current thing. So you're all good.

Correct.

If on the other hand, everybody is in, you know, I don't know what French date formats. What's that?

All right. Yeah, there's a French calendar and there's also French date formats just to be convenient.

Right. I'm gonna say French date format, not the French revolutionary calendar or something crazy like that. Journey probably only lasted for three weeks. Yeah.

Well, we do have it just for fun.

I understand, I understand. So we can cover those weeks. I'm just waiting for us to have regnal calendars.

You know, we have one regnal calendar. We have the Japanese imperial calendar.

Okay. We don't have, you know, the seventh year of King Aethelred.

No, we do not.

But presumably when we have leader data, we will be able to have that.

Yeah.

Okay. Any other things we need to choose here?

The initial implementation I have about this is basically built around our container dates. So like date object, time, object, date interval, and also the date arrays as we move forward. I was not planning on having an ability to say conform all of these to absolute times or date lists or date strings, but I want to do at least.

I think that's reasonable. I think we should be going towards, you know, our symbolic date representations.

Yeah. And then the other question was do we want to have some sort of traversal system for. I have a data set. I want to conform all the dates in it, but I don't want to pick them out specifically. Or do we say conform Dates should always take a list or a data vector or something like that. I, I see a decent use case for, you know, I have this data I got from a. A database or whatever.

No, no, no, I understand. A mixed structure with an association that has a few random dates in it.

Yep. If we're keeping it too focused on our symbolic date structure, then that's doable. If we're letting like.

Yeah, if it has a number in it, then we're tough, you know, that's hopeless. Right. We can't, you know, the only question is, does it walk and talk like a date? And then we can do something with it.

Right.

I think we should initially do this just with lists and we should check that we do not regret the fact that it doesn't conform things other than our symbolic data objects. And if that's okay, then we should extend it to deal with conforming things anywhere in a data structure.

Okay, that makes sense to me.

Okay. Why is this smooshed, this arithmetic? French?

No, that seems wrong, not ideal. Yeah, it's falling back to just the string for the calendar, but yeah, it should have a space there.

Seems wrong. Okay. Quantity Q. What's this?

So we have quantity Q and known unit Q already. A pretty common operation when writing functions or testing arguments is to check whether the input is a quantity and then whether it matches a certain dimensional specification. So I have an input, I want to know is it a valid quantity and is it a temperature or is it a velocity?

Yeah. Why don't you do the same thing as conform dates? Why don't you have quantity Q's second argument be a template thing? So if I say 30 Kelvin dimensions, it's going to check that it's a third, you know, that it's consistent with that. Yep.

That's basically what we're proposing here is that you can give a second argument to QuantityQ that is a quantity variable or a list of unit dimensions or a quantity. Right. Now I think the one bit I'm not 100% sure on is if we provide a string for a second argument. I think it's most useful if it's a quantity variable. So being able to say velocity or temperature is more helpful than saying like a unit name.

Why don't we support both?

I think the confusing thing is going to be what does quantity q of quantity 3°F, comma Kelvins mean? Am I asking is this a quantity in Kelvins? Or I'm asking is this a quantity that is compatible with Kelvin?

I see.

And same with unit Q, known unit Q at the bottom. We could potentially support both.

There's no, no, that's a reasonable argument. I mean, look, you can always. It's a little weird to have to give a. Completely pointless. Let's see. How do you go from Kelvins to the physical quantity associated with it?

This is not unique. There are units that are.

I see.

Meters, for example, are we talking length or height or depth, etc.

Yeah, or. Or for example, torque and energy density have the same units and things like that.

Okay, okay. So what is your proposal? For what? I mean, the question is, is this thing Q. A quantity with the same dimensions? It's not the same units. It's the same dimensions as meters per second, for example.

Right?

Correct.

So that's why specifying the quantity variable is more flexible. Velocity or temperature.

Yeah, well, I kind of think that's probably right then. And don't support the string form here, but do support, as a courtesy, support this kind of template quantity expression. Makes sense.

Yep. Yep.

Okay. Anything else?

Nope. I think that does it for today.

Well, we. We hit the time and we have.

Hit the whole agenda.

Okay, good stuff. All right, thanks a lot. See you all soon. Okay, bye.