Live CEOing Ep 727: Language Design in the Wolfram Language [CountDistinct and More]

Okay. Hi everyone. We are back after a bit of a hiatus for our summer school and high school program. And we are here doing a Wolfram language design review, part of our language completeness and consistency series. All right. Various topics today. Count distinct.

Okay. So this is a pretty small thing. We added a second argument, count distinctness. Right. And we didn't document it. And I think that was probably just an oversight. But I wanted to make sure we all agreed with that before actually de futureizing the documentation.

Sounds sensible to me. Anybody got an objection to this? This is the distinctness test. That is these things are considered indistinct if this test succeeds. Correct?

Correct.

Is that clarified in the. In the documentation? I think you should say they should be considered not distinct.

Okay.

I assume this assumes that the test is transitive.

Yes. Right. I mean it has the same issue as. As delete duplicates.

So you should make a comment that it assumes. It implicitly assumes it's transitive.

Okay.

Making sense.

Yes.

And presumably this is. I mean, it should also be described as. As the length of delete duplicates of list.

Right?

Right. In fact, it's. Oh, is that not. I could have sworn that was in the notes.

I was trying to read the count distinct by. And count distinct for the second argument.

Right. And so I have the, the relevant example right there in the agenda because.

I see. Okay, so this is a minor luxury to be able to have count distinct with that second with that test.

Right. It's sort of making it equivalent to delete duplicates, which also has.

Fine. Seems fine to me. Any objections? Let's move on. Part assignment to non existing keys.

Right. So you know, you can assign to a non existent key in an association and. Right. You can do part assignment to a non existent key and that will create the key in the association which is different from other types of expressions.

Lists do that, don't they? No, if I say list. Well, they, they. They used to, at least in some.

No, if you say.

The point would be if you had a list of length three and then you say list part four equals something.

I know. Ah. I believe there was a time, maybe it was in version point eight, before version one, where that worked. Okay.

I am not familiar with any version in which that has worked.

It's never worked since I've been here.

And the issue, you know what? It worked in smp.

Maybe that's it.

Yeah.

Because what the problem with it is if you do list of say max machine number equals D, you're in trouble.

Well, and also, I mean, what the hell would you fill in the intermediate elements? I mean, you don't know what to fill in. The only possibility would be filling in the. You know, we'll be adding the very next element.

Yeah, which is.

I think it's a nice feature. Which. Go ahead. What you.

I'm just saying it seems like a very weird operation. It doesn't seem very useful.

Yeah, but this thing does seem useful. I didn't know it worked, but it does seem useful.

Right. So.

No, I think the question is. Oh, I see.

Yeah.

Okay, fine.

The association case does not have the issue of we don't know what to fill in an intermediate elements.

Correct.

Right. Now that mean.

Well, that's why it's labeled completely crazily. It basically just ignores all the parts after the key and just assigns.

Wrong. That just seems wrong.

Right, right. So as I note right there in the agenda, I think the second behavior I've pointed out is, is just a bug. Now the first behavior is, is not documented, but there is code that uses it. And like, you know, okay, so this.

Case where it's not a part but a sub value, basically, that we all agree should work.

Yes, we all agree that should work. And, and that behavior is also documented.

What about.

What about.

Can't you provide. I'm not sure if this is documented, but can't you provide multiple arguments in the sub value and it will sort of go multiple levels?

I don't think so.

I don't think you ever create empty associations. So if you had a, with an, with an S association, you could assign with the nested down value, I think.

A equals this, and then we say and then that. Then A is this. And then we say A of quote C comma quotes D equals X.

And what if you do a.

Key.

C does not exist. What does that mean? If I say A of C equals X, then that better work.

Yes, but what if you do A of C equal empty association and then you do acd? I think in that case it would work. So I think, I believe that you can fill in keys, you can fill in missing keys, but you cannot create nested levels.

Which. Which I think generally seems reasonable.

Yes. Right.

And. Right. And these are always keys. So if you did A A of part of, of open bracket C comma 1, then that would look for a key named 1, I presume.

Yes.

Yeah.

Okay, so the behavior with sub values seems reasonable and should be documented.

Is it the, the fact that you can assign, you know, the fact that you can create a new key is documented.

The fact that you properly documented?

I'm not sure I would have to double check.

Exactly, but you guys will chase that. Okay, now what we're talking about is the part assignment case of this.

Right?

And the part assignment case will be specific to either the key wrapper or a string, correct?

Correct.

There's one useful case I can mention which is if you have an association which is nested in some stuff with parts, you could have, you know, expression part one, two, three, A equals something.

Yeah, I agree that that's sort of that first example that there's. That's why I did it as all comma d instead of.

Oh, ok, yes, okay.

Instead of all, instead of just d. And I think. Yes, the reason Julio is here is because I'm aware of Etienne having used this idiom and.

Yes.

All right, look, my take is that we should document this. Now, what's your concern about consistent with other different. Which is both documented, so. No, no. Okay. Right.

I'm saying that assigning creating keys via part and creating keys via subvalue assignment are not the same operation. And so why not? If. Well, because they, you know. Well, there are. If. If you attach side effects to that assignment. Right. Then one, one will fire in one case. And I suppose. What's that?

Why are they different? I'm presumably colon equals works in both cases. Why are they different?

Well, because. Well, because. Right. One is a down value type assignment and one is a set is a part type assignment. Right? It's right. I mean, you can't assign down values using part assignment. Right.

So if you look at practical usage, part is strictly more powerful because you can deal with more expression types, while the down values only works with the nested association. Then they have to be consistently associations.

Right. Which means we should support the part case. This case that randomly works seems just weird. And it should. Seems like it should fail here.

Yes, that should fail. Yeah.

I think we should do. For part, we should treat, you know, strings or key things with the key wrapper as referred to associations. And they're effectively equivalent, you know, at least at that level as, you know, association with just the single bracket.

I agree.

So the.

I mean, I would argue a bit crazy idea here. Yeah, like, so we have one reason why you want to use part, which is like more powerful in term of like, accessing both keys and indexed parts. What's the reason why you should prefer the other one maybe is that you can actually create nested associations using down values.

So let's listen, listen. We want to make minimal changes here. We've Got something that's basically working. We've got a corner case that's been identified. That's kind of weird. Let's get rid of that corner case and declare victory. Let's not build some elaborate subsystem that I don't think we need. What we have is working. You know, I'm telling you, the practical.

Use case is that if I already know that I want a nested association, then right now I have to say, you know, A equal empty association, then A of B equal empty association, then A of C equal empty association, and then A of B of C. Yeah, I understand.

So what's your. What are you suggesting?

That only for the down value one, which is a kind of, you know, that only works for nested associations if A is an association. And if you say A of B of C of D, you could actually create all the levels. So that would make the down value also useful in its own right, because it will have one power that the part doesn't have. I don't know if it is stupid.

Say that again. By the way, Sandra on the live stream comments that. That. Wow. Wait a minute. A assignment with up to. That seems pretty wild to me. Wow. We don't have any range assignments, do we?

So if I say you can assign to spans.

Yeah, span.

Okay. So if I say list of 5, semi, semi 8 equals x, then that'll do exactly what you saw us. Okay, that's nice.

You could even do something like. Like list of five common colon eight equals reverse of list of five colon, colon eight. And I think it will not basically subset map.

Right.

But there's a.

There's.

There's a bizarre syntax with this, that. That I think would run amok with up to. And that is if that list, I mean, five, eight, two is two elements. Right? So if you have a list of two elements, then it sequentially uses those elements. And I think that could get a bit bizarre with up to detecting those cases correctly.

I think that the span case is a good version of the up to. You can do up to like with span. I mean, with span. Okay, let's come back to our previous discussion. So Julio was arguing that there might be something nice about being able to have the multi case.

Like your. Your ac comma D working without saying A of C equal empty association. The one that fails above.

Yeah, I understand this. This one that fails without having this one here. That fails.

Yep, yep.

Well, I agree. That's not completely crazy. Boy, that's exactly the semantics of S and P. How horrifying. How many years has it been? 40 something years later we come back to exactly the same thing. Now then, in smp, there was this crazy thing with this attribute that I called tier T I E R that determined whether. Okay, so here's where it gets funky. Okay. At least in the SMP case. I don't know, maybe we've got a different case where you say this. Now, to us that is meaningless. Right. Because A of C evaluates, does it not?

Correct.

Is that right in the L value case that it still evaluates?

Yes, Which is why when you're defining sub values, you often need a hold pattern.

Is that documented.

I presume? I think so. I would have to double check.

I got very upset about documentation with them at the summer school, finding out lots of things were really under documented. Okay, all right. So then the question is whether this case here should work. Let's investigate making it work. I don't see a harm in. It seems useful. Even if it's very sort of retro. Retro in some level.

Just FYI, I looked it up. It's the second properties and relations example on sub values that you should use. Hold pattern to prevent things.

Okay, okay, okay. It might also be going to possible issues.

It could.

Okay, what else is there to discuss here? So I think we're concluding this case should not work. This case could work and everything else stays the same.

Yes.

Okay. What the heck is that? Oh, this is the prototype build, but somebody should screen cap and report that it's wild. See that?

Carla, do you have a screenshot?

Okay.

Okay, next item.

Okay.

Wow. Look at this wild thing. Okay.

Yeah, this is. This is another.

It was wild on the screen, but then before I was looking at the wildness of the content here.

Yes, Right. So this is. This is. Well, this was an internal bug report. The previous thing was an external user bug report. This was inspired by an internal user bug report which thought that the first input was a bug. But apparently there are some functions that if. If you go to part minus the length plus one, they treat that as the head. So you see here we have an empty list and you map F at minus 1 and it gives. It applies F to the head. And in the second input, if you, you know, we have a list of.

Length three even with heads. False.

Wait a minute.

These do not take the heads option, right?

Because this is map at, map at.

And it takes. That is part number minus one. And for some absolutely bonkers reason, it thinks that part minus one is the head.

Right?

Because length of it, length of list minus One is zero.

Right. So minus one would be the bug.

It looks like a bug. I mean, it's wild, but it looks like a bug to me.

Okay. All right. Yeah. So, you know, there was a certain level of consistency between a certain class of functions. Right. We found three functions. And if Danny. If Danny is here, he spoke up in defense of this behavior of just saying, well, you know, if part zero is the head, then if you start counting from the end, then, you know, minus one.

Crazy.

But no, we can't do that because we don't do that in part. If you give minus one in part.

Exactly.

Well, that's an adjustment in our part. Normalization should always be consistent.

Okay. So yes.

I mean, does extract also not return the head with minus one?

Right. Extract. Yeah. Part and extract and delete. All agree.

Can't possibly. But no, I mean, it just crazy.

Okay. I mean, I, I agree with you, but since there was internal disagreement, the. The. I, you know, Danny was invited. I don't know if he's just being quiet or if he. If he was not available. Reason. But, you know, Danny spoke up in defense of this behavior, so I, you know, I felt obligated to, you know.

Does everybody know what it looks like in the code? Is this just a bug? I think it's just a bug. Somebody just didn't think of this.

Yeah, I think that's. That's probably because there's special code in part that treats the positive and the negative case separately.

Okay.

And that probably just maybe we should make an inline function that does that correctly and then just use that everywhere.

Fine. All right, let's move on.

Okay. All right, so the save now buttons. And this is why we want it to be in the prototype build. Oh, somebody evaluated those things. I mean, you'll want to reevaluate them. Yeah. Right. So we. We.

Right.

So you see it. It says data not saved. Saved. Now, this was a whole discussion we had about three, maybe four months ago, and. Right. It used to say data not stored or data not a notebook stored. Now, but then the sub menus were, you know, two out of the three possibilities. Didn't store it in the notebook. So now we added. Per the discussion from last time, we reworded that first thing and we added a cloud privately. So you can. You can save it either with public or private permissions. One thing it struck me as a bit odd that we say data not. We changed the main thing to say data not saved, but we continue to use store in the.

I agree.

So.

Okay, let me think.

Audio is another place where this menu appears and they use Store not only for the menu items, but for the tooltip on the. On the icon change storage location.

Julio, I mean, Carlo is here.

Okay, can you speak to the audio case?

Yeah, I mean, we've established that that is a kind of different case because audio file and audio in Notebook are two valid modalities, while for the summary box, the case is different.

Right, Right.

So I'm not sure it is super relevant for the discussion for the summary boxes, right?

Because with the summary box case, right, even though you're saving it externally, when you actually use it as input, it's bringing it in core, whereas in audio, right, it's. When it's, you know, when it's storing it in a file or whatever, then it's now sort of a fundamentally different thing, right? You have a different input form, you're now processing it out of core, etc.

Sorry, say that again.

With audio, right? The difference between. With audio, if you change the storage location from notebook to file, say, then you've changed. You fundamentally change the object from in core to out of core. If you then apply any processing functions to it, it will take place out of core. So it's sort of a different operation.

Why did we change this from data not stored store now to data not saved, saved now. Why did we change that?

Because. Because when. When it was. Because ori. All right, so the history was right. Originally, there was no. No things, right? Objects were either.

Yeah, yeah, right.

Then we added the ability to the. It was data not a notebook store now. And there was exactly one action which was embedded into the notebook. And then we added, you know, local object and cloud object, and it still said data not in notebook store now. And that didn't make any sense because two out of the three options didn't put it in the notebook. And so that's why we.

Look, I mean, I don't understand that you have. I mean, nothing is saved until you save. What is really going on in the kernel right now. That data is happily there. Well, okay, is it the case if I pick up that.

No, it's not there. I mean, if you pick up that thing right now and try to use it as input, it will fail.

Right.

There's one maybe detail here. Why does the summary box not use the same mechanism? That is that then when you click embed in line in a notebook, this box can retrieve the data from the kernel. Why is that not used in the interpretation function of this or the equivalent of in the interpolation function. I feel like that operation would be possible maybe. Well, I tried it and it does work. It has a. It would be a kind of confusing behavior because then you don't know when the reconstruction would work and when it wouldn't.

Exactly.

I mean, but there are situation where it could work.

Maybe, but it's kind of annoying. I've been burnt by it many times where I copy that, you know, object there without having saved it, and it doesn't work.

Well, that's why I originally wanted that, that stripe to be red. So it would be very obvious that, you know, you can't use it.

Yeah, I mean, we don't want. Okay, what is really going on? Is it data not included?

Well, it's, it's, it's. I mean, but no, I mean, even if it's not, you can't pick it.

Up and expect the data to be there.

Right. Because it's not saved anywhere than in the kernel memory. Right. It's.

Right, but this object that you looking at here is data not included, like batteries. So I'm saying this object.

Right, but, but, but, but, but, but even if you, but even if you save it in a cloud object, that the data isn't included.

I understand. Right, well, so what is it. So stop. What is it? The data is not accessible from this object itself. Right. That's what's really going on. Is the data accessible to this object itself? Right. I suppose that's what it is, if I'm not mistaken.

I don't, I don't. I don't think that's a useful description for a user.

I understand. I want to understand what's actually going on. Then let's understand what we tell people.

All right.

Okay. So what's actually going on is the data is not available from this blob.

Correct.

It's kind of like underlying data is not present. But that's very weird.

Yeah.

Because then pressing the button would suddenly make it present by magic. And it's a bit confusing to me.

Well, I want to go back to this possibility that within the session that the thing is still alive.

I looked at that code and I think that is a potentially few line change.

I think this is something we hadn't considered at the time, but I don't see any reason why it couldn't work. Modular things like the large, you know, if it overruns the size thresholds or some such.

But I think you'll end up then with other undesirable behavior, like, you know, when you quit the kernel or when it's no longer available, then.

Then you're going to have the same thing as Dynamics and many other things.

Well, yeah, well, the output.

The output size limiter can avoid this because there is. It can look at dollar session ID to uniquely determine whether it's in a new session or not. I don't know that there's any such thing currently in these three things.

We could do that here.

Right. We could add that.

Okay, so let's think about that case. Okay, so really what this is is data not permanently saved save. Now. That will be less confusing I mean, if it said data not permanently saved, but nevertheless it did work within the notebook. Now. Ok, so let's think about three cases. Right? Okay. It's going to work within the notebook, but it says data not permanently saved save. Now. Great, that's fine. Next case you didn't save and you bring this back and you're looking at this notebook again and it knows that it doesn't know. Get what I'm saying?

Yeah.

In that case, it should just be rad and say this doesn't work.

Right.

Data not available.

And I think that is already the current behavior.

Right. I mean, if you quit your kernel, we can see what it does. I think that's. I know you might need to like.

Okay, let's try.

I think. Which should be enough.

I know. Maybe you need to.

Yeah.

And you'll need to.

If you click that. If you click the Save now, it'll.

Well, well, yeah, certainly if you click on it. But I think me dynamically.

I don't know, maybe dollar SESSION ID is not a sufficient dynamic trigger.

Yeah.

I think if you, like, close the note, save the notebook, close and reopen, it'll.

Well, okay, let's handle both of these points. Does dollar SESSION ID rev on a kernel quit? I think it does, doesn't it?

I think it usually does, yes. I. I certainly have built other interfaces that work and I can't explain why this one is not triggered.

Okay. All right. So let's just consider that a bug for right now. Assume that this should work. Okay, so let's go ahead and we'll reopen that. What the heck is this? Wow. Well, that's 13. I mean, that's the prototype build, but somebody probably wants to pay attention to that.

That's probably on me.

I think maybe. No, this is. That isn't.

Nope.

That is a stone.

What if you press the bug?

What's that?

Even if you press the button. Did you just try that? And it didn't do anything.

Okay.

Well, that's Fine behavior. Yeah, but I think it should do something. Even in this case here, It should say.

I mean, there is a state where it says the data is not available. So I don't know if there's some sort of caching issue.

So then there's some. Some bug here, so we need to track that down.

Yep.

Okay, but. Okay, so look, we got several actions here. One, do the few line change that Carlo is suggesting to make it so that this can be used. Change it to data not permanently. Permanently saved. Save now. I would encourage a small additional space here to make it easier to read after that period, because this is right now further away than this is from that. Get what I'm saying?

I didn't quite follow that.

Okay, so let me go through it again. That space there between Save now and the. And the icon is bigger than the space between the first sentence and the second sentence. That seems wrong. Do you understand what I'm saying? Put a bit more space there and a bit less space here. Does that make sense?

I understand what you're saying. I'm not sure that I necessarily agree with it, but.

Well, why don't you agree?

I didn't say I don't agree. I said I'm not sure I agree.

Okay, fine. If you think about it and don't agree, tell me why the. Okay, but otherwise, I think data not permanently saved. Space. Space Save now.

Okay, yeah, that's just getting kind of long, but.

Well, yes, it is, but I think, I mean, we could go slightly smaller graphically. Okay, so let me point this out again. Given the behavior we are now describing, this could go even a point size smaller. Am I making sense?

No, I don't think so. Because the key point per, you know, the fact that it's not actually saved is the key point. Right? I mean, people need to know that it's gone.

I'm just saying that because they don't actually need to know within the session. They only need to solve.

No, no, no, that is exactly the point I disagree with. They need to know that the moment they quit the kernel, that data is gone. Or, or. Or, you know, they'll be annoyed like you are when they haven't saved it and it's, you know, and it's no longer available.

You know, It kind of makes me wonder. It's probably too late by the time you're quitting. You know, there are two actions. You quit this notebook and then you quit the kernel. Or the other way around, I assume. Just like it says, you know, notebooks have not Been saved. You know, you've got an untitled. Whatever it is, is it obvious that we shouldn't be telling people on session, quit on application quit that there are things like this that are unsaved?

I don't really see how that's possible. I mean, right. How would it distinguish this case from, you know, any of a gazillion other dynamics?

But I mean. Well, it's not obvious the dynamics. You know, if we could tell people use save definitions that might make them very happy because they are going to be toasted if they quit. They didn't save enough state, they try and rehydrate, doesn't come back to life. Right. They're annoyed now. You know, in the past, like 30 years ago when we first had all these mechanisms, we just didn't have the option to say we didn't have things like the cloud and local objects and all that kind of thing. Am I making sense? So I just want to review this possibility. Is it possible, Lou, this is more on you than anybody else to tag things like this that are trailing unsaved items and make essentially a queue of those. Now clearly there will be weird cases like you close the notebook without having saved things, but you didn't close the kernel, so in principle you could still get those things back. Am I making sense? Right.

In fact, you absolutely can.

I understand that works now for things like cell labels, right? The in out labeling pops back to life if you save and reopen a notebook in the same kernel session.

Okay, okay, let's talk about a possible logic possible thing here, okay. That we actually try and make a are you sure basically thing that that does roughly the following. If it turns out you've only got, you know, two megabytes of stuff that it would take to save everything locally in your in the notebook, then just says, hey, do you want to save it locally in the notebook with save definitions or whatever else. In fact, this raises the question, should manipulate have another thing that says data not saved? Or see what I'm saying? Just like this, rather than manipulate, not having any visual indication that saved definitions is false.

I mean it's perfectly reasonable to have a manipulate can be fully working and rely on the global definition of a user defined function. I don't think that's a state that needs to be warned against. I think that's a state we should just embrace.

Well, except that when you bring it back, it's not going to work anymore.

Well, when you bring it back and decide to change the function definition, it will work.

No, no, no, no. Or you haven't run the function definition.

Or you haven't run and then you get that. I mean, those are errors.

That behavior is not good. That behavior is. It's pink. And you don't know.

That behavior might be exactly what the user wants.

No, no, no, no.

Because the developer has carefully decided that this, this symbol should only have a definition in certain circumstances.

No, no, wait. Wait a second, wait a second. What is it? The fact that the manipulate is just pink without an explanation is really no good.

There's no pinking in the examples that I'm imagining.

Well, what does it do when it just does the wrong thing? Right. If savedefinitions is false and it's supposed.

It does the thing that you've told it to do, it's not the wrong thing.

Wait a second, wait a second. If x equals 1 is set externally and then you have a dynamic of X, what does the dynamic of X do if the X equals one hasn't.

Been evaluated, it shows you the current value of X, whatever that is, which is X.

In that case. Sure. If X has a graphic, it'll go pink. I mean, I can demonstrate. I mean, we just say if X.

Is a graphic, then dynamic of X will not go pink.

Okay, Dynamic.

There are cases which will go pink. That is not every case, Stephen.

That's the point I'm making. No, I understand, but this case will go pink.

Yes.

Right. Well, when people see that in a notebook, they are very un. They're just clueless. You know, they have no idea what happened. They have no idea that there's something you're supposed to evaluate there.

And what does hovering over the pink.

Same. That's very bad. Cell context backtick. Rad.

Other than the cell context backtick. Is it still very bad?

It's pretty bad. I mean, it's not great. Certainly doesn't tell you if you're an un. You know, if you're just a user where somebody sent you this manipulate and you have no idea how it works, it's extremely unhelpful.

Right, but it's.

I mean, look, let's, let's. Okay, let's take this to the user assistance meeting. I think this probably isn't the right meeting for this. Not the right set of people. I'm claiming that we should do something different with save definitions, with the. With the un. Unsaved definitions. I mean, does it know. It doesn't know. You're saying, Lou, that it could be correct, that this simply says symbolic. Is that right?

Yes, absolutely.

And therefore that it doesn't know.

What is it.

The dynamic has no idea why it's pinking.

Dynamic doesn't know that it is thinking.

Right. Only the front end knows that it's thinking.

You know, in a case like this one here, there is an argument that it should say things like, you know, it should give some indication that there might be a definition that's worth running, even if it doesn't know that there is such definition.

Because, well, if, if it's, I mean, in, in the manipulate case, maybe because we have sort of a whole bunch of, you know, architecture around us, but, you know, this is, you know, at the level of dynamic. No, that, that's just not right. I mean, that's like saying that, you know, Robox should have a special tool tip that says, you know, maybe you need to, you know, make sure that the robox has a single flat list and, you know.

Let's consider the manipulate case at least. The dynamic case, I agree, is a little bit more under the hood, but I want to come back to the, the case of.

So for unstored things like summary boxes, I thought there was already an idea, a concept for when these things exist and need to be filled in. At some point the user could choose to fill them in. Whether that's proactively on the user's part or whether it's prompted by us. I thought there was already a concept.

I think that was something, something in the toolbar.

But I, I, I.

Which I see, which I see you have turned off. I mean, we'd need, we'd need John for that. John is, is. John would know what the. But he's, he's out today, so. Which is why he's not here. He normally would be here.

Sorry, would know what I would.

The status of that, you know, save things. Right.

We have a notion of a save things thing which could either come up on notebook close or session or application close.

Sure. But I don't know that it could come up on kernel quit.

I understand it probably could actually. The front end can watch for that.

All right.

Okay, but okay, about what we're doing here. Okay, so that, that's the thing we need to make sure. I don't know if it's in this meeting or if it's in User Assistance. Can somebody determine that? Can we pick, please, the discussion of those of closing actions and so on?

I'll discuss with John.

I think it probably should be in User Assistance because it has a bunch of UX stuff and we're not doing ux and these Meetings. Okay, so with respect to what we're dealing with here, I think we've agreed that this thing is going to be preserved. So it's just going to say data. All right. It can say data not saved here. Data not saved. Is that really enough? Okay, I have one more complaint, which is I don't understand why the font for these is so incredibly small.

That's. Because that has to do with that. That doesn't scale, and I don't know if, if that's.

Oh, yeah. So it's a potential bug, so we have to talk to John about it.

Well, I think it's a question of. For Louis.

I don't know what you're trying to say. Itai I don't remember any issues here.

So. So if, If. Right. If you set the notebook zoom to 100%, you'll see that the, the. The pop up and the stripe have the same font size.

Oh, I see.

And then when you increase the zoom, the, The. The.

I guess.

Should hang together.

This is something. What's that?

I think they should magnify together. Or not. I think it's a bug.

Yeah, fine. If they magnify together, I'm a happy camper. Okay, Can I just raise something? We should probably put in user assistance, which is a bunch of these ancillary things, not magnifying or magnifying. And I would kind of argue that there should be a preferences setting for whether these ancillas or whatever one calls them, amplifier, you know, magnify or not.

There have been lots of conversations about this. The toolbar, the predictive interface, the image editors.

But why don't we just make a preference setting? Because, I mean, basically what it is, is. And it could be something that can be set on a, you know, style sheet for a notebook.

You know, I agree that for accessibility, that might be a useful thing. I just don't know if the designs, as they are, are amenable to, you know, will just work under magnification or how much effort it will be to.

Make it just right and each. Right. And each one of these things, it would need to. Might be individually updated to opt into this preference, which, you know, I'm. Right, right. Yes. All I'm saying. I'm not saying it's not worth doing. I think that, you know, there are strong opinions on both sides.

I mean, that's why we have a preference setting.

Right, Right. Which is why I think this preference is. Use is a useful thing. But I'm just saying it's not going to be an instant Fix.

It's also possible that if John were here, he would say, well, then you should be interacting with your OS resolution and magnification settings and not any particular applications. I can't speak for John, of course.

Yeah, I don't believe.

Well, I can, I can speak from. You know, there was a summer school student who had sort of vision problems and who I know did not set their OS magnification because it broke a bunch of other stuff.

But that meant those other applications don't actually work with the OS magnification.

Probably including the OS itself in some places.

Yes.

What OS may mask?

I think that was Windows.

Okay.

Yes.

Okay. So we can.

Right. So, so, so, so the main, so the main question, I think. Well, there's the. There this. The output size limiter that we still need to discuss. But before we move on from summary boxes, are we keeping store or are we making it safe to be consistent with the stripe?

I think we should keep store. I really do.

I think, I think it's not right. I don't have a pony in this particular race. I just. The inconsistency sort of jumped out at me as I was reviewing Lou's changes and I wanted to get confirmation.

Well, because there's multiple options there. Embed versus store. I think it's okay.

Okay.

All right. What about the output size limiter?

Right. So evaluate the range to the six and you'll see that there's a store. I think it's store full expression in Notebook. So it's like the old button, which did not have a pop up and we never sort of explicitly discussed this. Do we want to turn this also into a submenu and does that, you know, do, do we use.

I think it's interesting to be able to store this in that way. I would love to be able to do the same kind of thing with images, you know, with giant images in a notebook. That would be super useful, actually working.

On something like that. Updating the image boxes to have compressed storage that potentially is optional. Using the embed button.

Sorry. Not just compression, but I mean the basic point is that I am regularly getting, you know, half gigabyte notebooks and things are getting very slow to save and all that kind of stuff. Yep, yep. And I really want those things to be externalized. And in fact what I would like is, you know, an option there to externalize in the same directory or in some specified subdirectory.

So. So Giulio is. It is. I mean, Carlo. Sorry, I don't know what. Give both Italian your Your indistinguishable bosons. The.

Excellent.

Except this. Carlo has a Scottish accent to Italian. Okay. Anyway.

The. Is. Is this for out of core? My understanding was you're trying to actually stick the tiff into the notebook or.

Yes to.

Yes to.

But the idea is to minimize the size of the images by compressing them within the notebook. What Stephen was kind of hinting at is kind of the opposite way. Not extracting them from the notebook and placing them somewhere else.

Yes. Which is what we want to be able to do. I mean, we've got some tools for doing that.

Yeah. As I said, notebooks, basically, I am kind of. We're trying to figure out exactly how this should work. And I have a branch where I'm prototyping all of these things. I think that should not be.

By the way, could somebody get added to the global Project list? This business about handling large images in notebooks, maybe that's Claudia's. I don't know whether you know how to do that yet. But. But do you know how to do that? Do you understand that? Do you not understand? Does somebody else here understand?

Yeah, I can keep track of that if you want.

Well, it's not for you to keep track of. It goes on the Global Project list. Right.

Okay, well, Avery can help her if. If she doesn't know how to do that. We'll take care of this in our weekly meeting tomorrow.

Fine. Okay. But the basic point is that should be a project because it's a long running, flapping creature.

Okay. Right.

It's not. It's not to be handled separately by the project management. For this meeting. It is to be handled as part of the Global Project list, which. Which is a different thing. Okay.

And that has several tentacles. One, what is the representation in the kernel? And two, what is the representation in the notebook itself? Essentially, image of a file.

Sure.

At the moment is not supported.

Okay. And three, you know, we've got to have a way of having those assets like, you know, like web browsers do, like website, you know, of having those assets in separate files. Am I making sense? And I know that John has looked at this extensively for, you know, assets. Separate assets for notebooks. Are you aware of that, Carlo? Yes.

And those at the moment, they are in. Yarn has an implementation in prototype. They are still within the notebook, as far as I know. So when you save the notebook, that will have the size of the images or whatever asset added into that file.

Size, but somehow they are put in a different place in the notebook file. And so it doesn't slow storage. It doesn't slow saving down or what?

The last I heard from Ian, and maybe Lou has more recent information, is that this still has some size limitations.

Okay, that's not going to work then. So, I mean. Okay, let's get a separate meeting specifically about this thing about image size and notebooks. Fair.

Yep.

Yes. Okay. And because I think we should build something analogous to the data not saved, save now mechanism for images which could go in the image toolbar.

You know.

What I'm saying, and the image attached cell. Did you understand what I'm saying, Lou? Do you understand?

I understand, yes. Although that toolbar is no longer maintained by the front end group.

Okay, I don't care.

Right. I'm just.

Yeah, I understand right there. Okay, let's look at this thing that you're raising about the full expression thing. Okay, so it does seem useful, does it not, to be able to store the full expression in any one of these places? Yes or no?

I think so.

So I'll point out one thing. When you store it in the notebook, the browsability of this expression is. Is remains, and the browsability is now depending on the thing that you stored. So if you were to store this in a local object, then you do Show More show. All those buttons would hit the local object. If you store it in the cloud, those buttons would hit the cloud. And that's all. That's all.

Okay.

Hitting the cloud is fast enough for this kind of operation that we're happy with.

I mean, if it needs to have a, you know, a spinner that says it's doing something, then so be it. All right.

I just want to make sure that we understand.

I understand that fully. Okay. I mean, it's the same thing for any of these things.

Right. But these other guys, it's just the. There are two states during parsing here. The time would be spent interactively.

Yes, but I mean, if Show More has a delay saying loading from cloud, which I doubt it will have, because I don't think these things are that big because a single Show More is not showing that much extra unless there's some problem with having to load the whole thing on Show More.

I mean, so I. Well, I think if you're still in the same session.

No, no, no.

But not in the same session. It would have to load the whole thing because. Right. That's basically right. A short of shallow. It's like. Right. How. How does it know? You know? Right. It's showing you the beginning and the end. Right. So how does it know?

I Don't think it can do anything other than assume. I assume it's going to save in wxf. It will probably be fairly fast to come back in. That's how it should do it, right? I mean, it managed to save it in the kernel once, so it can save it in the kernel again. Am I making sense? Yeah.

Right. And so should. Do we want to change store to save?

No, hold on, hold on, hold on. Why aren't we using the same wording?

Well, this is matching the. The previous. Oh, you mean, well, data now. I guess we could do that.

I think it's the least confusing to just do that and standardized on that.

What about. What about full expression not saved? Just to sort of make it.

Yes. Better.

Okay. And that. And then the menu items are the same.

And then save now. And then the icon and use the same menu items.

Okay, perfect.

And. And then down at the bottom at the footer, that's the thing that says expression size. Might as well just say it again, even though it's going to be probably the same as the size in memory.

I would hope so.

Yep.

Or do you think it's not necessary as the footer in this case because it already.

I'm sorry, I think I missed something.

Okay, right now it says at the beginning of that line, size and memory, eight megabytes.

Yep.

Okay. @ the bottom of the menu it would say again, expression size 8 megabytes. Unless we don't do that. Maybe we should cut it off from there. It'll be. It's confusing. See what I'm.

Saying?

Yeah. Well, certainly having two different text labels for the same information is.

Odd. Yeah. So I think we should just not include it in the menu here on the grounds that it's already included in the footer. Makes.

Sense.

Yep. I'm okay with.

That. Okay. All right. Should we move.

On?

Yep. Okay. New key or key value to check.

Arguments. Right. So this is a specific problem that, you know, we've run into with using check arguments to implement operator forms. So some operators like map, some functions like map, allow options when you're using them normally, but don't allow options when you're using it in as an operator form. And so you see, you get a different message in between the first output and the second output. That reflects the fact that, you know, options aren't allowed in the second case. And so check arguments certainly currently doesn't have a way of getting that behavior. Right. And so there were a couple of proposals, only one of which I actually really like, which is right we have this options mode key to customize check arguments, which can be longest or shortest, which has to do with, you know, our rules treated as. As option names or as.

Arguments. And this was setting that to none means don't allow options at.

All.

Yep. Sounds reasonable to me. Okay, let me just have a quick look at what. How check arguments handles that. Check arguments has an association in that.

Case.

Yep. I think I could go either way with this. I don't really know. I mean, maybe it's clearer to have two different keys. Or is it just obscure? Because if you give options allowed false, then the other key is just flapping.

Around. Right. Then the other key is sort of flapping around. And also, you know, option, you know, options allowed. Sounds like it might be a list.

Of. I see. Which options are.

Allowed.

Exactly. It will be allow options.

Right? Ah.

Yes. But I think Options mode to none is.

Okay. Okay. I wonder why is the heads not. I feel like. Oh no, I'm sorry, I'm confusing it with arguments options. Never mind. Okay, so this is a big thing which, you know, we may not want to get into now, but. Right. There was. There was some discussion and you know, there was an email thread and you said we should discuss and I mean. So the basic problem.

Is. Let me try and state a version of this.

Problem.

Okay. First point is, right now, background is just a.

Color.

Yep. I have been asking for like a decade, more than a decade, to have background be able to be an image. I.

Agree. I think that's the direction we would like to go. I think this was me thinking out loud that maybe an infinite sphere that was textured would be a thing, but I don't think it's the right approach. I think a background. The background option could be more flexible. I don't know what's involved in implementing that, but that's the right direction. Perhaps even support.

Tiles. Right. Well, I mean, this question of how background works versus how texture works on a polygon versus those kinds of things is subtle. And maybe we don't have the right set of people here. But I think the correct thing here is that you know, background quite globally, whether it's for a cell in a notebook or whether it's for a graphics object should be capable of being an.

Image.

Right. So there's two things. One is just background supporting images when it's 2D, which is fine. Then there's background supporting images when it's.

3D. By the way, even in 2D, it's not obvious what to do because what is the size of how does it scale the.

Background?

Sure. Right. Yes. Right. So certainly if we're going to start this more general discussion, then I would say we should postpone it until John is actually not on.

Pto. It's not only John who's needed. Brett is also.

Needed. Well, Brett is.

Here. Okay, well, maybe he could opine on this because I think we should start thinking about this, because background, that is an image that is, for example, square. But then it's going to be the background for a plot that is, you know, golden ratio, aspect ratio. What does it do? Right. There's four or five different approaches that things take. Right. And I mean, we already have you stretch, you clip in one way, you.

Tile.

Right. So we already have some options to other functions that do those kinds of things. Don't we. Have some image paddings? You know, image padding, I'm pretty sure has that. And doesn't image padding actually allow a background image to pad with image padding is an option. Is it pad image or image resize or. What am I.

Talking. There's also the maybe related question, which is in a 3D case, what projection do you assume the image is.

Using? And orientation, indeed, you need to be able to rotate. This is the case here. Image pad.

Yes. I argue this is actually analogous to the projection and rotation.

Issue. Let's talk about that issue for a second. So the question is, you rotate the object, what happens to the background? Is that one.

Question? Well, no, no. So there's. There's two quests. Well, okay, I guess that's one. I mean, the background could be fixed or could move. I was assuming it would move.

But. But.

The. I'm assuming it was the fixed.

Stars.

Right. Okay, so if it's the fixed stars, then there's two things. One, the texture you give for the background, what projection are you giving your texture in? That's one question. But then the second is which part of the texture is facing do or.

Something? Right. Well, how we define that? I mean, so, you know, this is what spherical image and other related things were supposed to deal with, I believe. And it's something for which Astrographics has, you know, Astrographics paints the fixed stars on the celestial sphere, so to.

Speak. Correct. Yes. So I would expect that we provide an echo rectangular image and that then it's converted effectively into something that looks orthographic because it's at infinite.

Distance.

And.

Yeah. So that means. Wait a minute, does it distort or does it not.

Distort? It doesn't. I mean, it doesn't distort with respect to the view that you expect in.

3D.

Okay. Because what we provide originally was distorted because it was a standard equirectangular.

Thing. I.

See. But I don't see any other easy way of providing it. We cannot provide, you know, a circle image or something like that. That would be too.

Difficult. Right, right. But I mean, this then relates closely to the spherical image.

Problem. Exactly. Yes. And for example, the in wfr, we have a spherical image function that gets a rectangular image and puts it on a.

Sphere. Okay. What's that function.

Called? A spherical image, I.

Think. Let me just set up for a second.

Okay. The different thing in Astro is that we want this fear to be at infinity. Rather than the.

Geocase. How is this useful? I don't understand this.

Picture. So that's. I mean, if you run that in a notebook, then you can click on that image and move.

It. You know, one thing that just drives me bonkers is the fact that the click to copy here isn't. Doesn't have this. Inclusion. Click to copy. Inclusion. What's happened to that? That should be another thing on our project list, by the.

Way. But you just mean that it copies both.

Lines. Yeah, yeah, that has an.

Option. The person who made the notebook should have just included them both in one.

Cell. Well, except we don't need. We don't want that. We want a multicellular implied copy. Okay, so now what can I do here? Oh, wow.

Cool. Okay, so your rotation mode's probably not quite right, but it's the basic.

Point.

Yeah.

Yep. It's kind of reversed. You know, you're moving your mouse up and the sky's going down so.

Well. But, but that's probably what you. If you imagine There was a 3D object in the middle of it, then isn't that the direction you'd.

Expect? This is like a trackball of some kind. I mean, I think it. It doesn't seem wrong to.

Me. There's different rotation controls through the method option that could be specified to not have the reverse behavior. But you know, if I think of this as like looking through a.

Telescope. That's. Telescopes flip things around, don't.

They? No, I. I'm talking about a non reversed image. But the point is, you know, you. You move your telescope right, you expect to see things to the right, and you're getting the opposite of that here. You drag your mouse to the right, I expect to like grab a hold of the sky and move it off to the right, and you get the opposite.

Here. That's a minor.

Configuration. It is. It is. It's. It's a minor.

Thing.

Yes. Where's the actual telescope in this.

Picture? It's inside.

Those. Inside.

Structures. Inside those.

Structures.

Okay. I don't. I don't know which telescope this is.

Actually. There's the Southern European Southern Observatory. I thought telescopes usually had domes so they could point in different.

Directions. Yeah. These days I think they're not traditional.

Domes. Okay. In any case. Okay. But the point is this thing. I don't even know what the heck this thing is. This thing, you're saying the original image. If I look at.

IMG360. Yes. So that's an equirectangular.

Thing and. Oh, Okay.

Yeah. So I think this is the right thing. You provide a rectangular image, and then something takes charge of projecting it.

Appropriately. Although there's a focal length issue. Right. Of how much of the image you're going to see and things like.

This. Yes. So it's an.

Aperture. Yes. There's the field of view, but then there's also. Can you look straight up in the spherical.

Image?

Okay. The question would be whether or not, I mean, the pixels would all be crunched at the. Near the.

Top. No, this is done differently. So these. What is down here is to have tessellation of the sphere into many polygons, and then pieces of the image are projected on those polygons. That's why there are no effects of accumulation at the.

Top. I don't understand up here, this thing. Why doesn't the same star get copied many times at the.

Top? Well, what I'm saying is that if we do that, if we did a parametric plot 3D, then we would see effects of many pixels accumulating at the North Pole, which is, I think, what Christopher was.

Commenting.

Yep. This is avoided here by using a different method. So first the sphere is tessellated into thousands of polygons, and then pieces of the image are projected onto those polygons. So nothing special is happening at the North Pole. The North Pole just has another.

Polygon. I'm confused. Which. Which piece of this top line is. Is the North.

Pole? The whole.

Line? But. But this whole line has different stars at different places. So why doesn't that, you know, why doesn't that confuse the question of what's at the North.

Pole? Well, if the equirectangular projection of this image is. Well done, then it should happen that the very same star is at the North Pole. It's just that probably the sky doesn't have any exact star at the North.

Pole. I see, so you claim it's black there. But if there were a star at the North Pole, the whole of that, that the light of that star will be, will be spread across, will be copied across that top.

Line.

Exactly. So when that star, its brightness would be, you know, it'd be shattered into pieces and then its brightness would be averaged over those.

Pieces.

Right. So it would not accumulate. Right. Because you would have. It wouldn't be that the brightness is. There will be a copy of the star many, many times.

Here.

Yep. Yes. What I'm saying is that this function has a method of dealing with that that is non trivial and better than a simple. Let's just put all those pixels at the same.

Place. Yeah, I get it. Okay, well, okay, so. So then we basically we've got to take image. Okay, thinking about background, first question is there are a zillion functions that have a background option, I think from, you know, from framed to all kinds of things. Like should framed allow a textured, you know, background? I think yes. An image background. Actually, when I said textured background, what about things like filling patterns and so on? Those are part of graphics. So in general, one could imagine the background could have either an image or graphics. Do people.

Agree? So presumably with additional metadata though, to say what to do with that.

Image? Well, exactly, but that's what Image Pad has to do right now. Right, That's Image Pad has to tell one. You know, does it do this, this, this, this, this make.

Sense?

Yep. And I mean, and it's also got to have like inset. It's got to have a way of deciding, you know, where to place that image relative. Because Image Pad doesn't have that issue because it's just putting padding on the sides. All right. This probably is another thing for the global project list because it goes across a bunch of different groups and a different bunch of different places. This would be a lovely thing to have. I mean, to me, even doing it in the 2D case is highly interesting. The 3D case seems. What are the immediate use cases of a 3D.

Case? For us it's just Astro having an easy way to put this background image of the.

Universe. Now, perhaps slightly related, maybe not would be, Jose, for you to have a Geographics 3D, but that's kind of like more like a Google Earth thing where it's not a background. In that case, you're kind of looking down on a textured stuff.

Sphere. That's very different, Jeff, because the sphere is finite It's a finite distance and we need to control where it is. And we want to have things just in front of.

It.

Right. It's not really a.

Background.

Right. For a future geographic.

Study. All right, well, something to look at. So we need to discuss this with John and others. But a good thing for us to have. And yeah, I mean, there'll be a bunch of subtle issues, but you don't need this for Astro, because Astro Graphics already has a mechanism for dealing with this, doesn't it?

Yes. The two DKs we handle ourselves. Jeff is asking. Jeff is thinking of the future 3D.

Case.

Right. I have a kind of a prototype of an AstroGraphics 3D in the form of a WFR called telescope view 3D. And you can say, you know, telescope view 3D of Mars, and it shows you the current 3D orientation of Mars as seen from Earth or from the moon or from wherever you want to be. But there's no background. It's just a black sky. I don't have an easy way of putting stars behind Mars in.

3D. I think you should solve that as a custom thing before. I mean, to me, the spherical image thing is really its own thing intended for VR type.

Thing.

Right. And I mean that that format, whatever it is, the USD format or something, is that relevant to this or is that something different? The USD format is what we need. You know, what we can now do with, with AR publish and so on. I don't know whether that relates to this. This may be yet something different. Do you know I'm not from USD. Well, that's what we just added in.

13.3. So my understanding from Charles is that, you know, this sort of. You know that. Yes, the, the. Right. This sort of, you know. Right. They call it a skybox and it sort of exists in USD and sort of game.

Engines.

Okay. This image at infinite distance, obviously.

If we were able to put our background on that, then people could orient their mobile phones and see the stars that correspond to the.

Orientation. Well, let's see if we can figure out how to do that. That would be very nice. All right, I should run here. Good stuff. Thanks, guys. We more or less got through the agenda. Good.

Work. See you.

Soon. Thanks for the comments on the live stream. Bye.