Live CEOing Ep 803: Language Design in the Wolfram Language [Annotation and More]

Foreign.

Hello, everyone. Welcome to another Wolfram language Design review. A suspiciously short agenda here. Somehow the short ones end up not being the shortest.

There's a good number of subsections also. I kind of cheated in condensing it. So both of these topics, I believe, actually came up in brief during the image processing and signal processing reviews. And the first topic is kind of a large one, which is we have a number of different systems or attempted systems in the language to try and annotate various things. So we have lots of wrappers that potentially need annotation, potentially some string objects. And we've had various methods of trying to do them in the past. And I think there's a couple of discussions about we need a slightly more general system for how we approach annotation. And so I think there have been a couple of meetings in the past. I tried to kind of summarize the approaches that have been proposed. We have some instances of things where we have the annotations containerized very close to what they're annotating. Things like text element or XML element, where we're very heavily nested. We have other things where we try and have annotations either through options or through an extra wrapper function like annotated. And there's some benefits to each approach. It's a little bit tricky to have something that works holistically. There's some examples up a little higher of how Ian has attempted to do some things like using trees for annotating. So the tree section shows how annotations can be used by nesting trees together. So, for example, yeah. XML element is something where we have annotation at a very granular level, and then we kind of nest them together to build a larger annotated object.

I mean, by annotation, you mean this sub value thing or what, Right?

Yeah.

I mean, this is fairly grotesque, as we all know, right?

XML element is basically impossible to read.

At a glance, but now is the tree version of it better?

So there's two versions of it there, basically. And I've done the same thing with text structure, which is another thing. I think were it implemented today, it would probably naturally output a tree rather than the nested text element version. But this, there's kind of two versions there with expression tree. One is maybe your more traditional sentence diagram, and the tree fold version where it's actually inserting the annotations on the nodes themselves.

Right. That is something, by the way, that we should look at is elements for text structure that actually give trees. Right?

Yeah.

Yeah.

Okay.

You know, there's other element we've talked about with quantity. For example, oftentimes it's important to be able to annotate a quantity with a physical quantity that's associated with or potentially have a physical quantity that has a quantity inserted inside of it. And we don't have a general framework for how that works on, you know.

All right, let's take a look at these different cases.

Yep.

Okay, well, I mean let's remind ourselves what this actually looks like. I mean the text structure thing is not completely stupid and. Okay, I don't understand. I don't know what the, what the input forms of these things are.

Yeah, text structure is very similar to xml.

Yeah, this one is not a great heck is that this is a WFR function, but it's showing another case where it's potentially nice to be able to take something much more raw like a string and provide annotation without going straight.

Into a. Yeah, I mean that's ultimately a post annotation of some kind.

Right. The way we'd probably do that is have an annotated string with some sort of information that says, you know, these characters or these positions should be highlighted.

All right, so look, look, here's what I think. I think all of these approaches to annotation are useful, however, I think it's like expressions like we have expand, we have factor, whatever else. What we need are some standardized functions for converting from one form to another. So for example the cat and the hat here, let's say I had got as a list of words and I've got wrapper type things indicating those highlights. I'd like to be able to convert it into a annotated string with an option as a second argument.

We have a prototype for that in the proto build. You can do highlight string and that will do exactly that. But there's some non trivial stuff about how the front end renders boxes and new lines that.

So we started my design idea rather than saying one of these approaches is better than the others to just be able to interoperate them by having a bunch of translation functions.

Yeah, I think that makes sense because ultimately the application changes. Which approach is better for you? If you're working with strings, it's really nice to be able to keep them as ASCII strings or at least string expressions rather than something more complicated most of the time. But sometimes you may want text structure out of it or you may want to turn it into a tree.

What do you propose? What are you saying, Julio? What was the.

If you just Copy that thing and replace the head instead of highlight text with highlight string function. Oh, you have double brackets. Yeah. Surprise was working in my motor.

Okay.

Yeah. So.

You see, this is garbage.

No, it's not garbage. I mean, this is prototype, Stephen. But this is an annotated string. Yeah. So we cannot annotate a row object. So this is an object that stores the string and the annotation. And the job is actually done at typesetting. So this is not actually altering the structure of the object, it's just storing the string.

The question is, do we want a function like annotated string, or do we want a general annotation wrapper that can annotate a bunch of different things? That's not obvious to me.

I don't think a general annotation wrapper can work in any way, shape or form.

Okay. But I mean, some of these things like this. Let's look at some of the other cases.

We have annotations, but I don't think it's worth having an annotated string. I think it needs a more general string object that one of the things it can carry is annotations, because all of these things lead to better workflows, basically. That's the main thing that they do.

Yeah. The big problem with string is that string is a row object that you pass around that I'm not sure that we can have a different string without massively impacting.

I mean, the issue is what happens when you say string reverse on an annotated string. What happens when you do all of the string operations on annotated strings?

Nothing. This is a prototype. And also, this was. No, no, I see your point. But the thing is, this is what we called for images annotation calculus. Like if I add a bounding box and then I do an image reflect or an image take some of them, I might be able to propagate, like image reflect. I could go inside and apply that. Transformation is well defined to all the annotations.

But let's say that we annotate images right now.

We don't.

We don't.

Right.

Okay.

There are prototypes in prototype or in one branch. I don't even think it's in prototype builds or we never finalize that. Okay, but the way we annotate audio, which is the idea we wanted to use for images as well with you, was by using an annotation rules option in audio objects.

Right. Which we don't happen to be able to do with string because string is deeply raw.

Right.

But, you know, it's Not. We could also have a thing where we say something like, you know, if all we were doing was string object or perhaps annotated string, I mean we could imagine what are our raw objects. I mean, we've got. We could have an annotated bytearray in principle.

I mean, at this point the annotated head seems to make sense in the way that it could apply to any object.

I think an annotated bytearray would be completely useless.

Why? I mean, it's like here's the beginning of the. This is a byte array that I got from an importer, here's the beginning of the content, this is the header, blah blah blah. I don't think it's the most useful, but I don't think it's completely useless. I mean. Okay, let's remind ourselves how graphs work with annotation.

I think possibly a good ref page to look at is annotate. It has a lot of good and easy to understand annotation graph annotation examples.

But we got rid of the thing that was just called property, right?

Yes.

I don't know why it's listed here.

I mean, and by got rid being renamed to annotation. Yeah, I think it doesn't behave drastically different from the old property.

Okay, okay, okay. So this is the notion of annotate. Which returns.

The same object.

Yeah, I mean this requires a structure object.

Right.

Because it needs to understand what is that on the inside? What is that thing on the inside?

It either translates with the graph object with annotation rules. If you have special annotations.

Oh God. Evaluate G. Yeah, I don't want to do that.

Or, or in this case of graph where we have edge style vertex labels that are quote unquote standard annotations, they end up in other options like what you have in there.

I can't read this.

Well, the thing you care about here is that you have this edge style undirected edge 1, 2 goes to RGB color. And this is because edge style is already supported by graph. So it ends up just dumping it into the edge style.

Right. So in other words, this did the auto transformation of what was said that we said annotate. Actually I don't use this functionality. But okay, so this is a quintessential example of something where this seems to be happening automatically.

Yeah. And likewise, if you use wrappers on input, when you create graph on vertices or edges, those get translated to annotations.

Okay, if I say graph, graph of, you know, one goes to two, two goes to one, one Goes to two. And now can I put on this style of two, comma red or something?

Probably not there, but in the vertex list you could do that. You could definitely do style on the edge in the edge list. But you can't style the vertex in the edge list.

No. Well, yeah, the problem is that those are two different vertices at this point. Yeah. But if you. In the first in the vertex list.

Which you didn't give, this should work.

Yeah, yeah.

And if you now had a vertex list like 1, 2, you could style the 2 vertex with a wrapper. If you look at the input format, it still stores them separately. See?

I see.

Because when you compute on this graph.

I know you don't want to care about the edge style.

Yeah. You don't want an RGB color hanging around on your edge.

Right. So it's the same type of thing as if I say, you know, X, X, X and I color that X, you know, you know, green, let's say. Right. So those X's are all the same. That's a styling annotation purely in the front end. Now, we could imagine. Well, okay, we could imagine something that was an annotated X that was green, where the evaluator specifically over, you know, doesn't care about the annotated part. I mean, that's a bit dangerous.

Yeah, I don't think that's a work.

It's, you know, it's one of those things where it's like angels, you know, immovable forces, you know, what's the length of the thing? Well, is it one or is it two type thing. Okay.

So.

You know, I should be using annotate more seriously. Okay, so now talk to me about annotation rules. So that's Annotate.

Now if you scroll down to scope and then open maybe vertex, vertex annotations. The beginning examples are just those standard annotations. There should be a last example in that section that says add a custom.

Why there aren't separators here in our documentation style. Anyway, minor point. Roger. Do you understand?

Oh, one big example that reuses G.

Yeah, we shouldn't do that. Shouldn't do that. Should just have that graph.

That's just because the copy paste rules on the web are so funky. But go down so you can see a custom annotation. So all these are standard annotations.

Let's see that last one. Set a custom annotation.

Oh, there it is.

Material goes to wood.

Okay. And that's stored in annotation rules. Is that correct?

Yes.

Okay.

And where to Indicate where. For graphs, we have a way to give vertices and a way to give edges. Those are the possible wears and the whole object as well.

Right.

Now for other objects, the where is different. Like for audio, true.

It's either a timestamp fix, a one time or a time interval.

Right.

For regions it's again different.

And for image, it would be again different.

Yeah.

Okay, so why does this annotate function not work on audio?

It does.

Why isn't it documented then?

I think there might be tentative examples that I forgot to untentit. I'll look at it.

Okay.

So I think whenever we have an object that's not absolutely minimal, we can build these. It's sort of not super hard to do to include annotations like an image.

What about graphics?

Graphics is a little bit complicated because it's not. I mean, we would have to think about that for a sec.

Well, you didn't put this in, so this is not adequate. Okay.

Audio annotate function should have examples. I can just port them from there.

Why is there a separate audio annotate.

That says automation inside? Yeah, that is the one that is.

This is the one by property presets. Right. If you open details.

Annotate is neutral. You have to write manually what you want to annotate with what this thing here knows about audio and can basically create the annotations for you.

And then for images it would be.

And in principle, I just realized a cool use case. If we line up the workflow. So, you know, this new speech to text whisper thing, we know the time intervals for where the sentences are said.

Yep.

So in principle we could had the actual text as annotation to an audio and it would be an annotated audio, if you like.

Yeah. Which would be good.

And then where this topic again came up in.

Now we can do karaoke.

Yeah.

We could also do bounding boxes, presumably in.

Exactly. In an image.

Yeah, yeah.

And that. That is the workflow. I mean, this is a typical radiologist or dermatologist workflow. Right?

Yeah. Right. So look, what we need here, presumably, is generic annotation rul. And then specialized things like how do we indicate, you know, highlighted regions in an image? We have highlight, highlight, image.

Right, exactly.

Which today produces graphics. So it would be really good if that. That actually was, you know, an image. We just need to work at it more and we'll be able to do that.

So, okay, so the idea would be. Yeah, that's kind of wrong, presumably.

I mean, I changed this like many years ago. Simply because it used to, in the first version was just an image.

No, I understand, but you know, that.

Was terrible because it would rasterize lines and stuff like that.

Right. So really what it should be, there is an image wherever we had that. Well, I don't know where we have that. Okay. It should be an image with, I don't know, you know, highlighted regions or something.

Why not annotation rules?

I mean, I think annotation rules can be even more general. Right. Than.

Yeah.

What I was thinking is that there's a specialized thing here. Just like you say edge styles, there's a thing that has an immediate meaning to the image object. Whereas annotation rules is just a carrier that you can have anything in that won't be interpreted by image.

Yeah, this is very important, I think, because a highlight image could produce that new rule. And my reasoning is as. Let's say that you annotate images with a bunch of things. You want to store them, but displaying all of them is going to be messy. So the highlight image could have a fast syntax that looks at the annotation rules and just you highlight, you promote to highlighting one of the annotations, like, you know, the cat bound in boxes.

Right. And then highlighting rules will be a thing that actually renders, just like edge styles actually renders.

That's going to be like epilogue, like the model is epilogue for plots.

Right.

It's an overlay, highlighting rules, overlay. That would be things like. Well, I don't know whether it would either be a bounding box where it might be. Hopefully we're going to get shapes other than some shape.

In principle, you can have everything. If you go back to the highlight image that we have. Points.

Yeah, points.

Then we have any 2D geometric region. Yeah, exactly.

Okay. So I mean, all of those things will be parts, left hand side of highlighting rules, right?

Ideally, yes.

What do you get when you export it to a file? Does it drop the highlighting rules or does it rasterize?

Yeah, almost no format supports the. I mean, if you have.

No, no, but the highlight rules element.

It has to be, it has to.

Be an export option. What to do with these. And I would say most of the time. Yeah, right. Most of the time we would want to include them and rasterize them. I think it'd be infuriating to have an image that you've, you know, you've carefully annotated and you export it for something and it's high enough resolution. Doesn't matter about the, you know, raster artifacts and you suddenly Your beautiful annotations that you put all this effort into just disappear.

I mean, sorry, please.

I would say we can. It's easier to opt into that rasterization by just calling rasterize of an image and expect all of those highlightings get rasterized into an image and then export or with an option like include annotations.

Yeah, which should by default be true.

I also have a feeling that it should. I mean, in a sense that is what you look at. Yeah. The image object should be clean by.

Default and you know, people can set it off if they don't want that, you know. Okay.

Stuff is not going to be highlighted by default. Yeah. If you annotate it, you need to highlight it.

So that's a good point.

Right, so, okay, so let's take a look now at. So in this model, so audio, how does it do its annotations? How does, how does it represent an annotated audio?

It's always in the annotation rules. And again, the where is a single time or a time interval. The what can be anything.

Okay, but there you're putting it in annotation rules. Whereas what we're saying is things that actually take action on the object should be in some specialized option, if that makes sense. Right. The annotation rules is a generalized carrier that.

Well, there's one distinction here that we notice while looking at this for like three graph graphics versus image and audio. The first group is built up of separate items, while image and audio, they are ideally a continuous region that you sample either by pixel or by sample rate. So annotations for image and audio live in this real principle.

Imagine image data. You get out image data and you start spraying style block red onto the points in the image data.

That's true, but then you'll be limiting yourself because you will be back rasterizing your annotation. You can imagine that, but I don't think it's the right mental model for image annotation.

Yeah, that's kind of interesting, isn't it? Yeah. Okay. I don't use that as much as I should. Okay, so let's keep going here. So now we've got text structure and we've got strings. And I have to say I, I am in favor of an annotated string construct. Am I making sense?

Yeah, I think with anything low level.

I think that you should think more general, Stephen, because, you know, like the string object you had. I think it's a better idea with annotation in, you know, and then it's sort of Future proofing it for other things that we might want to do.

Or an extended string or something.

Some. Yeah, something.

But.

But even if we just brainstorm half an hour or something, we would come up with more things that would be nice for sort of a text slash string workflow.

Look at this. Has head string.

It's kind of interesting that that would break a lot. Very. I mean, extending string in that way is very dangerous, viewed as insane.

There.

There are lots of functions that are just looking for like underscore string and not checking string queue.

Right. And. And they would all break if we did this.

Okay.

Which would be a great way to fix all those functions.

All right, I volunteer you, Julio, to go and fix all of 30 years of formatting code to do this.

Yep, of course.

Yeah, that's what we should spend.

Okay, let's keep going here. So text structure.

So. So, but that means too, that there could be other annotation, not just highlighting for a string.

Yeah, yeah, I agree. But that's what this is doing as well. That's what the text structure is doing. And maybe we don't need. Maybe we shouldn't confuse it and say it's a string, but rather say it's a piece of text. Maybe not. Maybe that's wrong.

I mean, the reason why I don't like text structure is because text structure is doing in input form what annotated string is doing in typesetting. So the input form of that is basically xml. That means it is very cumbersome to go back to the string.

Hold on. What we're saying is. Okay, remember my general principle at the beginning, we've got several different ways of doing annotation, from nested symbolic wrappers to options to whatever else. My initial comment was that we should have ways to convert between them. So, for example, this thing here, we could in principle turn this into an annotation rules option to something which will be convenient for some purposes and not for others. Do you see what I'm saying? Yeah.

So I mean, today, does string operation work on text structure? I mean, like, if I do string length, does that work?

No, I don't.

No, no, no. And this is an end of a computation. I mean, this is display, basically.

Right.

Okay.

Not that you just display, because I mean, this is actually like xml, so you can do pattern matching and operations like that.

Yeah, yeah, I got it. Let's review a couple of things. Okay, so image and audio and things like that are, as you say, special because they have. They have things that are sort of a blob that can't be broken apart. They do not have the possibility of in place symbolic wrappers. Right. Whereas some of these other things do have in place symbolic wrappers. Like text structure has an in place symbolic wrapper. So does. What are the other ones that have in place symbolic wrappers like GR. Graphics.

Graphics.

Mesh region.

Mesh mesh region, which are not listed here for some reason.

I would include notebook and xml.

Well, I don't know to what extent notebook and cell and so on have the ability to have symbolic wrappers.

I don't think they do.

I'm sorry.

I think they. I think they are the symbolic wrappers.

Okay, so here's the question. Question. Is this in. In, you know, for example, in list plot, we have the possibility of having symbolic wrappers that say, you know, if I say, for example here, if I say list plot of one, style two, red three. I get that.

Yeah.

All out are labeled.

Yeah, but so the way that that works, which we sort of introduced in for charting 7, is that there is a parser that sits on front and strips them and creates the structure.

I mean, this is very nice. It's very nice. And the alternative here of small things.

This is super nice, I think. What's that for small things? This is super nice for big data.

Yeah, I understand, I understand. So what we need here is we don't have for list plot a way of saying here, you know, we could imagine something which just says, you know, two, wherever it is, you know. And then here we could say, let me, let me make that seven or something. Just so we clarify what's going on. We could say something like, you know, not annotation rules, but you know, something like labeling rules or something. That's not that, that's not quite the right term. I don't know what it is. Annotation rules. Whatever. Whatever it is. Annotation. Right.

No, but, but the specification from afar, basically.

Yeah. Annotation labeling goes to blah, blah blah. Element two goes to blah, blah blah. Or it could be by coordinates. I don't know how that would be done, but. And this would then be. So the question is, say we've got these two things. So in the case of graph, you're moving things like this out into the. Because graph is a computational object, you're moving these things out into the sort of styling options. Right, right.

Effectively, for actual implementation purposes, we are doing the same for all this visualization functions. So there's a front end that moves them away.

Okay. So you have internal options that are the equivalent of My annotation labeling option.

Well, we call it a parser.

No, I understand, but, but, but you.

Are to strip them and, and, and remember where they are because they're going to be reapplied towards the end.

I get it, I get it.

Right, but, but, but they're stored in edge style, right? They're not, you know. Right.

There's no, no, no, but not the list plot.

No, no, no, no. It's completely internal description. There's nothing.

I thought he was talking about graph.

No, I wasn't, I was talking about list plot. So Roger was talking about list plot. And what Roger is saying is that internal to list plot, what's happening is something which could be revealed to users like this.

Yeah, I mean it would be complicated because we allow it so many levels. But, but yeah, absolutely. We would have to.

Right.

You have to have a way of specifying the left hand side here. Just like in audio. It's just like an interval or something like this. Except here there would have to be a language that represents what sub part of this we're talking about.

Exactly.

Right, right.

I think, I mean, so I.

Right.

I think it's more useful in less plot. Right. For having a way of, you know, representing data series, you know, styling for the different series than individual points.

Well, it might be an individual point. I don't think we can say that for sure.

I mean also the future of sort of visualization is typically to have many more dimensions and you often have kind of some property being like if you think tabular in some other column. Right. Sex.

Does that work? No, that didn't work. But that's the kind of thing we're doing that will be action at a distance. That will be the analog of the edge style.

Right.

Well, but also positional.

Right.

If you did, if you wrapped the red and the green in lists, then you would get that.

Well, no, I mean if I, if I didn't say the one arrow, the sparse representation, I just used a dense representation here, presumably I would just get what I want there. Yeah, yeah.

But also a sort of a future way of doing this for plotting and so on is to have basically more columns of data. So you have higher dimensional data and you read off some other property that you want to map to maybe color, maybe a label, maybe shape. Yes, I'm saying like.

Yes, but that's probably a super function to list plot. I mean that's not probably listplot itself or. No, actually what it probably is is a list plot with a, you know, visualizer or Something of data comma, you know, slice, slice there. Yeah, something like that.

Maybe not proceed too much on this today because we are.

Okay, okay, okay, okay. But I mean, but just to get a sense. So this is another example, you know, the plotting concepts are another example.

Right.

You can wrap, right, that seven in, you know, style of black and make it black.

Right?

Which one in that? In of 114.

Yeah, I could say here, style of this comma, large, I suspect. No, that didn't work. Why didn't that work?

Probably point size large or something. Point size.

Okay, wait, wait. I could say black here. Okay, okay. So there are these sort of. As you said, there are these things that are, you know, in place wrappers. There are action at a distance things. You're calling them in content wrappers. I agree. I mean, the two things we know here are in content wrappers and referent options, so to speak.

Yeah.

Now, is there another.

An external wrapper was kind of what I was calling the third category, which is just trying to have like we were talking about a super function like we have with annotated, but that implementation basically goes into the second.

Right. So I think we have that function which is annotate. Yeah, and annotate. You know, an interesting possibility for even list plot is to be able to say annotate. I mean, so. So, you know, a question there would be annotate. Oh my gosh, a list, 1, 2, 3. And then we say, you know, two arrow. I don't know what it is. I mean, I don't know what that. That doesn't really make any sense. What the heck is that? Message text not found. Well, bug, why doesn't QA automatically find all of these message text not founds?

Because you don't know what messages will be issued short of managing to trigger all of them.

But can't you just look up in the source code every place where a message is called for and look at what its tag is?

But this excludes you. I mean, most of us use helper functions to create messages. So it's not.

You can look for the tag. You can look for where the tag goes. Anyway.

Don'T forget, most messages don't actually appear in the message list because they're attached to general.

Okay, anyway, I'm. I'm just saying, can somebody please feed this to Stefan and just comment that. I mean, this is a specific bug, but just make the suggestion that we have a general way to do this.

I have passed on along.

Okay. All right, so let's come back here. Let's just do a quick survey of the different cases here. Right. And how we like or don't like what's been done. Okay. Text structure is a very. Is using is. Is really going to town on the nesting of in context rappers. And the whole point is we need a way to. I mean, you know, I would imagine something like. It's like. I don't know whether it's flatten annotations or something. I don't know whether that's quite the right thing. But. But.

It's a case where the annotation rules would be fairly complicated because there's multiple levels spanning. I think that having. Having some system to do the transformation definitely makes sense, but know.

Right. And it probably needs to be a somewhat controlled translation.

Yeah.

Maybe the correct thing is two annotation rules or some such other thing. Does this make sense? I mean, so this is. This is. And then it will be like from annotation rules.

So there's no where with text element or it's implicit because it comes from the nesting of text elements.

It has to be constructed. I mean the where has to be constructed out of. You know what it would be in the end is. In the end it would be something, I would guess that one thing is a nested text element and the other would be something like maybe this is wrong. A single text element with a very nested set of annotation rules. You see what I'm saying? One level text element. It's like a flatten.

That first argument would basically just be a string.

Yeah, I think so. I mean.

So yeah, I mean the other case I was thinking of that is sometimes nice for annotation of strings is things like, you know, we were talking about with this, with the linguistics functions of sometimes I have a string and it is IPA data. And just looking at it, it might be an ASCII string or it might be a Unicode string, but it. Unless something is telling, like the audio system, this is IPA data. So you need to handle it as such. It just gets treated as a generic string and that may fall into a more general, you know, annotated string or.

Well, I think that that's exactly another thing that string objects could gradually, as we accumulate workflows, slash use cases. That's the kind of thing that. That can absorb.

Carry character encoding or other things too.

Yeah, Sorry, say that again. I got distracted here for a second.

Yeah, we were just saying, you know, rather than, you know, there may be a case where rather than this being a text element with a string and then A bunch of annotation rules. Having a string object where you could say something like, this is IPA data, this is a string encoded with this character encoding your annotations and combine all three of them.

Right. And arguably. Yeah, I mean, so. So I don't know whether we can have something which is a general way of untangling sort of in context annotations, whether it'll be specific. But let's just look at other cases where. So, I mean, I think this. I don't think we should be attacking the text element mechanism. I think the text element mechanism is perfectly fine. It's just not always what you want. Do you agree?

Yeah. Once you have a text structure, there's not a lot you can do with it because it's highly nested, but you can do things like turn it into a tree.

Sure.

But if you just.

I think it's. Yeah, go ahead.

If you have a text structure as your starting point, it's hard to go back to a string. Having some sort of way to decouple the annotations and get the underlying string might be.

Yeah, right. So this is an untangling. I mean, the same thing is probably true of XML that being able to just go through and cut all the garbage and just get the basic, you know, sequential elements from the bottom of the tree, which would be presumably a tree map thing. How do you get the elements? Tree leaves. Right, yeah. So if I said of this, if I get the expression tree here and then I say to this expression tree, It's a bit crazy. What the hell is it doing with it? Why is it formatting like this?

Those are XML object options, which are different from the elements. There's a declaration at the beginning of the, you know, you have an XML.

Object, XML object declaration. If this was a piece of HTML, would I be able to read the plain text here?

In many cases, yeah.

Right.

So that's what I'm looking for. And I don't understand why there's big empty space around these things. What's the reason for that?

That's how graph lays out things. Currently I'm working on tidy tree visualizations that'll help with that.

Okay, all right, okay, fine. Okay, but, but then. So for trees, what is the annotation mechanism for trees?

There isn't one last time we talked about this, Metadata annotations, things for trees, specifically, we talked about having a payload wrapper that you can put inside nodes and put the annotations in there.

I thought, okay, but Berman, we have the payload is the first argument of the tree object.

Yeah, so the. And this was just a suggested name, but it would be like a capital P payload of. And you'd have the data, but then you could have metadata as options.

All right, so that's. That is an in context wrapper type of thing.

Right.

Which may or may not be the correct thing to do because the alternative would just be to have an option of which we have many in the tree objects. Right.

I think that for the nested kinds of objects there's some benefits to having the annotations close to.

Well, but then I think we want the generic annotation wrapper because. Which we don't currently have, do we?

It's a content thing. If you go to annotation page, it's a very vintage function that disappears in typesetting.

Vintage 2007. Yes. I mean we might be able to enhance this.

Stephen, can you open the source ref page for payload?

And if you see the annotation page, this is the same essentially as the usage.

I think it's almost the same. I mean I think we could just use annotation for this.

Well, annotation maybe with some extensions.

Yeah, yeah, yeah, yeah.

Like multiple.

I mean we, you know, a re. Modernized annotation.

And I think the last usage is already remodernized. As you can see, it is eaten by graph and mesh.

That was the new addition in annotation. Yes.

I see. Well, that's exactly what should be being done in tree. Right. So annotation is the generic, you know, so annotation rules is the generic option at a distance type mechanism and annotation is the generic in place wrapper mechanism, correct?

I think so, yeah.

So there are specialized versions of that as well, like style or something like this. But that's the generic way to do it. So what we need is something that goes between the annotation mechanism and the annotation rules mechanism. Right. We need something which for every one of these functions will try and move from one to the other. Now does Graph, for example, accept annotation as. Yes, it says it does. So I mean it, it, it. We're going to have to wrap for right now here, but this is a good discussion. I think we're making good progress. I do think some of the ones we haven't discussed yet. I mean quantity. I would have thought we should just have the possibility of an annotation rules option inside the quantity object. Am I making sense?

Yeah, we potentially can. There's a bit of a question about what whether the, the physical quantity has an annotation that is a quantity or the quantity has an annotation that's physical quantity. But that's something. Michael has strong opinions.

Yes, we should deal with that. But I mean, in. Okay. For notebooks and cells, one of the issues there is. I mean, we really need to have a look at notebook programming, which is like, you know, it's older than many of the people working on it. Yes, the. And you know, it was. It was done at a time when, I mean, a lot of things there are not as symbolic as they could be.

So I have a question about. So you said we have the in place notation and the action at the distance annotation where the current proposal for the ladder is the annotation rules option.

No, I would say that's the generic. But many functions will have a thing. I mean, the annotation rules option is just a carrier most of the time, will not actually do anything.

But the idea is to have something within the object structure itself rather than.

Yes, that's correct.

Yeah.

So I was wondering what did you. Does it make sense at all to have the wrapper live outside the object like annotation does and perform in a similar way?

So it does, right, I agree. Yes, I think it does. I think that there is a case. Look, what we've got is Annotate. That drills, right? Annotate actively drills and sticks things in places in annotation rules, for example, or in edge style or whatever else it is. Annotation can be a hang around outside and have an effect on the inside or be inside. Okay. So the question is, could there be something which is not like Annotate, which drills things into the inside, but is a wrapper on the outside? Yeah, I see.

I think that's what Giulio had in his prototype for Annotated. I don't remember the exact name, but the one checking prototype.

The string, you mean.

So that's what I had in mind. Something that acts, does something on the typesetting level, but is stirred in a separate way, not within the object itself. That seems to me like the complete analog of annotation rules, but without having the need of throwing stuff inside objects themselves.

It's an externalized thing which could also work for things like string and integer and so on.

Yeah, and the advantage of that is, sure, we need to teach the functions that eat that kind of object to ignore and potentially propagate the annotation wrapper. The annotated wrapper. But at the same time it makes it clearer where the annotation lives. And where it doesn't. Which makes the whole expert issue a bit clearer in my mind as well.

Well, you're saying something always goes wrong when you split them. If you.

It would be a single object. Right. It would be an annotated object that you carry around without doing. Without having them separate. I'm just saying, to be clear what.

We'Re saying, and we really need to go in a second here. But what we're saying is there is annotation, which is an internal. That's the case of, you know, in place wrapper. There is annotation rules, which is, you know, option inside structure. And there is annotated, Which is externalized annotation.

Yeah, complete.

I would say a completely equivalent to annotation rules. Just lives in a different place.

Well, it lives outside the object.

Yep.

This could have annotation rules in it or it could have other kinds of annotations in it. Now annotate is a driller. You know, drills the annotations, plants the annotations inside expa. It Making sense. Yep.

And I guess if it can't, in the case that there is a string.

Puts it and it wr. Annotated. Yep. And the issue is. So arguably, you know, we could make annotated with one argument. Pull out annotation rules. I think that's probably not a particularly good idea. I kind of think that.

So if I'm guessing what triggered today's meeting was text and trees. Is that right?

Or segmentation.

Oh, was the impetus.

Okay. This is pull out annotations as an option now. I mean, it could be that that thing is actually. This. That just returns the options in audio.

We do have audio annotation lookup. It's a lookup function.

What does it do?

Is it like annotation value? Sorry.

Yeah, more or less annotation value. We haven't discussed that, have we?

That already exists. And for graph.

Right.

That is where that. It's the rename property value.

Right.

I have to say it's not always very convenient due to having to take the object as an argument. It'd be nice if the processing functions could just semantically deal with annotations.

Let's talk about annotation value another time. Yeah, okay, but so these are the. I mean, it's. It's. That's not a comment. That is a syntax we could use for something. Okay. Any case. Okay, well, we've. We've opened this topic. We have not closed it. I think we'll need another hour or so at least to talk about it. By the way, this doesn't seem completely. Oh, yeah, yeah, yeah. This poll business.

Yep.

In 30 seconds.

I have not seen this before.

I think this also came up in the. The signal processing.

All right.

Review. But, yeah, I'd like to be in wdr.

But that's. That's okay. We'll discuss that.

Yep.

Okay. All right. Thank you.