Live CEOing Ep 794: Design Review of Sound & Vision Features

Okay. Welcome to another Wolfram Language Design review. This is heading into version 14.1. I just noticed that this agenda had a date from September of last year. But lots of things here are new, I believe.

Yeah, that's just when we started out. Okay, we start and we start, you know, 142 now. So it's going to have a April date.

Okay, great. All right. We talked about speech recognize and major enhancements to speech recognize and language identify. Okay. Is that now working Video transcribe. Okay, that's cool.

That's a new proposal based on our Monday conversation.

Well, it seems like a good idea to me, but I think maybe it needs a property argument that I mean in. Well, I guess if you just say speech recognize of a video, it will give you the transcription, but video transcribe will actually insert the transcription as. Into the video.

As a subtitle track. Exactly.

Yes. Sounds right to me.

Okay.

Okay. You figured out the interval business.

There is one decision that you need to make that is in the next little section.

Okay. Video intervals. Okay. Seems right to me. I mean, what was it returning before?

A list of lists of rules where the first part of the rule is start time, End time.

You mean where the left hand side of the rule is start time?

Yes, yes.

Video intervals, audio intervals only.

Oh, sorry. Never mind.

There is no rule intervals.

What are you saying?

No rules. These are just intervals of time assumed to be in seconds.

How does this relate to how we're doing? Sound note and God knows what else? I mean, I'm kind of. I'm kind of not convinced that this should have interval in it. Is it possible to. I'm kind of thinking that we could have a syntax where it says video intervals of V arrow and then what you want the return type to be. See what I'm saying? So we say something like video intervals, Video arrow lists or arrow intervals or something. I don't know if that's crazy.

It reminds me, it always requires a second argument, by the way.

Yeah, and I assume Stephen means that whatever additional arguments come next, that's why we're sticking.

Yeah, yeah. I'm just saying this rule will have to be not at the end of the. Whatever call is happening.

Well, no, no, it's right there. Video error, whatever. As opposed to muddying itself up with the criterion.

Yeah, no, no, that sounds good. I just wanted to make sure that you were aware.

Right. But I want to say that I think there's a value to just having Pl with seconds in them without fancifying into quantities and intervals and such.

Like, yes, and that was one of the reasons why we defaulted to that. But this seems like a good way to support both and not break.

Not sure. Remember, we have so many functions that have properties. Sometimes this formatting gets embedded into the property specification.

I know, but this one doesn't have properties.

Doesn't have properties. And then some other cases I think we have just even as an additional argument for what that formatting is going to be, I think we discussed that two, three versions ago that maybe an option, a unified option among so many functions that specifies what the format should be. Could be useful.

Could be useful. I agree. But I think, I think in these cases, unfortunately, you know, there are enough different situations I would not change what it's doing right now. I would support something which says arrow intervals or arrow quantities, for example, or quantity intervals, arrow intervals and arrow quantity intervals.

Do you want us to leave them without that arrow goes to something 14, 1. And look at the bigger picture in LCC.

Maybe, maybe. But I, I think this is the right way to do it. And I think don't change the return as it is right now.

Okay, that would be good. I prefer that.

Okay. Okay, So we're going to look at these things. So where do I want to start here? You, you had talked about NEST video before and. Okay, I mean, does this fold have to be individual frame by frame?

It can be blocks of frames.

Okay, so if I say why doesn't.

NEST video there, that has conflicts. And prototype would manipulate video, unfortunately. So we couldn't squeeze it in.

To this prototype build.

Right. I mean, I think it has been out of prototype for at least a week. I think there are a few examples in the ref page, but this is one of those functions that I'm sure you're going to come up with great ideas. For example.

Well, I mean, I do think that you need a time scale here. I mean, if this is frame by frame nesting, that's, I mean, I think it's much more likely to be wait five seconds, then do this, then do this, then do this. So I'm saying I would, I would say that returning in each. I mean, what's really kind of weird. Is what I'm thinking here. By the way, what is the minimal way to turn a 2D cellular automaton into a video? Let me, let me give an example.

I think it's already as an example down there.

No, but it won't be Nest because NEST isn't the minimal way to do it, the minimum way to do it. Let me, let me give an example here. So let's say cellular automaton, Game of life, random integer one. Let's say 100 by 100. And let's say we go for 100 steps. Okay. That's going to make a big structure. It's just a list of lists of lists of lists and so on. How do I make that into a video? So that's a list of frames. Do I need to array plot it first?

Map image or array plot, whichever you like.

Well, let's map image onto the list image. I mean, let's colon a gate of image mapped onto this. I'm going to make. Be nice and make it image object. That was nice and fast. Okay, now what?

I wonder if color negate is listable. In that case you want to check the result. But then after that it's going to be a frame list video. Okay.

Whoops. Why? I'm trying to type A2 in it and it just. What the heck? What is happening here? You see that white stripe across my screen?

Yeah.

Who knows what that was? Okay, well, now it's. It's spinning. Oh, my gosh. Kulasaurus. Now what if I want to make this bigger on the screen?

This. You need to reconstruct the video and specify an image size. This is something that.

But then with that frame list video. Does that take a raster size option?

I forget.

Can't you do video of video to just set that option?

Yeah, yeah.

Video. Video of that previous thing, comma, image size goes to.

Why is my. This is spinning like crazy. Something is very, very wrong here.

Could that be because the history has a lot of data in it and every time it tries to fetch something from the.

Okay, that was nice and fast. So let's try this. It's fuzzy. It's very fuzzy. You see how fuzzy it is? Can you see that through?

That's I think the interpolation. Raster interpolation. Carlo. Right. That's something. No, that's something we're experimenting with in prototype. Which should we have this? We discussed it. It's going to go away. That's wrong.

You've got echoes there.

Echoes are most probably the video encoding.

Yeah, well, okay, but it's wrong. I mean, look at it. It's really bad. Look at this.

Can you mind pressing on the hamburger menu at the bottom right of that video and open in system player and see if this is something that depends on us or is baked into the video.

That looks terrible.

Yeah. So I think this is just the fact that, you know, as Shelley said, when you Create a video. You pass it to something that compresses it.

Yeah, but this. Okay, so how do I get a decent looking video out of this?

Maybe you just specify a bigger raster size while you're generating the video. So then these artifacts appear at a much smaller scale.

Well, maybe it's just the wrong encoding. I mean, we've got 700 encodings, why can't there be one? Right?

No.

Well, so, okay, the only encoding that doesn't have artifacts are the ones that are completely uncompressed. But those would get you insane. Completely insane.

Okay, so let's say, I say video frameless here. Raster size.

Yeah, that might be better. Raster size 500, let's say.

Still looks fuzzy.

Yeah.

Now look at that.

It's.

Look. Can you, can't you see that? Maybe you can't see it so well.

No, no, I can see. I think. Okay, I have one. So the problem was. Well, problem. This is caused by the fact that we generated images that are 100 by 100. So those edges are the pixel edges and we are enlarging those pixels and however we enlarge them either with the raster size or just making the player bigger, some interpolation is going on.

All right, so let's try.

I'm wondering if you do array plot instead of.

I'm going to try.

And that is going to.

And then I say pixel perfect. Pixel constrained arrow three.

Wow. I don't know what that is.

It makes it pixel perfect. Okay, so there's my thing. And now I say video.

I would specify the raster size because that will still be important.

Well, how do I know the raster size? Because you see, that's pixel perfect. Can you see that?

Well, let's try without it and see what happens.

Okay, that will be 300, I guess in raster size. Taking its time. I don't know why it's taking so much longer here.

I would expect. Yes. And I would expect 600 in raster size.

Well, what do you know? Much sharper.

Yeah. The problem is that we generated the image at the beginning and to make it bigger, interpolation was happening.

But image resize should use a good resampling there, I think Interpolation in image.

No, it isn't really the resampling in image resize is good, but doesn't preserve edges. Right?

Yeah, but. So that's an option for the resize. But I mean here, if I say in this, the image. Can I give an image size here?

Yes.

You think it's 600 by the way.

Not 300 or image size should be 300, but then it rasterizes to a 600 because. Because you're on a high resolution display.

Okay. I was nice and fast at least. Why isn't this bigger? How do I make that bigger? Again?

I would probably just specify a bigger image size when you generate the images.

I did.

I would go to 600 because I think this is not for that.

True, Carlo. I think image size is not for that. True. It's not passed over.

Well, what's happening there? I don't know why that's.

Oh, the percent. The percent.

Oh, yeah. Okay. Oh, no, it's spinning. Fuzzball again.

Yeah, I think that this.

Well, yeah, I, I think I know what's the problem. I. If we'll try to get that result, and if we couldn't get that resolved, Carlo, I may be wrong, then we'll figure out what's causing it.

Okay. And then we need to document this and we need to figure out how to make it not happen. And we need to, if necessary, we should just run a classifier to figure out what kind of an image it is and make a decision about compression based on that.

Again, having the no compression version would produce really, really outrageously gigantic videos that are mostly not.

I don't think it's necessary. I don't think it's necessary. I think that we just got a bug here or we've got use of a, you know, a thing which is expecting continuous photograph type images, not discrete digital images. See what I'm saying?

Yep, I see.

So what I'm saying, and I, you know, we should make a little classifier that should be trivial to make, that decides very fast what kind of an image it is and decides what kind of compression to do based on that. And in the, in the progress monitor that says it's making the images, we can describe what compression it's using.

Is there even a codec that's particularly good for those?

I. I am not sure. I will investigate, but among all the.

Guesses we have, I have to believe something is.

My guess is no, but I am not sure and I will look it up.

Yeah, I suspect wavelet codecs, for example, will be better than cosine transform codecs and so on. Just because they have a, you know.

Could be. Could be. As I said, I'll look it up and see if there is anything that performs better than this.

Okay.

Talking about progress monitor of generating videos, Carlo just put together an ICE PR that shows the progress of frames. I put it in Chat. It's a screen capture of my screen generating a video that takes a little longer. Yeah, I noticed. That's a chrome, right? Chrome doesn't show that you have Firefox, maybe.

Well, yeah. Okay, now I can see it. All right. What are you showing here?

Showing that generating frame. The cow in that progress monitor. You show the progress of inflating cow.

Yeah.

That's cool.

Yeah. Especially since a lot of these take a lot of time. It's nice to see a progress of what's happening completely.

And the other thing that I added I think is nice is if you see that your video is pitch being weird, I added a stop button and that will just output a valid video that is just up to that frame.

Why is the cow turning its head to the side?

Good question.

It might be upset about all the gases. But wait a minute. How do you generate it? Is this just a radial expansion?

Yeah, I think so.

That's from somewhere.

I think the cat. The cow probably has its head misaligned relative to the center of expansion.

That's what I'm thinking. Yeah.

Yeah, exactly.

Let's just talk about the appearance here. I don't see that the parens are doing us any good. Yeah.

So this is the default thing that the progress evaluate with progress does. It's equivalent to the current item display, but obviously anything else can be used.

Well, I just think generating video frames. I'll just say generating frame and then no parentheses.

Okay.

I'll ask Julio what he thinks about if this is a change that applies to everything or just the video functionality.

Right. Okay. So look at manipulate video. How do I am I just look at documentation?

Yes.

That's a very obscure first example.

That's a bad first example. I told Roger I'll replace it this morning and then I ran out of time.

Look at the application examples.

Yeah, I would scroll down.

I'm just going to make myself.

Okay.

Want to see the whole workflow? That's spinning a disk here. Is that what it should be doing? Elapsed time zero remaining.

The elapsed time zero is a bug that is fixed in a pull request.

Why is my we don't see the.

Spinning thing, but you know, it should not be spinning.

How old is your prototype? The most recent, I suppose.

I assume this is dead. It's dead. What do I do? Just kill the system?

Yeah, for square.

Do I have any choice? Well, it wasn't a great demo. When the thing comes up. So I assume that when it comes up, the cow is left.

That that was your quicktime player. I think.

I know, I know. Hold on. I've got to find my.

The cow blew up the system.

Was that a revenge?

Yeah. Over inflated. Okay, hold on, hold on, hold on, hold on. 09. Right. Okay, so my question is do the. To the. Do the controls show up on the manipulate video or is it just the content area?

Just the content, but I think we do have a. That's the default. We do have the appearance control. Appearance option, I think.

Appearance actually.

Is this the right document?

This looks like the old one.

Right. We need the one that says updated. All right. Back down here. Okay. And we want to manipulate video.

If you cannot run it, maybe scroll as Roger suggested to the application exam.

This is. This is a special case. This is a different thing, right?

That's new from Manipulate itself. Yes.

A manipulate has no meaning for. I mean when you've had other tour videos and so on. Does that syntax work in those tour videos?

Yeah, time goes to some spec. Yeah, yeah.

But does it work in an association like that?

So let me just check. The reason why this is an association, not the list of rules is that Manipulate does have a list of rules syntax. What does it do?

What does that mean?

Labels. I don't remember. Let me just show.

That's not on the right hand side there. That's the left hand side has labels. The right hand side is not using labels there.

The right hand side is right hand side value. Right hand side is the label. If you open the Manipulate CBS page. It's not in the main usages in the notes. Well, in the details. Sorry. That's why I would have supported the. Yeah, that table. The 1, 2, 3, 4.

I see. Yeah, that guy over there to set a bar.

Yeah.

Well, you don't have a SATA bar in the case of a Manipulate video, do you? What sense does it make to have a. In other words, a manipulate video as it has a continuous time variable, even the dt.

It could be anything, Right. It could be a discrete variable. It just would be.

How does it drive drives it from that? So if there's a dt, it drives it in discrete intervals. Yeah.

And same if there's a list of values. It's like Manipulate does the same when you do the outer run. Right. If there's a setup bar, it will go through all the values in that setup.

I'm not thrilled about this this association hack because I think in fact had we had associations when we were designing Manipulator originally for this label thing, we would have used it there too.

And the interesting fact about manipulate video is that most of the examples that so far we have created, we haven't created many are not very happy with that. Second argument, the things that. The thing that seemed to be even more handy is that new time marks option that we are proposing both for manipulate and manipulate video. The reason I say that is that the second argument defines how we vary a single argument at a time. And then you have to think going back in time and define it for the second variable, third variable. In time mark specification you think about time zero, give all the settings for all variables at time zero, then think about time one, time two and so forth.

Is that what they're called? I mean in stop motion video, what is it called? So my question. Sorry, do you understand.

I get your question. I don't know the answer.

Okay, we need the answer.

The reason why these are have this name is just for similarity with bookmarks, which is what this is based on, which is an existing option. Manipulator.

Yeah, let's find out. Videos.

Yeah, sounds good.

Go to. I mean I don't think that there is enough applications yet, but go to applications and manipulate video. This one doesn't have much.

That's pretty wild. I don't understand this at all. How do we know 01? I don't understand. Are these. Are we giving integers? Are we giving just numbers?

Those are times. Those are. The left hand side is a time and the right hand side is the setting.

What happens if you want to interpolate between two times?

It does interpolate.

Well.

What happens if you don't want it to interpolate?

What happens if you want to specify a different interpolation order where you can either specify it within manipulate that already has an interpolation order or if you want to in the variable specification. So Instead of list x0 comma 1 you could do x0 comma 1 and specify an interpolation order option just for that variable. If you have multiple variables and each has a different behavior.

Okay, what have we done in Tour video and things like this with these time. I mean this is now a much more complicated thing with. I don't even understand the.

Yeah, I think what Roger says. Maybe clearer examples might. You might find them in the manipulate video page in the application so section.

Like look at that rotation. Okay, stop.

Okay, okay, okay.

So let's say that that's the manipulate there and then here is a script. This is what you want to achieve. Right? How do you translate that script to code?

Yeah, It's really weird that we've Got lists. I mean, why isn't it just a function to be running with semicolons and things at that point?

I mean.

What?

What?

I mean, what fun. What are you talking about? What's the function?

That list returns your.

I. I don't know why. There's a bunch of assignments there you.

Set your video on. I don't know if you want to.

I don't know why. Sorry, my bad.

These.

I mean.

No, you understand what I'm saying? That list is going to return the right hand sides of those assignments, which is pretty weird. That list has a value. But what all you're trying to use is side effects.

I mean, I think that it's time marks because of the way manipulate was written. I don't think there needed to be assignments at all. You just want these particular parameter values to be your landmarks that you go between.

Right. But I mean, so that they could just be a list. Yeah, where we know it's px.

That would be fine. This is just for compatibility with.

Well, how to bookmarks.

Manipulate bookmarks exactly like this. They just don't have. The left hand side is a label instead of the time. They don't have a notion of time.

Right.

Performs actions. Wow.

Yeah. This is ancient. It seems like no one has almost ever used it well.

And look at how pathetically it's documented. Just pathetic. Yeah. Well, I think we should just do this with. With the settings and call it something out of the settings.

I think that's what I call it in the time marks documentation. But even the syntax you don't seem to like.

No, I don't think it should be like that. I don't think. I don't think we should have the. If what you're doing is setting those variables. We know the order of the variables. Just set the list of variables. Just give the list of values. Don't say thing equals that.

But in the case of one argument, we know the order of variables. In the case of three arguments, I think we still can't.

We will try the order of variables we always know because we always know it there in the manipulate.

What if there's nested controls?

Nested controls or honestly if as soon as it gets to 10 controls it might be more explicit to be able to specify what exactly I would like to change.

This also allows you to not specify some variables in some steps.

Yeah, I was going to say that.

It looks clunky at first, I understand.

But its name is terrible, so I'm.

Not arguing with the name. Yeah, yeah, I agree.

Timed Actions. Now that's terrible too.

Key frames at some point. Yeah.

Keyframes is usually keyframe animations.

Yeah, right.

I mean these are key parameters in our case.

Well, maybe what we want to call it is keyframe settings. Keyframe rules.

Keyframe, you mean keyframe.

I assume Keyframe is one word.

I like the settings idea, but roles is also okay. I think.

Settings or actions.

Maybe that's even more explicit.

Keyframe. I mean what we. We call it button action or something. Right? Key frame actions. That sounds. That's about right. That's what it's actually doing.

And so I mean, over time, I think we need to be able to elaborate on these. To have. Let's say that you want a label to come in in one frame or maybe.

Well, you can have an action. I mean you have an action, you have an arbitrary. The only question is whether that thing on the right hand side should be a function by default.

I, I think, No, I think that's super hard to use.

I mean, at that point you're just implementing video generator with a pure function that no one likes.

I see, I see, I see.

So if you want to do that, it's fine. But we do have the.

Already this, this seems good. Okay, so keyframe actions and there are series of actions that you take setting variables to things.

I want to point out that this same time marks option as it's documented, we're suggesting it goes into manipulate itself. And just like we could animate bookmarks, we can now animate these timed actions as well. Keyframe actions still sound. I just wanted to make sure you think that seems right in that context as well.

I mean, we're talking about an animation in both cases. It doesn't seem completely crazy when you manipulate.

And I think also that it suggests if you look at the list of options and you see keyframe things, you know, it's an animation related option.

Sure. That I agree with. Yes.

And we've worked it out with Lou and John. No need to change box structure. This will be backward. Well stuff. Yes, this will be backwards compatible and will not break anything even though it touches the manipulate code.

Okay, so Nest video, I think that an important piece of that story, by the way.

Stephen.

Yeah, this.

When we mature this a little bit, I think that this is another reason to revive demonstrations because you could have experiments and other things that are there for the different manipulates and authors could create those and those would be great for X posts and linkedins and.

You mean videos that come from the thing. Yeah, exactly. Yeah, right. That's a good idea to have videos on the, on the demonstration page.

Yeah.

As well as just static pictures to have videos. It's a good idea.

Right. And then, you know, and, and so the thing you had there that you saw the rotation was a kind of like a learning situation, kind of like when you get a lab compendium, when you go to do a real physical lab.

So this is basically you're suggesting demonstra demonstrations also become a collection of static videos that people could use without the player and so on. Yeah, fine idea that.

Or even just manipulate video to create videos to promote those hundreds, thousands of demonstrations, right?

Yes, yes. Well, we should just run manipulate video to create demonstrations by default from all those. From all those. From all those manipulates in the demonstrations.

But I think that there is that what makes it interesting is when you actually tell a story, not when you do random.

Well, right, but I mean, and, and then, yes, there are elaborate videos that can be made. I mean, there's a lot of stuff that needs to be done with your teams having, you know, demonstration examples of machine learning stuff and so on that hasn't been done and needs to be done. But I mean, we're just about to have a big modernization of demonstrations just coming through. But that's a good thing to add to that. Okay, let's look at this. End time. This is not grammatically correct. Default duration per frame. Is that per. Per nest? Is that correct?

That is the duration.

I see.

However, I don't think that makes any sense. I think that's wrong. I think that got copied over in the doc page. I'm not sure there should be a default duration.

Well, what's the duration per nest?

Yes. So I think in that context it makes more sense if we interpret the default duration as the default duration per nest.

Well, no, but, but, but I mean if default duration otherwise is the total duration of the video, we can't do that just in this case.

I, yeah, it is that case.

This is, this is what I, what I'm saying. I think, I think interpreting default duration as the duration per nest is the. Would be the correct thing to do.

No, because we use default duration elsewhere for the total duration of the generated.

Well, then we need some other way because default duration doesn't particularly make sense here.

Well, the option documents frame rate as well, which could specify. But then there's a conflict between the two. We'll have to decide which one takes which one supersedes the other.

We have three things, right? The frame rate, the number of iterations and the default duration.

Don't get completely confused here. I'm sorry that by the time you've got that fourth argument, why are you not just generating the nest and then doing a frameless video of that?

Because if. Yeah, if you want 100 of those, fine. If you want a thousand of those, you're going to run out of memory.

I see. Well, show me an example. That's actually these interesting. I mean, I don't know if these are interesting.

Some of those random walks I think was very interesting. Look at applications. There are that event's second example with the angle path. Well, I'm not seeing that up in the basic examples. There was. Yeah, all of these are nice as well. You should be able to play most of these videos.

Frame rate. So wait a minute, there is a frame rate in the Nest video? Yes. Well, so that gives you the duration.

Yeah, that's where I am saying you should download the video. Frame rate and default duration can be conflicting and we'll have to decide which one takes the precedent.

Shadi, I really don't think Nest video has a default duration option. I think that was just copied over in the docs from some other video generation function.

Right, okay, then we just think it.

Should have a default duration. Why not?

I also think it will come handy.

Yeah. And the frame rate supersedes the default duration.

Okay, fine. That's okay.

Click it, Stephen. Click the arrow. Click the arrow. Yeah, I don't think that that was.

Okay, so.

The one above is interesting as well.

Hold on, hold on.

Sure.

Basically you've got a bunch of data there. You know, this is a general category of thing where you're writing data out as it's generated.

Yes.

And is it really right? Is this really the right design or is a better design to have something that's more like so.

Right.

Because then you could do it for anything. You're just saying I'm going on. It's kind of like so video frame. See what I'm saying?

We kind of use that internally. Not exactly, but yeah, we create some iterator that opens a file and then whatever construct, you can write a frame. But it's a very low level though, kind of way.

What I'm suggesting is something like this. Okay, let's just say reap video of so video. I mean, yeah, as I said, this.

Is pretty much what we have right. Internally. It's just I. The impression that we had is that this is a bit too low level.

I just think that by the time you've got that fourth argument, it's kind of like, why do we have this?

I would not be against having some construction like this because then any iterator construct the language we can use. However, I'm not sure we can wrap this something like that up for 14.1 just because the current version of so video frame is a gigantic monster with 1000 options to make sure that.

Well, why don't you look at. Because I just don't think I. I'm unconvinced about Nest video and so on really am. Because of this fact that you have to have the. The. That processor, the thing you're calling Writer, which I don't really like the name of.

Again, I would be excited to expose something in the reap video thing. I'm just saying that I am not sure if.

Okay, okay, fine. But I'm saying I don't know whether we want to have video given, you know, maybe even put it in the function repository for this version or something.

Okay.

Because I just don't think. I think it's very undefined. What? I don't know. Roger, what do you think? Well, I. I think.

I mean, I think that there's been a couple of demos there. A Nest video that I like. Like one of those things that look at nearby things. Was it this nest one or fold one that sort of create like motion blur, that kind of thing?

That was the fold one.

Okay, okay, but I'm sure these things are nice, but I'm just. I'm wondering why isn't this just fold with a. So video frame wrapped around the. The target. I mean, in other words, if so video frame returns. If so radio frame of Expo returns expo, then that does everything you need. Well, I mean you're saying that so video frame is going to have to have a function in it anyway because what you're going to want to do is you're saying so video frame of EPA comma processing function.

Exactly.

I mean, not necessarily.

Right.

That could just be processing function of exp right?

No, it cannot be. It's got to return expert otherwise it can't do the nesting.

Okay, yeah, that's fine.

Right, so then what you could do is you could say reap video of blah blah blah blah blah comma proc fun. Which could then just do the processing for everything. Right? So in other words, when it comes in, it processes and that reap video.

Could be a table of something, could be a nest of something, a fold of something, etc.

Yeah, exactly. It could be a do. It could be anything.

Yeah, exactly. It could Be an arbitrary expression, I would argue.

Yes. Which then has a SO video frame in it. Where the SO video frame either has a processor in the video in the individual video frames, or there's a bank processor in the reap video. Yeah.

As I said, we. This is equivalent to what we use internally. And I am. Would be excited about something like this. The one thing that I'm not sure would work out or that we need to think about maybe is a better way of wording it. How do you figure out the progress monitoring of this? But I guess the changes that Julio just made would make. That would make that trivial, I think.

Okay.

Okay.

I'm just thinking it's even sufficient that says if it says, you know, running reap video, you know, collected one frame, two frame, three frames. That was enough, I think.

Yeah.

Saying how many frames altogether there are is luxury and might or might not work.

Yeah, sounds good. As I said, this is a function that I would be excited to have.

Fine. But what is the right name for SO video frame? Is it so video frame? Is that the right function name?

I'm trying to think.

It's not obviously wrong.

So the difference between reap and so and what you're proposing is that so video frame does not wait until it is collected by REAP to write the frame. It does it immediately and then loses it. Reap reaps the resulting video rather than.

No, it doesn't so's immediately into some data structure that's out in the wilderness, so to speak.

Yes, yes, this is what I'm saying. But reap video doesn't get what SO video shows. It gets the finalized video rather than the list.

I'm sorry, what do you mean it gets the. Look, if there's a proc. Fun here, it applies the proc Fun to the expert.

Yes, but it would be silly to store that in memory. It just needs to write it in the video file.

Of course. Of course. Of course it does. And if. If there wasn't a proc. Fun here. Well, whatever proc. Fun there is here that needs to get run on whatever reap video receives from every so.

Yeah, yeah.

Before it's written into the video file. Yeah.

I'm not sure if. The thing that makes me uneasy about it is that REAP and so are not the most obvious functions to start with.

What do you mean they're not the most obvious functions to start with or what?

I don't know. They are very specific to our languages. Someone else that hasn't had to it. Okay, well, I'm Fine with that. That seems fair.

What else would we have?

I was.

I was thinking about raise an exception and have. Come on, what do we do?

We have streams. Right. As well.

We're not going to use streams. They're completely obscure.

That's fine.

Streams nobody uses in our language. They use them in, you know, in. In Go or something. You probably use streams, but not, you know, for systems programming.

I mean, that's fair. Again, I don't have a strong feeling about the name.

No way are we going to say open a stream right into the stream. No, this is nice. A nice construct that I'm definitely not arguing. It doesn't. Doesn't work.

Yeah, I think the syntax of this is excellent. I think this is. Covers most of the cases for video generator that I can think of.

Well, then I think it's a win because I think it's easier. Okay, so I would suggest nest, video, etc. Go in the function repository.

Okay. Okay, that's a Good idea.

Okay. FFmpeg. Improved MPEG installation. Is it really clean or is it still messy?

It will be clean for the release. It is work in progress, but you've seen the workflow. It's just a matter of clicking a button and it will do the thing.

Okay. All right, fine. Video plot. Really? That's a weird concept.

It's just a utility.

I would not call it Video Plot. I would call it something like Video Summary plot or something. Because it sounds like two.

That's fine. That's exact name. I suggest.

Okay. Does it have a million options or not?

It has a bunch.

There's another appearance which. Where you can see the actual frames.

Where is it?

I don't know. I saw it in examples.

I think it is. Yeah, it is in the options and then plot layout.

Yep.

That's incredibly obscure, I have to say. I think I would give that as a second argument. I think it's very major issue.

Okay.

For it. And what are the possible settings?

Apparently it's raw and stacked, but like one can imagine a grid.

Yes. I could imagine it with or without the audio. Blah, blah, blah. How big the audio is.

The audio track selection already exists and you can set it to none. So you can kill the audio plot, essentially.

I forgot to add it to the list of options in the notes. I'll add it.

Okay. Well, this is interesting. I mean, I think there's an argument that given the video player, there should be some kind of button to get the summary video plot from the video player.

We could do that.

I mean, let's look at where that would go. Because I think that would be useful.

I mean, there's the usual menu that we have at the bottom. Right. Seems like a natural place.

All right, but then where would we put the video summary plot? Would it be in the frame, I.

Would say next to all the copy?

No, no, no, no. But where would it go? Where would the actual resulting plot go?

Either clipboard or paste in the cell underneath.

If it's a copy summary plot, I think it just copies in the clipboard.

Okay. Okay, that seems useful. I think that's, you know, I can see using that routinely, actually.

Yeah, like for your book. For books or blogs or something. I mean, or something that doesn't.

Among other things. Right. That reminds me, did we get implemented the. The torn appearance for expressions? I don't know who's. Which group is doing that. Do you know what I'm talking about? Where we say, you know, range of whatever, 10,000 or something. We want to show that, but we want to show it torn after the first few lines.

Oh, I see. That would be cool. I do not know who.

Okay, can we. Can we ask John Fultz and see whether he took action on that? Otherwise we need that. I mean, I know we have WFR functions for it. Jeremy Davis has written them. I think we just need to build it in, I think.

Can you follow up?

Yes, yes, thank you. I'll get that. Yep. Okay. Video image resolution. Is this something I need to look at?

I think it's something that you had a beef with and I think rightly so.

So, so what. What was the thing that got done here?

So in 14 oh, the bottom two results, a plot that was showing nice and small got passed into a video generator or any other video generation functions.

Yeah, right. And became giant. Yes. It's just a bug.

Well, no, it is not, but it is something that we can work around if we add an image resolution option to video. And, and because the video file doesn't have an idea of resolution, that doesn't have that metadata, unfortunately.

Which means the video only gets that good treatment in our system. As soon as you open it in your system player, it's going to look big and ugly. As soon as, in fact, you get the file path to that video and wrap a video object around it again, it loses that notion of image resolution.

But this will everything in a notebook.

Yeah, sorry, I was. I got distracted for a second. Could you say that again? So the problem is video files.

Yeah, there is. Video files cannot store video resolution, unlike image files. There is a Very standard way of storing resolution for images, not for videos.

That's just wrong. So you're fixing it?

Well, we're fixing it. We're not proposing a new international standard.

For biggest in the process our video wrapper can do it does have it, yes. If you have an alien MPEG or something, then you're out of luck.

Yeah, correct.

Exactly.

But if you generate using our video generation functions, then that will be baked in hopefully at the correct value. And if it is not the correct value, then that is definitely a bug. But it shouldn't be.

Okay, Okay, okay. Image segmentation. Okay. What is this?

Segmentation object is a new object. Remember our label matrices from all of segmentation functions was just a giant ugly integer matrix. We're wrapping it into a head. But also we're.

How is that being done? Compatibly.

So that's a part of the discussion that comes at the bottom of this notebook, at the bottom of the notes. I'm suggesting that image segmentation components that was just introduced in 14 can get upgraded to return segmentation object. Especially because we can get rich information about what kind of an object we're looking at when we're looking at segments. But then for things like morphological components, it doesn't have any notion of this component is a dog or a bolt or something else. So we can just wrap the result into segmentation object if you want it displayed nicely so that we don't break backwards compatibility.

I'm sorry, I got distracted again. Could you say that again?

So I'm saying the very old functions like morphological components were not breaking backwards compatibility. If somebody wants a nice new object, they wrap the old result into segmentation object.

But does it have information?

It doesn't have any additional information other than the integers corresponding to every pixel. But then from newer segmentation routines we can get information like this component is a tomato, the next one is a potato. Right. And then that would be the interesting information to include in a segmentation object in addition to just a label integer matrix.

Sure. Well, I don't understand the segmentation object versus things like text recognized that has. That can have, you know, regions that correspond to text and things like this in.

Yeah, text recognized. We. We only have bounding boxes. So you can say the text corresponding to bounding box one is.

I know, but you could, you could equally Well, I mean for find faces, for example, or facial features. You'd also could do, you know, you could have a segmentation object from that as well.

Yeah, that's another note I have in details. I think all of these functions that you mentioned and a few more can feed rich information into segmentation object.

Okay, so let me just comment that this is very similar to what we're trying to do with text annotation.

Okay.

We've got these various text element constructs and it's also somewhat relevant, you know, in general for an image. There's sort of the annotation of the raw image.

Yes.

And how are we planning to do that? So the tomato thing, how do we say, how do we indicate an image that it has tomatoes in it.

We do have annotation rules in some of our funk. Some of our objects like audio rafts and I think some of the.

What we need to do is it's a general thing where you have an image and you have added metadata. It's like, it's like closed captioning tracks, you know, subtitle tracks. You're adding some meta information to the object. I am much less convinced. And then, then that object, that annotated image can have various display options. Like it can have something where it has, you know, purple boxes around the things that. It's true. I don't think this is right. I don't think the segmentation object, this disembodied segmentation object independent of image is quite right. I think this needs. I, I'm sorry, I, I suggest going back to thinking about this.

I suggest you want that bundled with annotations.

I think so. With images. I mean, I think that that's what. And I think then what we do is some of these functions like, You know, we have to rethink how these things, which honestly they are not my favorite functions, all these morphological components functions and so on, they're pretty hard to use. You know, we're going to want things that essentially map over components and things like this. Am I making sense? I think it needs a bit more global thought and I, I would suggest, I'm not certain, but I would suggest that what we want is a sort of composite extended image object that contains those annotations that. Where things are kind of like living on the. I mean, I, I think we're going to need the same kind of thing for GEO as well. Audio, geo, text, etc, they're all kind of a related thing. We need sort of meta information that hangs off it where you can do things like say, you know, visit all the places that have been segmented out or those kinds of things. I would recommend that we do this as a, I, I don't know, ILD or LCC meeting and we try and collect. I. I thought we tried to do this. We had an annotation summit, like, six months ago, didn't we?

I think older than that.

Yeah.

But yes.

Find the notes from it, because I think that's what we need to do. I don't think this is right.

Okay.

But it's great to see all these things that got faster. That's all fine. I wouldn't put this part in yet.

Okay.

Text recognized. That's just making it better, right?

Yes.

Okay. All right. Those are all good. I mean, that's importing into a tabular, which will not be the default, but it'll be something you can ask for.

Correct.

Okay. Data connection.

That's, I think, very much.

I think let's bring that up in one of these tabular meetings, Stephen.

Okay, fine. Okay, then I think we made it through the agenda, did we not?

Yes.

Okay. Thank you all very much.

Thank you.

Bye. See you later.

Yeah, bye.