Live CEOing Ep 696: Language Design in Wolfram Language [Video, Image, & Audio]

Okay. Hello, everyone. We are here for a Wolfram language design review. This is part of first review, I think, in our thirteen. Three version series. And today we are looking at sound and vision, so videos, the new feature of the last year or so. Okay, so tour video updated to support geographics. You're going to show me examples of these things or what should we do to. To get to those?

If you open the ref page, there is updated syntax and I believe at least one example.

Make sure you're up to date, Stephen, when you. The source, at least in case you change things.

Yeah, I definitely change everything, but let me make sure I'm up to date. Okay, so manipulate video. Wow. Controlled version. Manipulate to video.

This is going to be a little bit more of a discussion. It's not fully cooked yet. I'll tell you the background story to it, which I think is really compelling, which is that we can easily do basically keyframe animations from our R manipulates if you create basically key parameter values, if you like, and then you interpolate between them.

There's some error with the update here. Let me just turn that in. Okay, keep going. Sorry. So keyframes.

Yeah. So then you can animate between those and we can make it really behave. Even if something takes time. When you generate video, you can make it really spiffy, but you can also create a story because often if you use a complex manipulate, there are many experiments that you can derive from one Manipulate that has a story.

Okay, hold up. Let's look at this geo to a thing first. Okay, I'm sorry, where is the example?

If you scroll down in scope that. There. There was a tentative example. You just passed it. And that. That video should be on the cloud if you click on it.

No, I'm more interested in seeing it actually run. You know, I hope we're fixing this geostyle thing. That's a su. Okay, this geostyle again, that. Please, whoever's PMing, will you please take that to. That's Jose and team. I assume that they're continuing to improve that for 13.3, but can you please poke them on that? Waiting. We're waiting. Waiting. Still waiting. Okay. All right. Well, we'll see what this does. But now manipulate video. It's a very weird name. It's a very, very weird name. Wish there was a better way of saying that that didn't. Wasn't totally confusing with the verb noun form.

Right, I agree. I have a note about that in the tentative ref page, but.

So.

This is. I mean, really, it's a manipulate keyframe video. Is that true? Yeah. Or it's a keyframe video derived from a manipulate. Correct.

Yes, yes. The point is that we'll make many kinds of keyframe videos. We. I mean the one that you're doing right now, Tour Video is such a thing.

Is it? I mean. Yes.

Okay, you specify two key frames, one center around Paris, one center around London.

Right. But the interpolation there. I see. So the manipulate. Okay, let's see this run. Could I have told it to be cyclic?

Go back to Paris. Yeah, yeah. You could add another position there. Paris, London. Paris.

I see. Okay, fine. This is pretty cool. It's pretty cool. So this is in a sense an alternative to a manipulate as I move the video around.

Correct.

In this case we'd guaranteed performance because there's no compute.

Right.

And so if you, if you have something very heavy duty, that's, that's, that's compelling. Right. But, but the other thing that I, you know what I think, hold on.

I just want to make another point. I mean, so this is always a one parameter thing because time in the video is one parameter.

Correct.

But, but for example, a one parameter manipulate in the cloud could in principle be done this way.

Correct.

So it has no server transaction at that point or at least well, you know, it's a very well optimized server transaction. But you know, whatever a streaming type video. Actually I don't really know because I mean streaming videos often do all kinds of crazy buffering. You often see a lot of buffering.

Happening, but at least there is no computation. As Roger pointed out, if there is any expensive computation that's done once.

Right, okay. All right, so go ahead. Roger, you were going to say something? Sorry.

Yeah, so I think that this is for ourselves and for many of our users. Unfortunately the manipulate video is not fully cooked. But let me continue to tell the story because I think we have a bunch of other things that affect you and the rest of the company. I think that one of the things that we can do is to re energize the demonstrations project from this. So for instance, we can engage the contributors to really tell a story, you know, so right now we have fixed screenshot and we have this kind of slightly stupid auto generation where it just twiddles all controls which is not usually correspond to any kind of experiment that you want to do. But, but let's say that you go back and you tell a story and you show. These are like the key experiments that this manipulate can show or some somewhat of voiceover you mean? No, you could do it with voiceover or you could do it with, you know. Yes. And so I kind of, you know, that's, that's interesting for demonstrations and re energizing that story and to start to produce shorts that for instance can go out on, you know, like YouTube shorts or other short form video like tweets. Right. And so I kind of think that like, like for many of our areas we, we do want to produce these videos because they can give you kind of like a summary of something.

Like.

Like in many cases you spend. The government spent like say 100 million for a decent supercomputer. Maybe 200 million these days. And then what is the deliverable that comes out? It's usually a video, a short form video, you know, from a simulation. But I think we can do these things, you know, our own, you know, like our users and at least a more compelling learning material. More compelling?

Yeah, all around.

But as I said, I don't think manipulate videos cooked yet to that level. But, but I think that, that that's the potential.

Right, hold.

On.

And so one of the things that we've done now is gone back. Most people did to kind of look at bookmarks because there was something there already. Although when we sampled people, turns out that 90% doesn't seem to know about the bookmarks and manipulate. Yeah, they were not widely used.

I mean. But my question is, okay, so let's say we've got a demonstration and we want to make a video out of it. So you know, I have to say, in modern times with the whole alarm business and so on, maybe there are different ways to think about how to set the bookmarks because I, I don't think, I mean, you know, the method we had to set bookmarks before is not particularly bad.

Well, I think that there are two situations. One is kind of like the bookmarks where you just manually set the controls to something and then you go and bookmark. Sometimes you want a little bit more precision and you want to actually set them, you know, to very precise values. Because it's a real experiment that you want to perform.

Yeah.

And then a programmatic setting is better.

Okay, but I wanted to talk about the whole workflow that ends with a video with, with voiceover. Okay, well, so I mean, because that's going to have to be something.

I think LLM is an alternate. You know, I think that that's another path. I don't think it's the only path. And it's.

No, it's not the only path. But I mean one could imagine things like here's a, here's the code for a. Manipulate. One could imagine something where it's like, okay, LLM, here's the stuff I want to show. You know, do this and explain what's going on. Not quite possible yet. But with multimodal models, something that is definitely on its way. I, I mean it would be bad to spend a lot of effort designing something that is for a world, you know, moments before sort of the LLMs walk in. So I mean, I'm just trying to understand what.

Well, I mean, my impression of LLM, it's kind of low precision and in the sense that, that you get sort of roughly what you say. Like the same thing that happens with natural language generally.

Sure, sure. But, Right, but, but in making, you know, that's the point that in making a video from a demonstration roughly is already very useful.

Yeah, I think that's useful too.

Right. I mean, you know, being able to say just here's this. Manipulate. Just make me a video of this. Now, which is.

But how many sentences do you think. Let's say it had 20 controls. How many sentences do you think it takes you to say what all the things are that you want?

I think with very high probability that you can give something which is, you know, if you're going to explain it, you're not going to explain all 20 controls in every part of their explanation.

Right.

So it's going to wind up being something that is, you know, where you say, I want to go to this limit and that limit and then it's got to figure out how to settle the extra controls. And I think it has a reasonable chance of being able to do that. So I mean, let's, let me understand what, what. Okay. For this, this looks rather uncooked so far.

Yes, unfortunately.

Okay, but so I, I want to.

Point out that one of the main things that we were planning in to support how this works is updates to the bookmark system within manipulate. Because the specification of these keyframes syntactically needs to live within manipulate. And I think Shadi has, in the, in the design notebook mentions a time marks page, which is a proposal for an update.

Well, let's see how it works right now. Yeah. So where is the bookmarks option right now?

That's right there.

Bookmarks. Okay. Bookmark settings.

But it has, it has several limitations. That's what.

Okay, we're not going to redo it just because it has limitations. We're going to first understand it. Yeah. Okay. Which perform those actions I see. So it's essentially procedural. A bookmark as a function to run. These are named functions that you're running.

No, it's an expression. It's an expression to run.

You're right, you're right, you're right. This was obviously done before we had associations.

Correct.

I mean, it's.

Yeah. I don't think it needed to use assignments to do it. I don't know why it did that, but it means it's slippery to manipulate.

Well, okay, so what's your idea for how to do it? No, no, I. I'm not.

I don't. I didn't particularly want to redo. Bookmarks are bookmarks, but I think it was a bad idea to use assignments.

Okay, but wait, no, no, no, no, no. Roger, Roger. We're not going to do this thing about introducing another name for something which is incredibly similar. No, no. Okay, so what is this now?

So this is a proposal for an evolution.

Well, you need to store different data. You need to know at what time something happens.

Yes. Okay, so this is why Lou is here and discussing with him. There were several possibilities. Adding an additional option, as you said, it is something that is very similar to bookmarks, or obsoleting the bookmarks name, updating its name and adding new syntaxes while maintaining.

But basically what you're saying is you can give a proposed time for the bookmark. I mean, the time might be minus seven, and then, you know, absent time travel, you can't get there.

Correct.

Right. So, you know, the times.

Is a crucial element. Like when.

No, no, I. I understand that, but I'm trying to understand in the case of a tour video, for example, or an animation video. Let me, Let me remind myself how animation video works.

Animation video is just a single variable. Manipulate, essentially.

Okay, but, but doesn't it have. Okay, how. How does. In the case of tour video, for example, how does it. Doesn't it have a way of saying. Yeah, T. Yeah, T. I Step I.

Absolutely.

Okay, so one model that we have, step, step I at times ti.

And unfortunately, this is not particularly viable for us given the fact that we use expressions to specify these key scenes in manipulate.

No, no, wait, wait, wait, wait, wait, wait, wait, wait. In this case, this thing here is a tour video with, in this case, two elements there, and it's going to go between those two. God, I hate the way that this is done with the I and so on. But anyway, never mind that. Okay.

That'S a relatively large image.

Okay, but, but. So why are there. No.

You can specify the Times or you can leave them to be automatically computed. And in fact, if you open up.

This is a very silly way to say this, if I might say so. I mean, isn't this, What is this doing? How does it integrate this? What is it doing?

It goes between two rectangles.

And this is the alternative form. Yes, correct.

Where you could specify a time as well. If you added a time key as well.

I don't think it should be called steps. I think it should be called something like frame one, frame two or something. Okay, something like that step seems just super confusing. I see that, especially at steps. I mean, it's just not. That doesn't mean anything. But it generates a video by. What the heck does this mean? By taking certain, by visiting certain frames within graphics or something? Yeah, it's poorly written. We should fix that. Okay, all right, okay, fine. But here, how does this relate to the manipulate syntax? So this is saying at time ti you get frame I, so to speak.

Right.

Okay, so now in the case of manipulate, what's happening here is the, the kind of, the knitting of the frame specification with the original graphics is anything I don't fully understand. It's essentially a show. What, what really is it? What, what is the function that is applied to the graphics?

So if you're going to have. Yeah, so if you're going to have the camera be one rectangle and now you're going to move to a different rectangle and that might mean just translation or it means translation and rotation basically. And so you need to interpolate both of those when you go.

But you're explaining a different thing than I'm asking. What I'm asking is when you get to that frame, when you get to that time, what you have is some kind of image trim or something of the image. What is it exactly?

It depends on the data. If it is a graphics, it's some variation of show with a specific plot range.

What if it's an image, it's some.

Variation of image trim and image rotate.

It's kind of unfortunate that there isn't a single function that we can say, you know, take this cookie part, cookie cutter, you know, cut out. Take this cutout in this graphics or image.

That's fair. And including rotation also is a non trivial part as well.

No, no, I understand. It's algorithmically non trivial. I'm just saying that.

Yeah, I'm just saying if a new function were to exist, I feel like including the rotation would be important.

Yeah, it's like image cutout or something. Or Image. I think we should have such a function and we should know that what the tour does is it applies to that first thing. A. So, I mean, the generalized version of that would be. Even. This is very confusing to me. Does it need to know beyond the keyframes? It needs to know that those keyframes. I mean, it's using the actual internal information about the frame specifications to do the touring. Is that correct? Yeah. Yeah. Okay. So it takes apart.

So it gives you as a user, you kind of get like a relatively minimal spec, I'm saying. Yeah, but still very relatively precise.

Right. So we couldn't have a generalized, you know, function video that takes.

You have one and you hate it. And we all hate it.

Except developers Video generator.

Yeah.

Okay.

It's very generalized.

Okay, what's the thing for.

So to the point, that is very hard to use, but very nice to control. I mean, I understand.

Okay. Why is this slideshow video not doing its times in the same way that this does?

It's a good question. It has. Go back to it again.

I mean, I think this is more useful. What?

Yeah, I think that is. That was a mistake in hindsight. Yep.

Which was a mistake.

Well, one of the two. I'm not entirely sure.

Well, I think this one for slideshows is.

Is useful.

I don't want to be computing, you know, I want to be. Don't want to be doing arithmetic, and when I slip a new slide in, I don't want to be re. Arithmetizing everything.

Yeah, so this. This kind of has. This is. Okay, so this is like a different. Right, because these are not timestamps, these are durations.

Well, I understand that, but I mean, the question is, why is it really applied based on durations?

Well, because.

Well, in the other cases, you're not going to linger on a controlled spec. You're going to be moving all the time. So you specify the events rather than the here in a slideshow, you're going to linger for a long time and then switch very quickly to the next one.

In tour video, we have dwell time as well. Okay, but apart from. Yeah, but that comes in the more complicated association syntax of steps or frames.

Just a minute. It. Okay, I'm sorry, back to another little emergency going on. Okay, go back to this. Excuse me. Okay, so. Sorry, what did you just say? I. I missed that.

The move is to point for the other ones.

Okay.

We just have snapshots where you stop.

Okay.

Here, dwelling is the this domain activity.

Okay. So that's not silly. Okay, let's Talk about manipulate. Okay, so we want a way. And what are you proposing?

So one thing that I want to make clear is that all of these key scenes need to be specified within the manipulate expression because the scoping wouldn't allow you to specify them, given that you have an expression that sets the variables of the manipulate. Right.

What is the setting here? Is this an expression? X equals 3.

This works like bookmarks. It's just an expression.

No, no, no, no, no, no, no. That's not how Bookmarks works. Bookmarks, as we just saw, has an actual. An expression which does an assignment. Is that what you're doing here as well?

The same. Because we're, we're kind of updating the framework that Bookmarks is using and.

Okay, but I don't like this. I don't like having two names here. I don't like this, I don't like this. Yes.

So the other alternative, that would be to just upgrade bookmarks and upgrade as well.

Let's look at how Bookmarks works.

Yes. So the updated page addresses the limitations that we think are important for video creation.

Okay, we're fine. But I mean, what we have here is a label and an action for bookmarks. Okay. No, no, we're not going to do this. No.

Roger, I think you haven't committed your update from this.

Yeah, we just looked at this this morning and I removed it.

Okay.

All right.

Okay, fine. Didn't I commit it or whatever.

So one, one useful thing we were talking is that if bookmarks have labels, maybe time marks or whatever function could be an option of the video generation function and then we can refer to times. Goes to labels in a sense saying that I want to be at that setting at time five.

Yeah, that's what we suggested too this morning. The right hand side could be a label string label, basically.

Alternatively, in bookmarks, those could be annotated with their times. I mean, or the label, if it's a number, maybe that's not a good idea. Could represent a time.

I feel like that this time marks page could just. That functionality could be included within the bookmarks option if you look at the syntax.

Well, I mean, it depends on what. Okay, look, let's talk about how this is in practice done. Yes. It doesn't really matter a lot how elaborated the syntax is if it's all going to be done through a GUI anyway.

Yes, this is what I'm counting on as well. We just need an internal method of storing this information and producing the video. But I agree, the input form for all of this, the Most practical one for sure will be the GUI related one.

Wait a minute. I'm having a big deja vu from like 15 years ago. What is this? Lou, help me out. There's some dynamic refresh something or other thing where we also have. What the heck is it? It's some. There's some control or something where we also have some kind of time marks. Help me out. What? What. What am I thinking of.

With that description? I don't yet know.

Okay?

Are you talking about animator and all?

Yes, I'm talking about animator. Things like animator. Because animator has the same. Is it animator or is it something? Is there some analog of animator that runs to keyframe marks? Do you understand what I'm saying?

There isn't. In manipulate, you can animate between already.

I understand, I understand. But there's. Isn't there something here that does this? Lou, help me.

I'm not sure what this.

Well, what this is is something that has times and settings. Okay, so look at the ref page for slider.

There is one calling signature for slider that lets you specify values and the interval of relative widths. The last clause there. Perhaps that's what you're.

No, it's not what I mean, it's something dynamic that we did about 15 years ago.

Okay, I don't remember. I. I don't know.

Okay. We shouldn't get it out because. Because it's something related to animator, manipulator, something like that, where we've got the same idea of. Of having, you know, a time mark. I mean, also we've got, you know, the model that we have in sound note. And sound is another thing, a bit like this, where we have these symbolic objects that represent a note that should be played for a certain time, which is a different thing than we have here. You know, the analog of a sound note approach to this would be a, you know, some kind of video frame wrapper and then an overall wrapper that says interpolate between these frames. Am I making sense? But maybe that's not possible because, like in the case of Tour video, you have to know what the underlying graphic is that you're walking around. It's not just a question of interpolating between the frames. It's a question of knowing the underlying mechanism, which same in manipulate.

Yes. I mean, manipulate is simpler because it doesn't have very advanced controls.

Right.

Like, when we come to 3D graphics, we're going to have to interpolate 3D rotations. It's okay, we have what we need for that. But Those are kinds of things one has to do.

Right.

Stephen, are you thinking of the Autorun sequencing option?

Perhaps? I am, yes. Let me look it up.

Yeah.

No, it applies to Autorun, not to bookmarks.

Yep. Yes, I may very well be thinking of this. I mean, I remember that we discussed a lot, sort of an early version of dimension reduction here. Because what you've got to do is you've got this high dimensional space and you're going to find this sort of space filling curve. Oh, I wasn't too bad on my 15 years, give or take. Okay. In any case, yes. This is what I was thinking of. So explain the relationship between this and what you're now proposing for manipulate.

So this doesn't have any concept of frames. This just when they say run variable I, it just means explore the whole domain. Okay. This is very different in terms of what happens at a specified time. And this is why this can specify durations rather than specific times. Right. It says run variable I for T seconds, which is not what the bookmarks thingy, whatever that will end up being called. So it is related, but definitely not the same thing.

Okay.

This is more automated animation. And what we're proposing is give control to the user on video generation rather than.

All right, let's talk about the workflow.

Yes. So I'm not sure, Roger, if you want to go ahead and do that or if you want me to.

I was trying to define this. Okay, so I'm going to go to a random demonstration and I want to basically make my animated version. And what you're proposing is that I could give a script.

Exactly. I even have a mock up where I did it.

Okay, but so, so where there's also an audio or a. I mean, where you either give a closed caption or you give audio or both. And maybe the duration is the length of the speech synthesize of the audio. I think that, I don't think everything.

Needs audio, but yes, we could think of that too.

Right, let's talk through that, that workflow. Because.

Because audio makes it sort of tied to a particular language.

Yeah, yeah, yeah, but. Okay, fine. But we're going to need explanations, so.

Sure.

You know, we can have those explanations as, as, you know, captions on top of the thing and then those can be translated in different languages, et cetera. Right, but I think now that you.

Say it, we actually do have, in our video codecs we have subtitles.

Ok. In different languages. Right, yeah, they have a language tag.

Yes, it could be in different languages.

Right, yeah, right, right, right. I mean, it's taggable for different languages and presumably you can switch on the subtitles. Yeah, So I mean, we could have a, a voiceover and then subtitles. But, but. So here's the way I would think about the workflow, maybe. Okay, so one possibility is speech synthesize, which, you know, today is still a bit clunky just in the world at large, although it's getting better and probably this year one will see very realistic, you know, audio synthesis. And the alternative would be that you essentially record the voiceover by doing the following. Where it says, you know, where, where it prompts you. For example, we have some function, maybe in the function repository where, you know, it goes through, it shows you the manipulate, it says, it says, here's the next, you know, waypoint that you've defined. Now press record to audio capture the thing that is going to go between this waypoint and this waypoint. Then it takes that audio, it knows how, what the duration of the audio is. It fills that in and it puts in that voiceover. Does that make sense? Yep. I mean, maybe. Actually I said that a little wrong because what you want is at a given waypoint, you're going to dwell on that waypoint probably, but you might want something that both is the dwelled, you know, the dwell audio and the motion audio, if you see what I'm saying. And you could also have an audio that spans a dwell and a piece of motion. Am I making sense?

Yeah, I think it's important that there is like a minimal version that doesn't do that. But optionally, if one specified this, it's, it's. It's also has a very streamlined workflow just like you described.

I think that to make this really useful, I think it has to be annotated. You know, I just don't think it's going to be that useful to just see the thing sort of jump around from here to there without explanation.

Well, you may be right.

But I.

Think annotations can also be textual.

Doesn't necessarily mean there's three options here, four options. No annotations. Annotations that are burnt into the video as a kind of an overlay. Right. Which also will be useful and which we can also do speech, synthesized speech, where the source code is purely textual. I mean, that seems like it could be super useful because you can easily edit it, you can translate it, you can do all kinds of things.

The other thing that's useful is to. Sometimes you want to remove the controls because the controls are there for a.

Human yeah, right, right, right, right.

But when you have the video, you may not, I mean it can be.

You may not want the controls. There may be no reason to show the controls, but I would guess most of the time you don't want to show the controls. Controls.

Yeah.

I mean, sometimes the controls may be useful as an explanation of what's going on. Right, but often not. Right, okay, so then. Let's see. I mean, but, but see, when we.

Think now you're cooking, okay. And now we're starting to think of this as a storytelling tool and now demonstrations can become a storytelling tool.

Yeah, yeah, yeah, right. No, I agree, but I mean, I'm just, I'm just thinking through, you know, the actual workflow. So let's just enumerate. I think there are four cases. You know, the no caption, the burnt in caption which is image overlaid. The, The thing where the video. I don't know how useful it is to have it silent, but closed captions, I don't think it, I mean if you don't want the, the voiceover, just mute the damn thing. Right, but, but probably the case where you have closed captions is the case where there was voiceover but you might choose to mute it, have a different translation, translated language for the, for, for the closed captions, et cetera. Right, okay, so there's auto generated voiceover and there's human recorded voiceover, am I right? Yeah. And I mean anybody want to give their opinion about the quality of the speech? Synthesized voiceover.

I mean, we can, I mean the one that we have now is not good enough. We'll come to that a little bit later. The one we have now stock burnt in sounds robotic. It's, it's.

Yeah, I understand, I understand. But I mean, I think we all believe that it's a, it's a technology that's going to arrive fairly soon to have good, you know.

Yes.

Okay, well, so I'm, I'm not sure how much of this. Let me see. Okay, well, I mean, so, so maybe this is as far as we need to go on the design of this. Right now there's a question from the live stream before the about can these video functions accept an interpolating function?

I mean that is already a hidden option in manipulate, but yes, that sounds like a very useful thing to add.

Yeah.

One important thing for many controls to make them look nice is that you really don't want just straight interpolation because Chris kind of jerky.

Yeah, I agree, I agree.

So it needs kind of like a Spline, you know, spline fitting thing that comes in with a nice derivative.

That must be in the world of dollying and, you know.

Yes, it is.

That must be a known. Yes.

We're talking about manipulate. Manipulate has controls of dramatically different domains. It can be discrete, it can be whatever. So the interpolation function needs to be tailored to the control type. And this is some of the things that.

Fine. Well, then we should have, you know, a typical control that I would say moves quickly and then slows down at the end and then comes to a halt. You know, like.

Exactly. That's, that's, that's like those spline like endings.

Yeah, right. But I mean, it should be kind of like what you, what you have become used to with a car where you have, you know, where you're going at a constant speed and then you have a deceleration and then you come to rest. Right. My guess is that that will feel fairly natural. Right, okay, well. And yeah, clearly there's a whole bunch of work to do on sort of curating the way that this, this happens. So now the question of. So there's quite a bit of control on. Yeah, I mean, it's. It's rather confusing because the settings thing doesn't seem quite right. Here's why. You've got two parameters, X and Y, and the end result is at a particular mark, x equals 0, y equals 10. Okay. But as you get there, the X coordinate, let's say the X coordinate is discrete. Let's say it's N and X or something. The N is discrete, the X is continuous. The whole process by which you slow down and approach, you know, n equals 0, x equals 10, or whatever it is, is not obvious, not trivial. It's not a pure end state. It's not a function of state, so to speak. The path matters. Am I making sense?

Yeah, but I kind of think if it's discrete, then you will arrive there. There's no, there's no xenos paradox with discrete stuff.

No, I understand. But let's say it's X and Y variables, both continuous.

Okay.

Okay. You can just like you've done already for. Is it to a video.

Yes.

You know, you can have the things spiral into that point. Right. So similarly here, there's no reason that the manipulate with its two continuous variables shouldn't be spiraling into a certain endpoint as opposed to just moving in XY space. I mean, what determines is it moving on a JD stick in XY space? What is the structure of XY space? Et cetera Am I making sense?

I mean, I agree with that. I think what we're aiming to provide is a good default behavior. And if the user wants to spiral in, they will have to do something more custom.

I understand, but we need a specification which allows spiraling in. And what you had defined there is something with marks where basically you've only got the function of state.

You could define a spiral. Right. You could just define several bookmarks along the curve. Not great.

But how does Tour Video work in that regard?

So with Tour Video, you have the option of specifying a pure function as well, and that allows you to do this pile.

Let me see. I see a function there to generate step specifications for each frame. So that function is fed a continuous parameter, a single time parameter. Is that correct? Yep. Where is the specification for that? Where is the specification for that? Don't tell me it's not in here. Come on, guys, you know the rules. We can't have it only in the examples. Where is it when a step is provided as an association of properties? Okay, I'm kind of annoyed. We've been through this many times. Okay.

Yep. We should add it.

Yeah. Can you go through all the functions from your team and make sure that that mistake does not occur, please?

Yes.

What the heck? There's nothing here. Where is it? Where's the thing specifying how you use the function there?

Last one.

Okay, that's just not good enough. Just not good enough. I mean, I don't know what. So it just takes a time and I don't even know the range of the. I mean, the time is presumably in seconds.

Correct.

Okay, but notice that in this case, this is returning rectangles. Yeah.

So any.

But now could return any of those supported step specifications.

Okay, I understand, but so what that would be in the case that you're now describing is something which returns, in effect, an assignment. Am I making sense?

Yeah. Then, yeah.

It's a little weird that it's. Well, I mean, you know, it's such a heavyweight thing to generate a video frame that doing an assignment is peanuts. Right. If that's how we want to do it with an assignment which we may have no choice about. Did you understand what I'm saying?

I mean, yeah, I feel like. I'm not sure. Then the manipulate video would be the best tool for that, rather than just using animation video with that single variable time and then just making sure that that time is used in the expression that generates the frame.

Yeah, but no, you can't get away with that because it could be the case that in the story, right, you're going from here to there to there to there, and then you spiral in on something else.

Okay, in that case, yes. Okay, fine.

Now, just so I understand it, you know, we need a full workflow where the thing has a titling slide. You know, where you do a slide show video of a manipulate video and of a slide and then a manipulate video and then something else. Will that work?

I mean, we do have functionality to combine videos.

No, no, no, that's not what I'm asking for. I'm asking to use the slideshow video mechanism so that you can put up. Well, maybe. Maybe I'm. Maybe you're right.

That's a good idea. Adding slideshow video, adding video support for those inputs.

Right. But I mean, you know, you might also be right that all that we want to do is just take a slide for a certain time from a slideshow video and then ram it into a manipulate video. Because after all, it's. Although you might want dissolves and all kinds of things like this. But you know, I think when, when movies are made, right. It's like, you know, they have this idea of a shot, you know, like. Like a movie consists of, you know, 1500 shots or something. And each one, I guess the camera can move around during it, but it's kind of. Each one is a kind of a setup and then you do your thing there. If I'm understanding correctly, that's kind of how we've been.

That's kind of why I've been thinking about this in terms of these creators, that we have certain kinds of mechanisms and now we build easy mechanisms to do those shots, if you like.

Right. But then the combining of the shots is just a video join. Is that correct?

Yeah.

Okay, well, that's not unreasonable. And then. But one thing that I think is a little complicated there in Video Join is voiceover.

Yes.

Because you might want the voiceover to span multiple shots. I mean, I think in. In typical movies. Right. Doesn't the music track. You know, it's. It's. The music is continuous even as the. You're going from shot to shot.

Yep. And I think it's very common to create the audio track separately. But I believe we have to go back to our audio track injection subtitle track injection at specific times and make that a little more seamless.

So, I mean, maybe we need. In this meeting, maybe we need some of our people from the video team who know how typical video editing software works and so on, and maybe we have that here. Do we have that here? Do you guys know that well or not?

I don't know. I do. No.

Okay, fine.

I've talked to our video people though, a couple of times.

Well, why don't we have them in the next one of these meetings.

Sure.

And suggest that they could watch part of this meeting recording to get up to speed. Yeah.

I think what this enables, the kinds of things that we're building now are videos that we would never build before or we would have to be strongly, strongly, strongly motivated.

Oh, sure, I agree completely. But I mean, what I'm asking for is, you know, as I understand it, when people make a movie or something, they, you know, they're literally the, you know, the orchestra is playing in front of a, you know.

No, I mean, we should think of that.

No, no. But I'm just trying to explain when, you know, I think that, you know, the way it works is you have the silent video and then you literally have the, you know, then, then people know these are the time marks you have to hit in the thing you're playing when you, you know, when the grand, you know, you know, the grand symbol, you know, thing happens. And the, you know, has to be when the video does this. So in other words, I think people think of it as you make the video and then you add a soundtrack to it. But I don't know. You know, in our case where it's.

Explanatory video, I think our video team doesn't add soundtrack. I mean, we don't compose our own music.

No, no, but we do put music beds on videos. But, but that's mostly muzak type music. As in it's not, it's kind of rare.

Well, we should done it.

We've certainly done that.

We've certainly. True, we have done it, but it's kind of rare if you.

I, I know, but we're not. That, that music isn't well coordinated with the video because to do that you really have to have, you know, all kinds of fancy work done. Right. Which we've not been doing. But so in this case, the problem is that we want to make it easy for the particular case we're dealing with here to, to add, you know, voiceover that is somehow coordinated with what's going on. Okay. All right, we should wrap up fairly soon here, but let's take a little look forward. So I think there's more to think about here. Another meeting for this with video folk and so on. Now let's just take a quick look here. Okay, that's nice. So you can say an image of a graphic with. With what kinds of options to the graphics? Does. Does pixel constrained work? For example? Does pixel constrained work?

I haven't looked into that specifically.

I think you should. What other options? What's up?

Strained is obsolete.

I'm sorry?

Pixel constraint is documented to be obsolete.

Well, what's it. What really? Because people use it all the time. Really? Well, I would talk to Jeremy Davis about this because he uses it all the time.

Okay.

In fact, pretty much anybody who does stuff with like cellular automaton images uses it.

Yep.

So something is not working that's causing that to be necessary. I don't know if it's an anti aliasing issue or what. Okay. Anyway, that's the thing to explore. But what did you have in mind here now, except most graphics options.

Thanks. The few options that I documented that I found them missing things like frame grid lines, epilogue.

Missing? You mean if you say image of the graphics, it just drops those features of the graphics?

No, no, no, no, no, no. So image, image object itself can have all of those graphics options. Now you can construct an image.

Oh, that's nice. Okay.

Yeah. So the key use case is that various workflows like medical or our own highlight image adds annotations, basically. And now it becomes a non image. And in the future they can can still be images.

So you're saying frame arrow. True.

Correct.

I see. That's a framed owl, special species, right?

Well, in some cases frames are good. Like if the tumor is, you know.

2 centimeters, this is already useful.

0.1 centimeters.

This is very useful because this is in pixels, right?

Yes, that's already useful.

Great. And epilogue is surely useful, right?

Yep.

I mean if I say epilogue arrow, you know, disk 100 comma 80 comma 20 or something. That's lovely. Very nice. And presumably I can do picture in picture by putting another owl there. I'd have to do an inset, presumably.

Yes.

Right.

Yes.

Okay. All right. That's good. Make sure you show an example with an inset.

I'll do inset. Yeah.

The. Find image shapes. Are you using Christopher's library mechanism or is this actually burnt in code?

No, no, Christopher, no. That. That still is not working with most libraries this is building. This is separate code. What's not working with this is Christopher's code. We checked it in fact, for a very simple library early.

I hope you complained. Yes, okay, fine. What's not working about it?

We send a whole list. I can.

Okay, all right, fine.

Look it up.

Fine. Okay, interesting. Roughly, what's what's not working about it?

Some special C. Oh, yeah, right.

It was, it's for C. Yeah, right. The stuff he built is primarily for C. Yep. Right. Okay. So that's a, that's, that's the issue there. Okay. Find image shapes. Let's take a look at that. That sounds interesting. Sounds a bit like image graphics. Is this Image Graphics just with. I'm confused here. It's.

It's recognizing simple geometric shapes within images.

Right. So it's a subpart of Image Graphics which would try to turn the whole image into shapes.

Yes.

Image graphics is. No, I mean image graphics is very general in terms of, you know what it does. These are for simple shapes.

No, no, I, I got it. But this is. Image graphics would presumably include all these shapes and many others for an, for an image. Is that correct?

Correct. Currently we only have anything that is a contour, so lines, circles and so forth. But, but then planning to have.

This is a filtered version of Image graphics. True or not true?

Not true.

Great.

Could be seen as that. Roger, though.

Sort of. But I think you're both missing the point. So this one is to recognize regular simple shapes within images.

I get that.

Image Graphics is basically vectorizing our graphics and so it's running marching cubes to recognize borders and it has nothing to do whether shapes are simple or not.

I see what you say. I see what you're saying. So, so this is attempting. I mean, this is essentially a model. You're saying image graphics. There are two different models being used. This one, the model is, you know, has an emphasis on low numbers of parameters in the, in the model. Whereas the other one probably has an emphasis on fidelity.

Yeah, you're right. That's a way to see it too. Yeah. So you can say that Image graphics and it literally turns it into graphics. It's like a non parametric model. I don't know why people call it that because they have massive numbers of parameters.

I know, but I know what you mean. Yeah, right, Okay, I understand. So this is essentially a fit here.

Yes.

Fitting lines to.

The two key features in the image. Yeah. And some simple shapes occur a lot in, in, in the man made world.

Right. So I mean, if I take this image here, for example, and I.

Which is not so man made.

I know if I say find image.

We don't have it. We don't have it in today's prototype.

Yeah, okay, but, but in principle, if I said disk here, it would. Yeah, and I would suggest strongly that you do that. And I would even suggest in the world of the future that you allow things like disk underscore radius 2 or something, or is that crazy?

I proposed similar things and we did similar things for distributions.

It's incredibly costly to do, but still, as a design, it's, you know, it's.

A nice design, but it has exponentially numb.

In the case of image, what should I do with color styling and all of that?

Well, this one doesn't make a statement about that.

Right, right.

But you're saying find the red disk or something. Well, but are you doing.

And then the almost red. No, no, no. This is. This is working on a grayscale image where whatever kind of a shape you want to find is highlighted.

So for today, it would just be this. But in the future, it might be a pattern that involves, you know, color distance and all kinds of stuff.

Okay.

It might be, you know, find a disk radius less than 2, radius greater than 2, for example. That might be a very common thing to want to do. Right. Say here we want, you know, blank question mark, hash greater than 2.

Makes sense.

Be very cool to be able to do.

I mean, actually, so I think would be really difficult.

But.

Well, I just realized the way that we can do it, it's brute force. But it's kind of in the spirit of what Stephen is saying. Like, our cases, you know, our pattern matcher is kind of stupid. It just looks at everything and then filters out.

Yeah.

Let's say that this one did the same thing. It found all the disks and then just filtered out. Then. Then code wise, it wouldn't explode at all, actually.

Right, right, right.

That's what I would suggest.

Okay. That's how we were thinking about it.

Right. And then. Then we could also do something like.

And so then in a way, what you're proposing is the same as just like getting at all the output and then running case, you know, cases on it.

Exactly. Just a way of. So here it will be color distance, you know, hash, comma, red, less than. No, if we look the color distance, it's black and white. What do you mean? Well, by the way, did that semicolon. Did that ampersand. Oh, before it goes in the wrong place. Okay, never mind. Okay. All right. Anyway, I mean that. This seems like a nice. Very nice conceptual syntax, so to speak.

Yep.

So in some sense, this is almost like. I mean, we've got image cases, which is a little different, but this is more of a cases kind of. Well, what does it return as output?

A bunch of lines, circles, disks, eventually.

Okay, so it's the it's the primitives.

It's the primitives and then you can get properties for what has. What has been detected. Like.

You color. If it was a style, you could potentially pull up that. Or are you just dealing with geometry currently?

We were just thinking geometry. I'll. I'll have to see if we can really squeeze in style subsets as well in there.

Okay.

I mean shapes are different than geometry.

No, I, I know that.

And what I. I think as long as we call it fine image shapes, it can just be geometries or shapes and then find image subsets or something like that.

Possible image cases. Right, right.

I understand.

How does this relate to the image cases?

That's a. Image cases is a higher level construct. I think we're looking at more classical template matching, pattern matching. If we talk about something like find sub images. Maybe, maybe not. We also have that. I'll have to look into that. Think about it more.

Okay, fine.

This is just shapes, no color for now.

Okay, That's nice. So the text recognize is going to get better? Basically, yes. Oh, we got an audio page.

Audio GUI is getting updated if you want to open that subscription.

Yeah, open that, Steven. It kind of summarizes it really nice.

Where is that?

The little subsequence section? Yeah, that one.

Yeah. See. Okay. What's the. This is the before, this is the after.

Yeah, yeah. And the comparison with the video GUI.

Controls, which is down.

Which is down below. Yeah.

That'S too big. That font is too big. That font should be this kind of size.

I agree.

Yeah, that's just a mistake. Make sense.

You also have the eyes of a hawk.

What font? What font eyes. And what about the icons which I think is somewhat useful. What's that?

Those are dropped. We don't have them in the new version.

Is that globally true for how we're doing the data in whatever stuff?

I guess so. I think they were custom in the old audio gui. I'll check it.

Icons in any other thing, icons will be present in the store in Notebook or is the change storage location button the one on the lower right? So that that opens up a menu and.

Okay, but the. The current state is not indicated by an icon.

Correct.

Okay, So rather than the waveform you want to go to this kind of thing? Yes.

That is not the curve. That is the depiction of the full waveform rather than the previous one. That just gives you the waveform at a specific time. Like are they.

I know, I have no opinion about this. That's the amplitude of the waveform.

Yeah. This Gives you a better way to differentiate between audio files. Well, the previous one was useless and less well was used.

That looks very much like SoundCloud.

Correct.

It's nice.

But is. Is that again, my naivete and lack of cultural knowledge? I mean, in the case of something like SoundCloud, is that showing audio as a function of time or is that showing what used to be shown on like, stereo things? Well, maybe.

No, no.

Which is.

That's the wafer.

It's the way it just lights up as it plays. You can see it sort of turns color as you progress along the waveform.

I see.

That's.

And we do that too.

Yes, yes.

Okay.

All right.

We should wrap here. But this is nice stuff. And clearly this is not the last meeting about this. Oh my gosh. Seriously? We've got a chess playing system now?

Yeah.

Okay. Why not? I guess it's part of the human condition, so to speak. It's part of. I mean, it feels like if we're doing that. Yeah, I mean, it's an interesting prong, let's say.

That's exactly how I think about it.

Right. I mean, it's an indication to me.

This is a little bit. It's driven by great enthusiasm, a few of our developers.

Okay, fine. But so one thing to think about there. We got. I would say your force. Not relevant for tests, but there's the whole game theory. What has happened to the game theory project, by the way?

Anything at the moment, we. That there's nothing to. If you ask, there's nothing to send out.

If.

So, you know, because there's no. We need. We need to start over. The solid. The foundations of it is okay, but the code sucks.

Okay. Be a good thing to get done.

But that's very different from actually.

No, no, I know that. I know that. The. The thing that this reminds me maybe a little more of is the virtual machine framework, which also is partially done. I'm not sure. Does that. Does that resonate? I mean, does that analogy make sense to you? Roger, I think that this is a little bit.

Well, this is too specialized.

Virtual machines can be very specialized too.

Why would you say that?

Well, I mean, you're poking at a device, a thing.

But. But isn't a virtual machine like a whole architecture? It supports. It's usually the case.

Well, it could be a solid automata. It could be a register machine. It could be something quite idealized and simple.

Oh, okay. So is it. I mean, I don't know what use cases you guys went after when you were sort of doing that. But one thing is sort of from.

Turing machines to, you know, IBM 1790s to early video games.

Okay, I was going to get to video games. Was that kind of part of it?

Yes. All right, we should wrap up here. Go ahead. You were going to say one more thing.

No, no. It's a longer discussion. I have a bunch of things to say, but I. It's like I don't want to.

That's right. Okay. Sounds good. Lots of stuff happening here. Right. We'll need another one in this series, obviously. Another meetup in the series. Okay. Thanks a lot, guys. See you.

Thank you.