Live CEOing Ep 742: Language Design Review of Sound & Vision Features for 14.0

Sa.

Hello, everyone. Welcome to our third Wolfram language design review of the day. This is part of the version 14 season, and right now we are looking at video. Image, image. Do we get audio as well? Oh, my gosh, we got audio as well. Okay, video. What's happened to video in the cloud, by the way? Because it just wasn't working.

It should be work. There is one corner case of files that are within the layout that is still problematic. But as a general statement, it should work.

It should be working now or it should be working in 14 now.

It's been working for a while.

Didn't work for me.

As I said, there is a corner case of files that exist within the mathematical layout. So unfortunately, that covers some of the example data that we have. But in general, if you create a video or grab a video from some other source, you don't have any issues.

So if I say something like manipulate, plot sine X plus a X0 10A, I don't know, 05, then that hopefully will create for me. And manipulate. Okay, now I say. What do I say? How do I make a video from this?

Just video of manipulate. Should work.

Video of that.

Yep.

Well, we'll let it grind for a while. Is this going to work, do you think?

I think there is some issue with progress indicators in general in the cloud.

Can somebody trace that, please?

No, no. The details of that.

Could somebody trace that, please? Who's PMing here? I'm PMing.

I'll make note. Eli, I believe that there is a ticket or a bug report for it somewhere.

Right.

Trying the animation video example too.

I'm not trying anything.

Oh, that crashed.

It crashed. Oh, all right. So it's not working. I think this is what happened to me before.

Okay, Carlo, could that as well be the progress indicator?

It could be.

I mean, these guys just need to trace it. Okay. Okay. What am I supposed to look at here?

2 or 3D video. This is just general statements about there. This is probably a little bit overstated from where we are right now. Should have probably paired those boys back. Sorry about that.

Okay, so we look at the cow from every angle.

Yeah.

How do we determine? I mean, so if I say so, if I make a 3D object here, this is going to save this at a reasonable place. That's not what I want to save it. I want to save it in Storm Notes. Where should I save this? Image processing. I must have an image processing.

Gosh.

I guess I'm just going to make a folder here called Design Reviews, but I really would appreciate if people would actually put it in here. Get some PM to put it in here so that we keep it in the same place.

I prefer you pull them when we have the meeting because.

All right.

They're updated to the very last.

Yeah, Okay. What's it doing? My God, this is so slow. Should I be. I mean, is it just because we're doing a lot of stuff all in one go that this is so slow?

Most of it is rasterized.

So how do we make.

What resolution machine are you on?

Very high resolution.

Yeah. So you're getting high resolution rasters.

Well, is that what we want all the time?

You can control it with a raster size option.

I don't know to control it because. Sandra on our livestream is commenting for tour video as a recommendation. When you do a 10 times zoom in, what you see is that the size is now changing linearly over time. But that means the zoom rate is not constant, which looks unpleasant. What if you do a 10 times zoom in? Yeah, let's just see what this does. Oh, for goodness sake. Okay. It's amusing. It's not dramatic, but it's amusing. I think you should understand what Sandra is saying. 10 times zoom in for tour video.

I think I understand zooming in. The. How we zoom in is not linear zoom. Right. If you. If you specify width of the frame at time 0 and a width of the frame at time T, how we get from one to two is just by, I think linearly interpolating the width. But that means the zoom level is not linearly.

Sounds wrong, right? Take a look at it. Make sense.

I think it's specific for case to case. I'm sure it is wrong in some case, but it is not wrong in other cases. I'm not sure that is a general statement I want to make before I understand more.

Okay. Just look at it and see what. Okay.

Testing the cloud quickly to loop back for a second animation video of hue of T and T goes from 0 to 1. I'm upset that that crashed and I don't understand why we'll look that. We'll look back on that. But I think that is kind of a corner case. Yeah. So that's what I expect.

Why does this have. Why does it have the banding here? Why isn't it a fixed color at the top?

I think that's the control, the HTML element that we rely on. That is how we can play back video without involving the kernel.

If you play and move your mouse out, it might go away after a second or so.

Oh, I see. Well, it's cute. I mean, that's a microscopic video, but.

Okay, so look at these tours. I haven't. I mean, I've seen them in prototype mode, but see up there, there were sort of pre tours named. You can specify, of course, you know, every. You know, key points, you know, key camera views and whatnot.

Those other ones don't work.

Those are still be ironed out.

Oh, man, that's. That's the easy ones.

Those are the easy ones. Right. Only Rotate is implemented in your version of protocol.

What's it going to do by default?

I think Rotate is fine.

Okay.

Pan would be useful if you have a very, very wide object. And that's not most of the cases. Yeah, Zoom wouldn't be that exciting. Yeah, I think Rotate is the best.

You know, I'm about to make use of all this video capability for some video I'm making for some talk I'm supposed to give. So you're probably hearing from me.

Well, that will be good feedback, right?

It's going to be a little complicated because I'm. I'm trying to make. Well, there's several different things going on. I'm. I'm actually. I'm surprised you haven't heard yet from Richard at the Institute, because he's making some very complicated movie with hypergraphs and etc. Etc. Etc. Well, you'll hear from him. He starts. Should I tell him to contact you, Carlo?

Yeah, why not?

I mean, if anything, general mailing lists would be the best. Stephen.

And if there is need for like quick progression, like the TV processing, Rocket Chat is usually pretty good.

Okay, can somebody communicate that? Do I need to communicate that? Eli, could you please send. Write a little email and I'll forward it on?

Sure thing.

Yeah.

Okay. Subtitle Edition okay, What font does it use for the subtitles.

We can style things don't have. Right. Necessarily a default. It's usually either stirred within the file itself or as a side.

I see, I see.

And it's symbolic. It's not like it has. We can store style information, but you know, if the font doesn't exist or whatever, some default will be chosen by the player.

All right. I have no idea what these fades look like.

Scroll to the option section. I try to visualize it in that transition effect. So no transition just goes from one to the next. And then if you have fade, which is the next one, you have some kind of an overlay like that.

And then I have no idea whether it looks reasonable or not.

Click to download the resource. The File. You don't need to reevaluate. And now play again.

Can I resize this or not interactively?

No.

Why did that flip so quickly stop.

Just before the end? There was something weird that was happening there with interactivity.

Well, looks reasonable, but I don't. I don't understand fades very well. I mean, it seems like you have multiple many fade options here.

I don't think those will be necessary just because we're not dealing with something like sound that has a logarithmic perception. We were looking at pixel values, so I think the linear one will be the default choice. And I'm not sure there will be much need to tweak that. But we do have support for several curves.

Have what?

Several curves.

We have support for all of those kind of fadings of the intensity of the two curves.

Yes, images. The intensity of the images. Yeah. Sandra is asking why is it possible to rotate dodecahedron at 200 hertz in the front end, but rasterizing it isn't even close to that speed?

Oh, that's a very good question. Unfortunately, the front end is not very fast at rasterizing. It needs to. First of all, it needs to figure out how big the thing is, which, from what I understand, and keep in mind, I'm not a front end expert here. Figuring out the size of the cell that is being rasterized is expensive. And then all of those pixel values need to be stored and transferred over MathLink, which is also not the fastest thing. There are some.

We're doing the wrong thing.

I mean, look, I think that there's something fishy. There's something fishy about that whole thing because if we, if we, you know.

We should talk to John False about that, he will be sure.

Okay. But first of all, you know, if there's something about finding the size, we know the freaking size here. If we're rasterizing a whole image, we know the size.

Well, I'm sure that is true, but clearly there is some subtlety that John will know better how to explain than.

I do you understand what I'm saying?

Oh, yeah, yeah.

I also. That was my assumption too. That's what I asked John as well. Can we skip this calculating this size?

Well, also in modern times we have all of these foreign function interface mechanisms and so on. Maybe we don't need to transfer anything through, you know, WSTP or whatever.

Yeah, these are questions.

Where is it?

I haven't.

Where is it transferred? Transferring to an external file anyway.

No, no, it's transferring data right from the front end to the kernel through MathLink.

I know, but why doesn't the front end just open some FFI connection to something which is dumping data into a file?

I would agree. And those are all rasterized improvements that we would definitely welcome.

But that's not rasterized. That wouldn't be rasterized. There'll be something special for videos.

Sure, sure.

But all that. Open the thing because you're going to throw it into a file.

Yeah.

An encoder through a video encoder.

So I'm just thinking something there that.

Renders still need to be not. No, it would not go through rasterize ever. It would do the same thing.

It still would need to pass through the kernel. The front end doesn't have access to encoders. It cannot write directly to a video file.

Well then. Then you may need to write something that has some separate little program that just goes through the encoders to a video file.

Yeah, I mean, that sounds potentially good.

I thought we did write something, actually. Link for the.

To the front end, right?

Yeah, that's what I was thinking.

That is for reading, not for writing.

Well, but then I guess we can add writing.

Right. So then the idea would be don't go. Don't ever go. I mean, it's crazy to go through the kernel. That's why this is so unbelievably slow. Come on, guys, we can avoid this. Thank you.

Something that definitely we need to talk to John about. I. I feel I'm not qualified enough to see. Say, can.

Can we get this. Okay. This should probably be on the global project list. It's kind of a critically important thing given that we're trying to support video and you just spent, you know, a couple of years working on this. Let's make sure we don't. You know, the final link isn't just totally broken. Makes sense. You can. Why don't we bring that back to another meeting with me?

Sure. I mean, we do have also a meeting with John regularly. We can see what he thinks of this avenue.

I think so.

I. I think it's going to need me grinding on the engineering because it sounds like people are confused.

I think the idea is pretty clear.

Maybe, maybe not.

Well, it is.

Well, if you have, you haven't done it for the last two years, so I have to believe it wasn't that clear.

No, I mean, the idea is clear. Just what we. The conclusion was that is not a viable solution.

Okay, I don't believe you. That's why I think I need to be involved.

Let us have a chat with John and we can loop back with you if needed.

Thank you. Okay. If you haven't actually solved it, loop back with me.

Sure.

Okay.

I'm certain this can be done. Okay. Segment anything. Okay.

That's Sam we've mentioned in the previous meeting.

Yeah, yeah. Okay, so what should I look at here?

All those functions should have document.

You see the examples below.

Which examples? Oh yeah, example.

It is hard to locate in that big.

This is. Yeah, these are background removal examples. So currently what you see on 14 SAM based background removal, we're grabbing a hold of the boundary around the image. Ignore the artifact, the one pixel raw artifacts. But from that we are trying to segment outside from inside.

Yeah, well, it's doing really well.

Yep.

Wow. Yeah, it's great.

But this is one flavor of sam. The other two functions we're introducing or we're proposing to introduce. One is a global segmentation image components. The other one is really similar to remove background. But instead of returning an image with an alpha channel, returns the mask. Just the alpha channel.

Okay, so.

This is really old. Stephen. You didn't update.

I did update. So I don't know what's going on. So you tell me. I just ran an update. Okay.

I don't know how your scripts work, but my version has an illustration in the notes and starts with.

Then I. I just asked in the last meeting for somebody to please check this. Apparently they didn't do it. So what should I be seeing? I mean. Okay, unless you. An image. What. What was it? Image components. What was I looking at? Well, that's what I've got.

This is very old, maybe more than a week old.

Okay, well, I just updated. Can update again right now.

Roger, can you confirm you see an illustration maybe there?

Yeah, let me double check. Oh, I need to update too, because you just did it.

Right.

So.

Okay, it says it's up to date. Okay. Says it's up to date. So I don't know what's going on. Okay, so you want to email me the file or what? Something. Something's wrong with this. It's not correctly updating. So let's not waste time on this.

You want me to send you a file? I can.

I want you to send me the image components documentation file. The one that is not updated.

Yeah, I have the. I get the updated file from.

Okay, I don't know what the heck's going on, so can somebody please get with the other Roger?

I can work with Roger if you.

Want me to please do it.

Okay. Okay.

Yep.

I just sent you the file.

Okay. I don't know how this could possibly be wrong, and I worry that it's going to spray all kinds of stupid old files into the layout.

Yeah, that's happened before.

I know it's happened before, supposedly because, you know, one of the machines had a disk that filled up. But let's make sure it doesn't happen again. I'm not getting this mail yet.

Now Sam is slow. We're working to see if he can make the neural network faster. The first time we need to download the network.

This is. Is that Sam? This one here?

Yes.

Okay, so it's unbelievably slow.

Yep. We're hoping we can cut that by a factor of two or something, but it is still going to be so many times.

So how are we going to deal with this?

We ship an H100 to each customer.

No, no, seriously, Roger, how are we going to deal with this?

I think that they're doing, I mean, well, so I haven't seen the.

Obviously have a performance goal. Yeah, but the issue is how's anybody ever going to know that it's not just bad.

Is there a progress monitor for it?

Yes.

But what are we going to have as the default?

And. And are we using GPUs? Are we. Are we using GPUs?

So why not? Why are we not using GPUs? I guess that I usual answer that in 14. Oh, the Onyx implementation will support CoreML, which means we can use it on Macs then.

Then it will use it by default.

Yeah, that will be great.

Yep.

Yeah.

Okay, so.

So please get CUDA and Core ML. So CUDA for non max. Core ML for max. I think we should try that sort of a faster inference.

So is this the CPU timing or what?

CPU timing? Yes.

Okay, so it'll be much faster with GPUs.

Yes.

So, but. But I think it would be good to invent a scheme for progress monitors. So one of the lessons that we've learned in recent years is, you know, reevaluation works in the following sense. For example, you. You load a wfr, you've got an out of date version. Okay. It says. You've got an out of date version. It says press here to up update. Okay. But it gives a result anyway with the older version making sense. But it seems to work just fine for people. You press the update, then you run it again if you want that. See what I'm saying? So in this case, what I would suggest is it has something which says you know, performance goal. And then it might have a pull down in the, in the, in the progress monitor or something. You know, if it was quality, it could be like a, a button that's like stop and run at lower quality.

That would be interesting. Yep.

So I'm saying, so that way we could default it to high quality, mostly people. When people are using it interactively, they'll just see it. You know, it might grind for a while, but they'll at least see what's going on. And that way they'll know to change your code and so on. All right, what else do we have to see about this image components?

Nothing much here, design wise, is controversial. Unless you want to discuss.

Wait a minute, wait a minute.

This is a new function.

Wait, wait, wait, wait, wait, wait, wait. I don't understand here. This, this is a labeling function, but just like it reminds me of, of connected graph components versus. You know what I'm saying? One of them actually gives graphs and the other gives just the labels.

Right.

I would like to get something which is, you know, the image component images, so to speak. I don't want to get all my horses out as a list. I mean, I understand they'll be incomplete horses, but that's okay. See what I'm saying? Rather than a label matrix, which I then have to mess around with.

Would we wait for that four annotations to come. I'm guessing 14.1. And then maybe we can even embed that into the same image. But you'll still have to mess with that.

No, no good. It's no good. That's not what we.

List of images.

Yeah, that's what we want. Because in many cases, in many cases, if those horses were better separated. That's exactly what I want. Just like in facial features, we get a list of faces, which is very good, very useful. Mm, I don't see any reason.

For one thing, I'm thinking it's going to be a very much bloated result. You're. You know, if you have so many components. I think I have an example of some tomatoes, maybe in another. But then how, how many images are going to be coming back?

Well, but remember, each image is smaller. I mean, they have bounding boxes, but the. Oh, the puppy is a small puppy, you know. See what I'm saying?

And then we lose the relative location of the puppy. Is that useful?

Well, it's the same thing as facial features. We should do it the same way we do it.

Right? A different set of properties and maybe some, some criteria for selecting what Components to return.

Yeah, I think that would be a good idea. But I think that's a really useful function. I think it's much more useful than this function with labeling. This is only for image processing experts.

You have proposals for the name. I wasn't able.

I don't really like image components being this, this thing that does something that's basically very, very expert based. What are the other names in this area? I mean there's, there's things like morphological components and so on, Right?

Yes.

What do we have there? What do we have?

All the segmentation functions that return a label matrix end with components. Now we have other functions that end with components as well. Clustering components, image foresting components, grow cut components.

Those are all image.

Image segmentation. One of those segmentation algorithms that honestly never works. Well.

We should probably obsolete this function.

I, I was thinking about the same thing.

I mean we should probably, you know, for some of these functions we should just at least nominally obsolete them and just say it's been superseded by image components or whatever.

Okay.

A lot of these things that were, you know, algorithm specific, so to speak, which can now be done better with machine learning. Why not plan to obsolete them?

Sure. Some of them still probably have value for two. Two reasons why 1 speed and, and 2 being so very well defined and deterministic. What they do, what criteria.

Maybe they should just.

Sure, I can see that.

Because I.

Think they're forex and then they get documented as method in the method framework for documentation.

Indeed. Which would be good.

Basically this page more or less, you know, just reformatted as a method. Yep.

I think it's a much better idea because I think muddying things up where somebody says I'm going to learn image processing. Oh my gosh, I have to learn foresting components. I don't know what the heck that is. They don't have to learn it or.

Sean Bezzy Binarize.

Yeah.

Yeah.

Roger's favorite.

Yeah 99.

Nobody knows what it is. When I ask them if they know what it is.

Yeah.

I've had 100 success rate or failure rate depending on how you view it.

I think Christopher likes that function. You could try asking him what it is. I think he would know.

Okay, I'm, I'm telling you that the functionality that segmentation works this one, I could never get a good segmentation out of it. But chanwise it works.

Right. Well, maybe it should be the default option for default method for Binarize. And, and I do think that this idea of, of when anything is at all slow. Having a progress monitor with this, you know, buttoning and so on to indicate that there might be a faster method. Okay. So we need to figure out and I don't know what.

So maybe I can. This is, this is more of a brainstorm thing, but maybe you guys have thought about this too. I wonder. You know, so several editing tools, kind of like, you know, Photoshop or something like that, they allow you to do like a little piece of an image very quickly so you can see what it is. But I don't know exactly how to reproduce a workflow like that. And I think that something similar might be useful for video. You don't do the full video. You do a little piece of it. You can say whether that's what you want. Type thing. Yes, but I don't know in a repl exactly how one does that.

Well, I mean the one thing that will be useful to see is things, You know, is seeing the thing like we do with image. Do we do it with image synthesize? We do it certainly with the version of that that I was using for that blog post about cats. You can see the image build up and I could imagine a situation where you know, where you'd see the image, so to speak. I don't know where, for example, the progress monitor would show the image building up. Does that make any sense?

Yeah, but I don't, I'm not sure.

That may not be doable here. Sandra is commenting that that a reason for having these method based functions is that you want something deterministic that doesn't change every year because of a new neural net. Point taken. Yeah, but, but I mean that's I think where you set a method option because you know, you've got a choice. Huh. Sandra's commenting that find image shapes is great, but it will be good to open up the hidden options which he appears to have found. Cluster threshold. Cover threshold.

Sander is digging in.

That's good. But I mean how, why those are. Those are controls for how the image shape is gets found.

I actually don't remember all of these.

Okay.

I, I'm assuming, yeah, I'm assuming they're all selection criteria.

This is pathetically under documented like.

But we have, we have plans to updating this anyways. All right.

Okay, let's come back to the. The. The image.

The segmentation label versus Yeah.

I mean it's so horrible to say component images as well as image components. That's just too horrible. But I think ending with images or features for that matter.

Image component features.

I Don't know. It makes me think I'm going to get a list of features. I mean, a list of numerical feature vectors.

Is there a way to use the label matrix to pick out that sub image?

Yeah, we have component measurements. You can pass to it both the image and label matrix and say, I want the sub image for Label 5. Or actually maybe the majority of the work would be put there to make sure.

I thought about that. I thought about it. But the problem is by the time you have this thing, you've got to feed the image in twice. See what I'm saying? It makes this from merged components. But this image here does not contain all the information. So you need the original image as.

Well as the mask component measurement can take that. Right.

It has to take both things.

It has to take both things.

Right, Right. So we already lost the game. I'm saying you can't do pure sequential function, type flow. So we really want something where. Where out of this comes.

But so label. Label is also category.

It's.

It's categories basically. Right. It's finite number of categories. And are they. They're always integers for us, obviously, since we stick them in an. In an image or an array.

Yeah, no, but I mean, look, facial features is a good case. Is a good sample of how to do this. Right. So I mean, I like facial features. I've used it, you know, See, isn't that nice? And here we could have things like, you know, we can have things like the location and so on.

Well, if you get out the sub image, we could. That's just a sub rectangle in the big image. Right. So we could say how. Where it's relatively positioned.

Sure, of course we could. Which is what we do for facial features. We do. We do. We have. I mean, there's the nose points, you know, hit it on the nose. Right. So somewhere here is bounding box.

Well, that's the bounding box, but you need to.

Of the face.

Oh. In the big image. Okay, then that's exactly what we need.

Yeah, yeah, it's. It's the exact same model.

Yep.

Now, I don't understand image contents. I thought I was gonna say like this.

Correct.

So I don't understand why we don't just put this into image contents. What does image contents do? We got two dogs here.

Yep. Gives a label use a data set of.

Well, that's my image. That's exactly what we need, except for the fact that it needs to use sam.

Oh, with sam, we don't know what each component is. We don't know. It's a double order.

Ask something else.

We could try, but it could still be unknown.

It might be unknown. So what?

Right.

This looks great to me. This has exactly what we need. It's got the image and the bounding box and even a probability that might be. There might be two probabilities there that might. The probability for the concept. I'm not sure that the, you know, the default here. Yeah. I mean this looks fine. I mean much of the time we will be able to resolve what it is. Some of the time we won't. This is surely the right.

Should we. Should we return components that we can tell what it is? No. You said it can be unknown.

I don't know. You have to try some things there. I mean it could be.

Let's try.

Yeah. It could be that. That most of the time you can resolve it. It could be that it's useful to. It could be that we should suppress by default things which we can't identify because they. Because they're kind of silly because they're just like the, you know, one, the left paw of the dog type thing. And that. That's not enough dog to consider it a content element.

I have one question that maybe I missed. So image components does the segmentation doesn't really necessarily just do rectangles. Right. It's correct segmentation. So how would that fit in image contents? Maybe I missed.

I don't know. I don't know. Maybe it needs a mask as well. I don't know whether image contents has a mask but it certainly could have with this technology.

Image contents doesn't. I was imagining if we inject SAM into image contents it would mask out that sub image exactly. The shape of the segment itself.

Right. I mean in other words, it could.

Be a list of masks or a list of masked out images.

It would absolutely mask out images. That's what we want.

Yeah. An additional element in this association that's a masked image or something.

Yeah. The image and the masked image. See what I'm saying? The image fragment and the masked image. Yeah.

And it could still have the bounding box because. Why not? And additional element.

I think this should be put in image contents.

Our SAM based development was longer than what you see now for 14 and I think we should continue on that. I don't think how much more we can squeeze into 14 at this point, but obviously.

Yeah. I mean look, if this is remove background. This is the result from remove background, right?

Yep.

Yeah. So that's nice. That's a good result. But I mean, you know, I don't know whether we can get this into 14, but this is where this should go. This image contents thing is the place to put it, I think.

Okay.

I mean think about the design, but it seems right to me.

And you don't want anything like image components that just returns the label matrix.

No, you can have that if you want to, but it's for experts.

Okay. Nobody else wants it naming wise. Is that a fine name to you?

I don't like it very much. I think it's way too general.

Alternatively, I think we're proposing, if you go back to that notebook I sent you, semantic components.

Oh no, that's.

That's terrible. I mean, semantics don't do that. It could be anything, right? Semantic from.

Yeah, don't do that. That's a stupid idea. Okay, so. Well, what kind of image components is it.

Anything.

Yeah, I mean, look, why does the dog. You know, why are the dogs a component and the stick is not.

Well, this one is in remove background. But if you look at. Yeah, if you scroll down examples of image components, things are separated better. Where in the notebook in the back center. Like that horse, the puppy.

Well, Yeah, I mean the nose got separated. I would not call that semantic components when the nose is separated.

Sure, We did think about. And then still maybe time to put it in anyways. Use the semantic segmentation networks that we have in the repo and put it also in image components.

I don't understand. What is this thing? I thought this was a semantic segmentation network. What is it if it isn't that?

It is a very general purpose segmentation network. The name is segment anything model.

Yeah. Okay, okay, okay, okay. Probably works like our brains work.

And I kept the syntax so far very simple so that it has. It allows us to add more things, more method methods for different networks or models, algorithms, whatever. As you try.

Try feeding some other things in. See whether.

Okay. Foresting components. Okay. Yeah.

I mean method, arrow, whatever. I just don't like these label matrices very much. I mean they're just, they're a horrible workflow. Except for people. I mean, this is, this is. It's the same thing as you're doing LU decomposition or something and you stuff the two parts of the matrix into the same thing. Because all you ever had in FORTRAN was a. Was an array, you know, was a matrix like array. This is the same type of thing. This is the same. You know, somebody has an, you know, their data structure is an image, so they stick numbers into an array. I just don't think it's actually the right thing.

But, but I was also thinking that we should, we should think a little bit about the workflow for these label matrices. Like if I could mask on five, you know, that's the only thing that shows in my image, then it was. Be much nicer.

Yeah. Be more useful. Right.

And I, I just think that the little tweaks like that would go a long way.

Well, so take a think about it. The other thing I'm wondering about is images that aren't rectangular, that maybe we should have a, A image should be capable of having an image region that isn't rectangular so that it becomes, you know, and I don't know whether that means in image collage and things like that. It could imagine doing something more like word cloud, et cetera, et cetera, et cetera. But I think the rectangularity, I mean.

We, I think it's interesting. Here's another related concept which is that often in many workflows you need, which is kind of like the same thing like, like in medical you, you need to sort of mark something up and you need to have a little bit of information about the thing that you mark up. You know, this is a tumor, you know, grade A or whatever it is. And I think the same thing comes up in almost all applied workflows, like remote sensing type things.

Right. You indicate something and I mean we have a region of interest mechanism for images.

Yes or no, it's not baked into them.

Masking option is here and there.

Okay, but it's not baked into the image. That's what Roger and I am both saying. Right. I mean, maybe that's a better thing than even having weird shaped images is just assume the image is going to be rectangular but have the possibility of picking out a particular sub. Element, sub, sub region.

It's been suggested a couple of times to use alpha channel for that. I don't like that idea. I think it has to be. I agree, I agree.

If we're going to use a channel, let's use the XI channel. Okay? Right. Let's not use a channel that has another meaning.

Exactly.

If we need it as an implementation hack, we can use a channel that, that, you know, some other channel that we, you know, for example, right now we have channels 1, 2, 3, 4. Why don't we have named channels? Why can't we have channels in an association?

Yeah, we've talked about that. It's come up in like spectral imaging context and things like that as well.

That'd be a good thing to do because that, that way we could have a region of interest channel.

I think it's just a little bit of an expensive implementation wise project and maybe it never got prioritized in, in my head.

Well, I think you should think about it because I think that that way we could solve this problem because that way we could have, you know, we could have the DOG channel and things.

One thing is about how, how to talk about this with the external world. Right. For example, many image formats only accept three channels of four.

I mean, I mean, I don't think that that's a problem. Like if. Let's say that. I think that there are many ways to do that. So let's say that I have markups on a DICOM image. Should you render those markups into the image themselves when you export it?

It's a choice.

Yeah. Or should you ignore it?

Sometimes you don't.

Right?

Exactly. It's like a flattening operation. We've got it in some data structure and to export it splump, you flatten it. Okay, let's look at shape detection. Find image shapes. So what was updated in this.

The thing that it's not yet in the builds, but we think we can make it. The thing that might be useful and we're just considering adding that to this release is to use label matrix as the.

How would you specify it?

Label matrix that you don't like?

Well, I mean option called label matrix.

No, as the input. So currently we have an image.

Oh, I see. So you take a matrix as input as well as an image.

Right.

And if that matrix is just a bunch of integers, then we recognize the shapes, the five shapes. The fives make up a circle or.

Because one problem in finding shapes when we pass it edges is, is to group the edges into like connected or related components, pixels. And once we go in with a label matrix, that problem is already solved.

Well, I don't think this, that look, I think it's. It's like a bunch about. I mean, I don't. What happens if I just say blur of random real 140 40.

You'll get a message on evaluator. What did you expect?

I wanted to take those like I wanted to do the same algorithm that it would do if I took this thing here.

No, you could do mean filter or, or you know, other filters. But.

Right, right, right. So I do this and then I want to blur that and then get.

The matrix back out.

Yeah, why not? I mean, since it works with mean filter. How would I know that it doesn't.

Work with this, I, I don't know. Blur always sounded very like visual.

It is, but I mean, I don't see why it shouldn't work with these. Right. And I mean just like, you know, and, and that way that would be an excuse to have this. The image shapes work with numbers as well. So that for example, even if I have like, like for example, if I say well, find image shapes a little weird, but I mean in, in find image shapes on a sparse array, for example. Oh, it's just got one, you know, I mean that would be kind of interesting to be able to find the two lines and you know, if you have some sparse array that came from some, you know, PDE ish discretization or something and has two lines in it, why not? Yeah. Sandra's commenting by the way. We could just use Gaussian filter on the random reels and that's true, but I just don't see why the other one shouldn't work. I mean it's not going to do anything else.

That one requires a radius Blur has a default I. When we talk about these icing utilities, I always wonder how many times people intentionally, intentionally want to blur data and how many times it might be some kind of a coding error and they might appreciate some kind of an error.

I think it's, I think I this particular case, I just don't think they're going to do it on image. If they did it on something, you know, it'll be because they had the numbers and they forgot to make it into an image. But you know, that would be my way of justifying making it work on label matrices.

Okay, but then with real matrices we probably have to do some kind of a gradient filtering to kind of segment and in an integer matrix the, the gradient is just different numbers, different integers.

Yeah.

And I don't know, I'll have to think about it.

Yeah, think about it. Maybe whatever we're doing for the image itself, we should do for find image shapes on a matrix.

M on images. The one behavior you didn't like, but we thought that's the correct one, is that we ask the user to compute the edges. And the reason why we say that is fine. Image shape. Yeah. We're planning to allow for different things to be highlighted and then we can find disks as well as circles. And then us automatically computing gradient filter might not be right at that scenario as well.

Where are you computing? Where are you asking the user to compute anything?

Expects under the table we say expects an image to be in which the shapes are Highlighted and yes, that gradient filter before you find image shapes.

That's horrible. That's horrible. But I mean, so we should be able to do. Oh gosh, you guys. Polish guys. I mean, maybe it wasn't technically possible before, but it is now. So let's try and make sure that we can use the same thing to do the segmentation so we don't have to insist on that. So that becomes a comment and a suggestion, not a necessity.

But then I guess. Okay, I still want to think about scenarios where as we were talking about deterministic behavior. When do people want to compute their outlines with whatever method and then say.

Now use and then. Right, but I mean so.

And then say something like pre processing none or something.

I don't know. Yeah, something like that. But I mean it may be that the pre processing is so I think it will be so good at noticing there's a purple ring there and ignoring everything else.

Huh. I think if we introduce some option like that pre processing, we can use it in many places. I don't know. Five versions ago we talked about it. Never introduced.

Yeah, you remind me of Oliver. He calls anything that isn't solving either pre processing or post processing.

All right, I've heard that.

So. But it's like for him it all means finite element related post processing.

But I kind of don't want to call this edge highlighting option or something like that. Although we can squeeze it under method option. Method goes to. Actually I think we do have pre processing sub option in some places. Okay, well but like auto edge detection goes to true or false on their method.

Yes, but look, I think. Okay, so several comments. I don't even understand why the compass is there by the time you've got the purple highlighting. Why don't get rid of it and just say find the. You know, if you don't want all the noise of the compass, just do your edge detection first and do find image shapes on the edge. Detected image.

Uh huh. As the, as an example.

No, not as in this example I'm saying just get purple here.

Huh.

Right. So I mean in this case just add graphics.

Sure.

Yeah. No highlight. Just, just. I mean what I'm saying is just. Do you know we. Oh Christ. You know, Drives me crazy. Drives me crazy. I'm so confused by what's going on here.

Here, Just, just.

Oh, there is a gradient filter here. So, so this is.

Yes.

I hate this. Oh come on. Why don't you just put the image in there? Because you have a bad workflow.

We was using it twice Yep.

Okay, so, Okay, the ghost of the compass is there. Now if I say find image shapes of that, How many circles is that?

That's more than what's there. Do we have more parameters over.

Although we treat it.

Oh, we're just picking the two bigger ones. Maximal bioregion measure too.

Oh, come on. Come on guys, do a better job with these examples. I mean you got to show what it really does.

This function, I agree, should improve. And as I said, we put our focus on sam. Now I think the better edge detection, even from SAM is going to help better finding my shapes. And then we talked about some other ideas here.

Well, I mean, I also am not sure that the right answer for this isn't some kind of data set play again.

Oh, I agree with that. Sure.

Where you show what, what the actual thing was and the picture and then you know, and then you give the actual. You give the form. I'm not, I'm not certain but this at least properties.

Maybe not a data set by default, but at least a set of properties would make sense.

Okay, what is inpaint? What's been done with it? We're going to need to come back to a bunch of this stuff, obviously.

Yes, I'm. I'm kind of have a hard stop myself here in just a minute.

Okay, so what are we doing with InPaint?

I sent you some additional notebooks that you don't have in the docs. So the thing we're adding here, just like Sam can pick out the component if we specify a position or a rectangle and then segment that piece. Now we are saying previously inpaint expected an image that highlighted what's being inpainted. But just look at those few tentative examples. You can give it a point and it gets rid of the segment indicated by that point by that rectangle, set of rectangles and so forth. So I. That's just auto segmenting again. The problem when we do that, it requires downloading the network and a few seconds of running the network to do the segmentation.

Well, downloading the network is fine. I mean in the cloud it'll be pre downloaded and we need to make sure that's the case. And you know, people, I do think that as we have more stuff that requires downloading that there's. We should discuss the possibility of a giant sort of download panel. Okay. Where you know, something in the preferences, for example, which says what to pre download. Because I can imagine a situation where somebody, I don't know, they're running some class or something and they want to send everybody a pre download sort of manifest so people can run it and it grinds away in the background for an hour and then they show up and then they can do everything.

Assets, resources needed for different function again I think was discussed a while ago.

Yeah, well, right, but I mean I think, I think. But putting that in the preferences as something that should be background downloaded to that computer makes some sense to me. I hadn't really thought of putting it in preferences before now. Look, I mean I think, you know, monk with a, with a umbrella or whatever the heck this is, maybe they're not monks. I don't know what they are.

I think they are. This is very low resolution, but they look like that.

Okay, but in any case, I mean presumably this could be a name here.

Of a thing, right? The prompted version of all of these is probably as well a 14:1 project.

Right, I understand, I understand but I mean the other thing we should be able to do is to once we have image contents and one of these monks is extracted as one of the sub images, one should be able to say inpaint this thing comma the monk because you should easily be able to find the monk in the image. And we have a function for doing that, don't we?

Right.

I mean if we don't.

I'm going to have to go. I'm going to ask to set up one more hour, right?

Yeah.

Because we have also audio, we have input export, not super much in either one. But wrap what we started.

Yes, okay, fine. Okay. All right, see you Roger. We'll wrap in just a second here. What, what has been modernized and text recognized?

Not much in the ref page, but if you open the benchmarks. More specifically, so far we have improved natural.

Was that on purpose that you have a typo in this.

Typo? It's. It always amuses me how quickly you see typos, Stephen. So both, both the improvements are in natural scene text detection or text recognition. But we are moving forward with easy ocr, a set of networks that are known to be good performing as well and then updating possibly printed text and other languages as well. Some of those that second number for fine image text is not that it's still a big improvement, but not as big as the first one you were seeing. Tesseract was performing like 21% accurate on natural scenes before and now we're Getting I think 70, 80 if you scroll up.

That's good.

Yep.

I assume it performs. I assume the new thing performs with the same time performance and as high Accuracy on a printed page, for example, on a scan. Right.

Well, we do want to run a benchmark before we switch the backend and that easy OCR that I said is also a very promising backend for printed text.

Okay. But we can, we can surely run a incredibly cheap classifier for whether we're looking at a natural scene or printed text if we need to switch between them.

Yes, That's more of the future stuff text related.

Can we do handwriting ocr? Is that. Is that, is that advanced?

You pause that project? I think a year or two years ago. We haven't got back. Gotten back to that, but I think it's something that we could potentially get back.

Well, I just don't know whether it will work yet. I mean, I've got tons of examples if you want them from.

Which one is higher priority. You think math or handwritten?

Math, by a long way.

I agree.

All right, well, we should. What is that? What is that?

Wait, that is not for that speech synthesize. That does not Right.

Recognize using the OS recognizer. The OS APIs is not a 14 feature anymore.

Likely not for 14.

Is that local or is it going to the cloud?

So I think on Mac it most likely goes to the cloud. But Apple is a bit fishy about how to do how they document this. They. They aren't really explicit if it goes to their cloud or not. So this is why we're kind of postponing to make sure we have all of these details more.

Okay.

Under control.

All right, we should wrap for now. Looks good. Look forward to the next iteration here.

Okay, thank you.

Okay, see you all soon.

Bye.

Thanks.