Live CEOing Ep 745: Language Design Review of Sound & Vision Features for 14.0 continued

Okay. Hello, everyone. Welcome to our second design review of today, part of the version 14 season of design reviews. And we're looking at sound and vision. And did we. Did we start on this or. We've got nowhere with this so far.

I think we've threw most things, actually. So I think in particular for the image stuff, I'm almost. Maybe we want to revisit, actually, now that I think about it. Shadi, what do you think?

Yep, yep. I think we can revisit it today as well. But it might be best for some other development purposes to go through audio and import export first and then come back to image because it might end up being a longer discussion.

Because we were talking last time about image components and the question of what it actually returns and so on.

Yes.

Label matrices versus other things. All right. Import, export.

Actually anything audio. That remains to be discussed. Carlo? I don't think we went through them, but there shouldn't be any controversial design either. That's just improved results.

So that first one. Yeah, this is a table of audio.

Hey, do you know why YouTube doesn't. When it speeds up audios, when it does time stretch, why does it not adjust the pitch? I mean, why does it. Why does it make the pitch go high pitched? Why do they not.

I think the pitch maintains the same. I think YouTube does like time stretching without pitch shifting.

Okay.

Sometimes I do hear pitch shifting, but it's one of. I agree with Carlo that it seems like it's trying. It may not work in all cases, but it's one of the better ones we've heard anyways.

Okay, what's that service and how do I get to it?

That unfortunately relies on API keys. Like we have OpenAI and all of that kind of services. It's similar.

So where are we putting it in the. Are we putting it in the preferences somewhere?

Once we have all the other services there, it should go there. But I don't think we have. Do we have preferences for Enable?

I am not aware of that.

We do. We do for LLMs, we have that. We have exactly that.

13, 3 or 14.

14.

Come. Okay, well, I'll make sure we get our services added. Do we want all services there or anything AI related for now.

Or what other services do we. Are we talking about here?

I mean, we have a bunch of service connections, right? Maybe not very much used at this point, apart from the AI ones. Yes, I think this will need to be.

Okay, so there we go. So there are things here. Now this isn't showing services yet, but that's Supposed to be coming.

Yeah, I'm not sure this applies exactly to the same thing or if a separate one needs to be designed for these.

Well, hop on it. Because I think that's. That's becoming a. You know, I mean, I agree.

If there is a way to have service connections handled in a more UI way. That sounds good.

Are there new characters here?

No.

So what does it say? Voice style data updated.

Not the dollar variable.

This is just. But also this is just a backend update. Just the OSSDK changed and there is no design update, essentially.

So what happens if I say voice style data? Why on earth is that not one of our entities? Carlo?

I think this is a newer change. I reported it a week ago or so.

Okay, yeah, we can take care of that.

But you see the use case down there, which is voice style data for a particular service like 11 labs.

Well.

There'S a bunch of other.

Each service, right. Each service comes with its own voices, so we can query for them and then use them.

What the heck happens if I said, you know, voice style arrow moniker and then I'm using 11 labs and they only have Gigi?

Well, in that case it would fail with hopefully an informative message. But the point is all of these are not stored on a computer. An Internet request needs to be made to figure those out.

Okay, but. But what? I mean, like with fonts, you know, we do try and fail over to, you know, a font with the same characteristics of serifs or whatever else. We don't have that font. We produce a message and then we fail over.

I'm. I think similarity of voices is not something we can really gauge at the moment. I think we fail. We don't substitute the voice.

Okay, but I think that's. I think with fonts we should learn from that because, you know, the gender is going to be there and the language is going to be there.

Yeah, but if those information are the only ones that we care about, then yeah, it would be easy to fall back on another one that exists.

Right, but that's my point. I mean, my point is if somebody set moniker here, okay, at the least we've got gender, female language, Spanish.

Okay. If there is enough similarity, then, yeah, that's fine.

There is a difference between fonts. Font swapping is not expensive. Using external services could be costing people credits and money. I'm not sure if we want to.

I think. I think what we want to do is the same thing we do with fonts. Produce a message and fail over to something reasonable.

Okay, so you Want to always produce a valid audio, essentially, is what you're saying.

I think so.

With the warning.

I think it's the right thing to do. I think it'd be super annoying if it's like, oh, no, because here's. Here's the thing I'm using. I might use, you know, 11 labs, or whoever they are, you know, because they are silky smooth and lovely and so on. Right.

Yep.

And I just know I want to say something in Spanish. Okay. And I just like their way of saying it. And I didn't do it because I just wanted. Monica, let's say.

See what I'm saying, Carlo, do we support language? Goes to Spanish?

We do.

And so, you know, if. If you really don't care about Monica, you could just say language, Spanish, Spanish.

Fair enough. Yeah.

We don't support agenda saying it might be an expensive generating stuff with the voice that the user did not intend to use. Might be an expensive operator.

All right. Okay, fine.

Gender is also, as we know, is a little trickier. And some services don't even return gender property for their voices.

Oh, gosh. Well, how modern of them.

Yep.

And how many genders do they use?

The ones that do return something are still male and female.

Okay.

I mean.

Or at least I haven't seen anything else.

No, no, it's good.

I just, like, they should just say, forget gender. Just quote, you know, formant frequency. Average formant frequency, high pitch, low pitch. Yeah, that would be okay.

Yeah.

Yeah. I mean, because that's really all it is. I mean, that's the only difference here. And I'm sure that there are. You know, I'm also surprised that, like, I think. I mean, you know, there's a. There's a definite mapping for child voices and things like that. I think. I mean, I think it's similar to, you know, just the formants move around.

Right. There's probably tonal differences as well. Not just. Yeah, frequencies.

Well, okay, so I don't. I don't have a very strong opinion about.

I wouldn't fall back just yet, if you're okay with that.

Yeah, it's fine. Fine. By the time they specified, Monica, let them just fail if they. If Monica isn't home.

Okay.

That was good.

What's this? Is this. Is this using some local service here?

No. Yeah.

So the OS services.

I'm not sure why that is that.

We'Ve mentioned that's not making it to 14.

Oh, it's not?

No.

And we forgot to update.

Okay.

Eventually, once we get to it, it is really going to use the local. Well, the operating system service.

It doesn't necessarily mean Local might make a request to Apple Server, but we.

Think it's faster and better than what we have.

We've been working with incorporating other neural networks, but at the moment we are at a benchmarking stage.

Okay. What on earth do you think that is?

Oh, it's one of the specialty voices from Mac os. It will sound bad and weird. I think that was there when I was playing with my.

Oh, well, we've got one here. We've got one here.

Yeah.

Okay. Okay.

That was there 20 years ago, that voice. I do remember it from when I was a child.

Okay, okay. By the way, I assume that Speak. Does Speak support these voices too?

No, we.

Why not?

It's a good question. It uses a completely different.

I know, I know, but I mean. But. But if it's. If these are Apple voices and things, it could. It could use them.

I'm not even sure. Does the speak have any other arguments?

I don't think so right now.

No, I don't think so.

Well, so I think it could have a voice argument just like this does.

Yeah, we'll have to talk with whoever has been in charge of that. I'm assuming that lives in the front end.

I think so. Did you hear that? Yeah. Well, I'm not sure I can set this to. It's amusing. Okay. But I've now learned it's called a trinoid. Not a trinoid, because it said it itself. Okay, what's. What's new here?

Video formats, the ones that do support subtitles. Now, we can. We used to have a RAW subtitles import element, but now we're adding a slightly cooked version of that as well. You can just give it raw strings or styled strings and export or import into. Yeah, exactly. You can give it strings and export a video with subtitles to a file, or you can import subtitles as in a format that is more readable.

What makes it more readable? I mean, it's just text, isn't it?

The RAW format was really unreadable.

Yeah, it was. It was completely a mess. It was. Yeah. If you opened. It was a complete disaster.

This just MP4 should have some examples. If you want to open the ref page.

New or old ref page.

The authoring source notebook. It's tentative.

Okay, where do I go for that format?

Formats.

O Formats. Yeah, I meant O. But where's the formats in the middle?

Two up, three up. Huh?

It just beeps. I'm trying to type into this field. It's just beeping. It's just beeping.

Well, if you. Yeah, I get that. Sometimes if you click on MP4 and then, then click on that dot that you just clicked, it should work.

Okay. Okay, where do we go here?

Examples. Well, keep scrolling to import elements there. Tentative examples. Now if you open those two and compare them with the following section that says RAW subtitle.

I'm just totally confused here. I'm sorry, I'm totally confused.

So here you're trying to preserve the formatting using text and style, and style is style comma white, and that evidently doesn't display ideally.

Did we talk to front end Carlo about that?

No, I was planning to talk to John about this as well in tomorrow's meeting.

Okay, I'm totally confused. Subtitles text Is this RAW subtitles? It says RAW here. Is that what you mean? Or is that not what you mean?

That was. That was a copy paste typo. The RAW subtitles is the next section that is not tentative. We have had that before. So if you want to see how stuff are stored in the file, it's kind of a script info, then the format of how stuff are stored in the. In the file.

I just don't understand at all.

Get rid of the data set.

Is this what we're talking about here?

Yes, and as you can see, that is completely. Well, it's very annoying to parse.

Yeah, I get it, I get it. Okay, all right, so then. But then there's some tagging information here which you are then losing. I see in subtitle text.

We're not necessarily losing it. The subtitle element, all of that information is preserved by style and text.

I understand the subtitle element, but in the subtitle text element.

Yeah, that is blowing all this stuff away.

Just saying. Haircut, please.

Correct, Correct.

That sounds sensible to me.

Yep.

Okay, I. I mean this is obviously important and the thing we've discussed many times which I really want to see an example of, is natural spreadsheet understanding. And you know, we've discussed using LLM type technology for this, which Sean seemed to think was not going to be necessary, but I'd like to see proof of that.

Yeah, and, and I think the last discussion between you and Sean, you said you'll suggest the developer for that.

I thought you guys were dealing with that. I thought I did suggest something. Is it, Is it. Are you waiting for me? If you're waiting for me, you better, you better tell somebody that. Because as far as I'm concerned, if nobody tells me anything Based on one of these meetings, if nobody gives me any reminder and it's just left to me to remember what to do, it's not going to happen. Okay, what is the status of that?

Probably then, probably then it was never relate to you. I can send you.

So you're giving me homework.

Then I'll send you an email though. Yeah. Yes.

I don't really like that. I thought we discussed it in much more detail in that meeting. Course of action, right?

No, no, no.

The project itself. I gave you a specific, I gave you specific information there but you didn't follow up or what?

I, I think really what was discussed was that the project. There are other projects Sean was working on and we, we just. I think. Okay, so probably this is what happened. What I recall is that this was probably right before or around summer school and we were thinking we could get someone from summer school and then I got into 140 and didn't remember to follow up.

Well, please follow up. Sure. I mean, I don't think, I mean I. Yeah, let's please follow up. I don't want this dropped. I was told, I said there's a course of action that involves using machine learning. Okay. I was told, oh, you shouldn't do that. Sean Etc will figure it out without that. I said show me that you can. So I don't want to hear it didn't get, it got dropped. Because if it got dropped, let's go to the default that I thought should be pursued. Okay, okay. Okay. I have no idea whether this makes sense. I have no idea. How elaborate is the format?

It's very much used by the R community.

I'm aware of that, I know that. How elaborate is the format? Is it a very complicated format or is it a fairly simple format? Is it like CSV or is it something with. With endless tentacles?

It's unfortunately very under documented, both the format and the library. I just committed something I thought earlier today. Well, I mean, I'm surprised you don't have it.

I may not have updated. By night. It should be awesome here. So I mean, what, what are you telling me? Is it. So I think initially we're using, you know, we have some data files in our format. Can we say, okay, great, we can import them or does it have footnotes 10ft long?

No, I think we can just import them. We're talking about numerical data, string data, boolean. And all of those will just come in as tables of data.

Okay. In the appropriate formats, in the appropriate types. Sounds good. What is this Docx?

The Microsoft Word.

I don't know what the format is, but what did we change about it?

It's not supported in our system yet at all.

Oh, Doc. So we're talking about Doc is, but not Docx. Seems useful. I actually supported for like 20 years.

I don't see it really.

Well, anyway, I thought it was. Shows how often I've used it. Maybe I've exported from Doc to something else. Anyway. Okay, well, that sounds good. So similar to the support for PDF you're saying here.

Right. Except that I don't think we have a renderer to bring stuff in as page images. But we'll first attempt. We'll have text, maybe formatted text and embedded images. Then we'll see where we can take it from there.

Sounds good. Okay, what should I be looking at here? I mean, I don't know what I'm looking at. There are no examples.

Examples all that. But really, as you're seeing, types of data support that, Boolean integers, real string timestamps and so forth. So we'll just get tables of those.

What is a timestamp? It's just a data object.

I. I haven't figured that out yet. It was kind of an unfortunate situation. The developer left and now we gave it to another developer. Dates and time. Dates and timestamps are done separately in our data. I don't yet know.

Yeah, most systems separate. Most systems have three kinds of things. There's date time and date time. Our date is date time.

Yeah.

Unless you use granularity, in which case.

Right, Fair enough. Okay, let's see. Is there other stuff we should go over right now? Today?

Back up to image components. Image component detect.

All right. What did you change from last time?

Image components, you can take a look at that probably has the biggest change.

You know what? My mouse froze. I thought this might happen. I'm telling you, my mouse is ill.

It's to rejuvenate it.

I think it was some update to the mouse driver. Okay. Okay. So we talked about this and we said we didn't really like that very much. And where was the thing that we talked about image contents as. As the way to do this? What. What happened?

I have. I have updates in image contents as well. Everything that's mark tentative up on the page is new since last week. Okay, so up in the syntax box as well, you suggested adding properties. And then I was looking at component measurements there we have criteria and format and things like that. There is one. One problem with criterion and component selection with more elaborate segmentation methods. The segmentation itself is costly. So we can still support criterion and then have people do the segmentation every time and then pick their segments. But the more reasonable workflow is for people to compute the segmentation, save it as the label matrix, and then work with select components, component measurements, and all of that for any other analysis.

Why don't you just do it for them?

Just cache all the results and then.

Well, no, no, I mean, just do. If they say, give me it with a criterion, we compute all the components and then do exactly what you said, use component measurements and so on and compute the, you know, apply the criteria.

That's what this design, this tentative design is suggesting. All I'm saying is that a lot of the times it makes more sense for people to save their results and then do additional computations on. On that label matrix because it's expensive to get the label matrix.

I understand, I understand. Because they want to. You want to try multiple criteria, and this is only allowing one criterion to be tried at a time. Well, I mean, you know, Roger, I would say the thing we were discussing in the last meeting, where we were talking about. In the last meeting, Shadi, we were talking about cases where symbolic computation is slow but numeric computation is fast. And we imagine what we're going to do, you know, up to two seconds of trying to do it symbolically. And if that doesn't work, we're going to fall back to numerical. What I was suggesting was that we have a progress monitor that says you're trying symbolic computation, you know, reverting to numerical or something, or falling back to numerical in 2 seconds, 1.5 seconds, 1 second, blah, blah, blah. And I could imagine here, if there's an expensive operation that takes a visibly long amount of time that we bring up a progress monitor that simply says, you know, computing whatever they are, component, whatever it is. Why is that useful? It's useful because it tells people that's an expensive operation. And if in the progress monitor that it brings up, it says what the function is that just directly does that people will be able to figure out, oh, I need to do this multiple times. Let me do that expensive part just once. Okay, that makes sense. Roger, does that make sense to you also? And I think it's a good design approach. I think, you know, the thing we invented there seems like a good, general. General thing to do.

Yeah, probably. I mean, it kind of relates to a little bit. Yeah. No. Okay, let's go down that path.

I think that's probably gonna image components already. Since last week. It already. It Already had a progress monitor. What Mark has added since last week is a stop button similar to Net Train. You'll just get an intermediate result.

That's cool. I think we need to look at this. I mean, one more time and you know, when you. When this has all come to equilibrium. Unless. Unless you think the image stuff is now at equilibrium.

The one weird thing.

You know, I really need to get going in a minute, so. All right then what. What do we. What do we have here that we can cover right now?

I think these are all, probably all of this image components and how stuff gets integrated to image contents is the real remaining thing here. And it's not super quick.

Probably fine. All right, so let's pick that up again.

Are you overall happy with what you see on this page so that we proceed with image components?

I think so. I mean, okay. Particularly. And you haven't listed image contents here.

You can open image.

No, I understand. But you haven't listed it here. Right. So. But you did change.

Yeah, the agent did. Yes. The agenda we haven't updated. We should.

I don't see any tentative things.

Scroll down. Mask bounding box and masked image.

I don't quite understand why that's not the default. Or maybe in this here. I'm confused by why that's in the default here.

In image contents. Be the default in image contents.

Yeah. I mean. Well, I don't know. I don't know. I. You have to warm me up on this again. I, I, you know, have forgotten some of the details of this. But I, I thought what we had, what we were concluding was that we have this new method for doing image components and that the whole point is you want to get the zebras, right? I want to get out of this picture. I just want to get a bunch of masked zebras, which could mean many things. I mean by mask zebras. I mean zebras whose image area is. Is masked off.

Right.

Anyway, sorry. I'm now having an image of, you know, like those blinder. The things that they have for horses to keep them from getting distracted. Anyway, never mind. Okay, sorry. Focus. Okay, so you understand what I'm saying that that would be. That was what I understood was one of the things that could come out, you know. I want a list. I've got a picture. Show me all the zebras in the picture. Just like with facial features can easily get the images of the faces in the picture.

That. That is in image cases. That's already the image cases function. Image contents is this one that returns the data set.

Okay. But so what's, and I think what was the change? It's a new segmenting anything, you know, a new backend which allows for some new functionality. Is that correct?

Yes.

Okay, what is the new functionality?

The one bigger function we're proposing is that image components function itself which returns a label matrix, a full segmentation.

Okay, but what is the new thing that image contents can do?

I don't know why I can try this in prototype build. But like image cases, image comma, masked image should actually give you the masked out zebras and elephants in that image. And then same thing with image components. If you ask for masked image as a property, it will include that in the data set.

Okay, so you're saying if I take this picture here, okay, and I say image, what should I do if I take image cases of this and now I say what the heck are these things anyway? Animals. I mean, let's just say zebras here. So what do I do? I say zebras.

By default you'll get all of the animals or all of the objects in the image.

Well, I got nothing.

I think you should say zebra singular.

Well, I think I should say control equals zebra. That better work. I hope that works great. Well, maybe it's a different kind of zebra. I don't know.

It's. That is a problem. We, I think still have to go.

Back to mountain zebra.

Do you see a concept entity?

No, no, I don't. I don't believe in concept entities, but I don't see one anyway. All right, let's try.

Don't. But you don't even need to specify that second argument. So this second argument problem will have to solve. But regardless of that, it gives you these zebras and elephants.

Those are garbage. What's happened here?

I don't know. I'll report that.

But this is what it's always been doing. Is that correct?

No, it shows a nice looking concept entity.

No, no, no, no, no, no.

But the output is. Yes, yes.

Has been doing that forever.

Yes, yes, yes.

Okay, but if I say masked image here, it'll do something different. Is that right?

It will give you the same images with alpha channel that backgrounds is or mask.

Okay, so how do I do that? Comma, what? Here?

Masked image.

I doubt that. That's right, because that's asking for a masked image zebra, isn't it? It's got to be automatic or all or something. There. I, I understand this argument.

I, I think we, we automatically understand whether an entity is given or a class is given or a property is given.

So the valid object category. I gotta go. So let's sort this out and we'll come back.

Yeah. As I said, I think this wasn't a prototype, but yeah.

Okay. All right, so one. One more iteration on this. Not for very long. Let's try and combine it. Roger. We should be able to have some kind of, you know, pickup meeting with a bunch of these different items in it.

Yeah, I just don't want to do it too quick because then we don't resolve them anyway. Yeah, that's probably first couple of days next week or something like that. And then I think.

Yes.

Okay.

Sounds good. Sounds good. Okay, Thanks a lot, guys. See you.