Live CEOing Ep 787: Design Review of Sound & Vision Features

Hi, everyone. We're here for another Wolfram language design review. This one is about sound and vision. So speech recognize. So this is running locally, right?

Correct.

And what languages does it deal with?

If you hundred languages. Yeah, it should be in the notes at the bottom.

That's the speed up compared to real time. So deep speech was slightly faster than real time. Whisper is like 65 times faster on my machine.

Wonderful. So can I run this right now?

If you have prototype. Yes.

What? This is what I'm running Prototype build.

Yep.

Hello there. See whether you can understand me well today. I wonder what language I should try speaking. Well, I don't know too many languages. So what's happening right now?

So that it's downloading the appropriate model should be not gigantic one.

So did it do a language recognized to figure out or does it use a single network?

Whisper is interesting in the sense that the languages are part of the tokens that it's transcribed. Okay, that was a tough sorry for the amusement.

I'm guessing you probably didn't set the microphone correctly.

See.

That is weird. Is this if you audio plug.

Hello, hello, hello, hello. That may not be the right microphone. How do I set the microphone?

It should be. Yeah, top right.

Okay, this is better. Hello, hello, hello. That's really weird.

I would try and maybe recording. I'm not sure if the the monitor should work even when it's not recording, but just. That is not the case.

Hello, hello, hello. Okay, that looks okay. So that's a bug. Yes, just found. Okay. And now we're recording some stuff so we can see whether it knows what we're talking about. Should just say running speech recognizer.

This is weird.

Okay, so this is totally broken.

Can you tell me what build you have or what version of prototype? Because that looks like the old. Okay, I will investigate from yesterday.

Should be good enough. That should be good enough. I'm saying that build is good.

I have it.

Always disappointing when the demo doesn't work. That's great that it has punctuation. That's really nice.

If you run that first example. Do you. Do you want to quickly try to make sure you get that result same as what's here? Yes.

I don't think it's going to be any different if the previous one didn't work. I'm guessing this I understand raw result.

Why does it say. Come on, guys. Okay, this is all totally broken.

There is something wrong there.

Yeah, look. Look at this. Come on. Be elapsed time. Remember the rules for these summary boxes for these progress boxes. Don't camel case. Right.

I think this is the old speech recognition. That's miserable. Well, no, but even not that, because.

Here, right here you've got.

No, no, I get it.

Okay. Get that right. Get the capitalization right. Don't be sloppy.

Yep.

Okay. And it's also failing completely.

Yeah, we'll have to see what didn't work on your machine.

I think the fact that it's an Intel Mac that is being problematic. I will investigate And F has qa.

Stefan, have you gotten your hands on this yet?

I think we're just starting to look at it.

He should have looked at it actually already.

Yeah. I don't know if he's been checking audio capture specifically, but the rest of it he has been testing. Yeah, it should be independent of the audio source. If it is dependent on the audio serves, then we have a bug. But it shouldn't be.

Somebody should report that for spell spelling. Should I report it? Can somebody else do that?

I can do it.

The transcription quality is not the same on all of these languages. These are as I said, we're using the whisper model and essentially we get what we get. About 50% of these languages are pretty good recognition results.

Now how are we doing this again?

The model outputs timestamps.

No, no, I understand, but what's the data structure that we're using to represent this?

Oh, it depends on the second argument of speech recognize. The third argument of speech recognize.

I don't see a third argument documented. Where is that? It is in cbs in the source. Ok, so there's a bunch of design here.

Yes.

And we kind of followed what.

Text, by the way. You weren't surfacing that for this design review. You should have been. Am I making sense? The fact that you just give an example here isn't really good enough. You should have said we have new, new options for speech recognize. Right.

Sure, we can try to do that. I was mostly. Yeah, yeah. This is how we make symbols and listed on.

I mean, we knew that this would.

Be a discussion point. Stephen and I expected us to look.

At the reference page. What the hell. I mean, it's just illiterate. Uses the OpenAI speech to text. Why does it say the there? I don't understand. It's just illiterate. I mean.

Yeah, I usually count on things like that.

What's that?

Sure, these are the things I expect DQA to fix, but.

Okay, fine. Well, maybe they haven't looked at it yet.

This is an old one, but sure.

This isn't the right way to say it. I mean, what you're saying is you want to either get a single string or a list of sentences or a list of words. Is that correct?

That is correct.

Maybe a list of paragraphs one day. So now this reminds me of some other functions we have, like word counts, things like this, if I'm not. Well, no, no, actually. What?

Text cases, what's.

Yes, okay, how does that do its thing?

This design is really similar to that.

I see. Okay, let's take a look here.

And I totally agree with you about additional levels down the road. One could be broken up by speaker, for instance.

Yes, for sure there is some backend functionality, but we haven't tested it enough to feel confident in it.

Okay, but that's going to require a more complicated data structure, right?

Yeah, most likely.

Right, so we need to think about that now rather than. Right, because. Okay, so here, these are the things that are being returned in what structure?

The structure is defined by the third argument. What is return?

No, I know that, I know that. What I'm saying is if you say text here, then each one of these things will be returned as text. If you say audio, then, for example, you might get back breaking. I mean, it's a very weird thing you'd say, you know, this is really. It should be kind of. I don't know what function we have. It's for breaking it up into elements. But in other words, you'd be saying speech recognize, sentence comma, audio would be a way of breaking the thing into. Right. Now we should have a direct way to do that because that's a useful thing, like break the thing into a sequence of words.

So if you ask for intervals, for instance, you get time intervals that can be passed to audio trend.

I understand, but I think that we should have a function that simply says take this audio and turn it into words or turn it into sentences, for example.

So I have to preface with this with a note. So the timestamps are produced by the network. These are part of the tokens that make up the vocabulary of the essentially LLM part of the network, which means that they are usually decent, but they can hallucinate. Right. These are not something that is generated.

Of course, it's an LLM. It's some kind of, you know.

Yeah, I just wanted to make sure that that is understood.

We know that, we know that. But you know, what we're trying to do like, like image identify can say it's a cat when it's really an elephant.

Yeah, yeah, right.

These things can happen.

Yeah.

But what we Want as a thing that is, you know. Well, I'm. I don't understand what subtitle text is.

That is kind of a convenient, maybe convenient combination of the intervals and the text. So it's a list of.

Okay, but can we feed that in? So let's imagine you want to translate.

You can do video combine of speech recognized subtitle text and put those.

There should be an example. Or maybe I didn't put it in, but yeah, we, Carlo and I worked out an example exactly for that.

So. So hold on. So let's say I want to do translation of a video. How does that work?

The first question is, do you want to translate it from a random language to English or some language that is not English? I'm asking this.

Well, let's say, for example, random language to English. I don't know. I mean, I don't have a specific use case.

But the reason I'm asking is because Whisper has a special power which is translating any. A spoken language to English and it does it automagically and we do have a maybe ugly head. Well, no method sub option to control that.

Oh God.

Otherwise. I know, I know.

Don't hack it. Stop hacking.

No, this was just an interesting side note on a capability of the model. How you would do it is most likely you would have the model return the sentences and the text and the time intervals, do whatever functionality that you want to use to do the translation and rejoin those.

Okay, so look, in the end, what we're going to want is some kind of video map functionality that allows you to, you know, have some function that you're essentially mapping over the video to do the translation of the audio in place.

Audio or audio to text?

No, I'm talking about, I'm talking about I've got a video in Urdu and I want to make it a video.

In English with sound or audio or with subtitles.

I pref. I don't know. I was thinking with sound, but maybe I should be thinking with subtitles.

So speech recognized is really, you know, primarily getting to. To text.

I know, I know, I know. But it's, it's a pipeline that will be speech recognized followed by speech synthesize. Although there might be a more direct thing that is the pure, you know, because I mean, like, like somebody just did a interview with me for some Brazilian podcast where they're going to run it in Portuguese, so to speak, using.

Maybe that should have a new name.

Probably needs a new name.

So we have speech synthesizer, speech recognize, and they, and they are just Sort of.

We probably need speech translation.

I mean it's almost like speech transcode or something. I mean it's.

Yes, it is. Right. It needs a new name that. It's basically like take it in in one language. I. Oh well, I mean apparently they sent a bunch of demos and it seemed to work quite well. Yes. So we're going to need a new function for that. So get my, my talking about this here. But I don't know how much, how far we should go. You know, maybe we should have a function that adds sub tools to a video because that's the thing people want to do and not have to have people go through seven functions and make a bunch of mistakes.

It's two functions, so it is not as bad. But I understand where you're going with that.

I mean it's a standard thing to want to do. Put subtitles in a video.

Yeah.

And also, I mean here's a standard thing that we've encountered which is to use a translated subtitle like we made Russian subtitles for a conference in Russia.

Right.

Even though we spoke in English.

Right, right. So I mean subtitling with a language option seems like a good thing to have. You know, in other words, video subtitle or something. How do you extract the subtitles from a video?

You can do video extract tracks subtitle and that should be enough.

Okay, that's not great. Look, I think.

How, how else would you imagine.

Well, I'm, I'm, I, I think we kind of need a function video transcript or something which is maybe, maybe the. Sorry, maybe the right answer there is speech recognize a video does it from the speech. But if it's already got subtitles, how do you extract those? Is the question. But maybe that, maybe that is a special case.

But, but also like that subtitle track could be completely unrelated to what is being spoken.

Right.

So.

Yes, fair enough, fair enough. So, so maybe it's okay. The subtitle track extraction is sort of okay, but. But then, you know, there's add subtitles to a video, translate the speech to speech in a different language and presumably speech recognize of a video just works.

Yes, not in this version that you got, but it will work. 14.1.

Okay. And my question would be, okay, what about speech recognize and with a language option, particularly if this model supports translation to English.

Yes, the language option, we have it. It's useful to give if you want to tell the. To help the model in there.

No, no, I'm saying a different thing. I'm saying the target Language.

Yeah.

So speech recognized. And the thing is speaking in Swahili and coming out in English, we can.

Only do translation to the same language or to English. We. We cannot.

I understand, I understand. So the one to English seems highly useful to me.

Yeah. Which is why we thinking of about exposing it, but couldn't quite come up with.

No, no. Okay, so let's come up with the design. Okay, so you're saying speech recognized, language arrow whatever. You know, that language arrow whatever really shouldn't be necessary.

It is. It is for this model.

Okay. But that's this week. You know what I'm saying? In the future, I can't believe you'll need to specify the incoming language.

That's exactly what I was thinking. I was thinking, is this a feature we want to bake into 14.1 or is it a little too early?

No, no, no, no, no. So what I'm saying is, what I'm suggesting is, and I don't know how it's compatible with other functions or not that speech recognize language arrow English would say I want the destination language to be English.

That feels like target language or something like that.

Yes, that is. That's more reasonable. Target language might be more reasonable. But what other functions are there which have language options which are defining. So. So for example, text recognize as a language option, which means the incoming language.

Yep. Yeah.

So you're right. Target language sounds like the right thing.

But also text translation has the second argument. Lang 1 goes to Lang 2.

Source to target.

I see, I see. That's nice. So you're saying that we could have a language option here that says source to target.

Yep, I think that would be nice.

And then we could say all arrow English, for example, or automatic to English.

Or all English. Sure.

It would be okay to release this, even though the second arg of that rule can only be automatic or English.

That's fine. Just give an error message if it isn't that for right now.

Yeah. Okay. Sounds good.

Okay, let's look at the rest of this. All right. Strength. What do you mean by strength of the recognized text. What does that mean?

This is what text recognize, I think uses. It is kind of the probability of the.

I don't believe that word. I don't believe in that word. It's incomprehensible.

Yeah, it's like recognition string, I think was what was intended originally.

I never reviewed this. I would never have signed off on this.

The. The thing with text recognize and strength is that we're not exactly returning actual probabilities. It's a combination of measures.

When did you put this in? Because I never reviewed it. When did that go?

If you open the source page, there should be an annotation there.

Kind of annoys me.

And I'm pretty much not sure we have looked at all of these together.

It shouldn't be called strength. It should be called something like, you know, certainty or I don't know what, or, or. What values does it take anyway?

It returns a value and it's.

What is that value? 97. What does 97 mean?

No, usually 01.

Okay. Certainty perhaps will be a reasonable term.

Confidence is another function that has it again, what? Find faces also uses strength. I'm just trying to figure out what places this needs to be changed to.

Well, I would think confidence would be a better term. Roger, do you want to comment on this?

Yeah, I think any. I mean they all sort of indicate probabilities. But you guys said it wasn't probability. So what?

It's not. Nobody knows what probability means, you know, is it a frequentist probability? You know, it's happening with frequency?

Yeah.

Okay, fair enough for choosing like strength. Right, because there's kind of a vague term that.

Yeah, it's a vague and incomprehensible term. Yeah, because it might be, you know, when I saw it for speech, it might be is the person shouting or not? You know what I'm saying?

Confidence sounds reasonable to me.

Is that consistent with other machine learning functions and so on?

I was trying to look at whatever.

So that would be my candidate for this.

Well, let me look around and see if that's going to be conflicting with anything.

Okay, what is the difference in text and subtitle Text?

Text is just a string subtitle.

Text is interval subtitle. Text sounds like it's a sub case of text and it's not. It's something like subtitle track or something.

Or it could just be subtitle.

I don't think so because it's more than that. And how compatible is it with the. With what comes back from the subtitle track track of a video.

This is what we this it, it. It. You can just plop it in video combine and it will write a subtitle track in the correct format. It's the same.

Okay, so hold on, hold on. So but what is this? So then we could call it what is the extractor from a video called for the subtitle track?

This is what it comes from, right? It's a subtitle text if you do video.

But is it in Fact text, when you get it back from the subtitle track.

So this is a high level interpretation that does kind of strips all of the formatting that is done to the text. This just has the timestamp of the text.

For video. If I extract the subtitle text track, what do I get back? Is it just text or is it marked up text with just text.

Interval goes to string.

What do you mean just text? Did you say interval goes to string.

Or just interval goes to string? It's just a text.

No, it isn't. It has the interval in it, right?

Yes, but it doesn't have all the formatting. Nothing about.

I understand. If it has interval goes to text, it's not just text.

Real writing. Its interval goes to string. Right, Goes to text.

So that's not right for saying subtitle text. That's not what it is. Subtitle text should be just the text. You see what I'm saying?

Okay, then that's another one that video extract tracks has that type or property. Then that has to be changed.

There has to be changed. Yeah, it should be changed. Subtitle text should just be text, which would be quite useful. Right, Just here are all the subtitles boom. As pieces of text. No intervals there. I mean what we've used elsewhere is subtitle rules. But I'm not sure I like that very much. That's what we've used elsewhere for things which are of the form, you know, thing, arrow, thing.

I mean that seems okay enough to me. As long as there's.

Yeah, that seems subtitle rules. Yeah. And then in this case with speech recognize, there is no subtitle text because it's just text. You see what I'm saying?

Timed text.

Well, there might be if you want to put in. But that means it's text. It's a string. Do you want it to be a string? That might be useful.

It shouldn't. The structure, as I said, is the useful structure is really time interval goes to string.

Well then it's subtitle rules.

Okay.

I think. Which means that that's what we should support. I mean for now we can leave subtitle text as doing this thing, but we should add subtitle rules to the video extract.

Sounds good.

Yes.

Interval containing the text. What does that mean? Is it a single interval or is it multiple intervals each of these?

The number of results will depend on the second argument. Unfortunately, if it is automatic, it is a single interval. If it is sentence, it's a list of however many sentences that input Might contain.

Okay. Subtitle rules is pretty weird if you say word.

I mean you'll just have subtitles go very quickly. Not very useful subtitles.

Well, each one will be a single interval. I mean how does it break up the intervals for subtitles?

So the default. So the network by itself has its own weird way of breaking up the input which is pretty optimal for subtitles. If you do sentence. Obviously that can become a bit more lengthy and problematic for subtitles. Subtitle Tact is trying to do something specific for subtitles that will look decent when put in a video.

Sandra on our livestream is asking, are the intervals interval objects with quantity objects in seconds?

They are not. They can be. They I think interval objects. I'm not.

What the hell are they if they're.

Not that, but they the quantity.

Oh no, no, no, no, no, no. They cannot be that. No.

What about list of Quantities?

List of 2 quantities? Yeah.

I am just. Interval does some strange things.

No, no, no. Then fix the bug. Don't not use it. I mean why are we not using interval?

That's fair. Well, okay.

And let me, let me give you an example of why we need interval to work. Okay. Because this thing, you should be able to feed it into timeline plot and have it produce a timeline plot.

Okay.

If that doesn't work with the format you have, then the format you have is wrong.

Okay.

Interval it is, but check that it works. I mean check that everything works.

It should. I think the main thing that is being weird has been weird in the past with interval is for stuff that is close to zero. It often it's the outside bound on the interval ends up being negative that wreaks havoc everywhere.

So you need to address that. I don't know why that should be. An interval is from A to B. There's no outside anything.

Oh, there is because interval was designed for worst case arithmetic. So it has to use outbound rounding.

For. For machine precision numbers. Yeah, but what does it do for quantities?

I think it treats them the same way.

Probably the same.

Which might be wrong. Right. But I think this is, it's really important that this be a thing and also that it format in some reasonable way here. Am I making sense?

Yep.

Sandra is asking about accents. And I think that's beyond the scope of what's doable now because we don't have any way to specify. I mean we are just about getting locales for spelling, for British English etc, but if you say, you know, cockney of this age and this whatever and this, this, this. Forget it we don't have. There is no, to my knowledge, there isn't even a canonical representation of all the various accents.

Am I correct? In passing his comment, like, if there is an accent for like a weird British accent, you still should be able to specify language goes to English and the Net is still will be optimized for transcribing in English despite of the accent. So how I'm interpreting that comment is the language specification should help with the understanding of stuff that is heavily accented.

Yeah, I see, I see. Yes.

Yeah, I think that that's one of the big pieces of progress these last years that, that it actually works very robustly.

And Sandra's commenting. Yeah, sure. I mean, these things have improved dramatically over the last decade.

Yeah.

For me, trying to imitate American accents to, you know, whatever it is.

And your interval does still do the outbound with.

Okay, so you need to be addressing that possibly for quantity objects. I mean, I'm sorry, you. We can't not use our system because it does. Oh, that's. That's crazy. That's just crazy. I mean, maybe we just have to format it differently or something. Okay, that's an issue. You can bring this up in LCC or talk initially, I guess, to Jose about this. Does that make sense? Roger?

Yeah, I have a feeling that we need another interval object that doesn't have this mission.

No, I'm sorry, we're not going to be able to do that.

Or we could abandon the interval. I mean, we have centered interval, which is sort of better for arithmetic anyway.

I think we should just make it work differently for quantities. I mean, think this is crazy, you know, look. Okay, so that works fine.

Yeah.

Wow. Yeah, that's very much wild and crazy.

That very much looks like a machine epsilon to me.

Yes, I'm sure it is. That's just crazy. There should be some other way to deal with this. Oh, that's crazy. Wow. Okay, we got to address this somehow. I think initially the way to address it might be to just switch this behavior off for quantity objects. Am I making sense?

Doesn't that sound a little hacky to you?

Well, we're dealing with a hack here, so we're going to have to put another hack.

It just had a different goal in its introduction and mission. So, you know, it's, it's.

I don't think it was well thought about. This wasn't, you know, this shouldn't have been set up this way. Maybe we should change its behavior. I mean, I don't know what it would affect better not affect too much because this is a, and you know, and use centered interval to be the thing that does serious numerical analysis. Am I making sense?

Yeah. I mean, it's not a quick, quick change though, Stephen. Just, just, it needs to be worked through.

I understand, I understand. I'm saying, so get it, you know, we need to get it worked through. All right. So.

Yeah, that method sub option. Don't need to look at it. We decided language goes to rule of language to target language is the solution for that.

No, no, no, we're talking about the speech.

Oh, no, sorry.

Non speech. Yep.

I think one thing that's this sentence here. That's not true anymore. Right?

That is.

Okay.

No, the next one, the next one is a new replacement for that.

Actually, this brings up another challenge and you might have an idea or we might have to see more of it.

Okay.

We're starting to have all these functions that support some list of languages, but which list for each function. They're all different.

I think we should put it in the data repository. Either that or we should have a. The other possibility is that it somehow goes with system services.

But how would you as a user, I mean, what. Think about sort of the workflow for a user.

I'm thinking it will be in the documentation. It says the languages supported are, you know, these, and it's a link to a thing that just gives you the.

List of languages for speech synthesized. We have made it a function voice style data that then for every method that is available, we can return a list of voices.

I know, I don't like it very much. I don't think it's very scalable. I, I, as I've said a million times, like the, you know, like that musical instruments thing for sound.

Yeah, yeah.

That should be in the data repository because you want to be able to generate, you know. No, you want to programmatically.

I get that.

Yes.

Right.

I'm just thinking of sort of.

I haven't, I haven't had time enough.

This came up now just literally days.

Ago, and it's like, shit, we got.

To think this through. We got to organize it.

Yeah. I'm concerned that now, let me think about this for a second. I mean, I kind of feel like the data repository is the right place for these lists.

But isn't that going to be sort of like a mysterious link or something?

Well, it's in the documentation. Our node is suggesting extending Wolfram language data to include supported languages maybe.

No, but it's different per function. So we need to know which function and sometimes it's even different.

What, what this really is is something that's like information. Yeah. On speech recognize.

Pretty much. Yes.

Maybe that's how we should do it. We should just have. I mean, I'm not sure if this is. And that would give a big.

A big properties to figure out what information you can glean from that.

Well, this would just give a big data set or something of everything.

Yeah, yeah, yeah.

And then you could say information, speech recognized supported languages.

Is there anything else though? Because it seems a bit silly to just have one thing. But if, if.

No, I mean that we could support this for. Look, a very obvious thing here would be which we. Which we've never had a good way to do it. And I, I think this would be a very useful thing. I don't know, Something like that. That's the thing we've always wanted. Give me a list of methods or even, you know, give me a list of options. I mean, I guess that. That we can get just with options of.

Yeah, yeah. But for anything that is more nested, for example, all the sub options, if you have a method that is complicated, you'll never be able to get what those are.

This is the involved in language data. But I think these are all generic, they're not specific things. I would prefer a mechanism like this for specific properties of functions.

Especially properties that are useful in computations.

Well, you could use it from here if you want to, but I like this idea. In other words, information we've. We've applied to objects, we haven't tended to apply it to functions. But this is a good use of it for functions, I think makes sense and we can investigate that more in lcc. But I would suggest.

Sounds good to me. It sounds like it would be good to collect a list of functions where this would be really, really useful and.

Right. Well, a really good case for this is cases where there's one of these weird tables.

Exactly.

All of this stuff.

By the way, why is this font wrong? I've reported this like six times. Somebody please get this fixed. That font is wrong in that table.

I'll bring it up again. Yep, I know which bug you're talking about. There is already an open old bug for that.

Why. Why couldn't it just be fixed?

Buried under so many other things, Graphic design and documentation.

I don't believe that that is. So what is that language? I don't. I mean, that's Kyrgyzst, presumably from Kurdistan. I've never heard of that language as spelled there. Yeah, I'm Just. Okay, I'm wrong. Okay, I'm gonna have to go, but. So we're gonna have to pick this up. Let me just have a quick look. That is unconvincing as a name. Nest Video. That's like video feedback or something.

Yeah, it is. Surprisingly. It has surprisingly interesting examples, I would say.

That's cute.

You should be able to play, Download and play.

I can't. Well, click.

No, no, you can. You can.

You need to download or not. Can I run it?

Not in this product, but if you click the thingy, it should do. Yeah.

Well, I admit that's a good use case. I admit that I'm surprised, but I'm.

Design wise. We have to look at the fourth argument here, but perhaps next meeting.

I never thought I'd seen Nest video, but it's an interesting idea. I stand corrected here. We're going to need an hour and a half at least for this.

Okay.

That seems complicated. Yeah, that will bring up its own. Right.

We'll bring those up in the other. In the tabular.

Okay. All right. Good stuff. Actually, I'm. This is. This is. Except for the fact that it felt like a bunch of things were trying to be railroaded through, which I didn't like. This has got some great functionality. Really nice.

We never try to railroad things, Stephen.

Ever. No, never, never, never.

We wouldn't.

All right.

Anyway, just a quick note, since we're on this section in a second. There's also. The new model allows us to have a classifier for languages, for spoken languages.

Well, we have language identify, don't we?

On text. But if you scroll back up.

Yeah. You're now providing it for audio, which.

We are naming it a new name. Right. Classify spoken language, silly.

No, don't do it.

No.

Language Identifiable Audio should give you.

Oh, Language identifier as a function. Yes, we can absolutely do that. But then classify as a spoken language built classifier. Sure. No Language identify for sure as well.

Right. That's the important case, I think. Good stuff. All right, talk to you soon, guys. Okay. Okay, bye.