What's New in Wolfram Video?

Hello everyone, My name is Carlo Giacometti. I work at VOLFAM Research in the Sound and Vision group. And in this talk I will cover what is new in video essentially since the last time we saw each other at the last technology conference. I will talk more generally about the updates that we did around the Video object and camera kind of backend and infrastructure, some updates around video editing and new functionality that covers video generation. Talking about backend and Video object, the new thing in version 14 is that we now support any FFmpeg version between 4 and 6. And you can see here on my machine I have installed Fempeg 6 that is perfectly supported and we also ship if you don't have it installed locally, we ship version 6 installed by default in the Mathematica layout. And you can see that is picked up by Mathematica. And this comes with an updated list list of encoders and decoders. As usual, this is pretty giant. Things that have been improved in this latest version, I think mostly are encoders that deal with the Avi AV1 codec. So this will be some small, hopefully noticeable, but definitely not a front seat I guess updates in performance. Another thing that we've done around the Video object is better subtitle handling. So we had supported subtitle for quite a while, more or less since video was introduced in the language. But we didn't have really any utility to read the item. You have always the. You always had the option to display them and the gui. In this case I just select the track and play and you can see the subtitle is displayed. But importing them and exporting them has always been kind of annoying, fiddly or impossible. In the case of Expert, something that we had previously was the raw subtitle import. As you can see, this is very. And we have a whole section of headers that specify what the style will be in a very obscure way. And then each subtitle has some text and some timestamps. But also here there is some weirdness about how this is specified. So what we introduced in 14 is a new format for subtitle import. You can see here I just put the background of the cell to orange. But you can see just to make it easy to see since the subtitles are white, you can see here the import returns a list of timestamps and a list of native mathematical representation of the subtitles. In this case we have text and we have style that I use to represent how these are displayed. I can lock the input form of this first one. You can see, a text allows you to specify position and the style has all this font information. If you're not interested in all of this visualization or how the subtitles are formatted, you can easily input just the subtitle text with its timestamps. And in this case, if you are just interested in the content, then you don't have to deal with unwrapping, I guess all these wrappers. And we have introduced more subtitle handling in higher level functions which we'll get to in the next few slides. One brand new function for version 14 is video extract tracks. We had previously video extract frames that would allow you to extract arbitrary frames on video. Video extract tracks is just a level higher. It allows you to extract and separate the tracks from an input video. So here we have. I can show you the information of the video. We have one video track, three audio tracks and two subtitle tracks. When I call video extract tracks on them, you can see how I get all of the resulting all of the tracks separated. I have a list of lists. The first item is a list of video tracks. The second item is a list of audio tracks. And the last item is a list of, in this case, subtitled text tracks. It is possible to extract all tracks of a specific type. And I will quickly show you the video track as well. Because this is kind of interesting. I will extract the first video track. You could make the assumption that this creates a new video file on your hard drive with just the video track. But in reality what we do, you can see we try to use the same original file and then use the track selection options to ensure that only the interested track is reported in the output. We try to do the same with the audio tracks, but that doesn't necessarily always work, mainly because audio doesn't support all formats and codecs that video supports. So in this case we end up importing the data in the notebook itself. And this is all the audio tracks. As we mentioned before, we can now operate on subtitles. We can by default we import the subtitle text element, just timestamps and strings. But you can also ask for the full subtitle including formatting. And again, here I have the cell has an orange background to make seeing the subtitles slightly easier. Video extracts follows kind of the same syntax as extract. So you can take out an arbitrary set of tracks, in this case the second audio track and the first subtitle track. But this can be completely arbitrary. So this kind of is the equivalent of the input of subtitles or any specific tracks. The convenient part of this is that you don't need to figure out which file corresponds to which video object. You can just pass it the video object. And that is all handled natively on a Converse. Video combine kind of does the opposite operation as video tracks operate while one takes out the single structs and exposes them. Video combine takes a list of items that could be video objects, audio objects etc etc and combines them into a single into a single object. So here I have a simple video that just displays noise and I combine the noisy frames and an audio object. And now we have a single video with a video track and an audio track. And this was the state of pre 14 Wolfram language. We now support writing of subtitles as well. And there are several ways that you can specify these. You can either send a list of strings and then these will be equally spaced throughout the video. Let me re evaluate that and you can see how the subtitles are in there. You can select them in the player, but small but they are there. You can control the time at which they appear. And you can see this fits and you can also exactly like we've seen in import and in video Excel tracks, you can specify the formatting of the subtitle using style to control the styling, text to control the positioning and rotate to control the angle which they are displayed. Now I want to put a caveat here. Style supports everything more or less. However, not everything that style supports is allowed for some of the subtitles and coders. So most of the common items for style like color, font size, font specification, font variation and all of these things will be respected. However, there is a limitation to what subtitle formats allow. In this case I just have, you know, color size and that's it. And you can display that in the player as well. And that works very easily. And you can have funny examples here. I have a bunch of random choices between chirp and wheat that are positioned and rotated really randomly. You can see a couple of them. You can see it's a text and a style. And then you can combine that with our previous bird video you have these subtitles circling in Twitter around the frame. And this, this kind of shows how wild subtitles can get with the new support that we have given them. Another function that has got an incremental updates is overlay video. Overlay video so far has been used to take a video or an image and overlay either images or videos on it or arbitrary content. Content really. And you can control the positioning and you can control. And that positioning can Be time dependent. In this case I am just taking a pretty boring video and in this case I'm just placing an image in the center of it. And the second input, I'm placing a video over that previous video in version 14. The thing that we've added is to control when the overlay is displayed. So in this case I want the overlay to appear at second one and to disappear at second two. And I guess we can do exactly the same thing with the second example. I want the video to appear at second one and disappear at second two. Look at the image case first and the video case second. And this all work without any. Without any issue. I am using here the placed wrapper. It could be any kind of overlay that was previously supported. If we wanted to have more overlays as well, all of them with timely information that could be done as well. In addition to that, another new feature is to control the transparency of the overlay. In this case I'm just specifying a value for the transparency. You could specify a mask and masking can be dynamic and I could. In this case I'm applying a masking that is extracted from a video. You can see how that masking changes dynamically as well. You could also have a pure function then in this case returns a number that controls the transparency for the overlay. And I've shown all of these with a video. But this could be any of the other items that have been previously supported in overlay Video. Another function that has gotten an incremental update regarding video editing is Video Join. Previously Video Join. We just put the requested tracks one after the other and the change is very abrupt. Here I'm generating a red and a blue video and you can see it just switches instantly from one to the other. In version 14 we add support for a set of transition effects that will be expanded in the future. It is possible to control the duration of the transition. And it is also possible to control the effect type. In this case, previously we had a fade kind of effect. In this case we have a push type of effect and it is possible to control the direction of the effect. In this case, the wipe will go towards the bottom. These are the same effects that have been supported in various front end related dynamic functionality. The plan is to extend support beyond these. One of the main goals that will not be in version 14 but will be most likely for the version after that is to have the user be able to specify an arbitrary transition, possibly by using a function that will take inputs from the frames of the two Videos and the time relative to the transition. Now, we've only looked at the transition between the frames for audio tracks. Those need to be. So what you need to understand is that these transitions operate on the assumption that that the two tracks will end up overlapping by some amount of time. So then in this case we either can wipe between one or the other or we can fade between one or the other. However, we always are operating under the assumption the audio tracks are synchronized with the video track and we cannot break that when applying these transitions. This is why the audio tracks are faded as well to preserve this alignment. In this case, I have one track with a noisy audio track and then the previous video that only has the blue frames and silent audio. And you can see how the transition applies fading on the audio data as well. It fades between noisy to completely silent. And this allows preservation of synchronization between tracks. Another function that has been updated between these last few versions is remove background. Now this has been updated to be able to operate on videos and has gained several neural net backends. I will take again here our favorite bird video that we post in the mathematical layout and I will not run this because it is a bit time consuming. However, the results of the background removal here I would say are pretty impressive compared to all the other classical methods that we have. The output of this, we need to know this. We use a different encoder for the video track because we need to host also the alpha channel of the image in this case, this is one of the few encoders that support transparent data or alpha channel data. So that is it for video editing. And now we switch to video generation. And the function that we are introducing in version 14 is called TUR3D video. So in previous versions we have introduced several functions to generate videos. One of them has had the goal to allow users to easily explore two dimensional constructs such as graphics, image, two dimensional regions, etc. Etc. The idea is that within the Wolfram language you are able to generate a lot of this content. However, in many contexts it is videos or shirts or gifs is the natural way of sharing this content rather than the, you know, a native Wolfram language notebook. So the idea is to provide a way to transform dynamic content that is either two or three dimensional within a notebook to something that can be shared on video based platforms. And we want to make this transformation between graphics or graphics 3D to rendered videos as easy as possible. So as I said, in version 14 we are introducing two 3D video which allows you to explore these three dimensional constructs I will use here as an example. This simple castle wall 3d graphics 3d object and we do have a single 1arg form of 2 or 3d video. In the default case it will just take the input and fly around. Let me reset this at the beginning. Fly around the object, do a full rotation and stop there. It is also possible to specify different kind of presets if you don't want to specify what path the camera will take. And you can specify some parameters for them here. I still want the rotate presets and I am specifying a radius at which the camera will rotate, the view angle which controls kind of the the length, the focal length of the virtual length, the virtual lens that you that we are using and how many rotations we will perform. And we'll have several other presets like swivel, pan and zoom. Like in Tour video, it is possible to control the path of the camera a bit. More specifically here I am passing a list of points. These are interpreted as points where the camera will pass through. These are normal 3D coordinates, nothing fancy about them. What will happen is that times will be automatically assigned to these points and then a path will be produced by interpolating between these points. And you can see here, let's go to the beginning of the video. And for a finer control. In addition, just specifying the points interprets those as the position of the camera. We can define multiple parameters that control the scene. In this simple example, I am just specifying the position of the camera and the subject position, which is where the camera points to. Other things that can be specified are the view angle, as we've mentioned before. That kind of controls the focal length of the virtual lens of the camera and the view vertical, which controls the tilt of the camera more or less. In this case, as I said, I'm just specifying camera position and subject position. So similar video to before, but the direction that the camera is pointing is different. And if you want a complete control over the generation at each time without leaving anything to the interpolation, you can define a function to control the camera position. In this case I have a simple function of time and I am displaying here with our subject. This is the path that the camera will take. So kind of an upward spiral. And we can generate the video for that. Let's see. This will take a little bit by the way. One thing to note is like all of our other video generation function, you can to speed up the generation, you can always use parallelize in conjunction with it. This is our upward spiraling video. Like before, when we specify the points or the keyframe specifications, the function can return an arbitrary association with all of the same parameters. So here I control the position, the position of the subject and the focal length of the of the camera. And we get a similar result, except that where the camera is pointing is significantly different than before. So given all this, I prepared a couple of examples. So here we start with a simple graphics 3D visualization of a cellular automata. This is just a bunch of cubes in a 3D environment. And we can define a set of a set of points in this 3D space that are equally spaced, that are visualized here as red points. And they are ordered. So using the find shortest tour function, we can now interpolate between these points using interpolation. The fact that this is now a sensible path is only thanks to the fact that they are ordered. Use Fine Shortness tool. And in this 3D visualization we can easily plot that trajectory as a function of time in this case. And we can use the function that we interpolated in addition to its derivative, to control to generate a tour of this 3D cellular automaton. And let me get this started. And this is what we get. As you can see, we visit all of those points and we control also at all times where that camera is pointing. Another fun example, I need to credit our intern Anika Patel for both of these last examples. These are modified from a couple of her excellent community posts that I urge you to go read. This is based on the protein visualization from the Packlet repository. And let me see. I will. We will start with this visualization. We're only interested in the graphics 3D structure of this. We don't care about all of the labeling, etc. Etc. So what we want to do is define a function that kind of spirals all around this molecule and then flies through this space in the middle. So we define a few functions. One that defines our spherical spiral, one that defines this fly through, one that puts all of those together. We also need to define a function for how to view vertical will end up evolving, just to avoid any funny business on how the camera is rotated. And we can show how the camera path will evolve over time. You can see that the path is in blue camera position is that there are a dot and where it points to is the subject position. You can see it spirals down and then flies through the middle part. And we can just use those simple functions to generate the final video. And that is it for me. I encourage you to visit our Correra page Linked in the Notebook the Notebook will be distributed after this presentation and I urge you also to attend our Sun Envision of Sauer in Loveless Lounge today. Later today. And yeah, feel free to ask questions now. I see two questions already. One is why it isn't parallel by default. Well, we don't make a habit of launching parallel kernels by default. We want the user to make a conscious choice there. Also, because there might be licensing limitations on the number of parallel kernels available all the time, Notebook will be distributed after the presentation so another question is about resolution enhancement. You'll have to apply any of the neural networks that are in the upscaling neural networks that are in the neural net repository on each of the frames. It's not really an in place resolution enhancement. You will have to map that function on each of the frames. And the latest question about two points of view. Actually, if you look at one of the community posts from Anika Patel, as I mentioned before, she does show a way of combining two different points of view. It will involve generating two separate videos and combining them combining the frames from them. Or you could do your whole generation using a higher level, a lower level generation function like animation, video.