New and Powerful Components for AI and LLMs

Okay, let's begin. Hello everyone. I'm Shadi Ashnai, R and D Manager of Sound and Vision. It's my pleasure to welcome you to the last session in the new Wolfram 14.1 webinar series. If you look at quickly the webinar series, over the past few weeks we have had a comprehensive overview of the new release and specialized presentations on the latest mathematical computations compiler, system modeler, external language integration, visualization of astronomical, chemical, historic and geographic data. You can access recording for all of these previous sessions right here in the Webinar series landing page. As for today, we're talking about new and powerful components for AI and LLMs. I'm pleased to present these new additions along with Giulio Alessandrini, Manager of Machine Learning here at Wolfram. Julio will start today's presentations by talking about prompt engineering for LLMs, as well as connecting to different LLMs. After that, I'll present updates and upgrades in audio, video and image computation. We also have Bradley Ashby and Timothe Verdier to respond to Q and A in chat or help with any specific questions. You can also read more about these features and a lot more in Stephen Wolfram's blog post or release announcement. To try examples for today webinar, you'll want to access version 14.1 of the Wolfram app will share a free trial offer so that you can get started right away. We're going to ask a poll question to the webinar audience now about whether you've already upgraded to version 14.1. We also have YouTube livestream audience as well today. And for those of you watching this webinar on YouTube, please feel free to type any questions, responses, comments in chat. We have Wolfram webinar and technical staff joining behind the scenes today to help us coordinate Q and A between the webinar and the live stream. So I really encourage you to type your questions as well. As the webinar is going along, our staff will answer questions and type the responses back. And we'll save some questions perhaps for the end of today's presentation. Thank you for answering the poll question. As more people finish up submitting their responses, let me point out a couple of things about the webinar interface. A download link with today's presentation notebooks appear on the top of the chat as a sticky note. There is also a link in chat that includes all previous notebooks. Thank you for your attention. Without any further delay, I'll hand things over to Julia.

Well, thank you Shadi. So hi. As Shadi mentioned, I'm the manager team leader of the machine learning group at Wolfram. And so if I manage to share my screen today, I'm going to talk you talk to you about. Well, here it is, dynamic prompting and connecting to a Lamps. So this is going to be kind of a quick overview our LLM stack functionality in the language, including what is new in 14.1 intermixed. So hopefully it's going to be useful also for people that have not familiarity with LLM functionality in the Wolfram language. Okay, without further ado, let's begin. So just to give you a quick idea about how my presentation will be structured today, I'm gonna start with the top level functionality to handle interaction with large language models. So we have like several functions to do that depending on what's your angle, let's say. And then of course there's a plumbing in term of supporting functions and options and parameters that you can specify to kind of truly customize your interaction with the Lamps and hopefully kind of make it much easier than writing requests yourself. And finally we are going to look at what are the new service connections that release in 14.1 and a sneak peek about what's coming in the next few months between updates and new versions. Okay, please, if you have questions, feel free to write them. And as was said in the Q and A panel, there's my colleague Timothee Verdier that can answer them. And we'll also have a Q and A session at the end of a session. Okay, so interaction with LLMs in the language. Since we started supporting LLMs last year, we've been kind of providing a suit of different functions to convey your input to the LLM and gather the output and process the output. The simplest one is LLM synthesize. So by default you have to do very little to have this work. So this for now still requires that you have at least one account with an external API. But once that is set up, and I did so for example, I set my API key for OpenAI and many others, but by default OpenAI and I'm on a Mac so I'm using the keychain and I just set my key there. So LM syntheyes can read that can read custom authentication options. So if I didn't have a key he saved, I could just, I could just type it here. Or if you don't have anything saved and you just try the normal one, you get a pop up. I'm not gonna get that now because I'm authenticated. But there are different ways to Fill in stuff. And with many of these, you have to do it only once. So once you are set up with one system, then you're done. I also bring up the. Bring up the Our preference panel that has been updated. So you see, I'm already connected with few services here. You can add your own. It's a bit cumbersome, but we offer ways to do that. So it's more like power users, but it's not impossible. And you can set defaults here. And if you don't care about some, you can even suppress some connections. Okay, in plumbing, we're going to look at a bit more about that later. But let's say that once you do this one stop setup, then the whole interaction, like setting up the connection, sending the API requests, interpreting the parameters, interpreting the result is on the fly and you can just do what you want to do. For example, here I asked give me three warm colors and I get the answer from the LLM. Of course you can modify this. So give me three warm places, for example, and this will connect. And okay, I mean, they look warm to me. I have only been here and it's warm. Now, this is a very simple interaction. Now you can just. Using a simple string as an input, you can have more complex interactions. For example, here I'm providing instructions to the LLM and this is just a sneak peek in the art of prompt engineering. So we try to do some of that for the most complex task for you, but you can do it yourself by injecting instruction to the string. So here I'm saying I complete the sentence. The present to the USA is. And I'm just kind of formatting something in a way that looks like the model has to do something. And this is actually completing the sentence currently. Joe Biden blah, blah, blah. But this is very, you know, it's a very broad task. So similarly, as if I would ask this to a person, there are different ways to answer here. Like if I just evaluated this a couple of times before the webinar and I kept a different one. So this is also perfectly valid, perfectly valid completion of this. And then again, in this case, the model is deciding to actually repeat this part here. So basically the key takeaway here is that we might make the interaction easier. But this, this is still like talking with a kind of very informed person that doesn't get much context sometimes if you don't provide it. So an interaction like this is a lot closer to what you might want. I could even hint that it has to be the name of the President here. So like the name of the president. And I'm just going to try it again. So I'm doing this five times and I got five times the same thing. Yeah, okay, this is still correct. So this is just to show you a bit what you can do right now and how you can kind of coerce it into proper behavior. There's one addition here that we added for 14.1 which is image expressions directly supported. So many models take images. Of course you don't want to deal with Bayes encoding by converting images to strings and pass them along. So this is taken care automatically for you by the framework. And I can just for example start a conversation with the yellow lamp by passing an image and then the model is perfectly happy to answer about giving me some generic information about what the image contains. And again of course it has some element of randomness in it. So if I ever read it again, you can see that this is not exactly the same sentence. I mean it tends to be close sometimes, but again no guarantees. This is randomly generated. Of course the function supports like a mixture of text and images. So you can for example put stuff in a list and say how many animals are in this image. And in this case here, let me say one again if you want to do proper counting and stuff like that. I'm going to show an example later. You might want to be more careful with how you prompt the LLM. But I mean in spirit you can intermix text and images freely. And internally we have a way to represent these different types. It's called multimodal input, which is kind of this verbose annotated message parts. So this of course is valid input and this is what we use to disambiguate between things. So. So I mean in this case it's kind of future proof. Like right now it's kind of easy to distinguish between text and images, but we all expect these models to be able to handle more in the future. So yeah, I mean the idea is to offer a pretty solid framework to you people. And yeah, I mean this is again similar idea of open interaction. Now this is a very basic function. It a one off interaction. You give some inputs and you get some outputs. Now what if you actually want to save some of what you have and reuse it in the future? In one way we offer to do this is LLM function. LLM function is kind of a delayed evaluation. So you can basically set up your interaction and wait for user input. So let's start from the similar input I had before. Like Give me three warm colors. Now this thing here doesn't do anything on its own. You can just apply it as a function, normal Wolfram language syntax, and you're gonna get the result. So in this case a pretty stupid function with no input. Also, the colors tends always to be the same. Probably there are some. We are pretty boring in defining warm colors. You can use string template though and start putting parameters in the string. This is a very fast way to create a parameterized input for the LLM. So here I'm saying give me three colors only names just to add some simplification to the output. And now this is a new function which is perfectly equivalent. I mean we wrap string template by default, so you can also write it this way, which is a bit shorter. Now if I say cold instead of warm, I get different results. And now this doesn't in principle doesn't look any different than applying any other warm language function. It's just backed by an API call instead of a local computation. For people unfamiliar with string templates, you can add more parameters, an arbitrary number of arguments that you can define for your template. So in this case I'm saying give me colors. And of course I'm going to follow the same pattern of number and type here, but you can do whatever you want. So here's four cold colors and one more if you want to customize this a bit in terms of how the function behave, you can use numbers to specify actually the order of your arguments. So now I'm swapping warm and cold instead of one and two is two and one or one that I very much like. You can give name to your arguments and then pass an association as an input. So here I'm saying give me number, type colors. And this is kind of a, kind of more self documented way of dealing with this. Now there's a second argument to a length function that can, can take an interpreter type. And this actually forces interpretation. So if I know I want colors in outputs, I can add this and this will try to interpret this thing as a list of colors. And now that you see that this thing here is actually if I check the input form here, I'm on Mac is like control command shift I. This is a list of colors, it's not a string anymore. Now why this is interesting because you can mix this for example with multimodal inputs that we support now and create something like an image captioning function. So here I'm kind of combining what I presented so far. Now this ll function takes a template object which is Kind of a superset of string template. I will refer to the documentation for how to use it. And here I'm saying like generate a list of objects present in the image. I'm passing the image and I'm kind of giving it some specification containing only a list of objects separated by new lines without any punctuation. So I'm just maximizing the. I could probably maximize it a bit more but I'm low effort maximizing the probability of getting just a list of. Of the content of the image. And now if I apply this. Well actually I can apply it live if I apply this on this image it gives me monitor, keyboard, mouse, notebook, pen, pencil, desk, lamp, blah blah blah. So I can also redo this properties could be very similar. Let's see back and forth. Oh yeah, desk, desk is going. And I don't know why lamp is lowercase uppercase but yeah, I mean there's some variation of course again it's semi random output but it's very easy to set up an embacked function with this sort of pipeline. Now this thing here is still a one off. So how can you actually have. In most common GUI systems you can have conversation with lms. The way we represent that programmatically, of course you can use chart notebooks, but we deal with the programming language side of things and is a chart object. So here for example, again showing multimodality, I'm creating a chart object specifying a multimodal model from OpenAI. Actually I could do a different one. Let's first see what it does. So if I break it, it's. It's still informative. Now I can start a conversation, you know, a persistent conversation modifying this chat object using the chat evaluate function. So I'm saying, you know, this is inheriting all the settings from chat object. More about this in the next session section. Sorry, I'm asking here what is this picture? And I'm just grabbing from our entity database the image of a cat. And you see this kind of enrich the chat object with the input and the output. So it's a cat outside, piece of wood, background, blah blah blah. And now I can continue the conversation here is a very free form. So the LLM is going to decide for itself what's the best output for this. I'm asking differences and similarities with this other picture. So I'm getting a new chart object which contains updated conversation and you see is doing some markdownish thing. Similarities, differences and yeah, I mean this is free form. So it decided on its own how to present this, but this is again how you can keep going with the conversation. And it's also a nice storage format. Like if you use chart notebooks, you can always export the conversation as a chart object to kind of manipulate that later on. So yeah, I mean, it's kind of a nice complement when you need history to be preserved to the LLM synthesize LLM functionality. Now, this thing is very simple to use, but sometimes it might be too simple for what you need. So we have several things to kind of personalize and augment this workflow that can enrich the workflow with. Let's start with simple things. Many of these services have options like parameters that you can pass. One of the most famous is temperature. So you can make the output more random by setting a temperature. How do you do that? The interaction with LLMs is controlled via the LLM evaluator option. And this is similar. I mean it's called LLM evaluator because it kind of mimics the fact that when you execute a computation in the language, you can pick an evaluator, which is a local kernel, a cloud kernel, a remote kernel. So here this is basically giving settings for what the evaluator for this LM request is. Now I'm just keeping everything default and just setting the temperature. So let's see if I'm asking random colors. So with high temperature is probably going to be very random. Okay, there's some slowdown in my connection. Let's try again just for the fun of it. There it is. Okay, now you very often have more than one parameter that you can set. So you can symbolically represent the collection of parameters using the LLM configuration option object. Sorry. So in this case, I'm creating this configuration that takes both high temperature and the max token value. That means generate four kind of words, quote unquote, and then stop. And this configuration can be passed to the lmvaluator. So now you see it's trying to answer. Sure, here are. But it's probably like 1, 2, 3, 4. I don't know how it's split internally, but roughly like this probably and then it stops. So if I change my configuration to 10, this is going to go higher. But yeah, I mean I can use to kind of really finally control parameters with LLM. And this is automatically mapped to different services that you might want to use here. One thing I can do live that I promised before I could do, service goes to Anthropic, which is another one I have a key for. And let's pick an Automatic model because I never remember, I never remember what, what's what. The models are called the full name. So if I run this, We may have problem with the panel here. This is OpenAI for saying anthropic, but you know, point, point is it's quite easy to specify a different service here. This works with the connection with support. Like if I do something like this, it's not going to work. Now how do you, if you really like your settings, you can set this $LLM evaluator that will basically make this configuration will be the default setting for LLM evaluator in every call. So if I run this and I run this, you see, I'm not specifying the option here, but it's still obeying that. And of course if you set it automatic, it's just going to be reset. Now we talked a bit before about prompting, so prompting is very important. So no wonder there's extra functionality to handle that more easily. Now one nice way to use package prompt is the LLM prompt function, which is backed by an LM prompt repository, which is like our data repository. And the function repository and neural network repository is something that people can contribute to and it holds a collection of prompts that are hopefully well crafted. They're typically reviewed. So for example, here I'm asking what's at the center of the earth? And I'm getting a pseudo, hopefully not so pseudo scientific answer. But let's say the disease is too complicated and I'm using LLMs to generate content for kids or people that don't know much about science. So instead of writing manually, please keep the answer simple, blah blah blah, I can just modify the evaluator with, you know, this prompt parameter is prompts because it can be more than one, of course. And I'm saying, you know, use explain icon 5. This is taken from a Reddit board as an acronym. And now the answer is very different. It's quite simple. And of course, you know, as I said, you can mix and match. So here I'm saying answer all requests in Spanish and use this, this prompt here. And I can actually show you what this prompt is here. So this is, if I click on this, I'm going on the web. This is quite simple. Answer question is but if it's a five years old child. So this is more for self documenting code than actually the difficulty of typing. But these can be arbitrary, large and complex and give complex instructions. Now here, for example, I'm saying make a plot of the sin function and love Python. So in this case it's giving me a quite long. I mean, this is short. I shortened this, but again, actually. I think I broke something in my system while playing with this. Let me redo this. I should have started from a clean version. There it is. So this last time I run this, this was a pretty long essay about installing matplotlib and writing some stuff. But I see that you're working here, so you want something that you can use directly in the notebook. So I'm using the code writer prompt, which is priming DLLM to answer with Wolfram language. And then you get something which is immediate usage here. Now, speaking of interaction with the Wolfram language, a very, very useful thing that you can do with LLMs is go beyond what information they have in the training sets and actually have them prompt live, you know, ask live for new information. Basically function calling. This is also called tool calling. We do support tools with a tool parameter in the evaluator. So for example, here I'm saying check if. Let's run this without. Without everything here. Okay, let's see. Divisibility, check, we proceed. Blah, blah, blah. Is it not a prime number? Okay, so it runs a bunch of things. Actually I should try with the prime number and see if it finds that it's a prime number to check the reasoning. But a much faster way to do this and be sure is just define a tool in this call. I call it checkprime. DLM uses the name of a tool to understand when to call it, which takes an integer or actually something called integer. I should actually make this an integer, like force it to be an integer. Probably is like this way, hopefully. And there it is. Now in this case it's a sureable check. And now we can check. I use all here as a second argument of LM synthesis so I get all the properties. And you can actually check here that there was a tool record. This is our model agnostic representation of tool requests. And you see this was. This was done by a string. So the model asks for if you call the check prime tool with this string value. And we answer with this. This thing was generated by the result of prime Q. And this is very good because I would trust prime q more than LLM abstract reasoning. And again, there's no 100% guarantee because the LLM could ignore the fact that we say false here, but is a lot better than just relying on the intrinsic knowledge of your len. Now you can use tool and prompting together to get interesting things. For example, here I'm saying create a new document with a Plot of a SIN function and I'm using the code writer to just get only code. I'm actually making it just adding a system prompt here to make sure it uses a tool and then providing this is from our tool repository, a documentation lookup tool. And this way even if the LLM doesn't know about the Wolfram language, it can try to search the documentation for stuff. There's a slightly better way to do this, but I'm going to show it later and now I get this. So if I do this, it's going to create. Oh, there it is. So create a new notebook with the same function inside. Okay, now let's go to something which is entirely new in 14.1, which is retrieval, augmented generation or rag. Now the point of this is not possible. It's not always possible to know beforehand what's the best content or tools that you can supply to an LLM. So one thing you can do is you can augment the prompt on the fly based on the conversation to give extra context to the LLM. Now the way we have packaged this for 14.1 is through LLM prompt generator. And actually, let me give you an example which is completely devoid of any interaction with the input, which is just decorating the input with. With a function that doesn't take any argument, which is basically telling the LLM, what's my cloud account name, what's the date and what's my city? My city is actually not Champagne, but I'm on VPN with the office, so I'm in Italy right now, so the time will reflect that. And here say hi there, how can you help me? And I prompt with always greet the person you're talking to. Of course you can imagine this is arbitrary, complex in terms of customer interaction and stuff like that. And if I run this, let's see if it says the same thing or a similar thing there. Okay, now it's not really greeting me, you see, it's not really respecting that much. Okay, I don't know why it decided not to help me. But before it was actually the model was more. More. I mean, as you see, I mean, I'm providing this information regardless. The model can choose to ignore it. So here it didn't choose to ignore it and says, hello, Julia Cindrini, how can I assist you? Is there something specifically looking for in Champagne? And this is because what I gave, I can just evaluate this generator outside is. I mean, with an arbitrary input it doesn't really matter. And you see, this is input independent, but it will provide this information. Now let's add more meat to this with a prompt generator which is actually doing something with the input. So in this case I'm saying this is generating a prompt and it says check for the input to be longer than 100 characters and only answer in that case. And then the input length is. So this again, this is very simple function for purposes, but you can run this. And now it says I'm here to help which inputs are longer than 100 characters. So because I just input the sky is blue, sometimes it's less accommodating. Okay, can you provide more details or expand your input? Yeah, here we are kind of leaving to the lamp how to deal with not answering. But I mean you could modify this if less answer with failed or stuff like that. Now let's see another kind of more interesting use case. You could. You could define special tokens, for example, in your conversation. So here am I using our string functionality to say that if we start the string of the input with and some letters and like a space or something, then I would grab this and add it to talk like a. Talk like a with B. Default was one of our old examples like a pirate. So here I'm defining this generator and then saying owboy, hi there, what you're doing? And let's rerun it. Okay, so now it's. I mean, if I don't run this, it should talk like a pirate. Ahoy is, you know, pirates are very fond of ahoy.

And.

Well, it's about me. So I can probably say about something like that. Okay, nice. This is talking in Italian, which is definitely like an Italian. Now this is nice and everything, but how can you make it interact with context in a more interesting way? Also new in 14.1 we release semantic search and vector database functionality. So in a second I will explain what this means, but the point is it will basically generate context in the prompt based on the semantic sense of what you write. So here we are doing something very literal like parsing the string for markers or evaluating functions of the string. Here we are trying to look at the content of a string semantically and provide extra prompts that are giving nice context to the LLM. So here I'm saying create a new document with a plot of the sin function. So you can imagine this is kind of enriched version of a documentation lookup is we are not leaving to the LLM to decide if or not and how to call a tool. But we are. I mean, here I'm using the code writer prompt, but I'm also Using LMP prompt generator directly on the semantic search index. Again, more on this in a second that is being built is basically semantically indexing the Wolfram documentation. And now the LM knows about functions, probably a function, and how to use them that it probably wouldn't know about because it gets relevant snippets from the documentation. So in this case it's giving me this document notebook textile. I also don't know how to do this without this stuff because I seldom use this thing. And if I evaluate this, this is actually creating this thing. And probably if I click here, yeah, it will open a new window. Now what it is that is powering this machine, let's look quickly from the ground up. So I mentioned vector databases. So the task of finding similar content. Almost everything in computer science is solved in the numerical domain. So it's immediately mapped to the task of finding vectors similar to other vectors. And I mean you can say something that you can do with nearest. I define a list of vectors and I use nearest on the list to find the most similar vector 233, which is this one in this case. And that's correct. But this would quickly get out of control if you start having tens of thousands, millions, billions of vectors you have to search which are very long. And sometimes you are not even interested in having exact matches. You are interested in something that is roughly similar. So this is the point of vector databases. You can create one with create vector database object just by putting the vectors in and then you can do the database search. So in this case it's exactly the same. The approximation doesn't really kick in with few elements. But you can immediately start customizing this saying use 32 bit reals like half precision, sorry, single precision to be faster and use cosine distance instead of Euclidean distance, which is the default. So now I get a different result which is the closest in angle. And you can imagine that you save half the memory doing this. And you can. And we're supporting integer types as well and other things. So we are going to constantly work on improving this for the upcoming releases. But this is a kind of solid foundation to start semantically looking for things in the large databases. Of course you can have some conveniences baked in, like what if I want to, I know how to represent stuff as vectors, Then you can use feature extractor. I mean, I'm using image features, but you can put an arbitrary function and I'm putting a bunch of images that I'll then store as vectors. And then I directly search with images here So I do a database search with this image and this closer squirrel, and this is squirrel to monkey. I don't know if it's too small here, but. So now we need to wrap this up. So there's one last bit that we need on the stack is how do you go actually from text to vector databases? This is what our semantic serve functionality is taken care of. So the idea here is that semantic serve functionality takes text splits, it embeds that in vectors. So it semantically creates. It creates a vector that semantically represents the text and then it performs the matching. So here, for example, I'm saying create a semantic search index using Alice in Wonderland, which is easy to get in our resource data. And then I'm saying search for something matching a sport game and giving only three items. So here I'm just framing them to make it easier for you. But here talking about boxing, and here it's talking about playing again. And the third one, okay, what is a calcus race? And here is also explaining what is the calcus race. So now this is going through the book and analyzing these chunks. So you can imagine this is automatic chunking, but you can also chunk it yourself if you know good semantic blocks, semantically coherent blocks. And the idea is matching the semantic context of these with this. So if I say like an animal, I probably have to recreate things here. Okay, yeah, mouse, deer, for example. Yeah, turn into a pig. What is this about a serpent? Okay, I mean, there's probably more. Oh, there's a pigeon here. But yeah, I'm just getting three. So you can get more if you need to. So yeah, basically LLM prompt generator can very conveniently wrap one of these indexes that you create to kind of inject the context. So I could ask questions to a DLM about Alice. And the idea is that the LLM gets this stuff and can create crafty answer. Taking this into account now very quickly, I promise you some extra information about connections. So we've added several new APIs to the system. So we have here I'm showing an example with together AI because I have a safe connection. And so here I'm not using this using the wrapper. I'm directly using the service connection framework to create a connection to an LLM and then use one of the requests that we support, like chat, for example, which is what is actually used by chat execute and stuff. So this is lower level, but you can build up stuff with this. And for example, we have embeddings with together AI. And so here I'm Creating and embedding using as input the string generated both, which is giving me a real 32 numeric array and it's actually studying also which model is using to do that. So now this is a complete list of what you added or updated. I mean Anthropic and OpenAI were there before. We also had Google Poem which was retired by Google and was replaced by Google Gemini. So we have added that and there's a few things coming up, I'm just going to mention them. We are going to launch soon a subscription service so you can kind of enable every LLM based functionality without having an account with any external entity and something which is coming in 14.2 which is run your local llabs. So here, I mean don't copy paste this because it's not going to work on any of your builds. It doesn't even work on my build because I'm using 14.1. But this is an example of how it could look like running something locally. So this takes place entirely on your computer. So of course first low computer would take some time but I think this is very nice to be able to do this and use the whole stack that we have with these local things. So I'm putting some links here and well thanks for your attention and yeah, feel free to put questions in the in the chat. Okay, now after me there's going to be Shadi speaking about improvement in audio video processing. So Shadi, you have a floor.

Hello, my name is Shadi Ashnai and I'll be Talking about new Inversion 14 updates in video on audio and image computation. Let's start with some video updates. There has been quite a bit of quality of life improvements in version 14.1 to begin with. I would like to mention that I would like to remind that FFMPEG is one of the key factors in our video computation capabilities. We have always had a version of FFMPEG installed and shipped with Wolfram language. However, because of so many patent issues the Wolfram language would benefit greatly from user installed. FFmpeg version 14.1 added support for adds support for the latest version of ffmpeg being version 7 and there has been a great improvement in installing FFMPEG for users on macOS basically through some dialogues. Next thing that I would like to mention and it's I think of quite a bit of interest and importance version 14.1 up until version 14.0 or up until version 14.1 as of in 14.0 as well video objects didn't have image resolution option unlike image objects that did have image resolution. The way image resolution works is given the fact that most people are on higher DPIs, the option is taken into account in displaying the image or as we are adding it in 14.1, the video or decides how large the objects should be displayed in the environment. In 14.0, if we had a plot of a sign, this is a graphics graphics objects. Let me quickly do something like let me grab this and say if I rasterize this plot, it would still show this object as large as the graphics was showing. If I look at options of this image that I've just created, there's an image resolution 144 which corresponds to the resolution of the screen. I'm evaluating this on. In 14.0, if we pass the same expression to a video generator function like video generator, we have more video creation generation functions other than video generator, but any of those, including video generator itself, all of a sudden the video shows twice as large. So in order to to make it look as crisp as the one above in rasterizeofplot or the plot itself, one would have to display it in half the size. Now, in version 14.1, we're adding the image resolution object to the video functions. So any function that's generating a video from graphics or images that are rasterized to the higher resolution, they are being displayed at the correct expected size. So that's our image resolution option set to the same 144. Now, it's important to note that video files, unlike image files, do not have a metadata for image resolution. So once we're looking at the file underlying this video object, there is no notion of image resolution. If you open it in an external video player, you're going to see it as large as you would have seen it in Fortino. However, as long as you are within Wolfram language, we are at least respecting the resolution that the images or the frames in that video were generated. As. A good utility function added in 14.1 is video summary plot. Basically, we're allowing for equivalent display of video content and audio content in a video. Here I'm taking an example video. A video summary plot shows a given number of frames and a summary of the audio plot. One can tweak how the audio plot is shown or how many frames are shown. For instance, the default is the same as a plotting scheme called stacked, but one can show them in a row. Probably a lot more of these can be imagined. We may add more frame stacking in the video summary plot. And as for audio, one can decide not to show it. And as I said, there are more options for both how the audio is being plotted in the summary plot and how the video frames are being plotted. There has been quite a bit of updates in subtitle Handling in 14.1 as one of the more noticeable updates, video extract tracks is being updated with a few more subtitle properties. One of them which is I think is going to be a really handy one, is a subtitle string which is going to give us back the whole subtitle without the timestamps and as a single string object. One can ask for subtitle string rules to basically get time intervals associated and then one can get subtitles which are completely formatted subtitles as they appear in the video file Here I had to change all of the colors to orange because subtitles of this video are white and white on our white background. Notebooks wouldn't show. We have more of the subtitle gymnastics in the works. Hopefully soon that's going to be something we're going to release as well. Another quality of life improvements when we are generating videos and a lot of the times those could be longer videos, we are showing the generated frame in the progress monitor. I think in general Wolfram language has come a long way of reporting updates in the progress reports, more functions supported. And here I'm just doing some hue shifting on a sample image and I'm generating a video based on the hue shifted frames. And it's really nice and handy to see what is going into my video. Especially given the fact that video generation a lot of the times takes longer. It's important for us to get a sneak peek of what's being generated so that if anything is wrong, we can stop it and correct our program. Okay, some of the small things I just mentioned, but some of the bigger things that have been added to 14.1 a good set of updates in video generation in general, constant video is simple from a single image to whatever duration one would like to specify. One of the good use cases, for instance, is when we need to have some title images. So we want to join one video with another. In between we want to say what's happening. So something as little as I have one second of the same text, basically nothing fancy. Another good use case of that is we have is when we have an audio and we would like to stick it on top of a still image and basically post it on YouTube or just use it as a video in any platform where a video is more attractive. And then in that case as well we Have a constant video, same duration as audio and, and basically creating. And let me make sure I'm sharing my sound as well. Okay, let's play this again just to make sure audio comes through as well. Okay, so that was our simple constant video generation. A bigger, much bigger function with a potentially much bigger use as well is manipulate Video manipulate in Wolfram language is a construct of version six, I would say, what, over 17 years, perhaps 16, 17 years of life in the Wolfram language, there are I think 12,000 plus manipulates just alone in demonstrations project. And people have created a lot of those in their social posts and experiments and whatnot. The goal of this project manipulate video is to get any of these manipulate objects, some of which have very many controls and define a user specific set of way of manipulating different controls. And how we are talking about it internally as well is to make sure that we can define experiments that one would like to show with each of these manipulates and basically record them into a video. Simple examples here I have a simple manipulate with a single variable, a disk with a changing hue. Manipulate video of that same expression changing hue. Just to save time, I've evaluated all of these. Basically just creates a simple linear evolution of that one control. Nothing really exciting or different that can be done. Most of the time once we have a single variable, we would like to just tweak that variable from minimum to maximum. And we have already had a function animation video for that, which was basically simply doing the same thing. However, still, in that case of a single variable, one may want to define some kind of a pause or a change of speed. For instance, here I'll stay at zero at time zero, then I'll go to 0.5 at time one, pause on 0.5 until time three and then keep moving until time four. So here that's my pause from second one to second three and then we change all the way up to hu going forward. So this is something that animation video was not able to do. Now manipulate video is capable of defining in this kind of short and simple syntax how one would like to change the variables over time. Now let's look at some multivariable manipulates. Here I have the similar manipulate with a second slider where the first one still changes the hue, the second one changes the size of my disk. Now even as a human, I would say I can change hue all the way back to the beginning, I can change s going down and then change this one again. So there are different types of experiments one can define with multiple variables. As a default Manipulate Video changes all controls at the same time simultaneously. So here the color is changing, the size is shrinking at the same time. However, similarly to our single variable Manipulate video, we one can specify exactly how the controls should change over time. In this particular case, I'm having the same behavior as I defined above for my hue and then for the size specification, I start from a specific size, go up at time two, I'm at size four, at time four, I come back to size two, and then this is the growth of the disk to four and then going back to two, and the hue was changing specific to the way I specified it up there. Okay, so let's manipulate video with just a second argument specifying how the controls are changing. We're also adding a keyframe actions, so one can think about keyframe actions. This is an option both to manipulate and manipulate video. And one can think about it as bookmarks, which have existed in Manipulate for years and versions. But now the keyframe actions are basically timed actions, timed settings for all the controls. If I quickly open this previous example I was showing and not this one. Oh, I forgot to show this one actually. So if I go back. Never mind, let me close this. Actually, it might be better. Let's actually just look at the keyframe actions and I could go back to the example from Demonstrations. So if I look at how we defined how these controls change over time, up here I was focused on a single variable at a time, and I specified at what time what value that control is getting for one variable at a time. And then the second variable. Now if I look at keyframe actions that I was talking about, keyframe actions is now specifying it in a different order. Keyframe actions is focused on time, and for each given time, it provides all the settings for different controls. So here I'm looking at this exact same manipulate object that I had before, and I can actually go ahead and say in the manipulate object, go ahead and say keyframe actions goes to true. And just like I was able to say bookmarks, specify bookmarks for different times, now I can say add keyframes, and I can add this one as my initial setting. I can then go to all the way H goes to one, add another setting, I can name this, or I can just say add, and then I can go ahead and specify another setting, add two keyframes, and I'm done. And just like we could animate bookmarks, I can now animate my keyframes. This one pushes H all the way to the largest value and then s to the largest Value and so forth. Now, assuming I have this manipulate, something like manipulate video is going to respect the setting of keyframe actions and create the manipulate based on how we have specified these controls to change. However, both manipulate and manipulate video can take programmatically keyframe actions as an option. So just like I had specified that with with the gui I can specify my keyframe actions programmatically with at different times. So here, as I said, I'm going to be focused on given times. I say at time 0, I want H to be 0s to be 1. At time 1 I want to have these different values for H at time 2 H and S. In fact, at time 2 I want these different values for S HNs. And then if I create this manipulate and I just look at the keyframes, I already have these, these obviously again could be named. If I say animate keyframes, I am actually following that keyframe actions that was specified above. And if I do something like manipulate video of M again, exactly the settings of keyframe actions is respected. So this is the equivalent of animate keyframes and I can pass that keyframe option specifically to manipulate video. And again the same video is going to come out. Okay, so these were some simple nice functions, nice examples. But let me actually, because I mentioned demonstrations, let me actually show this example as well. Inspired by a demonstration project like this one, where 3D rotations were explained using different controls and things like that, we took this demonstration here. I can in fact just import the demonstration in the Wolfram language and play with it and show different demonstrations, different normals there, changing the origin of rotation and so forth. So there were different kinds of demonstrations or experiments that were being explained here in that demonstration. Here I have created a much simpler example or form of that demonstration only basically three controls so that we can explain how everything is going to map to manipulate video. So here's my angle of rotation. And then the center of rotation is changing only at X and Y. And for instance, if I change both of these kind of to the center and then rotate around, I see how that's happening. Now here with the three controls that I have, I basically this is the same expression passed to manipulate. Now I'm passing this to manipulate video and then specifying basically ranges for my theta PX and py. And then I'm specifying a set of controls or the set of sequence of changes in controls for my theta PX and py. And if I just evaluated this for the sake of time, if I look at what that has happened I basically put PXPY to 00 and set the rotation to minus 2 PI. Then I stick to that 00, basically do a full rotation around the center. Then I change PX and PY PXPY to 10 instead of 00 and then do another rotation, then move to center, do another rotation. And basically that's what I've created in this video. So rotation around, rotation around and then the last one, rotation around, center. So this kind of a language is easier compared to even I would say, interactive specification of keyframes, but also easier than specifying px PY theta changes over time one at a time. Because I know I would like to have a full rotation in the first two seconds, another rotation in the next two seconds or three seconds and then another one until the end. So it's kind of time centric, time oriented and would be nicer and easier in my opinion to specify it like that. But in some other scenarios, in fact the other way around might be easier. Okay, so we covered manipulate video keyframe actions and updates to manipulate. We have however more functions in 14.1 that allow for video generation. One function that's really nice and fun and probably a very powerful one is fold. Video frame fold basically takes operates basically like fold list. Let's look at just that fold list example of a random function F on range of 5. Basically we return the first element of the list, then apply F to first and second, then apply the result of, then apply F to the result of the previous evaluation and the next element in the list and onward. Same thing is happening in video frame fold. We are applying F to the first two frame frames, goes to the first frame of the output, then get that result from the first frame, the first two frame evaluation and use the next frame in the input video to compute the next frame in the result and so forth. Examples that we have here for instance is some kind of a blending over time. Here I have some cars moving and basically I'm blending frames with each other with some small blending factor. This is basically creating some kind of a motion blur. Or the more fun example we have here we do some kind of a remove background on that frame. Nice because it has a nice background that's easy to separate. With the salient method we remove background, we compose the foreground on top of the previously composed set of result. So yeah, basically the movement of that person batting is going to show up as the movement is happening and the background is removed for all of those frames. So these are two examples. A whole lot more can be Done with video frame fold. Now, in the same spirit, we thought it would be really nice and useful to be able to sew videos or sew video frames that, that are of any kind of interest and then reap them in a function like reap video. So again, very similar to sow and reap. Sow video and reap video are basically saying, as I'm evaluating something, if I come to an exciting frame, I can sew it. I would sew it. So I would wrap a sew video around it and then reap video is capable of reaping those sewn frames and put them in the output. So in this schematic view, so video of that one frame is going to the output. So video of that one interesting frame comma n is basically repeated n times and so forth. A simple example. So video of one frame, not. So in a sequence of expressions that I have, the first frame, the yellow frame and the blue frame are shown. And so those are the ones that go into rip video. And if I extract frames, those are the ones that come back. Same with these other simple examples. Again, if we just saw a frame, one frame goes out. If we saw that frame five times, five frames go out. And one can specify for how long that sewn video should be repeated in the reaped video. This is, for instance, very interesting for cases where nesting operations are happening. So for this specific case, I'm applying pruning to that image repeatedly. And then frames are being sewn. And I'm only repeating this based on a conditional nesting operation where I would like to say, whenever the result of the pruning ended up being static and not changing, stop. And while that's happening, all the frames are being reeked. So let me evaluate this. And that's showing me the result of basically pruning. And it ends, the operation ends when the last two frames are exactly the same. No more nesting. And a lot of these functional programs and functions that we have in the system can be used along with so video reap video in order to create interesting experiments and operations. Another example that I'd like to show here is conditional sewing. Here I have a video of a person doing yoga. It could be anything else. And I'm basically sewing videos in a condition that I find any face in the video. And if I quickly separate these show the video itself, there are some poses that this person is, is at where no face is obviously detected. But with, with that condition, this video is basically only giving me the snippet of the video where the face of the person has been detected. And again, a lot of these kind of conditional things could could be imagined security system if a person is detected only only get that bit and piece of the video extracted. One other thing moving on from video generation that has happened in 14.1 is we basically extended many, many of our audio functions to support video directly.1 up until 14.1, one could have extracted video object, sorry, audio tracks from video objects and applied any of these processing and analysis functions. But it basically just makes sense to apply them directly. Why not? So we have extended audio functions, things like plot, audio plot, spectrogram, periodogram, which essentially just extract the audio object and do some kind of analysis on it, return something else. And as well as things like audio delay, audio pitch shift which are being applied to the audio track. Now all of those can directly happen on the video object. So here I have a video object of our known example data Wolfensh and I amplify the audio by a factor of five.

And louder audio.

Or I can basically have the video object focused on the second audio track, which is a spoken track. And then if I do spectrogram, that's what I get back. Just to maybe clarify, if I hadn't added that audio track selection. By default, these audio functions use the first available audio track. So this is the chirp which we were amplifying up there in Audio Amplify. Then other things like audio identify. Again this is applied to the first audio track and this is identifying that track as a chirp, identifying the second track as speech, and then the third track, which is a German speech as well, we can pass it to language identify and it's hopefully coming back as German language and so forth. So basically all of these functions listed above are going to be supporting video objects in version 14.1. Sneak peek into what's coming next. In future versions we're doing a similar development for more of the image operations. In one of the earlier releases we had operations like image resize, image crop and things like that, except video objects directly. Hopefully in the next release or one of the near future releases, we are basically going to have other analysis functions like find faces and more work on video objects directly as well. Okay, so to a good extent that graphs are updates in video, however, updates to speech recognition because speech recognize and alike also directly support video, also are improvements to video object. There has been a great update to speech recognize from what we had before, which was an older neural network to a whisper model, which is a great amazing model. And now if. And I'll iterate through what are the updates, what are the qualities or improvements given this upgrade in 14.1 here I have a 16 minute long. This is a speech of I have.

The pleasure to present to you Dr. Martin Luther King JR.

This is the famous Dr. Martin Luther King speech. Duration of that speech is 16 minutes. And I can do speech recognize of that 16 minute speech in basically a rather short amount of time, which is a great improvement compared to what we have before. What we had before, which was almost linear timing. It would have taken potentially close to 16 minutes or so to compute that speech recognition. This is now just under 20 seconds and the speech is there. So speed compared to real time is significantly lower. We used to use deep speech too, and now we're at whisper. So every minute of an audio can be transcribed in a second. Roughly punctuations and capitalizations are coming. So that's our old result compared to new result. This is a lot more parsable. Non spoken, non speech tokens are being recognized and returned by the new speech Recognize. Here I have a shorter snippet of the audio and things like applause are returned from Speech Recognize. If for any application we're not interested in those non speech speech tokens, one can say non speech replacement to empty string and those are removed. And that's again a comparison to the older result. Timestamps are something that are approximate but perhaps very useful as well. So I'm here splitting speech recognized into a level of a sentence and that's also that second argument and the third argument to speech recognize are new in 14.1 as well. I can say I want to split it by sentence word and I want to get back intervals, audio objects, text or whatever. Here I'm splitting by sentence and I have a guess for when that sentence is happening in time. And I can get.

I have the pleasure to present to you Dr. Martin Luther King.

Well, I just realized there was a typo or misrecognition there as well. But. But again all of these are things that weren't available before. And now for many applications these could potentially be very meaningful and useful. There are also a lot of languages supported with whisper. Qualities really vary by language, but a large list of languages are claimed to be supported. They are supported, but as I said, based on our tests, quality really vary per language. And it's probably worth mentioning that at least now we're geared up for any kind of quick updates to speech recognize with these newer models that are out there and we'll absolutely use them and upgrade them. Language identification in two places we have this spoken language classifier built in classifier as well as language identify as I already demoed it before. These are all now available directly on audio and video as well. So language identify of a speech that was above English, same as classify spoken language and it can be directly applied to something like a video. These are exactly the same examples I demoed in the previous slide. Now we have a new function using these capabilities called video transcribe. If I have a video object with a spoken audio as a track, I can basically transcribe it on the fly and add it back as a translated sorry, as a transcribed new subtitle track. So that Bullfinch automatically has had two subtitle tracks, the English and German and now that track three is what's being added by video transcribed to that. So because I used track two, I go ahead and show track two of the subtitle and then I switch back to the newly transcribed subtitle or newly the transcribed audio track which is added as this track 3 subtitle. So obviously the text could be slightly different than the exact subtitle that has been provided in the audio, which is supposedly the exact text of the audio. But and in this particular case probably you notice the style is different. But apart from that, this is a neat addition as well. In 14.1. I talked earlier in my presentation about subtitle track updates. Here is a quick example about adding transcribed or translated subtitle tracks into a video object. In this particular case, I'm basically doing something along the lines of adding a translated subtitle track back into a video object. The Bullfinch video is there. I'm basically getting the first subtitle track with the timestamps out. I'm running that through text translation to Japanese. I get Japanese translation back corresponds to the same times as were extracted from subtitles. And I'll put that back in into another track if I scroll down. That's now the new track three, which is the Japanese subtitles back into the video object. One can imagine if a subtitle didn't exist at all in this case it existed. So I use that to pass to text translation. But that same workflow can happen with speech recognition. So I can first get these intervals and texts for the recognized speech in a specific audio track translated to as many languages, stick them back into the video object for uses on YouTube or any other platform. I would like to finish or before I finish, I would like to mention some incremental, small incremental updates in the image computation Domain as well. We have upgraded our OCR functions basically text recognize and find image text on macOS only so far to utilize the operating system APIs. This has given us a lot of improvements as well. I'll go through some of the improvements and quickly as well. Let me just close all of them for reducing noise. Okay, so here I have basically a screenshot of a paper that's low quality, basically an index of a book, low quality and so forth. And if I use the combined diff view, this is also an amazing 14.1 feature of the Alt text recognize and the new text recognized. There is a lot of the new recognition that was missing from the previous text recognition engines that we had. So as in that example above on macros, operating system wins big against the old Tesseract engine that we had. When we have low resolution images, handwritten fonts and texts. Again, we're winning big using the operating system compared to the old engine. Natural scenes a big win there as well for many of these images. However, we did have our own trained networks for natural scene text recognition and detection. I didn't do the comparison there. Classroom projector, one of those use cases that a lot of people a lot of times want to just do a quick picture of the projector or something like that and then see this is a lot of the times. Poor lighting, low resolution, blurry and whatnot. And again Mac OS wins in that case mixed printed and natural scene. So that's a screenshot of our text recognized ref page itself kind of on par. I would say in some of these cases Operating system was confused more than Tesseract. And that's I think one of the cases where at least the result of the old methods is going to be more reliable, sometimes better. So we'll still have everything available. And then in the cases of code that I tried and this might get improved. Obviously the good thing about operating systems are that they're updating all the time. So we'll, we'll just get updates that come with the operating system as well. But in that specific case and some other tests that I did, I would say also Tesseract wins over operating system as well. This is. These are usually made for more of a natural scene settings compared to printed settings that Tesseract was really fine tuned for. The last thing I'm going to talk about before we close this session is that people segmentation has been one of the things that is very important for many applications for foreground background removal. And then we are basically in this small update utilizing a network in the same function, super function remove background to target people in the foreground. And and this is one of the functions that can apply to a single image. Or here I have a video with people in it and the same network can be applied to different frames of a video object. That's about it for today. We'll close it and hopefully there's going to be enough of updates in 14.1 for all of you to put your hands on it and and play with it. And there is going to be more in the upcoming releases. Thank you very much. Okay, perfect. So a lot of the questions that came in chat thanks to everyone here in the background got answered. Julio, you mentioned potentially you would be interested in interested in reiterating some of the maybe common questions or more important questions live.

Yeah, I mean a couple of things I can mention. Let me share my screen here. Let's share screen. Share content. Screen. There it is. Anti screen. There we are. Okay, so I mean let's get to a new notebook. So people were asking. I was a bit fast on the vector based functionality and semantic search. So of course there's more tweet that would have shown you can do maybe I have to make this a lot bigger. You can of course use create semantics search index on text 1, text 2, 3 file URL and things like this and this will maybe all go to the same index and of course these indexes can grow. So we have update semantic search index which can take if this is index can take index and you can put more things in this. So I mean people are worried of not being able to do this. Of course this is, this is supported. This is backed by a powerful C library in backend which is called usearch and it's very nice, it's very, very fast and so it has file support. The index can grow easily beyond what's your memory limit. You can use not here, of course here the generated asset location option to place this like in a file. You can specify a file, a folder where the file will be created, a cloud object, a local object. There's a lot of options where to store the index or the factory base. They offer the same options and I guess these were two points. One other point that wasn't the question but might be useful. You can of course tag each one of these with things like let me do it here. This can be tag 1, tag 2, tag 3, tag 4 and these can be expressions or they can be things like associations. So you could say that date is now all things like that. Oops. Right vowel. And then when you do semantic search, I don't have any interesting one loaded by now, but I could open the documentation and bring up some. When you do semantic search now you can. Sorry for the scrolling, just going right to the point here I'm tagging for example these. I'm creating a smart search with wolfram and URL HTTPs wolfram like www.wolfram.com and here I'm tagging with website Wikipedia website Wolfram and then I can use a function to filter my search elements. And this is, you know, this respects the number of items that you want. So if you want 10 elements, bring the 10 closest Wolfram element and ignore the Wikipedia one. And of course you can imagine that you can mix match this with all our programming functionality. You can use data arithmetics, data intervals, string content to tag and retrieve specific elements. So you mix semantic search functionality with like normal search function function based functionality. I think this is worth mentioning because it's something that is not using the LLM interaction, automatic LLM interaction that I've presented, but it might be worth it to use in your application and okay, I haven't seen much other answers that were questions that were unanswered. So I guess that's it for me probably. Shadi, if you have something that you want to.

There wasn't anything specific that I would like to answer live. I think I'll answer all answers in chat were comprehensive. Thanks a lot for attending today's webinar. Thank you to our behind the scenes staff. Please take a minute to answer the short survey that will appear in your browser. As you exit the survey, include a question about requesting a certificate of attendance from today's event. I know one person asked about that in chat as well. We hope you'll enjoy. You have enjoyed attending this webinar series. This is the last one. I would like to encourage all of you to let us know how you use all of these new features. Perhaps Wolfram community is going to be the best one or through other channels. Also let us know if you have any suggestions just. Just like you did here in chat in any other way you can reach us or if you find bugs and problems also let us know. Thanks a lot.