Get Started with AI: A Beginner's Guide to Automated Classification, Predictions and Computer Vision

We'll be starting just as soon as we've got the YouTube live channel up and running as well. We're having a few technical issues apparently. Okay, great. It looks like we're all connected. So welcome everybody to this webinar. I'm John McLune. I lead the technical services team at Wolfram Research here in Europe, where we do projects for customers using Wolfram technology, very often data science projects. So today I'm going to be talking about getting started with AI. The idea is that it's going to be an introductory session that will walk through some of the very basic concepts from tools that happen to use AI within the Wolfram language, right up to the modern generative AI with things like ChatGPT. And I'm going to try and cover that in the space of an hour. You can ask questions. Those of you who are watching on BigMARKA, you can ask through the chat panel on the right hand side. If you're watching on YouTube live, then just post comments into the, into the comment section below the video. I will probably take all of those questions at the end of the presentation, which will take probably about an hour. If I see the questions beforehand, I might take them early, but I'm not very good at watching the question channel and talking at the same time. So let's share a screen here so you can see what I'm seeing and we'll get started. So. Right. Hopefully that's sharing now. So what we're going to cover in this session is some, some complete basics of machine learning. What even, what is machine learning and how does that lead to AI type decisions? We're going to go through the basics of automated machine learning in the Wolfram language and we also going to look at the process by which that goes through how you extract features, classify, predict and test what you've done. And then we'll get into some more advanced neural networks and at the end we'll touch on generative AI and things like GPT. Now, as I said, this is metadata introduction, so some of you I know will be Wolfram language users already, many of you not. So I'm going to introduce a few basic concepts of machine learning. First of all, take types of tasks and then a little bit of basic syntax, just a minimum for you to be able to understand what's going on. So the first question is what do we even mean by machine learning and how that sort of gets muddled up with artificial intelligence? I mean, AI, of course, is a very vague phrase. What we've been doing for 30 years at Wolfram was called AI. Back in the early days when computers were able for the first time to do things like symbolic integration or solve equations in a way that humans used to do on pencil and paper, it was called AI. These days tends to be applied to things that seem to be fuzzy reasoning and linguistic type reasoning. But underlying all of those mechanisms are combinations of pure logical programming and machine learning. So the logical programming is all, in the end, human curated instructions that describe exactly how processes should work. Algorithmic type, things like symbolic integration fit into that category. But machine learning covers those areas where we are data rich and understanding poor, where we have lots of examples of what we want, but we don't really quite know what the rules are. And that's really been the underpinning of all the modern view of AI in the last 10 or 20 years. So what that is is taking data and building a model. So where in the old days we might come up with an equation that describes the motion of planets and then use that model to predict the future, now we're simply going to take data and get the computer to come up with a model. So the idea is we have some data points, and at its very simplest level, you can see, imagine, fitting a curve to some points of data is the very simplest kind of machine learning, because the parameters of that curve had to be figured out by the computer, by the machine, from the data. In a much more modern rendition, we're taking things like images and then building a very complicated model, much more than an equational model, that then can predict the outcome of whether things, in this case, the images are pictures of dogs or cats. Now, machine learning breaks down into a whole collection of different classes. But here's the most important ones. Supervised learning is where you are giving the computer the supervision of knowing what, what is known as ground truth, what the correct answers are. So in my previous example here, if this was my input data that I'm providing to the computer, the fact that I have labeled that picture as a picture of a cat is my supervision. I'm telling it, this is an example of a cat, this is an example of a dog, and that's the human supervision. It's particularly useful, of course, when you have labeled data things like class recognition and predicting values, where we have examples of what happens that we want to replicate. Unsupervised learning is the same thing, but without the labels. So you are simply providing a collection of data, but you don't really know what the true answer is. You're simply trying to look for patterns. So a typical thing that goes on in that space is to look for clusters to say people who bought products like you also bought these other products. No human has said that you are a IT enthusiast middle aged man, but it's identified that you are part of a cohort that has similar, similar behavior. And that's, that's a form example of unsupervised learning. And then there's a higher level which is reinforcement learning, which is when you get the model to gain its own feedback from the environment. So notable examples have been things like T teaching computers to play go where rather than just feeding it millions of examples of go games, it is effectively playing itself and learning how to play through generating its own data for doing its own supervised and unsupervised learning. And it's basically just closing a feedback loop on the other two approaches. So some of the typical tasks we have in this are classification. Can we say what kind of a thing, what group it belongs to, like whether pictures are of one thing or another thing. Prediction is another important task which is given some inputs, can we make some numerical prediction, some quantitative prediction out. And that might be a single variable like this, where we're maybe predicting the height of a child based on age and sex. But it could be multiple values that we're predicting out. But the difference is one's discrete. There are fixed categories, cats and dogs or continuous categories that can be any value. Other examples are things like creating mappings. A common mapping is text translation where we can figure out how to take one space of data and map it to another space of data being the other language for the same phrase. We can predict future time series. We can try and predict things that don't look like they belong. So almost the opposite of classification and prediction, you're trying to spot outliers. We're doing things like data clustering to try and form groups and cohorts that might have similar behavior or could be treated in similar ways. And more modern things are things like chatbots, where I can have conversations with a computer and it's able to give me human like intelligent seeming responses. So these are all underlying the mechanism in all of these is machine learning. In the end, we're taking some data from the real world to learn a model. And we're putting some data in and we're getting some data out. And that data can be any kind of data. It can be very high dimensional, like images and video, or very simple things like numbers. Now before I'm going to show lots of examples to have code. But don't worry, all of the code examples are extremely simple. But I'm just going to give you a little bit of overview of the Wolfram technology so you are not distracted by the notation. I'm working in a notebook, so inputs and outputs are in the document here. I'm not going to be working from the command line or running scripts, so you'll just see outputs appear below inputs. We have within that technology stack for Wolf language all kinds of functionality that is different from machine learning, all kinds of computational language technology, but it is all accessed in the same way that we have commands that have a name. So if I'm adding 2 and 2, the command might be called plus and then there's arguments that are separated by commas and they have square brackets around them. So when you see something that has this name, square brackets, that's just a function in the Wolfram language and everything is a function. We will see data in a couple of forms. The curly brackets are a way of denoting a list of numbers. Higher dimensional data is just lists of lists, so there's no sort of limits to the depth. Here we've got a list of lists of lists. So we've got three dimensional data and it doesn't have to be rectangular, it can be ragged, like this example. And we also have a key value notation like a dictionary in Python or a struct in other languages where we have data which is not ordered like a list is, where we've got a first and second element, but it's labeled by a key. So the key here is name and the value is jim and the little brackets here denote that this is a key value pair. So with that you've got essentially all you need to do hopefully to understand the code that we'll see. So let's get into the tools that are provided within the Wolfram language that make this hopefully very accessible for you to be able to just get started on. If you want to follow along, you can download the notebook. The link is in the chat panel on the right hand side. If you don't have Wolfram language, you can either download a a trial version on your desktop or you can open a free cloud account@wolframcloud.com and access it through a web browser. And that's free of charge, although it's got some limits on memory and CPU time and retention of data you store in our cloud unless you subscribe. But either way you should be able to play around with this after this webinar free of charge. So There's a whole collection of commands. There's 5,000 commands in the Wolfram language. We're going to focus on a few dozen today. The most noticeable ones we're going to look at first of all are things like classify and Predict, which are the complete automated machine learning commands. Cluster, classify and find. Clusters are for finding clusters and we'll talk about the rest as we hit them and the language here. They're all very task oriented. We'll notice they're called things like classify not, not K nearest neighborhood. The algorithm is a detail, so we try and label things by what they're trying to do. So the name of the function is just key to understand the purpose. We'll also look at a much richer and more complicated neural network framework and we'll just lightly touch in on that for more expert users. So let's look at some quick examples of using some of these built in commands. First of all, the classify command actually already comes with some pre built machine learning classifiers. So you can use it with the, with the argument of the type of pre built classifier in this case a language detection classifier. And then the argument we're giving it is this string. What is this language? And we're asking for the probabilities of what's going to come out and it says the highest probability of that at 99.99% is English and actually force it to tell me some of these obscure probabilities. Let's tell it to give me the top three probabilities. And there's a tiny possibility it's a language I don't know called Tagalog. Or it could also be Afrikaans at an even smaller probability. So this is, it's making its class prediction. If I don't ask it for these extra bits of detail, it will simply make its prediction and say that looks like an English sentence. And there's about half a dozen of these standard built in classifiers within the classify command. Things like profanity detection, things like topic and name gender detection, things like this are pre built in. Mostly where this becomes useful though is when you want to build your own classifiers and we'll talk about that in a minute. At a higher level we have a purpose built image identification function that switches between various different, more advanced algorithms for identifying what's in pictures. So here I have two images that I'm using entering into the system. I'm going to say apply that image identify function to each of those images and we recognize a gray wolf and some key limes. Now the neural network framework that I mentioned is a way of being able to fetch arbitrary neural networks from a repository so we can do the same image detection task by a specific NET model that we've called Inception. It's I think a Google trained model and I've mislabeled these things. See what Google has to say about image one and two and it's given the same answer on the first one, but the second one I would guess has got that wrong when they look like lines to me. But maybe it knows better when these are unripe lemons. Other sort of high level commands we have built into language are things like image restyling. If you're doing image processing, being able to take say source images like the Mona Lisa and then style reference images and say restyle the Mona Lisa as if it was done by Kandinsky and it's taken the style of one and mapped it onto the other or Monet version of the same thing. Another high level set of useful machine learning things that are built into the language are to do with natural language processing. So those can be at quite structural level. So I can take a sentence here and decompose it into what's the noun, the subject, the object and separate sentences. Or at a much more component level where I can take. In this case I'm fetching the Wikipedia page on the moon, but I want to search that text for all of the things that are mentioned that are of type quantity and I want to figure out what those quantities look like. So it's using a machine learning content recognizer to find the phrases that look like things that are quantities like 10 meters or 100 years and then interpret those things in the Wolfram language representation of units. So we're not going to get the word meters, it's going to turn that into a stylized M automatically. And this text case works over everything that the WolframAlpha system knows about. So I could search that for cities that were mentioned in the text or countries or notable people or rivers if there were any mentioned on. I doubt there's many rivers mentioned on the Wikipedia article about the moon, but all of these entity concepts in the real world we can recognize those from text and hopefully this is going to not take too much longer. Here we go. So here's all the quantities that were mentioned on that page together with the units that they're in. And another high level thing that's built in is anomaly detection. And the idea of that is I can give it some data here in this case a set of colors and it's trying to figure out which doesn't look like it follows the pattern of the rest. And since these are all rather secondary muted pastely shades, it's decided that bright red and bright green stand out as anomalous. And this works even better if you can pre train on data that you know to be correct. So here I'm fetching. What am I doing with this one? I'm fetching some data from the MNIST data set to build an omnally detector function for characters. That's because the MNIST data contains pictures of handwritten digits. And now I can apply that pre trained detector to a set of example images. And it spotted the two things that don't look like they look like handwritten digits. One because it clearly isn't and the other one because it's so corrupted that it doesn't fit the quality of the rest of the data set. And of course these are retrainable. So let's talk about the process and if you're going to make your own, the bits that you need to understand the first thing which we're going to talk about actually in simple cases you will not have to concern yourself with. But it's important as things get more advanced to understand the basics of the concept. And that is here in these examples I've been jumping around all kinds of different things. So I've dealt with things like images, I've dealt with things like text, I've dealt with things like, well, colors and more images. So very different kinds of data. And the way that the machine learning frameworks that we've built all work is that you first of all have to convert those into a vector of numbers. And that vector of numbers is going to be the features that we are going to look for patterns in. Now if the thing is already in numerical form, then of course you're already there. The numbers are, or vector numbers is the features already. But very often we have to do some level of feature extraction in order to turn something. Into a vector that we can then process. And we deal with this at a high level with this feature extract command. There's a choice of algorithms behind and lots of details that you can control. But the basic idea of this is to take some samples of data. So here I've got a string in color and a date and we've got several examples of that. And it's going to examine those and convert those into a simple feature vector of three numbers to represent each. Now this is normally when you're doing machine learning you're dealing with quite large amounts of data, so proper patterns can be found. I've given it hardly any information here. And so it's done some rather naive things. It's decided that this is a phrase of some kind and it's mapped out to a number that's clearly one number is never going to represent the entire richness of. The human language. And in fact this number doesn't necessarily correspond to this. This number is the first feature of a set of three vectors that represents the whole thing. So it's taken that whole set and mapped that to some kind of space that is in three dimensions and then converts that to a number. Now these things start to become meaningful when you have larger amounts of data. But we accelerate the process when you have small data by leaning on some pre trained feature extractors that make these things make sense. And the idea is that feature extract can work over any kind of data. So if I give it this sort of key value data set, then here it's converted these things into a set of pairs of numbers, or I've taken these much more complex things of graphs and it's turned those each into a vector of. In this case it's decided six numbers sums up the number. Now it's kind of useful to imagine that these things represent something that is descriptive. So you could imagine, for example in this graph, maybe this first number is something like how dense it is. This would have a higher number number than this one. Or it's how pointy it is, or how connected it is, or how central the most central point is, or something that we might have words for. One of the things you have to accept fairly early with machine learning is that it's inventing its own concept. So while we might look at a picture and say that's a cat because it's got pointy ears. Pointy ears is our version of extracting a feature in its world. It's going to invent some concept which might, you know, best be described as zigzaggy splotchiness or, or kind of prickly jagginess or something. But it's a concept that it's invented that may not even have a word that we would even recognize when we see the patterns. And it's building this, this language for itself that it then ascribes numbers to. So these things are important, but not very easy to understand. So one of those ways we can understand them a little bit better sometimes is to try and visualize them, because sometimes that can give you a sense of what the extraction means I also want to show you what this looks like when it goes to a bit more scale. Let's do something now that's got a semi real set of data to it and I should run this straight away because it starts will take a minute to run. I'll apologize in advance here. I'm doing live web image searching, so if the Internet throws back anything inappropriate for me, I apologize in advance. But what I'm searching for here is three characters from the Guardians of the Galaxy franchise. Rocket Raccoon, Root or sorry, four characters. Gamora and Star Lord. And I'm doing a web image search to get thumbnail pictures just to keep the speed of this, the amount of data small. For this demo example, I'm going to get 20 images of each. So in a few seconds, hopefully we will have generated 80 fairly random images that match the image search. We might find that there are some things that are not correctly found from Google, that maybe instead of Rocket Raccoon from Guardians of the Galaxy, if there's a picture of raccoons in rockets, then they might come up. But if you have enough data, those kinds of things tend to average out. So I think we are done. Let's fetch those. So let's have a quick look at what we've got here. So here's our data set. There's Rocket Raccoon, some Groots and so on. So let's hide those again. We don't need to see those in this case. I am going to. Oh, actually this was meant to generate a random sample that I was going to show. But so here is my random sample of just randomly sampled a whole lot. So it's reordered them in order to be not in an obvious pattern order. And we're going to do feature extraction on those. Now in this case it's going to start doing not just how do I map that to some numbers, it's going to also start doing a certain amount of feature reduction. So we're going to accelerate this feature extraction with things we've learned from looking at millions of other images. So it knows to look for things that are kind of say pointy or square or round or, you know, things that turn out to be significant features in the real world or certain textures. And then it's also going to reduce that. And the idea of dimension reduction is that we're trying to get fewer numbers that represent the essence of what we've got. So if in fact every one of my images has in it the same texture at the same Value, there is no point keeping that number because it's not descriptive of the differences between images. And in another context, if you imagine maybe I was trying to predict what kind of breeder dog it was by its height, weight and length and colour, you might find that height and length, because dogs are basically in the same aspect ratio. Once you know one of them, you know the other. So there's only. It's not worth retaining both numbers. It might be enough simply to keep the length of the dog because the height is kind of inferenceable from the length. In practice, reduction takes the whole space and squashes it down to lower space, combining different features together. So it's nice to think of it as throwing out the numbers that aren't very useful, but actually it's doing something a little bit more complex. But the basic idea is that we can sum up these images, which are all 100 pixels, by 100 pixels, so there's 10,000 values in each. Well, I guess 30,000 values because they're in color. And we have squashed this down now to a vector, which is. Let's just find out how long that vector was. A vector of 78 numbers. And those 78 numbers are starting to learn what's important to sum up this picture compared to the other pictures that we saw in the data. So one thing we could do, for example, with that is, is they do the same process, but build a nearest function that retains every image together with its. Its extracted feature value. And now when I take a new picture, I can find the feature vector, the 78 numbers, and I can find the vector in that list that is closest to those 78 numbers. And that point in that space corresponds to the picture that has the. This most similar set of numbers within that feature space. If we're extracting useful features and reducing the dimensions appropriately, what we find is we get something that looks similar. It has. It's not similar in the same, and it's facing the same direction, but it's got the same kind of features in it than our sample set. And again, I'm saying here, what's the nearest to this picture? So these are the two that are closest in that. That vector space of features. And now you can start seeing how that, you know, these rather abstract numbers start to become useful because we have a measure of how close something is to something else. We can find the nearest thing, we can start finding clusters of things that are near to each other. And all of these things come out of this idea of reduced feature vector for the content. Now, one Very useful thing that I do as a kind of first step is we take this to the extreme and we take those pictures and we squash them down to a feature space of two numbers and treat those as XY coordinates and throw the pictures onto a plot using those XY numbers. And when, as a first step with any machine learning thing, it usually gives me some sense of plausibility of whether there is something useful to be found in those features or whether the features are just irrelevant. You know, if, if all we took out was a feature that was the, the, you know, corner pix, top right corner pixel, that might turn out to be completely irrelevant. And what we can see at a glance here is we start getting clusters forming quite naturally. Where we can see down here, here's a whole collection of raccoons together, we've got a whole collection of Groots together over here. I'm not seeing Star Lords in the group, but I'm guessing they're in there somewhere. It seems like we've got a lot of things in cartoon form together. So when you start seeing clusters forming in a feature space, plot or things that you can recognize as being similar, appearing close to each other, that's a good sign that this approach is going to work. This thing itself is not terribly useful, except as a qualitative measure of are our features doing something useful? But it's a good first step and all that's actually doing is taking our vector space and squashing it down to two dimensions and plotting the points, labeling them. Right, let's go through the whole process now that what we need to do in order to do a typical machine learning process from begin to end, and we're going to do classification here on those images is first of all we have to prepare the data. I'm going to largely skip that out because all data science projects end up having to deal with the mess of the real world. Data has been badly entered, it's been labeled badly, and you may have to do some massaging of the data to get the labeling right or to throw out bad values and things that don't belong and deciding whether the data was fundamentally biased and whether there's some subsets of, or resampling of that that will deal with how badly it was collected and all of these sort of background things. One key part though is you should be splitting it into a training set and a test set. So, you know, just like the way we deal with exams in school, we, we want to see if somebody's understood their subject by giving them exam There is no point giving them exam on the exact question answers they've just revised. You have to give them questions that are slightly different or not because you don't want to test whether you've memorized a particular value. You want to figure out whether they can generalize what they've learned to something that they haven't seen. So when we get to testing whether our work has worked, we want to make sure we've not shown the classifier the exam before we run it. So we hold back some of the data. Typically 10% would be, or 5% would be a good amount to hold back. We're then going to create the classifier which is going through the process of, of converting the source material to vectors, training a classifier and installing the trained classifier and then we're going to test that trained classifier on our held back test data and measure how well it works. And if it works, it's good, if it doesn't, it's not. This isn't really about checking your work in this world. We have to kind of, we can't be sure what or why it's making its decisions. So we really have to base our confidence in it on testing whether it appears to do the right thing in useful situations. So. I'm going to take that data and do the prep step. So I've paired up the character images with the labels that we searched on at the time. Because we're going to do this supervised learning here, we're going to give it the answers to the question. The question is, what is this? The answer is rocket, raccoon. And then we're going to do our split. We're going to split this into the first 60 elements are going to go into the training set and the last, I guess 20 elements are going to go into the test set. And now we're going to train based on our training set. As said for when things are simple. And by simple I sort of mean things with a one dimensional output and particularly where we have a feature extractor that is already built into the system. So that's things like images, text, time series, sounds, colors, graphs, video, you know, these sort of standard day types. We have feature extractors already built. All we have to do is say classify and you'll see here that it's automatically figured out that the inputs were images, that there were four different classes of output. And there's some hidden things going on here where it's decided the algorithm it's going to use, it's chosen that automatically. So classify has access to about 8 standard machine learning approaches. It's decided here logistic regression is the most appropriate, and it's trained on 60 examples. And then we've stored that with a name so that I can refer to it later. Now we're going to take our test value that it hasn't seen. Well, hopefully this one it hasn't seen, and say, apply that classifier to the thing and it's got the answer right, and also give it the probabilities argument here to say, what does it think this one is? Well, it thinks that we near 100% is Star Lord and effectively no chance that it's any of the others. Now, it's important to make the distinction here that probabilities here is really a measure of the classifier's confidence, not a measure of the probability that the answer is correct. If we want to know the latter, we need to do further calibration on some of the unseen data to convert these numbers into probabilities of correctness, because it could be that the classifier simply hasn't seen enough examples to understand that there's ambiguity between different things. Or it's possible that it's actually seen this example before, which would make it very confident as well. Hopefully not. I could test, but it doesn't matter too much. Now we can take our kind of test set and. And I've asked for the first five examples from the test set and see how those go. So we know that these weren't in the original unless there was possibilities, duplicate images, which I didn't dedupe for, which is part of the cleanup one might want to consider. But here it's got this one wrong. We can see that is not Star Lord. That one has got right, this one has got wrong, that one has got right, and that one has got wrong. So it's done pretty poorly this time. There is a little bit of non determinism in the training and obviously very much non determinism in the images I've got When I play with this example normally it gets about four out of five. Right. And then if we try example six through to 10 or 15, it might do better. Give it another chance. Correct. Can't tell. Yeah. Not sure what that's meant to be. Correct. Correct. Correct. Correct. Correct. I think correct. Wrong. Correct. So it's done a better job on those examples, but we don't want to be anecdotal about this. We want to be systematic. And so the WULF language provides a whole collection of tools to allow us to take that Classifier. We built the test set that we held back that it hasn't seen and go systematically through every element of that test set. Calculate what the model says it is, compare it to what we know it is. Because the test set had truth values attached. Assuming those were correct, and we get back a set of important metrics here, we can access these individually, but actually the key ones all appear on this main panel when we get it back. So the headline thing here is 75% accuracy. That means that three quarters of the test examples it got right. Now that's very contextual. There's two things you have to think about to weigh up. Is that a good answer? One is if you knew nothing, then how many would it get right? Well, there were five classes and they were equally sampled. So if you just said Groot every time, you'll be right about 20% of the time. So we're trying, so yeah, 20% of the time. So we are trying to beat 20% here. Well, 75% is really obviously much better than 20%. And the other contextual thing that we need to think about is, is what's the purpose we're going to put this to. If we're trying to suggest tags for an image that seems pretty useful. If this is a self drive car, having a 25% failure rate is a disaster. So we have to think about costs and benefits of using a predictor. Actually, it's decided its accuracy baseline is 35%. I'd have to think about why that is not 20. That's the comparison it thinks we should be making. And there's actually about 40 different measures if you want to get technical. But the next most important one is this confusion matrix plot. Let me just make this a bit bigger so we can look at it better. Cap talks. That doesn't help. So this tells us the pattern of failures. What we've got is the actual classes on the left and the predicted classes along the top. So what we want to see ideally is everything on the diagonal that whenever it's actually Gamora, it predicts Gamora. And we can see here that for two of the classes we've got lots going on on the diagonal. That Star Lord, when it actually is Star Lord, it gets it right every single time. Although it slightly over predicts Star Lord because when it predicts Star Lord, actually six out of eight times it was correct. Sometimes there were things that weren't Star Lord. So this gives a sense of the nature of failures because sometimes you get very systematic failures and you find that there's a particular confusion between two different things. And that either tells you, again, from a cost benefit point of view, does that matter? Well, it depends what the failure is. If it's more likely to do something safe by accident than something dangerous, then you may wish to ignore it. But it also can guide our future data collection that if it's always getting confused between Star Lord and Gamora, then we need marbles of those two classes to help it learn the distinction. And we don't need too many examples of classes where it gets things right. We can also sometimes look for things like the things it was least confident on or the things that it was most confident on in case that helps to tell us that in fact the data's been polluted. For example, if there's a lot of worst classified examples and we look at them and go, actually those don't belong in the data set at all, that can send us back to improving our data collection techniques. So I said that this classify command has a whole bunch of different methods. So that raises the question of which does it use and when. We should use different ones. Just as a fresh example, some. Some numbers that mapped colors. I don't know why that maybe this is something like the geolocation on the world and that's the dominant color of the flag or something like that. And we want to make some kind of prediction. What we get is by default is we have method go to automatic. You don't have to say it when it's automatic, but say if I want to say it explicitly. That's the default value when we use classify, which is why it's decided on its own to do something in this case naive Bayesian. Now we can control that if we want to by saying, okay, I don't want that to be automatic or naive Bayesian. I'm going to use say gradient boosted trees as a method. And now we've forced it to use a different method which will have different characteristics. And if we look at. I have made this a little bit big on screen. Let's make that a bit smaller. From a set of original data, we're trying to fill in the spaces with our predictor. You can see that depending on the method used, you get quite different behaviors of what happens as you cross from one area to another. That nearest neighbors is basically saying, if we're looking here, we look at the things that are nearest and they're all blue, therefore it's blue. Logistic regression is fitting kind of trend curves to it. So it's trying to find Lines that kind of switch in a logistic curve in a in a true force in a smooth kind of way. Lines across data that switch between zones Support vector machine I'm trying to remember the details of. I forget the details. Random forest is building decision trees and in random orders and then seeing how many of them make good predictions and discarding the poor ones. Naive Bayes treats the two variables as variables somewhat independent that like spam filters work very much like this. If I get email that says buy cheap Viagra Viagra is a negative word. Cheap is a negative word for spam. If I get email that say Mathematica or or Webinar those are positive words but it treats them independently. So if I had a webinar about Viagra those things would cancel out is the naive base approach. And neural networks is the kind of the richest approach which we'll spend the last third of the webinar on. So the way you choose the method I'm going to skip through this example I think otherwise I'm in fear of massively overrunning on time. Is basically to try each one possibly on part of the data if you have very large amounts of data and then use these measure approaches to decide which one is doing best. And in fact that's effectively what Classify does when you use method goes to automatic is it starts to train a little bit of the data on each method measures their performance and discards the methods that are performing poorly until it ends up with one method left that seems to be the winner. And then it puts all of its training effort into that. For this small training set it probably trains all of them completely. But for a large training set it will do that with a reduced sample until. Until it does better and even all the sub parameters because all of these methods here like gradient boosted trees has half a dozen parameters I can say which is a boosting rate and you know gradient measurement function. All these things I can spell out those things again trial and error and measurement is the only real way to figure out how to tweak these things to give the best performance. Now prediction I'm going to skip through this very quickly as well because predict we have a high level command for predicting numerical values. But it works exactly the same way that Classify does. So I've got this data set of about 20 properties of house prices in Boston. This is a classic old training set. And here's the first 10 properties. And I do exactly the same thing. I split the training data into training and testing. I do exact the same thing as before. Except this time the syntax is slightly different. I'm saying I want to predict the house price. The medvert, I think was the label for the medium. Value was the thing that we're trying to predict. So we're taking our training data, but that's the value out of that data we want to predict. I've given some other property here to tell it to try hard, and now I can give it some unseen values and it predicts that this is a $19,000 flat. As I said, the data is very old, so you'd be lucky to find a $19,000 flat in Boston. Now, the measurements, again, the principles are the same, although the actual measurements are different for continuous data. So we have to look at things like the standard deviation that our predictions were plus or minus about two and a half thousand dollars. We want our data to lie on this line, if we can, so we can start seeing that, that the biggest outliers are for the very high prices, it would seem. And there doesn't seem to be a great asymmetry. So we can get sort of qualitative measures from that. But again, there's about 20 measures we can use if we really care about the details. So in that world where you've got a number to predict or a class to predict, and also, while I didn't show it, you want to just make clusters. Those things are really fully automatable. Unfortunately, the real sophisticated fun end of machine learning isn't fully automatable. And for those, the by far the winner has become the neural network approaches. And the neural networks are a way of describing a model that has a lot of richness, but also some of the elements of the model are architectures that are designed through deep human thinking for specific tasks. So this is the lexicon of the language of neural networks. Most of these things represent trainable layers that we can assemble to build a neural network. And some of them are very generic, like an element wise layer or a fully connected layer. Some of them are quite specialized, like convolutional layers tend to be very good for image recognition. I'm not going to go into how you decide design these things. I'm going to skip over that completely and we'll use some that have been pre built and show you how to use those and modify them. So we saw this one earlier. This is the Lunet model for image character recognition. It's got a kind of compact image in our notebook, but actually behind the scenes it's got 11 different layers. And these things are a convolutional layer is kind of image feature extraction. Some of these things are purely structural. Some things like linear layer are about combining features. Softmax layer is a probability layer. And so these things have been built by somebody to be a very simple image recognition network. Let's get some data to apply it to. So these were the characters that we had before and we can simply apply that to one of the images to get a prediction. This is, here is a. Let's copy this out. This is an image that come out of the data set and when I say apply that network to that image, we get back. Now a prediction is not a picture anymore, is now an actual number. So I could add something to it, for example. So in terms of using them, we try to make that easy by giving them names that you can fetch using this netball command. And if you go to our neural net repository on the web, you'll find that there's all kinds of neural nets for different purposes, doing things like image segmentation or image content recognition or speech recognition, all of these kinds of tasks as ready to download networks, you simply have to say fetch the thing by name and then apply it to the data. So from a use point of view, we try to make that completely seamlessly easy. Let's not talk about. Oh yes, let's talk about this. So if we want to retrain for the same task, then that's really a one step process in most cases. So take some new data, which is the blur of the old data, and apply it now. And it still seems to have done a reasonably good job with a little bit of blurring. But you can see that now it's not doing quite so well. That blurred 4 is looking more like a 9 to it and the 6 is looking like a 0. But if I want to train it with blurry data now, in a similar way to the way that we had classify, we have net train, which is basically doing the job of, of classify and predict. But it takes as its first argument the network that we're trying to train and the second argument is the data that we are training it with. So here I'm just doing a data preparation to label the data and it retrains. That network is a lot slower. So you can see without a gpu, it's taking a good few seconds here. And this is the error between the examples it's looking at and its prediction. And as it tweaks its weights and learns from it and makes the model better, that error gradually falls and at some point it'll reach some level that it decides to terminate. Or I can click the stop button and end it early. Oh, it's found made some breakthrough discovery and now it seems not to be doing any better, probably because I don't have enough data for it to learn any better from. And it will get to the end. And now if you apply it to that, it's now done a good job, partly because I've cheated here and used the same data. So I didn't do the test training split that I should have done. Now, by default, NET Train starts with all of the training parameters of the net as downloaded. So they've adjusted, but they haven't been completely reset. So it still does a good job on the original data. It hasn't kind of forgotten everything, but over time, by this approach, it will gradually forget what it was trained on, unless you mix your training data with the training data that it was originally trained on. So it preserves both knowledge together. Now we can modify these networks as well. So if you are dealing with a task that is similar to something for a neural network, but not the same, then these things are all adjustable because there's a symbolic object that is computable, just like any piece of data in, in the Wolfram language. So by way of an example, let's get a much bigger model out. This is the one that's used by our image identifier command. But here I'm getting the actual neural network. And you can see here there's a bit more to this one. It's got 20 odd outer layers, but actually some of these layers are graph layers within them. So if I click on this one, you'll see that the, this one layer itself is in fact got 23 layers within it. So there's actually about two or 300 layers in this network, more than I'm going to want to figure out how to code by hand. But that doesn't mean that I can't transform the thing. So one thing I could do, for example, is just take the first 10 layers of that so I can just throw away the bottom end of the network. And now it's only the first 10 layers. And now I can see what's going on inside at that point. By taking, applying that to the image, I'm going to have a bunch of matrices to understand the matrices, I'm going to turn them into images. And what we're doing is now getting a glimpse inside the brain at level 10 of the, of the network. And we're seeing some of the features that it's detected. So whatever this pit of the thing that it's looking for, it found no features that matched. You can imagine some of these things might be picking out the stripes of the tiger. I don't see any that look very like diagonal stripes. But you know, maybe if we were looking closer to the beginning of the network at level three, then some of the more raw features would be identifiable to things that we're really looking for in the real world. What that also allows you to do is to do transformations on this. So I can replace parts of this layer, this model with different outputs and let's just do this live here. I think I'm clear this. So the basic idea is you do net replace part of our network and we give a list of the layers we want to change. So I'm going to change. Let's get the whole network back here so I can see what I'm doing. I'm going to change this, this last linear layer to be size 3 so I can have it not predict from 4315 things like tigers and cats and dogs. But I wanted to do rock, paper, scissors for example. I would replace the linear layer there with a fully connected three layer. And I would have to do something else here with the encoding. Decoding. The encoding is okay because those are images for the decoder. I want to give a new output classification language for this, which is netdecoder class. And I'm going to tell it what those classes are and I'll just call them rock paper scissors. Red is telling me I've got my syntax wrong because I've got one too few curly brackets going on if I've done that right. And another bracket at the end. Nope, doing something wrong here. What I should end up with is now a new neural net that ends in a Vector 3 with my new classification of rock paper, scissors. You see, everything's black here, apart from the linear layer. So those have remembered everything they know about tigers and cats and dogs. The linear is brand new, so it has. It needs training, which is why it's red. And then I would do a net trainer that against pictures of my fist doing rock paper scissors symbols. And it would learn how to recognize that. So this kind of transformation allows you to take those things from the world of our repository, find things that are close to the task, tweak them to a task without having to understand all the details of why. There's a net graph layer in the middle with a few basic bits of understanding about the kind of last output layers. That softmax is the probability Layer, linear layer, aggregates all the features together to make a final decision from and then make that change. Have we got time to discuss this? I think we do, yes. So let's talk about. Yes, we've got time for this. So let's spend 10 minutes on one key problem that you have to worry about when you're building machine learning, which is concepts of overfitting or underfitting. Because what we're not trying to do with our model is to build a model that remembers what it seemed. There's faster ways to look up data and say, is this example one we already know? Than to build a neural network. We want it to generalize so that when it sees something new, it can take the things that it's learned from the features and the relationships between the features and make some useful prediction from that. So underfitting, you fail to capture enough essence from the problem and it just makes bad predictions that, you know, if you imagine trying to fit a straight line to very complex data like this, in fact, that's. I guess what this first example is doing is if we only give a straight line fit to this curve data, then it can't capture the, the complexity of that data. A straight line is just not good enough richness to it. And that will underfit and make pretty poor predictions in places like here and here. It might happen to get lucky in some places, but on the whole it does a poor job. Our generalization is going to look somewhat similar to the underlying model that generated our data, but it's not going to take into account. There's always going to be errors because there's noise and randomness in the real world. But if we give too much freedom, we end up with overfitting, where we get a model that is so rich that it can capture all of the data points or nearly all of the data points, but you can see in some places it's doing a much worse model to fit to than our good model here. So this is actually a terrible model for the opposite reasons. And this is a terrible model because it's overfitting. It's trying to remember the data rather than capture the essence. And what it's therefore trying to do is effectively model the randomness that it can never capture because it's random. Philosophically speaking, it's driven by features that we don't have on the input set and it can't capture those. So it ends up being kind of just a random fluctuations in the model that mean nothing, and those get exaggerated as you overfit so let's talk briefly about a few ways to approach that. And I'm going to make a toy example here. I'm going to make some data that's roughly an exponential curve, but I'm throwing in some random noise into it and there's not very many data points. And here is our data. And I'm going to make my own neural network. I'm just going to code this one up from scratch. It's going to be a linear 50 layer, a nonlinear layer that's got a tanch activation function. So that's what gives it the ability to do something complex, is mixing linear and nonlinear and another one and another one. And then we're going to have one output, the input scalar, the output scalar. So it's going to take a number, give a number, but it's got over 100 degrees of freedom in here that these linear layer 50 are turning this into a vector of 50. And this is going to be a 50 by 50 matrix. There's all kinds of controls that it can adjust and we're going to train that and off it's going to go. And hopefully, if my random numbers have been useful here, and also for consistency, I've deliberately fixed the training methods called RMSprop, because its default one actually is a bit too good at preventing overfitting. So I've deliberately picked a slightly weaker but efficient training method. And now let's see what this prediction does. Now, you could say this is a great model because when we take our underlying data, it's all very close to the curve that our neural net has generated for predictions. And maybe if this low feature here, right in the center, was a real thing, the fact that the curve dips downwards here would be capturing that knowledge. But actually, in this case, we know that the underlying model here is. Well, let's look at what the underlying model is here. Before I added random noise to was this exponential curve with it going from minus three to three. That's the truth that we sampled data from and threw random noise in. So this feature in the middle is entirely imagined because it's given too much weight to the noise. The fact that somewhere here there's. The data point was surprisingly low. It's assumed that that's a feature and it's captured it. So this is an example where it's overfitting. We're not getting the underlying essence that I happen to know is there. It's come up with a model that's capturing all of that noise. And while this might look Good. This time if I generate another set of data and compare, you'll probably see that it does a very poor job. So there's three basic methods I'm going to introduce here to deal with. This one is called regularization. There's different versions of it, but I'm going to use something here called L2 regularization. I'm doing exactly the same thing as I did before, the same network as before, the same data, the same underlying method, but I'm adding this regularization parameter and I'm training for the same length of time and we'll see the results. And while it's thinking, I'll explain what regularization does. Inside that network are these various matrices that are trying to find tweak their parameters in order to make the best prediction. So we've got a much kind of more generalized prediction that's trying to ignore some of those outlined bits of noise. Now what regularization does is ordinarily it decides whether it's a good or mad model by figuring out the difference between the training example points and the curve. And it adds all of that up. And that's in machine learning world, it's called a loss function. They can get a bit complicated when you're trying to do really rich things, but in this case it's simply the sum of the errors. Now in our overfitting example, it tries to minimize those by tweaking this line to try and go near every single point. So the sum of the errors got small. But what regularization does is it adds a bit of complexity to that loss function to say not only is it going to punish the errors, it's also going to punish the complexity of the model. So the more the parameters in the model are not zero, the more it will say that's a bad model in the same way that it will say it's a bad model if it's not fitting to the data. So it's taking both things into account. So basically it's trying to fit to the data while at the same time use the simplest model that achieves the best fit, because it's going to favor models that have very small or zero parameters. There's different kinds of regularization depending on whether we're trying to minimize the sum of the squares of the parameters or just the non zero parameters and so on. But basically it's going to punish complex models over simple ones at the same time as punishing errors. And that compromise ends up giving us something that fits to the data, but also is a is not unnecessary using all of the controls that were available to it. Now, a completely different method is to dilute the data. And this is a slightly counterintuitive one. We're going to make the fit better by making the training data worse. So what we do is we do a little bit of transformation on our network by inserting a new layer, and we're inserting something called a dropout layer. And a dropout layer is a layer whose job is that whenever it's being used in training, it randomly disappears. So it's a way of killing certain parameters within the model randomly. So, you know, imagine you're trying to learn the best ways to drive across the country. You might end up with a very convoluted route to get from one place to another that is technically shorter than everything else. But what dropout layer does is say, okay, we're going to randomly close certain roads sometimes. So if you sometimes drove a long way up a single lane track, then find that it's closed, then that suddenly becomes a very terrible decision for that particular example, because on that occasion it was an unreliable bit of the root. So by making parts of the model unreliable, it stops it over fixating. So if we go back to our analogy about cats and dogs, if one of our features that we were learning on was the length of the ears, well, let's say the length of the tail, and say long tail represents a cat. You want it to sometimes be told, sorry, I'm not telling you the length of the tail, even though it was known in this training sample. You have to do without that because otherwise you might find its entire model is, long tail equals cat, short tail equals dog. Forget everything else. And if you make sure that sometimes it can't rely on rely on tail length, then it would have to also take into account ears and fur and face shape and other things. So this dropout layer prevents model fixation. You see, the training is much noisier now because every time it's updating itself, that random dropout is changing things. So what was a good model on the previous step suddenly becomes a worse one or a better one randomly, but on the whole, over time, it ends up with a model that is now richer than the previous one. But you can see it's still trying to avoid overly fixating on things like outlying data points. So we've still got some level of generalization, but achieved by a completely different method. The third one is an interesting extension on the test training set, which is as well as having our test data and our training data that we need to decide whether the thing works. We're giving a third data set, which is effectively a test data set for the trainer. So we make some more validation data and this time we tell it to train the net on the same main data, the same method, the same parameters. But periodically it's going to take a peek at this extra training set and say, how am I doing against unseen data? So obviously, if it looks at this too much, it ends up becoming part of the training data and it'll end up just being a bigger training set and it'll fall into the same overfitting. But this idea that it validates it more periodically and doesn't use it for updating the parameters, only uses it for evaluating the model means that we end up with. You can see here two things going on. This is the best parameters being found, but this is the performance on the validation set in blue. So even though, and I'll run that again, because it's a bit slow to see, even though the model can be fit better to the training data, the fact that it's not improving the validation means that the best model it had was actually much earlier in the training process than the one it ended up with. So it goes on trying to see if it can improve a model in a way that improves the validation. But in the end it falls back to the simpler model ended up with earlier. That was the last one where the kind of test, the validation set performance and the training performance were at all similar, so that it knew it was generalizing rather than simply overfitting. And again we end up with another model, but one which has got some level of generalization. Now, these aren't distinct processes. We can do all three at once. Some of them are quite parameterizable. So the regulation level, we turn that down, we make a more sensitive model, we turn it up, it becomes a much smoother, blunter model. Dropout layers we can set to be of minor or major. And there's also parameters you can set with. Obviously we can change the validation sets to be large or small. So there's lots of things you can tweak with, as well as controls over how the training performs. But in the end, a lot of that comes down to trial and error and then validating. And you're trying to tweak the parameters of training to achieve the very best model that generalizes in the way that you need to in the real world. So in the last few minutes before we. I look at the questions, which I can see we've got some already, but Now's a great time to be queuing them up for me. Let's talk about very briefly about the modern generative AI that has got hit the news in the last year. I'm only going to touch on this very briefly because in a week I'm doing a webinar entirely on this topic. So I encourage you to come back for that. Now, things like ChatGPT, the generative AI models, they're just machine learning models. There's nothing fundamentally new going on. And in fact, the earlier version of GPT, GPT2, before GPT 3.5, which was when OpenAI went live, is a model that we have in our network repository. It's got a particular architecture that's designed for string manipulation. So it's actually quite small. There's a couple of graph layers and a chain here that's got more chains inside it that have probably things inside them. So there's still some complexity there. But it is just a neural network that has been trained in this case, this particular version has been trained on web text data, which is a corpus that's been scraped from the web. And what this particular model does is it's a predict model that will take an input string and predict the next string you'll get. So if I give the input string, Albert Einstein was a. Its best prediction is that that would be followed by the word teacher. So it's just a class predictor for what is the next best word. And that's all that ChatGPT fundamentally is GPT4, that is the kind of current state of the art model is just doing this fundamentally at its core. Now, if we can predict the next word, of course we can add that word to the end of that and say Albert Einstein was a teacher. Let's put it in there. Oops, I didn't want the quotation marks. And we can predict the next word that follows that. Now, oops, I could actually type correctly in a live webinar. Albert Einstein was teacher who. So we can string that together recursively to now have it write code. So here I'm just doing that, I'm nesting, I'm replying, applying, recursively that in this case 10 times. And we get Albert Einstein with respect to German mathematician primary, because he was in early and I could keep going. So we're now generating prose simply by making sequential word predictions. And that's all that ChatGPT is doing is a model that does this at a very large scale. So where this thing has a few hundred, maybe 100 layers with a few hundred or thousand parameters. The large language models have hundreds of thousands of parameters. They're very large and trained on very large models. So this ability to predict text becomes extraordinarily good so that it can predict the answer that would follow a question or the actions that would follow a set of instructions and they start appearing to be human, like intelligence, because it can start predicting the the conclusion from that would follow some reasoning statements and it starts looking like intelligence. But in the end all it's doing is predicting the best words that follow. Now unfortunately these models are so big that they are, or at least the state of the art ones are beyond the ability to run on a typical desktop computer. There's the new generation of mid sized language models that we're trying to curate to have in the whole kind of framework that allows them to be transformable and retrainable by our mechanism, but we're not quite there on that yet. But for the state of the art ones that you're not, you're not going to be able to run them on your home computer. So what we've done with the Wolf language is to integrate them in to the language as external services. So if I do this llmsynthesize command here, I can take an instruction here and call in this case the OpenAI API and get that the answer back from that remote service. So this wasn't executed locally on my computer like almost everything else was in this demonstration. This was executed in OpenAI's GPT4 I think is the default no. 3.5 I think I've called here model to generate limerick and that's not very good limerick. I might give it another go, but you get the idea. Now it turns out that from a, from a using these models to their full power, not simply generating sort of conversational chat or text writing, there's a few paradigms that make them really quite usable within, within the language. And unfortunately I've accidentally deleted this. I'm just going to recreate here what should be in the notebook and it's probably missing from the downloadable notebook as well. We have a notion of being able to make a thing that looks like a function in the Wolfram language, but is powered by the machinery of large language models. And the idea is that you are basically programming in English rather than programming in code. So what my function is going to do is to classify something that we haven't seen into either animal, vegetable or mineral. So this is what's called prompt engineering where I'm kind of giving instructions in English. But this will be fed to the LLM before the parameter and sets up the task for it to do. I probably need to give it a bit more here by saying you have to be very pedantic with these things. Return only one word and let's see what happens if I apply that function now to this list of data. And so I'll call that function four times for each of the four data points. And it's done catanimal, cabbage, vegetable, diamond, mineral. So it's done the right thing for all of those. And I've thrown in a curveball here of something that isn't obviously any. And it's decided animal is a better classification than vegetable mineral for a concept like love. And the other basic paradigm that we have is the same kind of thing, but driven by examples, which it turns out large language models are really quite good at generalizing from. So here I'm giving it some prompt engineering again and we're saying extract the name and age of a person from the text returnable from language association with the keys name and age, looking like this. And then here's some examples. Here's an example. John is 52 and I want you to return this data structure. A man called Mike is old. I want you to return this data structure that's given an example of what happens if the age isn't mentioned at all. I've said missing and I want you to parse that as a Wolfram language expression when we get it back. And hopefully if that works, I can now give it this unseen text. My manager, Jane is very young, she's only 30, and it's named Jane age 30. So it's correctly pulled out the parameters and restructured that data. And this is really powerful for sort of dealing with unstructured data and transforming data, doing sort of classification tasks where you have limited amount of data. So where I might have, for something like this, used classify, if I had thousands of examples, when I only have a concept and enough energy to generate, well, in this case no examples, but I might generate a dozen or two dozen examples quickly, but I don't have a proper data set, then large language models are quite good at using their innate knowledge to extrapolate on the description or the examples to be able to do the right thing and a couple of other paradigms within that world is the idea of providing tools. So this is bringing the best of the computational world together here I'm saying use OpenAI to write a paragraph about Chicago, New York, but I don't trust it to know up to date information about things like city populations. So I'm telling it there's a tool available which is going to find the populations of cities. Some prompt engineering say what it does the parameter it expects and I'm going to run the Wolfram language command. So this is part of our inbuilt data. I'll just show you what that does. If I type something like London city and then type nation, we're looking that up from a database within our system. What we're doing is giving the LLM access to that computational tool so that the LLM is going to happily go off and try and do this. But at some point, hopefully in a few seconds we'll get our answer back. With remote services is sometimes you're not in control of when they want to talk to you. It's having another conversation. So this has ended up being a multi part conversation between my computer and the LLM and I'm hopeful this time it will stop having a conversation backwards and forwards. Normally this takes a couple of seconds. So I think OpenAI is a bit busy at the moment. Now, what's it done here? So from here down to here was all chatgpt, but at this point it's realized it'll be useful to get some facts. It sort of painted itself in the corner here. The city of Chicago is a population of approximately and then it's called back to my tool and run through Wolfram language a lookup from the database of the population of Chicago which is then written into the answer and then it's carried on its way and it's called back again. So this is why there was a conversation where the LLM was doing some work, we were doing some work, the LLM was doing some work, we were doing some work. And then the LLM wrote the conclusion and we get our paragraph containing the best of both worlds. One other thing in this world is also in this generative world is image synthesis. But it's basically the same kind of idea. We're using big neural networks that have been trained on lots of things that combine text and in this case images. And we are giving a prompt engineering to describe an image that we want to synthesize. And it's got enough knowledge about the short and long term relationships of images and things like dogs and where a scar would go when you smoke it. And it's come up with this example and these LLM and generative things, they have a certain amount of randomness in them because that makes them seem more human. Like so if I call this again, I'll get a different answer. I don't see any sheep in the background, but maybe that's a sheepdog in the background staring at my the Crown Cigar Hound. So that covers what I want to talk about. We've gone from, you know, the basics of what these things are doing in simple cases. The fully automated commands in the Wolfram language like classify and predict and how you work, work through the training process of train, measure, refine and test and what the neural network framework allows in the same kind of paradigms, but in a much richer lexicon of language. We do have a longer version of the neural network talk, but I don't know when that's scheduled for next. But watch our webinar stream to see when that comes up next to get more detail how to actually work with those neural networks. So in our schedule we still have 15 or so minutes left, so I'm just now going to turn to the chats here and give me a moment here to find. Yeah, so text structure for further languages would be helpful and indeed is true. There is a bridge that you can do now of course, which is that the text translation is very good through things like the the LLM tools. So there is probably something that could be done to map to an English sentence, structure it and map back again. I'd have to think about whether that would make an interesting resource function for our repository. We haven't had the training sets up to now for the other languages to be able to build the sentence deconstructor for other languages. Okay, quite a complicated question here. So there's a problem with the anomaly detection commands in slide six. Somebody says it possibly I have submitted the wrong version in the notebook or it is also possible that it's calling on some data that. Where's the anomaly detection example? You will need to have Internet access to access the resource data. And this may also download neural network in order to work the first time. So if you're having trouble with that, I would suggest restart Mathematica, run them again and ensure you've got good connectivity the first time. Once they're downloaded and cached locally, they should run just fine. If you're still having a problem, post into our tech support because there's something wrong on your installation. Classifier can get much better with more data, but where can we get data? Well, that is a fine question. There are, if you look at things like Kaggle and various open source projects, there are some quite big training sets that people have built for doing training from. But in the end, one of the things we're trying to focus on is the tools to solve your problems. It's all very well to be able to download pre trained networks for certain tasks, but where this gets interesting usually is doing something new. And if you're doing something new that only affects your world, then you're likely the only source of the data. So if you're trying to predict how your customers behave, there's limited value in fetching third party data on customer behavior. If your customers have a kind of different bias and slants and interests than other ones. The calculations seem to be very extensive. Can the GPU be used if present? Does it happen automatically? The answer is it doesn't happen automatically. But many of the commands like I think this is going to fail because I don't think I've ever set this computer up, but this does. We only support CUDA and the way you do it is let's go to the classify example I did before and find out if I am up to date with my CUDA libraries on this computer. I'm not on my own machine, so I don't know what state this is in. Where's the classify example? Is the classify example the way that you would do that is you add the option and most of the machine learning commands have this sorry not method that's the one we're using for it is target target that defaults to cpu. If you have CUDA hardware and you target device goes to gpu it will use that and it so it's detected. I have got CUDA hardware on my computer but I haven't got the CUDA library. So again one time only it's now downloading those onto my machine and installing them and maybe if we've got the patience that will that will finish. And once it's when it runs, depending on your hardware it can be many, sometimes thousands of times faster. We also support things like batch submissions of things like Amazon and Azure hardware. So if you don't have big stacks of high memory machines and big GPUs then you can rent those things by the hour from Amazon and Azure and we've an automated mechanism for you to place this classification job onto Amazon. Once trained, the classifier can be run on CPU or GPU locally but generally is much, much faster than the training step. In this small example, probably there's not much to be gained when the data sets get large. We also have some mechanisms for training out of core. I've done the lazy thing here of loading all of data into memory and then training on it. When you have terabytes memory of examples you want to load them in batches and update the trainer and then load the next batch. So you're never using too much memory at a time. I think it is now probably busy installing CUDA hard software having downloaded it. So I might kill that in a moment and try and make sure I'm set up next time I use this machine. Would you mind to recommend algorithm for face recognition of twins There are some specific neural nets out there for face recognition. Whether any of them I don't know what the state yard is for twin recognition where I suspect twin recognition is extremely difficult. And you know these neural nets aren't necessarily magic in their abilities. We do have one built in so we have a high level command for face recognition. I forget what it's called, It's something like face recognize or. There we go, face recognize. So we that's simply backed by one of these custom design neural networks. And what face recognize does is just package up the training process to make it simpler. It does things like facial identifying where the faces are in images automatically and things like that as part of the workflow. But you can see the mechanism here is you train with examples and then you apply it to unseen and it makes the prediction. Is there support for hybrid models using partly models and partly data? We've got nothing built in for that. It is something I've given a little bit of thought to that about trying to make a utility function for this because one approach you can take is if you've got let's say you've got 10 input data points and you've got a model that gives you some idea of the physics of the situation. What you can do is build into the encoding process something that encodes those 10 numbers instead of to a vector of 10 numbers to a vector of 11 numbers where the 11th number is the output of the physics model you've coded up. Now if that physics model is really good, then when you do the training on those data points it will start to learn that that last data point is the most valuable data point of all. If it is a model that isn't very good or it breaks down in certain situations, at some point it'll start learning that sometimes the better prediction comes from ignoring the model and it'll start to learn the situations where the model is good and situation when the model. So I think that could really be packaged into a kind of neat single step train with model function for the function repository. But that's only the approach I would take and I don't think it's a complicated function to write. Right, So you discuss a function of a single variable. Is it possible to study a function of two variables? I mean I want to distinguish between one variable output which is where commands like classify and predict can work. And when you need multivariate outputs then you generally need to start moving towards building neural networks so that you can encode the dimensionality of what comes out. So if you want to generate a picture for example as output, classify and predict can't do that for you unless the class is a pre decided picture. When talking about inputs, I only used one dimensional input for my my overfitting example because it's easy to visualize and it helps it make it easy to explain. So all of those examples do with things like the inputs dimensionality and measurements and controlling things like overfitting and underfitting. All of that stuff will work in arbitrary dimensions in exactly the same way. How to determine the number of layers for a large language model. How the very large language models, or at least the main ones are not open source so you can't actually see for sure what's going on. And so you can only rely on whatever people like OpenAI tell you they have. There is, as I said, there is a whole collection of mid sized models now which are open source and where you can actually get your hands on the model. And those are studyable. We haven't got them into our neural network framework yet. If you want wanted to simply count layers in the network then so we could take that image identify thing and we just convert that into a list of layers. Maybe this was a big one to use. Oh, we're still waiting for that CUDA installation to happen. Let me kill that. I don't know the keystroke on Windows for killing evaluation. I'm just going to kill it the hard way. In fact we just use length if we want to know the top level layers. So I think length of NET model of what was I using? I was using the Wolfram image identifier network. Will link. Give me the answer. There we go. So at the outer level the thing has 24 layers, although we need to reach inside and write a slightly richer pattern to be able to collapse all of these nested levels. But basically all of those things are interrogatable if they're in the. NET model representation. If they're external, then you have to go look at the source code or read the documentation for those models. Yes, I will post so I was asked, can I post a link with the additions for the notebook? I will fix the missing example and save this and we'll share that in follow up to anyone who's registered with their email address. I would like to implement some of these on a supermarket sales data set. Can you offer some guidance? Yes. So here's just a, you know, few things to. As a, as a first step. First of all, you probably want to start with using. Even though machine learning kind of offers the promise of not having to use your brain too much, start by thinking about where you're likely to get the strongest signals. So you could throw all the data you want into it and then try and get it to produce something. But you're better off thinking about, you know, if the socioeconomic class is known to me, that's got to be a strong indicator. The age is known to be, that's got to be a strong indicator. The color of their car probably isn't. And you know, pick out the things that you think are going to be strong features that you have good clean data on and that have things that you can train against. And start by trying to do the simple cases like that. Obviously do what I said about holding back test set. If you can get the simplest cases to do something useful, then you're onto something where you can start feeding it more data and giving it more kind of challenging things to predict against. But if you can't get the simple things to work, then you're going to be wasting a lot of effort not knowing whether you've overloaded it with irrelevant data, whether it's the quality of your data. So I would start simple and, and build up. I mean of course I should mention here I'm supposed to be helping the company here, not just talking about things I'm interested in. We do have our technical services teams who can help you. So if you have a project like that and you want some assistance, then we have a team that can come in and do the work for you as an external project. Can it predict the functional form of the underfitted, good fitted or overfitted model? So we, so the model itself, you know, has a functional form in the end, but it's, it's going to be massively complicated combination of all of the multiplications of the nonlinear elements in your network. But one thing that you could do is to take your model resample from the model. So just do it like equally space sampling or something, marching through the expected range of the model. And then we have a command that Will take data and try and predict find function. Maybe. This is not one I use very often but it begins. I'm pretty sure it begins find. So let's all the fine commands here. Oh, that's not very helpful, is it? Find. There is a command on this list. No, not find equational proof. There is a find model type command and I can't see it and I can't remember what it is that will predict the equation. That and it's reasonably sophisticated. It isn't just kind of like that's a straight line or a quadratic. It'll say oh, it's the sum of a sine of 6x plus a linear function. And you would apply that to the data that you've generated. That's probably the best guess for doing it. If it's a probability distribution we have fine distribution. But. So that's but for an equation it's a different command. Maybe it's linked from the find documentation page. Findformula. That could be the one I want. Yeah, here we go. So the idea is that FindFormula takes a set of data and a variable and generates the. The formula that comes out of that. And you can see it can do reasonably sophisticated things. Find formula of that data. And it's come up with this formula. So that's where I go about it. Generate some data and then use find formula on it. I mean in the end findformula is a custom neural network backed and heuristic backed bit of prediction. But we've got that already packaged up across this set of standard function. Do you think it would be possible to train a model for selecting audio recording takes based on intonation, sound quality and rhythmic precision? Absolutely feasible. There are already there's, you know, some aspects of voice. There are already pre built networks. We have ones built in for recognizing the the word spoken of course, but also the speaker. So if you have two people in conversation partitioning it into. Into which speaker we have those built in already. You would probably again be best to try and find some labeling data where this is somebody angry or this is somebody happy. And I don't know if those sets exist well enough but if they do, then the same kind of techniques would apply to be able to say something qualitative about the thing with rhythmic precision. It's worth pointing out that my line about use machine learning when you're data rich and understanding poor does have a converse that if you're understanding rich and data poor then there are other techniques. So if you are trying to detect can somebody speak On a very rhythmic pattern, you might be better to look at that using things like Fourier analysis or sort of wavelet transforms in order to try and get the underlying rhythmicness out of the sound regardless of the voice. That might actually be more accurate. And similarly, for things like pitch detection, there are ways that are both neural network based, machine learning based, but also signal processing, numerically based and sometimes hybrid solutions are the way to go. So don't always leap to machine learning as the first step. The find fit somebody was suggesting when I was stuck there trying to find this find formula. Find fit takes a given formula and then tries to find the parameters that optimize it. So that's basically regression. It was fine formula I was after. Where to find the info on getting any ONNX model, convert it to Wolfram language and select the GPU as a device for training inference. So there are two things to say about onnx, which is the open neural network exchange format and Wolfram language. So if you are wanting to use our framework, we have ONNX import support and export, so we don't have 100%. So it depends on exactly the details of the layers of the model used. But for many models you can just use import on the ONNX file and you get the neural net model in our. Net language where that is not possible. We also have something called external NET model. I think it's called an external net model is a way of. I've misspelled that external External net model. It's net external object. Okay, that's probably what I wanted. So there so. Well, I'm not seeing a good example here because that's importing an ONNX model. But the idea of the external model is that where you can't import it, you can reference it and then you can do certain operations from within our framework using that external model. So you can retrain it, you can execute it, but you can't do net surgery on it. If you can't get the thing into the symbolic representation, you can't do these kind of net replace part type operations. But if all you want to do is retrain and use the model, then net external object allows you to point to an ONNX model that's non importable and and treat it as if it's a built in command. Okay, what's next? Can I utilize the Wolfram technology for an object identification project on a robot? So in terms of object recognition and identification, yes, that's exactly what all of this machine learning framework can do. There's a little subtlety of The Honor robot. Because if your robot has embedded hardware, then we don't at the moment have the ability to do code generation. We can do embedded hardware at the level of Raspberry PI. So if you've got something like a Raspberry PI on a Internet of Things type object, we can run Wolfram language on that. But if you've got other custom hardware, not a kind of proper full computer, then you won't be able to run this on the robot itself. So you either need to generate the. NET model by developing it within our framework and then exporting it out to something like Onyx or MXnet and then using one of these executor libraries to run the thing on some non computer hardware on the robot. Or you want to have the robot call an external service and you can anything in the Wolfram language you can mount as an API on a cloud server so that the robot would send the image to the server, say you know where in the image is the box and get the number back. So there's several approaches to where you execute it internally in terms of solving the intellectual problem. Absolutely. The only question is for a robot, where is that model going to run exactly and what hardware is it running on? There is a subtlety there. Okay, so if you have SEM image of nanoparticles on the surface of a support material, how can you instruct image classification to give you the population of particles with different sizes in the SEM image? Again, one thing I would say about this is that there are non neural network models for doing all kinds of image partitioning. So you might be better to use one of the image filters to filter it by color or texture and then and then morphological components to segment the image into blobs and then measure the blobs and find their sizes. Another approach which I didn't show today is and I've got a terrible memory today for the names of commands. There is a content extractor command, let's see if I can look it up, which is trainable for. So the image identify command I used was for the whole image. But content detector function is a version of that kind of technology for sub image recognition. So here's an example where we are giving it examples of things you might see in an image. Oh, that's not the example. Here's the example. There's the image and we've got a rectangle that says within that rectangle there is an apple. Within that rectangle there is an apple. And in this image within that rectangle there is a Storbery. So we're showing it not just what's an image, we're also showing it where in the image and that is going to be capable of dealing with different sizes as well. So we might have small features and large features. We would want to give it examples of both. And then when you call the thing in in anger it, you give it the test image and it can now identify the thing it recognizes, but also the coordinates of where it finds that that feature. So that's the way you would go about that possibly, maybe as a pre step towards then using kind of classic image segmentation techniques, you identify where it is and then segment just that box to try and get the perimeter more accurately. You also might want to look at some of the image segmentation neural networks that are available in our network. And I'm going to give you a little glimpse here of a, I remember the name of it, a command that doesn't exist in the Wolfram language today, but we are shipping version 14 very shortly and there is a, there is a new neural net based image segmentation filter and I'm just going to get that quickly by and. By going to image segmentation components. Image segmentation filter. So image segmentation components is doing classic image processing type operations like segmentation, but using a segmentation neural network to achieve that in quite robust conditions. And the filter is a way of doing things like for, for a given pixel that we're interested in, what is the component that is around that. So can we pick out for example the carrot or aubergine or whatever it is in this image. And it's fairly robustly found the extent of the segmentation of the image. So that might also be a useful first step that is crossing between the two worlds. The machine learning features that everything I've shown is available on the Raspberry PI as well. But bear in mind that Raspberry PI is probably a terrible place to do the training because it's slow and has fairly limited memory. So in practice you would probably want to do the training side of what we've done on a desktop computer or in the cloud, and then copy the trained model onto the Raspberry PI to execute. You could do it on the Raspberry PI. There's nothing to say that it can't do it. It's got the capabilities, but it's pretty underpowered hardware for doing serious work. Okay, so Malia, your problem sounds like something to do with something wrong with your installation of Mathematica. So drop a, an email either to me or to TechSport and we'll get somebody to try and figure out why that stuff isn't running on your computer. I don't think it's a general issue and unless I am missing any from the YouTube, okay, they've been copied over. So I think we're now running out of questions and time simultaneously. Ten minutes past the time due. So this seems like good point to stop. So let me just wrap by. Thank you for all who've stayed to the end for listening. I'm back next week talking about LLMs in more depth. This is the link to the webinar if you want to go and register now or you can if you've got your phone on you. There's a QR code that you can use to get to that same link. Do come and join that or any of the other webinars in our series. And are we doing a short post questionnaire? So it'll be really useful when we close the webinar in a few seconds. If you hang on just for an extra 10 seconds, there'll be, I think, a three question questionnaire that helps us to understand whether we're giving you the right kind of content. It's really useful to us to please stay behind for a few extra seconds and click three buttons to say what you thought of today. Thank you very much.