Calibration and Validation of System Models

All right, welcome. Today we're going to talk a little bit about calibration and validation of system models. So just to give a brief recap of what I'm going to be talking about, first I'm going to give like a very quick introduction to System Modeler. I'm going to talk a little bit about how having we, we've been working on introducing more and more avenues for data to be part of system modeling. Then we're going to talk about our new functionality for calibration, which is in the form of the function system model calibrate. I'm going to talk about also about the function that we use for visual validation, which is called system model uncertainty plot. And then at the end I'm going to talk about an example that I've worked through which uses these two functionalities. Right? So first, for people that are not super familiar with system modeler, System Modeler, this is a nice tool that is used for component modeling. So it allows you to leave aside equations and to just take care of connected components. By default. You already have a bunch of libraries that cover a lot of domains like thermal modeling, fluid modeling, mechanics modeling, magnetic modeling, electrical modeling. And all of these components can interact with each other to create nice, interesting multi domain models. Now this tool is very nicely connected to Mathematica as well. So these models live in Mathematica in the form of system models and these two tools talk to each other in a nice way. So for instance, here I have a model, a simple model of a NRC circuit and I can visualize it nicely in System Modeler, work with the components, see the parameter values, et cetera. Right. And then of course we have a ton of functionality around these models which has been growing and growing and covering more and more interesting user requests. For instance, here we have a very basic function that we use which is called system model plot, which lets you pass a model and then you can just plot variables in your model. The advantages of working with components over equations are kind of, they can be seen very clearly in models like this where I can just have components for a full motor or a component for a, that I made personally with just a cart and a pendulum at the top. Then I just have to plug them and then have an input for the voltage and then I can just run down with whatever parameter values and inputs I want and then I just can immediately get simulation results, which is very nice. Of course, not to be said that the equations are not important, but they do become a little bit tricky if you want to handle These kind of problems with equations because you have to be very careful about all aspects. Here I have 20, for instance, be very careful about the moments of inertia of these three of the objects that I'm computing. Instead, system model here is taking care of this, all of that automatically. Yeah, of course I can set parameter values and use equations and solve them in Mathematica. But it's always good to have like an overview, like a big picture overview of your model when you are working with system modeler. That's what we had. That's kind of a very quick introduction that I always give about system modeler. Now, going a little bit to the past. Last year in my presentation I was talking a little bit about the things that we have already available for data and system modeling. Obviously you can set some data for interpolation, you can set statistical speeds to system modeler. And last year we were showing how you can use already trained neural networks since it's the modeler. And then we had this small indication here which talked about parameter calibration or the fitting of parameters in your models. And then what we said back then was that we really, really wanted to improve that workflow because it was possible, but it was very troublesome for the user. So that was one of the motivations. And today we're kind of fulfilling promises with system model calibrate. So the physics function does exactly what you would expect it to do. You give it real world data and you have your model and the function fits the parameters in your model so that your model now behaves as the data. Yesterday we had a question asking about the arguments that you can assign to systemoire calibrate. So I'm gonna go a little bit into detail about that. In this most basic form, you can just have data and then specify which variable in your model is supposed to follow that behavior. Then you specify the model that you want to calibrate and then you specify which parameters you want to calibrate. And in this basic form, this is going to produce the calibrated model as output directly. So it's just going to work the optimization problem. Basically it's just going to minimize the distance between the data and the simulation results, which is often called a loss function. Now what? But since people were asking about like the full spectrum of things they can send to systemoac calibrate last yesterday, let me go a little bit more into detail of other things that you can add to the arguments. You can add. Obviously you can add parameter constraints, you can add initial guesses to Your parameter values. You can add additional specifications like which model name you want to give to your model to your calibrated model. And you can specify a whole list of properties that you can extract from this calibration process. Because of course you want to say how good is my fitting? What kind of. Well, the parameter values that I obtained, the confidence band, the confidence intervals for my parameters, all of this information is stored in a new object that is called calibrated system model. And then you can just query from that object all of those properties. You can also send validation data to compute validation scores. We're going to see this in example right now. And obviously you can fit more than one variable. We are going to see it also in examples right now. This function was listed in 13.3, so you can already see this in the online documentation for system model calibrate. For instance, here we have a model of a heating system. So let me show that in System Modeler we have all sorts of fluid components. We have an input and step input, we have measurements and tanks and it's a relatively complicated heating system. And then we have data for the temperature of the components. In this particular case it's just the temperature of the partner. Of course we accept data with units because this model store all kind of information and we just take care of all the unit conversions for you. And yeah, so to call systemal calibrator you just have to specify that this data that I have corresponds to this specific variable. I specify a specific constraint for my parameter. In this case it's just the amount of heat flow in that component. And then I just specify that I want the object calibrated system model from which I can query information. You are going to see when you run this. I'm not going to run it because it's in the screen sharing is a little bit slower than it is normally doing. But you're going to see a project reporting like this. Basically we're going to. We show by default the current estimates of the values of the parameters in this minimization process. And we show up a logarithmic plot of how the loss function is doing while you're. While we are doing the minimization. And then obviously you have a stop bottom here. If you're already happy with the with the values that you have, then you can just stop the minimization and we obviously return an object with all the corresponding guess values at that moment. So I mentioned that we have several properties that you can extract for that if you. If we can compute the sensitivity for this variable in the model, then we can compute some limitations. That is called the prediction bands plot for the minimization process. So we have here in blue, we have the final result of the simulation. We have in green the data that we use for calibration. And in yellow we have the confidence bands for prediction. You can specify also confidence levels here. So you can put by default, this is just a 95% of confidence level, but you can change that as well. And here, of course, you can plot for instance, the residuals which could be of interest. If you want to see the validity of the fitting, this is what you are going to see. Another example. So I promised an example where we would fit more than one variable. Here we have real world data of a biochemical reaction. So there were 50 measurements of the concentrations of two reactions in a specific chemical process, which symbolically looks like this. And here we use what we do. What we do of interest in this process is that we try to be fit both concentrations at the same time first. The other thing is that we calibrate three parameters at the same time. And also we set some of the data for validation. So in this particular case, first we select just most of the data. We select, I think, 49 of the 50 samples as the training data. We pass the data to system model, calibrate, we specify the constraints, the parameters that we want to calibrate. And here we specify that we want to run the simulation for 20 seconds. If you don't specify the simulation interval, we just deduce it from the data that you are providing us. Right. And as you can see here, we obtain the single prediction. Well, we run the calibration, we obtain the nice fits for the parameter values, we obtain the prediction bands. So as you see, we have returned the plots for both concentrations that we fit. And then you can specify, here we specify the validation data and we ask for the root mean square error just to get some validation and training scores. You can specify the validation data either before, during the calibration or afterwards. That's fine. And we return the scores that you want us to compute. Yeah. Another interesting application of parameter calibration is, for instance, small simplification. In this particular case, what we have is a model which is a little bit complex, but in the sense that you have here a model of a speaker. But we want to. Instead, what we want to do in this case is just make this much simpler model which just throws away the speaker and replaces it by a simple inductance. We want this simple model to behave as closely as possible to this. So maybe you want to reverse engineer a product, or maybe you just want to do some testing in a very cheap product before actually testing in the full final product that you want to sell. Right. And if we just can, we can just first see how the two models behave before calibration. As you can see, they're very different. But then what we just do is just we take the simulation results from the complicated model and then we pass them for the calibration of the simple models. And then we specify here that we want to calibrate the inductance of the new simple model. And then we obtain a calibrated model with this value. We plot the two results, and then we see that now they behave much similarly than before calibration, I should say, by the way, that we have in the handouts, you can find the notebook that I'm using. So just to keep in mind, right, so that's for calibration. Now the new guy in the block is System one Uncertainty plot. First, let me talk a little bit about the functionality that we already had before about plotting. So for instance, with System one plot, which is a much simpler function, you can pass a model and then specify that you want a specific variable to be plotted, and you can specify also parameter values in the process. So that's pretty straightforward. Systemwide plot also allows you to give several parameter values for a single parameter. And in that case, what system well plot is going to do is going to run one simulation for each parameter value and then it's going to plot each each results. In this case, we are given the values 1, 2, 3 for this particular parameter. Other thing that you can do is if the model knows the computation of sensitivities, you can simulate to compute sensitivities with respect to that parameter, and then you can see how that specific variable changes when that specific parameter changes at first order. So that's a tricky important thing about sensitivity. That's a first order computation. And then we also have a function, a repository function called System oil Manipulate. System oil Manipulate creates automatically a manipulate that. Let me just run this manipulate automatically for your model. And you can specify which variables you want to see which parameters you want to control. Basically just like a regular manipulate. Right. So here you see we can see the same model that we've been talking about, but just changing the specific parameter, however we replace. So that's kind of an interesting repository function that is very useful. So that's the state that was the status a year ago, but now we are interested In a new thing, which is we want to explore all the possible ways in which a model can be uncertain. So a model can be uncertain because maybe the parameters are uncertain, or maybe the initial values are uncertain, or maybe we want to pass inputs to the model that are uncertain on their own. So maybe they are random processes. And of course, what this is going to do is it's just going to give us a full rainbow of simulations. And this particular case, what we're going to do is just, we're going to just run a bunch of simulations trying to sample all of these uncertain intervals or distributions or random processes. And then what we do is by default, we compute quantiles for these simulations. So we aggregate all the results, and then for each time point, we compute quantiles, just to see which percentages of our simulation results are where or which kind of behaviors we explore once we explore the full range of possibilities for a parameter. So as a simple example, for instance, we go back to the same model that I've been using, the motor model, and what we're going to do is we're going to plot the same variable that we've been talking about. But now we're going to specify that the parameter that we've been using, it just lives in this particular interval. And then we're going to sample that interval. By default, we perform 200 simulations, and then we compute these, quantize for each time point. So as you can see here, we see in blue, we. Well, in solid blue, we see the median, and then we see the range from 25% to 75% of the results. And then the full range from the minimum to the maxima of the results and the amount of things that you can pass to this function is crazy. You can pass intervals around distributions, regions, geometric regions. We can sample full regions for a parameter value, and you can give us a list of inputs or random processes for inputs like noisy signals, for instance, just to see some examples. Okay, maybe we can skip that because it's a little too basic, but maybe. Let's go back to the speaker. When you design circuits, you are very. Your interest is you're probably interested in having components and be really aware of the tolerance in your components. So for instance, in this case, my resistance in one of the resistance in my model of a speaker follows a normal distribution. And then I want to see this uncertainty in the resistance. What kind of behavior is going to return for the behavior of my speaker? And yep, here is what we see. So we see that there is some visible deviations from, from, for the current that goes through the speaker. We can also be interested, for instance in seeing what happens with my speaker if I have some noisy high frequency signals. So let's say that the voltage that I'm passing to my speaker right here, it's very, very noisy and it's. Yeah, there are some high frequencies voltages that may be affecting my components. So I do that. I simulate 200 times my model sampling the inputs that I have given them and then I can see how at small times scales, how the currents are going to impact my speaker. Maybe this is dangerous for your product, maybe it isn't. So that's on the side of system tolerance. In the other aspect that you want to do is validate your models visually. Maybe you have a control system like this. Typical archetypical example of a control system is a ball and beam model. So you want to control the ball on top of a beam and then just try to keep it as stable as possible, maybe against some disturbances or maybe against other factors. And then you can test this controller that is keeping my ball on top of the beam. What is it doing, how bad, how well is it behaving against the stress factors? So that I can do, for instance, I can push my ball away from the center as much as possible and then see if my controller can handle that or how far can I do it. Here I'm plotting the position of the ball on top of the beam when I push it, when I start, the position of the ball as far as far from the center as I can before it breaks. And then as you can see, the controller does struggle at some point, the ball almost reaches the end of the beam, but then it just manages to bring it back and put it back into the center central stable position. And then I can see how the different ranges are covered depending on the initial values of the for the position of the ball. What I can also do is just give disturbances to my control system. So for instance, here I'm giving it right here I'm using white noise as in disturbance. So this is just going to be a torque that is going to keep pushing. Annoyingly my control volume system, just to see if it can throw away the ball. Here we can see what's happening. So I decided to sample that white noise every 3.3 seconds. So here is the first attempt of the noise to throw away the ball and then the controller reacts to it with some torque against that noise and then just brings the ball back to the center. And then again there is something pushing the Beam again. Then it brings it back to the center and pushes it rigs to the center and so on. Right? That's for visual validation. Now we can go to an exam pool, which kind of puts this together. So, for instance, something that you can now you can already get by the system modeler is you already have components for a ton of stuff. One of the stuff that you can find already there is things like chip. Well, a chip in a gravitational field. For instance, this could be a satellite or the International Space Station. And let's say that I'm interested in the problem of doing orbital maneuver. For instance, maybe I want to move it to an orbit. So usually you want these things to be in a circular trajectory, but then I wanted to move it to a different radius. Maybe I wanted to push it a little bit away from some space trash or something like that. So how would I do that? How would I guess how to move my satellite? What kind of stuff can I do? I don't want it to orbit it chaotically or maybe end up in a elliptical orbit or just lose my satellite in the process. So one of the simplest things that you can do to do an orbit transfer is something called a Hohmann transfer orbit. So what you do is you first do a small boost of the satellite at some point in its directory, and then once the satellite is on the other side, you push it a second time. So the first boost is to change the trajectory, and the second one is to bring it back to a circular trajectory because otherwise it's going to be in an elliptical trajectory, and you're not going to like that. And what I'm going to do is, okay, I'm going to use my calibration function. So just to see, just to deduce what was, for instance, what's the magnitude of those boosts that I have to do there in order to keep my satellite stable. So I have already in system order my model of the chip in a circular trajectory. That's fine. That already does a lot of my work for me. What I'm going to do is I'm just going to create a very simple model that just does the two pushes, the two boosts, the boost on this side and the boost on this side. But I'm just going to leave the parameters arbitrary. So I'm going to create that model. I'm just going to say, well, what is that model actually doing? It's just doing a small push here, and then it's doing a small push here, and then it's doing nothing. More and then I'm just going to connect that model to my chip and then I'm going to try some values and see what happens. So if I try some for values that I have tried, that I tried here, as you can see what happens is here I push satellite a little bit too far and then somewhere here is doing the second push. But as you can see, it kind of brings us back to, brings it back to a circular trajectory. But it's definitely off center. So it really isn't really a circle, it's more an ellipsis. So it's not exactly what I'm looking for. But I would like to calibrate now those parameters and see what I can get. Yeah, by the way, if I plot how the radius change for each of these guys, I can see that the radius change from being something to something that is definitely not constant in time. These guys are being too wild. I'm gonna do something, the most basic thing, I'm just gonna create my own, the most basic function that I can think of for the change of the radius. I'm just gonna think of something that just changes linearly from the original radius to something new. And then I'm just gonna keep the radius like that and I'm gonna use that as the data for my calibration. So I'm just gonna create this artificial function, I extract some data from that and I'm gonna pass that to system model calibrate. So I pass the data, I'm gonna maybe specify some constraints for my parameters for my boosts. Maybe maybe with the exploration, the exploration that I did that maybe I guess some values, some guess values for the start point maybe as well. And yeah, here I'm specifying for instance that my new model should be called like this. So this is one of the specifications that I can give. Then I run system model calibrate and then it gives me some two values for my parameters. Interestingly, if I plot the values then it turns out that using this gross linear target, I did actually find something that does stay a circular, relatively circular. Like in practice. In practice most of these directories, like in real life, most of these directories are in exactly circular. There is always some chaos and effects from other bodies in the solar system. And then I'm just going to put together these two models of my, the original model and the calibrated model in a single model. And then I'm just going to run the animation, which is what I have here. So here we got the first boost for this guy and then this guy Goes a little bit wild. And then somewhere around here gets the second boost so that it doesn't get stuck in an elliptic orbit. And then this guy seems to be doing well. Let's see how well it does after a full orbit. Yeah, good. Yeah. Seems to be in a circle. So that's the new orbit of my satellite. And then, yeah, I can obviously plot it in System modeler. Sorry, Mathematica. This is what we are seeing here on the right. I can also do an uncertainty plot. Now let's see what has happened. If I had guessed the values of my boosts wrong, so what if I have been off by 10, then I can see how wild my trajectories will have become. Also I can use my previous functions. Yeah, I can use, for instance, let's see, I can take this plot that I have already produced. I can call System Wild Manipulate. That produces the manipulator I told. Then I can just use my uncertainty plot as a background. So now I can change the strength of the boosts and then see how that will have affected my model. It might be running a little bit slow today, but I think this is all because of the screen sharing. Right. So this is all now possible. Oh, okay. You woke up. Right. So yeah, the guesses that System Accelerate made weren't so bad. Right. So this is kind of where we are now. We are trying to explore other avenues where data affects system modeling. We are also very interested in exploring all these ideas of uncertainty in system modeling. So these are things. These are the new promises for the new year. So every Discovery conference now, I'm going to make promises for next year. We are planning to keep exploring designers, keep doing more of this field, seeing what comes and yeah, to keep connecting these two worlds of system modeling. Mathematica. Let's see if there are any questions. I'm going to wait a little bit for questions. Okay. Maybe the time is about to run out. But yeah, if you have any questions, you can always contact us, reach out to us. We're very open, very friendly. Yeah, thank you.