Multi-framework Neural Networks

Hello everyone, my name is Matteo Salvarezza. I am a developer of the neural network framework of the Wolfram language and today I'm going to present about NET External Object. So Net External Object is a new symbol coming in Warfare Language 13.2, which basically wraps external neural network evaluation engines to run neural network models from different file formats and different frameworks. So before actually starting. So unfortunately, if any of you is using the WTC 13.2 beta build, that net External Object might not work properly on all machines. So in case you want to follow along and evaluate things, I suggest you run this code. So these slides are on the presentation page. So I suggest you run this. This will download a 20 megabyte patch and load it into your session to ensure a smooth experience. It takes just a few seconds. That's it. Okay. Obviously this will not be needed when 13.2 finally releases. Okay, so Net External Object is created via import by specifying the NET External Object property. So this is an ONNX file, I can run this code. And this is the Net External Object. This is designed to be a generic interface to wrap different evaluation engines, each associated to a particular file format. Obviously for 13.2 the only format will be ONNX, but more will be coming if you are not familiar with what ONNX is. Onnx is a cross platform file format and specification for machine learning models with a focus on neural networks, and its natural evaluation engine is provided by ONNX runtime. So this has the same look and feel of neural networks in the world for language, but there it's actually not. There are analogies. I mean, in the panel here you can see similarly to neural network, it shows input and output ports with shapes and you can run it as if it was a neural network and it produces the results in the same form you expect from a network in the language. However, this is basically the only thing you can do with this thing. It's not a part of the neural network framework, so it cannot be combined with other layers. Because it's not recognized as a network, you cannot modify it and you're going to train it. So this is a generic black box container. So the scope of this functionality is as large as the scope of the evaluation engine itself, which, which is generally very, very wide, obviously. So it can do anything. Speaking of the ONNX format, he can do anything that ONNX runtime can do, basically, which is a lot. So I'm going to show here just a simple example of for a face detection pipeline using an external ONNX model, which can be found at this Ur sorry. So yeah, so here we have our network. So this is a phase detection network. So it basically takes an image and draws bounding boxes around faces. So here we have an image with some people. First step is to pre process this image. So in the neural network framework, the natural object used for pre processing is NET encoder, which is normally a touch attached to neural networks. But in this case this cannot be attached because again it's not part of the framework, so you cannot attach it. We might unlock the possibility to attach encoders, but for now it's not possible. But still you can still use a. NET encoder as a standalone preprocessor. So these are preprocessor and we are going to encode the image, producing an array with the same shape as what the network wants. Then we're going to run the network and get the output. So what is this output? Well, this particular model produces 17,000 something bounding boxes together with the probability that they contain a phase. So we have 17,000 bounding boxes and these four is obviously the coordinates of the corners. And for each box we have a probability that the box contains a face and the probability that the box doesn't contain a face, which is redundant because you need only one obviously. But this is how this model works. So yeah, so we can convert the boxes produced by the network to actual rectangle objects and extract the probabilities. Now obviously these are candidates, we're not going to detect 70,000 phases. So probabilities and boxes should be aggregated using the maximum suppression algorithm, which I'm not going to discuss, but it's a standard object detection post processing algorithm and it's available via a resource function. So resource function of maximum suppression, we got our nice compact function that we can run on these inputs to aggregate rectangles and probabilities and get our final detection. And we can show our final detection and you can see that it works pretty well. So this was just a simple example, but a wide array of applications are possible from NLP to video to many machines. Okay, so let's keep discussing net external objects. So it can be seen as an alternative way to import ONNX model in the Wolfram language. Because since version 12.1 it is also possible to import ONNX models as proper networks in the framework. So in a chain of graph and layers, and this is just done by calling the import on the ONNX file, this returns a net graph object. Now okay, there is another problem in the beta. This panel is not showing everything that is supposed to be shown, and it's only a problem with Linux. So if you're not running Linux, you should see everything here. But we don't need this to show everything for the sake of this presentation. Obviously this will also be fixed by the time 13.2 releases. So basically this code is equivalent of running import on the file using the. NET property, which is the default property. So using the. NET property I get the. NET graph. Using the. NET external object property, I get the. NET external object. So this might seem redundant, like why there are two ways to do the same thing. Well, in fact, they are deeply different, and we're going to explore this difference in the next few slides. First thing that should be said is that basically with this code, the ONNX file is analyzed, is looked into, and every single layer in this ONNX file is converted to a neural network language layer and they are assembled into this NET graph, which is something which is integrated into the framework. And you can run it and you can, sorry, you can modify it, you can attach it to other things, you can look inside it, etc. And it's also, when you run this network, this network is run by the evaluation engine used by the neural network framework, which is IMAXNext. In this way, the process is much simpler. We don't analyze the model, we just take it and feed it directly to ONNX runtime and. And it's treated as a black box. And also, obviously, when you run this thing, it is not run via MXnet, but it's run via the evaluation engine specified by its particular format, which in this case is an X runtime. So one important difference in the two approaches lies in the treatment of input dimensions. So you might have noticed that the input shape of the converted nets and the net external object is different. This is 3 and this is 2 3. And why is that? Well, basically, in onnx and most other external frameworks, some common layers like linear convolution or pooling. In external frameworks, these layers are commonly defined as acting on a sequence on a batch of examples, so on a list of examples as opposed to a single one. This concept doesn't exist in the word for language, because in the word for language, layers can freely act on both a single example and a batch, which is a property known as listability in the language. So if I define this linear layer with input size of 3, I can either run it on a sample single example of length three, or two examples or three examples or whatever, and it just works. External frameworks, for the most part, don't work like this in external frameworks, the number of these examples is fixed, is hard coded into the model. You can only run it on a number of examples that the model specifies. As a consequence of this, the inputs of these models and of these layers in external frameworks have an extra a dimension with respect to the Wolf language counterparts, which is called the batch dimension. So these two here is the batch dimension which specified that this particular net external object can only be run on lists of length 2 of examples of length 3. So when converting these layers, these operators to and from between onnx and the Warfare language, this batch dimension is automatically added and removed. So for example, let's check this pooling layer with an input shape of 2, 3. In the Wordflow language, I have exported this pooling layer to an Onnax model and now I'm going to run this magic code that I'm not going to discuss here. But basically this code looks into the file that we just exported and extracts the input shape of the shape of the input and the shape of the input has an extra one here, which is the batch dimension that was added by the warframe language. When the file is converted back, this batch dimension is removed, because again, we don't have this concept, so we remove it and we recover the original layer. So if I round trip the layer and convert it back, I have the same input shape. But when the file is imported at net external objects, we see this one. Obviously both the converted layer and the net external object, they do perform the same computation. So if I create a random input and I feed it to both, taking care of wrapping it into an extra list for the net external object, I'm going to see the same output with also where the net external object has also an extra dimension, the output, you can see it here, there is one here and one in the output. Now this size of one is just a convention used by the ONNX exporter, but it can be anything. So here I am creating an onnx file. This byte array is just the content of a legal ONNX file. And if I import it, converting it into the Warframe language, I will see that the input shape is 144. But the underlying model has actually batch size of 10, which shows up here in the NetXL object. Now, this extra batch dimension doesn't always appear, it only appears when layers of this kind are involved, which I'm going to call as batched layers. So for example, aggregation layer is not batched. And by this I mean the ONNX counterpart of this particular Layer is not defined to act on a batch of examples in onx, it's kind of a convoluted sentence, but that's what it is. So in this case, if I export this aggregation layer with input shape 2, 3 and import it back as an external object, I don't get the extra batch dimension. So it depends. This is a non trivial machinery because if I take an unbatched layer like this and combine it with a batched layer in the workflow language and do the same and play the same game, I am going to see now an input batch dimension because it's coming from this guy, while if I import it back, converting it back into neural network, proper neural network, I don't see it. Okay, let's keep going. Another difference is the fact that external frameworks, including onx, they treat weight, they treat weights as inputs. So and this can be seen. So consider these convolution layer, if I again if I export this convolution layer to onnx and import it back as an external object, I'm going to get an external object with an extra batch dimension, as I just discussed, but with a single input, just like the convolution layer. But if the layer is uninitialized, you can see there's netinitialized call here, which initializes the weights and biases arrays inside the layer. If I don't initialize the layer and do the same thing, the net external object is going to have an additional input for the weights and additional input for the biases. And this is going to happen for every uninitialized layer in the network. Now in these simple examples I'm just round tripping single layers, but generally no. A network has many, many layers and if the network happens to be uninitialized, I'm going to get all these extra inputs for all the missing arrays, weights and or biases. And we can obviously check that. Plugging the weights into the convolution layer and feeding the same weights and ETXEL object as input gives the same result. So here I'm defining weights, biases and a random input. And here I am plugging the weights into the convolution layer and running the convolution layer on input versus just running the object on everything, treating everything as an input. And again I get the same results. So why is this happening? Well, because again, OnNX, as most of other external frameworks, effectively treats weights as additional inputs. They don't show up here if the layer is initialized, because in the model file there is Information which basically provides the value of the weights as a default value for these inputs. So they don't show up because they have a default value attached to them. This default value in ONX is called initializer. Okay, so yeah, we are at the end of the exploration now I'm gonna discuss a few important points. One point is like one legitimate question that might come to mind is why are you adding another way to import onnx model as black boxes when we can already import them as proper networks, which is obviously better because you can do more things with them. Well, first of all, with respect to specifically to ONNX model, conversion to work for language, as I discussed before, is a more complicated process because you have to go into the network and convert every layer and it's quite error prone unfortunately. And it's mostly because of the differences between ONNX and the word for language I've discussed previously. So the fact that sometimes there are extra dimensions somewhere if there are particular layers which are defined differently in ONNX and the Warfare language, and the fact that it treats weighted inputs, while on the other hand importing to net external objects, it is way easier to develop and maintain because you don't have to look inside and convert things. You take the file, you feed it straight to ONNX runtime and it just runs. And so it's also much more reliable and it's basically as solid as using ONNX runtime directly via Python or via C or whatever other language ONNX runtime is bound to. But also do not forget that NET external object is not just about onnx. So we're going to add a support for more formats for which there is no dedicated importer at all. So one of them which is very easy for us is mxnet, because mxnet is already part of the framework, is already included in our system because we use it. The top level framework uses MXnet to run networks, we have a dedicated inverter for it. So does the same thing of going into the model and converting every layer, but we deprecated it. But aside from xnet, we're also targeting the other major which are currently PyTorch and TensorFlow and possibly other things. So and there is a lot to gain from that. And lastly, these wrapping external frameworks into net external object is the natural entry point to including these external frameworks into the language, which is a first step towards adopting them as additional backends to the neural network frameworks alongside the next Net. So this is a very is a long term project and we haven't started yet. But this is what we're aiming for first. We're aiming for, like, including these frameworks via Net External Object and then slowly starting transitioning to adding additional backends. Okay, so, yeah, this is the end of my presentation. We have four minutes. So if there are questions, please ask.

For one of those external models that you are able to import and bring into the Wolfram frame system, does it execute faster or slower than when you ask the external runtime to do it for you?

Yeah, so. Well, this highly depends on the model because, like, you know, it's not one thing, it's not just mxnet versus ONNX runtime is which particular models, which operators, which layers does it contain. And we don't have comprehensive benchmark data on this. It's very hard to get because again, there are a lot of operators and a lot of things that you can possibly test. I have done a few tests and sometimes mxnet is faster, sometimes MX runtime is faster. But one thing I'm going to say is that you generally might expect it to be a little faster on average if you go the Net External Object routes. And that's because basically every time you convert a neural network model from a framework to another, there is going to be some efficiency lost, because in general, one single operator in the ONNX model file does not translate to a single operator in the warframe language. And so the more models are converted, the more the number of these operators grow, which usually makes the evaluation time longer. And this is also obviously true if there is a previous step. Suppose you have a model that you developed in Pytorch and you export the model in onnx. You might lose something there because a single Pytorch node can become more than one ONNX node. And then you go from ONNX to orphan language and you do it again. So the more you do it, the more you can expect to lose performance.

Okay, Matteo, and one more quick question before we wrap up.

Under the other category, have you considered the Ludwig framework developed by Uber? Well, this is way too early to really consider things like we've just started, I've put other here, so I mean, we're open to everything. We haven't really went beyond these two major players here. We'll see in the future if we are able to add more.