Neural Networks for Audio Effects

I'm Carlo Giacometti. I work in a sound and vision department at Waltham Research. And today I'll talk to you about neural networks for audio effects. Now, I'll start with a hopefully brief and clear and not too confusing introduction. And what are audio effects and how historically they have come around and why we like them and why we use them. And then I will look at one specific paper that tries to emulate some audio effects using neural networks. So, talking about audio effects, we need to start to talk about recording sounds. Sound as acoustic waves. Recording acoustic waves has been done in different ways across history. A big start happened in the late 19th century with the invention of the phonograph, where sounds were transduced from acoustic waves to mechanical vibrations. These mechanical vibrations were transferred to a stylus that would then record those onto some soft medium like wax. So you would have a way to transduce acoustic to mechanical. And then you could do, with the phonograph, the inverse process. You could listen to the recording. In the 20th century, we start a completely different way of recording with the introduction of transducers like microphones and vacuum tubes. We have means to transduce an acoustic sound into an electrical signal. And this signal can be then manipulated, amplified, changed, processed by means of electrical circuits, initially based on vacuum tubes and then transistors. And then whatever after this step happens, there is an additional transducer that takes an electrial signal and still writes it onto some mechanical medium, still like a wax master. And again, this can be pressed onto a vinyl and then reproduced. And you can listen to recordings as well. This is some old timey microphone, some old timey console, and there's some example of circuit that involves some tubes that is essentially just a preamplifier. The next jump in recording techniques came with the introduction of the magnetic tape, which was invented in Germany in the 20s and then kind of had bigger success in the US after the Second World War. Signals could be now recorded directly on tape rather than on wax or some soft medium. So it was a magnetic recording rather than a mechanical recording. The level of fidelity increased dramatically. There were very innovative recorded techniques. You can listen to Les Paul and Mary ford in the 50s, doing experiments with multiple tracks of recording different speeds of tape. This unlocked a lot of creative techniques. And then finally we jump into the digital era from the 70s, where compact and more affordable digital recorders start to appear. So we take an acoustic signal, transduce it to an electrical signal, and then sample it with a analog to digital converter. So we lose it's not a continuous signal anymore. We abandon that and we have discrete signals. This discrete signal can then be stored in a file that can be pretty compact. And then the recorded file can be played back through a digital to analog converter and gets transduced again to an electric signal that then can be played back on speakers and you can listen to it. But the fact that we now have a digital signal means that we can process it by digital means. And this unlocks a whole universe of possibilities for signal processing. Right? However, there is one interesting bit of information. All of the music that has been had been recorded prior used analog effects based on tape, based on transistors, based on vacuum tubes. And people had grown accustomed to those sounds. So those kind of processes, those kind of effects were still the sounds that people would to listen to. And this is true still to this day. People still use analog effects, still use tubes, even though it's a very ancient technology. But those things still sound good because they are in the collective sonic memory, I guess, of humanity at this point. So one big potential for digital effects was to try to emulate the sounds of magnetic tape, of vacuum tubes, of specific circuits that were built and were popular in the past. And there have been many different strategies to achieve this descent. One was just to build transverse functions from scratch and tweak parameters until things sound close enough, good enough. Another strategy has been physical modeling where you build a model of the circuit of the effect that you're trying to replicate. You know the differential equation for a resistor, you know, the differential equation for a transistor, capacitor, whatever, you can build them together and simulate the circuit using those means. And by the way, we have some very excellent examples from Wolfram employee that works at System Modeler. He has used System Modeler to build simulations of several analog circuits. And I have linked in in the notebook a few of the blog posts that he makes that is excellent content that you should 100% read multiple times. It's very, very fascinating. So this is very successful. However, it involves a lot of knowledge and effort and software. Another direction that hasn't been explored that much is, as you probably can even guess, neural networks. So I will look at one very, very small and specific example. Last year, year and a half, this paper was published Efficient Neural Networks for Real Time Modeling of Analog Dynamic Range Compression. And I will look at this paper and see the architectures that they use and the issues that they overcome and why this specific kind of architecture has been proven valuable for emulating effects. And I'm not saying that this is the way. The interesting thing is this is one avenue that can be taken through neural network to achieve this result. So the problem is. So we have this big guy here is a compressor. Essentially it takes an input signal and if at some point the peaks or the volume is high, the peak gets reduced dynamically. This specific one is a fancy one, very expensive, still uses tubes. It is still being produced today, costs $7,000. Something pretty ridiculous and it is pretty subtle, right? So okay, we don't want that volume. This is an example of a signal and this is the result of the processing going through that unit. And here you can see a plot of the loudness of the two signals as a function of time. And you can see how the yellow one, the compressed signal, has the peaks produce. Anyways, this sounds good to music producers. This is a very sought after audience item and there are solutions that simulated through physical modeling. This paper wants to try to answer the question can we build a model of it through neural networks and can it be as minimal as possible? And can we run it in real time? And can we still have. You can see how this guy has these knobs that change its behavior. Can we have a neural network that has knobs that we can tweak in real time? So these are the kind of things that this is trying to achieve. So requirements, as I said, fast allow conditioning for the knob positions and minimal training data. There is a data set that's called signal train that has input, output pairs of unprocessed and processed signals. And that's what the paper uses. I will give one caveat. The implementation that I have in the vulceworld language has some issues in terms of speed. So this is kind of more educational rather than achieving a goal. I think there are some issues with the neural network framework that we will address and make this fast. So what goes into the neural network? Many audio nets take high level features to minimize the amount of data that goes in and needs to be processed. But this implies a giant computational cost and reduces the quality of the output. Because for example, if I feed a MEL spectrogram and the network spits out ML spectrogram. Now I need to somehow reconstruct the raw samples from this high level feature and that is an approximate operation. It's an expensive operation, we don't want to do that. So in this specific case, the network is fed the waveform samples directly. There is a lot of data, but the architecture makes sure they can handle it. Some networks reduce the sample rate of the signal to Minimize the processing that needs to be done. However, again, that is a loss in quality of the result. We are doing this potentially to use in a recording studio. We want the best sound quality possible. So full sample, we do not make any compromises. The proposed architecture in this paper and a whole ecosystem of papers around it, there are a bunch of references that you should check out at the end of the notebook. As I was saying, the main architecture is centered around a stack of temporal convolutional networks. These TCN blocks here and then here on the side there's a silly multi layer perceptron that handles the knobs that we want to control our net with. So let's see how this TCN works. This is kind of the central element of the architecture. At first glance, it doesn't seem too strange. The crucial element in the processing is this first convolutional layer. It's a one dimensional convolution. It can have padding that can be all on one side or spread between the two sides of the signal. And these are called causal and non causal pattern paddings. They serve different purposes. If you want a real time application, or if you want to do things like time series forecasting, you do want causal padding, which means the output at time T only has knowledge of inputs at time less than T. However, if you have access to samples in the future, why not take advantage of them? So again, this convolution, another important characteristic is that we want so for example, the sample at time T, we want that to have kind of knowledge of the past up to a certain point. And this certain point is called the receptive field of that specific layer in general of that network. These audio operations usually have a memory of the past and they need to take into consideration what happened in the past. In general, a good estimate for dynamic range compression is that you want a receptive field of a few hundred milliseconds. You can achieve this by having convolutions with giant kernels or having thousands of convolutional layers. Either of these solutions is terrible because it will take ages to compute, because convolutions with big kernels are very expensive. So the solution that is used in this specific case is using convolution with dilation. Dilation essentially injects holes in the kernel. So a convolution with weights, for example here, 1, 2, 3, with a dilation factor of 2 is exactly equivalent to a convolution. A normal convolution with the kernel that has 1, 2, 3, but these zeros interjected between the elements and the operation is kept efficient. We don't do the multiplication by zeros, we just do the multiplication by the actual weights. So we increase this receptive field without incurring into a computational, into the computational cost of having bigger kernels. And also the other thing that you can do is stack layers of this kind with exponentially increasing dilation factors. So the receptive field will grow within layer index. And generally this dilation factor increases, at least in many architectures, increases as part as 2 to the n or 3 to the n, which correspond to the blue and yellow curve. In the specific paper that we're looking at, they are attempting at using a dilation factor of 10 to the N. So you can see we reach a receptive field of 300 milliseconds with just three layers. So this reduces the amount of operations that need to be performed. The second interesting bit of this TCN block is the feature wise linear modulation layer. So remember how I said we want to be able to tweak knobs for the network? Well, this is what does that. For each of these TCN blocks, this film layer is embedded. And what it does is applies in a fine transformation to deactivations of that block given two parameters, gamma and beta. And these parameters are derived by a multilayer pass appliance that takes the input of the knobs, transforms it to a vector that is then dimension reduced and scaled for each of the TCN blocks. And you can see how this works here, right? So these are the values of the parameters. This outputs a giant vector. And then for each of the TCN blocks, there's a small linear transformation that transform the giant vector into the two gamma and beta parameters. This was introduced in an image question answering network where the same operation was applied to each of the residual blocks there. And this is doing essentially the same thing. Another interesting part is that the activation choice, you can see this is a parametric relu, which is a RELU nonlinearity that contains a trainable parameter alpha, so that controls the slope for inputs that are negative. So if the slope is zero, we get a normal RELU non linearity if it is less, sorry, if it is different than zero, we get what is called a leaky relu. And this, as I said, this alpha parameter is trained during the main training. And the last ingredient in the TCN is this residual connection. The input to the TCN is then passed by passing all of the layers there. Simple scaling is performed by means of a single one by one one dimensional Convolution. So that's just a scaling parameter essentially. It doesn't do any weird stuff. However, this convolution in the TCN loses some samples. The padding happens only in the first layer. All the other ones drop a few samples. So we need to do some cropping of the skip connection to ensure that the dimensions are the same. And that also, as I mentioned for the padding also, the cropping can be done causally or not causally. So we can drop stuff at the beginning or we can drop stuff both ends. Again, the causal one is usually the one that is used for real time applications. So this is the tcn. How are the processor parameters handled? Well, I've kind of already mentioned that the parameters in our case it's just a vector of two numbers are passed to a multilayer perceptron. This produces a feature vector and this gets scaled for each of the TCN blocks. And this multilayer perceptron, these linear layers are trained in conjunction with the whole network. It's not a separate process. The last ingredient, and I will try to go very quickly here, is the loss. Judging how the result of the network is different from the target is kind of tricky. We can just look at the waveform and take a mean absolute F error between the two waveforms. That does not perceptually tell the whole story though. You can have two signals that sound extremely similar, but the waveforms could be out of phase or completely different. So this is one part of the loss of disease. The other one is more perceptual. It is called multi resolution short time Fourier transform. And essentially it consists of two elements. A spectral convergence that compares two short time Fourier Fourier transforms, locks at the norm and divides by the norm random and then the spectral log magnitude term that Locke just looks. So what we do here is look at this kind of combined time frequency representation and compare those. And we can, in addition to that, we can compute that at different resolutions to make sure that we are not training the network to only satisfy one specific kind of short time Fourier operation. So this is kind of the more perceptual loss that is used in training. An additional metric that is used in evaluation, not in training is the comparison of perceptual loudness because again, we are doing a evaluating a dynamic range compression operation. So this is kind of the metric that people are accustomed to. See, I have here a wolf of language implementation. As I said, it has some small issues that need to be solved in terms of performance. But we will talk about that with Julia and see if we can figure out what's going on. I will not go through the whole thing because it is just the code for what I've been talking about. But you can see the finished net. We have the multilayer perceptron, the linear layers that feed the parameters to the stack of TCNs. And then the output. And I will present the result, The input signal. This is the actual analog unit. And this is the TCN network produced output. This has been evaluated by human testers and had had very similar scores between the target and the tcn. So the whole point of this is to show you that neural networks can indeed emulate the effects, emulate audio effects. And there are many different avenues that you can take. One is based on TCNs. Other papers have been published using LSTMS. WaveNet has been one very big item in terms of processing raw samples. There have been several papers talking about differentiable audio effects. And the research in this has kind of exploded in the last three to four years. So I encourage you to have a look at the resources that I'm linking here. Follow the rabbit holes, follow the citations in all these papers, look at all the Wolfram language capabilities that will allow you to implement all of these things and have fun. I think that's it. Thank you very.