Wolfram Student Podcast S1.5 Ep 1: Accelerating Simulations of System of Particles

Hello everyone, and welcome to a special episode of the Wolfram Student Podcast. I'm Sam. I'm Roshank and we're your hosts.

This is the Wolfram Student Podcast where every fortnight we dive into a new innovative project done by high schoolers using the Wolfram language.

For this episode, let's welcome Nicolo, Logan and Theodore who will be discussing their wealth project on the accelerating system simulations of systems of particles with the transformer based machine learning architecture. Hi, guys.

Hi. Hello.

Hello.

So, yeah, it's great podcast. So do you mind just starting this off with your name, grade, where you're from, etc. Just so we have a better idea of who you are. Logan, you want to go first? How about yeah.

So, hello, I am Logan Gilbert. I am currently a junior in high school and I was a part of the 2020, 2021 and 2022 Wolfram Summer Camps.

Hello, my name is Theo Mustakis. I am currently a senior in high school and I have been a part of the 2021 and 2022 WSC camps.

Hello, I'm Nicolo Monti and I'm in second year of high school and I attended the Wolf from 2022 High School Summer camp.

Yeah, sounds pretty cool. Can you guys maybe give us a little bit about how your whelp was? Because obviously you all just did whelp this semester.

I think this year whelp was. It was really fun. Obviously. I got to know both Logan and Nicolo during camp this year and it was exciting to get in a group together and just pursue our interests ourselves.

Yeah, sounds nice for our listeners. For context. Welp is short for the Wolfram Emerging Leaders program, which is a semester long program where people from camp work in like groups of three or four and they pick a topic and they just dive deeper in it.

Yep. And for. For your information, there's also a well Pew program which is. Which got started a bit more recently, but it's the same as well, except for undergrads. So if you're interested in either of those, feel free to look it up and visit the wildfrom websites. So, coming back to the team, this is pretty good question we would like to do. Do you want to tell us about your favorite Wolfram function if you have one?

I would say it's kind of a basic answer, but I really like the map function simply because it's so different from other languages that I've used in the past. It just enables so many unique aspects of the of the Mathematica language that I don't think I'd be able to use Wolfram language nearly as effectively without map.

I honestly agree with. I honestly agree with Theo. I haven't really thought too much about my favorite function, but MAP is just very powerful and sometimes even when I'm not doing Wolfram language, I just think of defining things in terms of a MAP function. So it's just kind of become a part of my brain.

So.

Recently I've used the external evaluate function a lot and it's really, really convenient to use because it allows me to integrate my Wolfram language programs into like a wide. Because it's like integrates programs for await ecosystem into the Wolfram into the Wolfram language. So it's really convenient because it allows me to glue together all of my code.

Yeah, those are amazing answers. And I agree with what you said about the MAP function. It's. There are these things about the Wolfram language that are really convenient when you get used to them and then you wish they were in other languages. You. So. Yeah, tell me.

So getting into your project, can you first give us a quick summary about what it is? Because your title sounded pretty complicated.

Essentially, our project applied machine learning to a simulating scenario called the electron two stream instability. The two stream instability is a scenario in which two beams of electrons have in a one dimensional environment are traveling in opposite directions and when they collide with each other, the moving charges of one stream affect the electromagnetic fields and apply forces to the other stream, which creates a unique instability in the velocity and density distribution of the system. And so our project looked at using machine learning to both accurately model the system as well as significantly reduce the computational expense and time taken in order to achieve the same results.

Okay, sounds cool.

All right, so our project is divided in three main sections. In the first section we explain what the problem is and what is the motivation for our project. The then we go ahead and describe how we gather the data and how we processed it. And then we explained how our architecture works and our system accomplishes its task.

So yeah, starting from the top, we already talked a little bit about the two stream instability, but that is the scenario which we focused on modeling in our project. In order to train a machine learning model, one needs to first have.

Some.

Data to train it on. And so in order to get our data, we found this open source code, epic, which is a well known and trustworthy code used by the plasma modeling field that uses the particle in cell method for modeling the particle dynamics. Particle in Cell is a molecular dynamics simulation method that Uses explicit individual particles like the electrons and nuclei of atoms to generate systematically generate electric and magnetic fields on a grid lattice and then use those to calculate the forces and motions of pseudo particle particles which are essentially just clumps of the explicit particles. And our particular simulation was in a one dimensional environment similar to like a particle collider. In order to generate our data, we wrote a Python script to run the particle and cell simulation on a Linux machine and just collected data overnight using a specific predefined starting condition.

So now I would like to talk about our machine learning architecture. Now the first thing to understand is why we then applied machine learning to this problem. Problem is that to run the simulation we had to use a very powerful machine which is obviously not accessible by anyone. Because this kind of computation takes is really computationally expensive. So we, we decided to like approximate our plasma. Our plasma involves within your network which results in like a great performance boost. Our model is divided into part which were initially blended, but then we decided to separate. The first part is an autoencoder. What is not encoder is very simple. It's a neural network that compresses the data into a bottleneck and then tries to reconstruct it. And it uses the difference from the input and the output as a loss so that you can learn an hidden representation of your data on distribution. In particular, we use a convolutional encoder so that we could. We could take data about the position of the various particles, what the densities in account to have a better result. And then for the actual prediction we are using a decoder only transformer. That for someone who is not in the field might seem like scary name. It's actually something very simple, very neat. That also the same model pretty much that powers ChatGPT for example, how this model work is that we have self attention running. We pass to the model. Okay, the first thing to do is we apply something called a positional encoding. What is a positional encoding? A positional encoding we can call a procedure that allows a model to enrich embedding vectors with information about where their position is in a sequence. And we basically do this by creating indices, creating learning embedding for those indices and simply adding those embeddings of the indices to the embedding of the embedding of the data. In this case semantics is the particle state of the system at each time step. And how the transformer work is that we basically apply this thing called a multi self attention in which technically self attention is really a Weighted sum. And this allows the model to learn correlation in the sequence. Because with self attention, each embedding can look out for certain information that it needs from other nodes, which in this case are the embeddings of particle states at each time step and can enrich itself with such information. And after that a simple multi error perception is applied and this block, this procedure is repeated n times. And after that we apply a simple classification head which will basically try to predict the particle density distribution. At the next time step we can see that we first project our hidden our particle states to the hidden hidden dimension. We then apply positional encoding to enrich to reach the vectors with the position. Because here is the formula for self attention. If you see this is permutational invariant. This means that if you swap the vectors the result will be the same, which is obviously not good. If you're modeling a sequence and we basically train the model to predict which what will be the next distribution of particles at a nice standard. And we can see that it approximates fairly well the distribution it's updating. Sorry for that. And obviously this is still a small model, but with more data and more and a bigger model, we think that it might improve even more and more does offer a good alternative to running those very expensive simulation.

Okay, interesting. So that was a wonderful explanation. Do you want to sort of elaborate upon. You know, I think you mentioned somewhere in the beginning of the document that you started with a different architecture and then you repurposed the so you change the architecture and then you switch to a decoder only transformer instead of your original architecture. So do you want to elaborate a bit on the reasons for that?

Sure. So our original architecture was to be honest, quite similar to the current one. It is. This is basically a Scala decoder only transformer, but operating on our latent space learned by the use encoder. So basically encoder is inside like it's on the decoder only transformer is inside and it is between the encoder and the decoder. For example, this is somewhat similar to the way stable diffusion works, but stable diffusion uses a different model. The synthetratransformat is a diffusion model, but for example, stable diffusion operates in a latent space similar to this architecture. What we found is that since we couldn't use the same data for I mean we found that the model like for some reason didn't gave optimal result. And then we tried learning just the decoder only transformer directly on the raw data and surprisingly it did better. Our guess Is that since the dimension of the data was already very small when compressing it, there was not very much to compress. So it was basically just losing data. And then we repurposed this autoencoder for evaluation in which we took. What we did is we predicted the next state from our model and the target state. And we saw if in the Latin space the points were closed. If you can see, even though that's a projection. So that's not nearly sacred because it's reducing down to just two dimensions. But if you see the points are mostly grouped at group of stores, this means that the target projection is close to the real one to the predicted one. And this means that the model is learning how to predict.

Another quick question. So you mentioned sort of going back to the architecture again that the encoder and the decoder pipeline was basically, you know, lowering the dimensionality. Do you have any sort of, you know, number on what exactly was the reduction? You know, what was the original data and then what would the dimensions got reduced to?

Sure. So the original dimension was what were these 400 numbers? Because we are for simplification, we're assuming we're working on a one dimensional space and the decoder this time is actually outputting at 8 by 25 matrix.

Okay, cool. Yeah, that. Yeah that's. That's provided in the notebook. So I. Yeah, cool. I guess again for the people listening, feel free to look at the notebook. We'll be putting the link in the description. So if you have any other questions. I think a detailed look through the notebook would help with that.

Yeah, this looks really cool. I have a. I'm a bit confused about your decoder only transformer architecture. I don't know if it's maybe just me, but I can't read like the text on that image super well. So I'm not quite sure like. Yeah, I mentioned you had multiple heads. Sorry.

Yes, sorry. For the text. That's a problem with Mathematica. Strangely like the image was at very high resolution but strangely it displays it this way. If you download the notebook and try to resize, that's for the viewers. If you want to read this better if you try the notebook you can either save as the picture so you'll get full resolution one or you can try resize it and Mathematica should fix resolution yet. Sorry.

Okay. My question was more about like your. That equation you had like attention qkv.

Yes.

Could you explain that a bit more?

So before moving to multi attention, I think it's better to quickly explain what is normal attention? So what happens in normal attention? Imagine that each vector, each embedding is a node. What you do is that you basically ask for like other nodes which are the other embeddings from like a particular information which you need. And basically what you do then is you blend. You blend each vector by the like by the attention coefficient, which is how much, which is basically the dot product after applying a transformation. So what you do, Q can be in like, in self attention are exactly the same. They're a copy of the embeddings. That's why it's called self attention. There is also for example curve attention which is used for example encoder decoder transformers, which are commonly but not always used, for example in translation systems. And basically what you do is that you basically do qkt. So you basically do the product of the embeddings with themselves. So you find how correlated are those embeddings and then you scale them by the animation of K so that you like your numbers don't get too big and you do again the product with B so that you can do the. So it's basically like doing a weighted mean, a weighted sum. It's also important to say that QK and V are. They're not just the copies, but they are the copies after applying a transformation. Learn transform, free transform transformation. Why is this important? Is important because in multi attention, the problem multi attention tries to fix is that each node, let's call them nodes, might need like more than one information. For example, let's take the example of a sentence of a sentence. If like the padding associated with a verb might not look just for the subject but for example might also want to look for the adverb. So what you do is that you apply and learn transformation and you repeat basically in parallel self attention on those so that each embedding can actually go and search for let's say multiple information that needs. And this is very important because you are not just looking for one feature. Now with multi ad self attention you can actually take various information from the sequence which are important to determine, for example the next predicted state.

So yeah, thanks for clearing that up. If we gears for a second. First of all, thanks so much for the author explanation. That was pretty clear. And again we'll be putting the link in the description for anyone who wants to take a look further. Great. So do you want to sort of. Logan, Nicola, Theo, do you want to talk about if there was a part of the project that was particularly tough for you to code or you know, if not something that was tough, perhaps something that you really enjoy. Either of those.

Well, maybe a tough part might be that training always the model was quite slow and because we had to iterate a lot and we had, I mean, quite a lot of data for the hardware we had. Obviously it's not much if you take consideration better hardware, but we had what we had for each iteration of the model. We had to retrain from scratch and it took some time. So iterating was kind of painful. But after all, it was really satisfying because we think we've made something which can be interesting and maybe with further development even to sum up to someone.

Okay, yeah, that makes sense, I guess. Logan, maybe do you want to explain what like, I guess what was the most enjoyable part of the project?

I think for me the most enjoyable part was just learning all the physics and all the science involved. Especially given that in college I planned to go for a mix of physics and music. But especially for the physics part, learning all of this now is especially good to just kind of understand ahead of time what I will be getting into in the future.

Yeah, that makes sense. That's pretty cool. Physics and music is an interesting degree. So have fun with that.

Okay.

Okay, thank you for sharing. Your entire project seemed really interesting. I guess just to wrap up now after doing like this one kind of warframe project, do you guys have any final thoughts about the warframe language? Kind of what appeals to you about it? What's interesting stuff like that?

I think for this project in particular, it's appealing about the Wolfram language is like the ability to work on large sets of data like this, to be able to handle it and process it as necessary in order to get it into the right form as when we exported the data it was in. It was just essentially in like a spreadsheet. And so using the Wolfram language, we were able to import the data and sort it into the right categories and labels, everything in a way that I think would be much more difficult in other coding languages. So that's certainly an advantage for this project.

For sure. That makes. That makes a lot of sense. Logan and Nicola, do you want to talk about your own experiences? And, you know, while you're at it, do you have any ideas for what you might want to do with Wolfram in the future, perhaps as a team or as individuals?

For me, I think one of my favorite parts of Wolfram language, just independent of really any project, is just how flexible it is. Like having done multiple other languages. Wolfram language makes so many Things just easier. And there's just so many things that whatever you need, there's just a function already there for it. And it's much more approachable of a language, I would say, and just more fun and kind of easier to work in. But you can still get impressively powerful results out of it. As for future projects, I'm not completely sure yet, but I will be a TA for Volfram Summer Camp this upcoming summer, so I'll likely do something there.

Hey, that sounds super cool. Yeah, I remember when I was at the summer camp, TAs were super cool. So it's nice that you're now in ta. Yeah. Nicola, do you have any future ideas that you want to do either like continuing on this project or a different one entirely?

Sure. I'm currently exploring like Transformers and sequence modeling in the workflow language. I'm continuing to do this not in the realm of physics, mostly in language. So that's something I'm working. I hope to continue working in the future with the work from language. And as for my thought about the work from language, the thing that I like the most is the fact that it's really coherent. Let me get an example about our project. When we went to develop our project, we had there were like many pieces. The machine learning architecture, data pre processing, visualization in the language, like Python, for example. We would have had to use a lot of external libraries which for example might have had dependencies because those are dependencies which have other dependencies which have recursively other dependencies and they're just not coherent inside of the fourth language. We just could like user language just fit together perfectly. It was like really simple because we just had to think what we wanted to do and like translating our thought to the workflow language with just like a query process which made like the our whole iteration and creation of the project really fun and also really reconvene.

Great. Those are some very German thoughts. And honestly, I'm excited to see how all of you take this experience and integrate it into your future work. And yeah, good luck with all of your future projects. Once again, thanks so much for joining us today and we enjoyed having in the podcast. Thanks a lot.

Been great hearing from you guys and we look forward to maybe having you again with some more interesting projects. Thank you.

Thank you for having us. Thank you.

Thanks for having us.

Yeah, thanks for listening to this episode of the Wolfenstein Podcast. We hope you learned something new and take away insights from today's discussion. If you want to be featured in a future episode fill out the form@tinyurl.com WSP S2 interest and be sure to.

Tune back in two weeks. Time to hear our next episode. Once again, thanks for listening to this episode of the Wolfram Student Podcast.