Researchers used diffusion models to reconstruct videos directly from brain fMRI scans, moving beyond earlier image-only methods. The system handles spatial and temporal details in a two-module pipeline and outputs 256×256 video at 3 fps on a single GPU, though some frames still fail or show false positives.
This is beyond crazy. AI can now literally read our brain and can see what we are looking at. A few weeks ago there was this paper, Seeing beyond the Brain and everybody was talking about this paper. In that paper the authors proposed an approach to convert brain signals into images. Now we have all seen text to image converters, such as Midjourney, where you provide a text prompt and the model is able to convert it into beautiful images. There are even text to video converters. So you simply provide a text description and the model is able to generate videos out of it. But in that specific work, they converted brain activity into images. Now the way they did that was they showed visual stimulus or images to people and then recorded their brain activity using FMRI and using this diffusion based model, they were able to convert these brain stimuli into visual representations and they were able to create images, which was great at that time. We were talking about just a few weeks ago, but now they took it to the next level. Now in this new work, they're proposing a new method called Mind Video. It's cinematic mindscapes, high quality video reconstruction from brain activity. It as you can probably guess, in this work they are able to reconstruct a video just by looking at brain activity using the FMRI data. And so the way it's done is a person is shown a video and while watching the video, their brain activity is being recorded by the FMRI scan. And then they use a diffusion based model to reconstruct these frames from just the brain activity. Now you might be thinking that if you're able to reconstruct images from brain activity recovery, reconstructing video from brain activity might be easier. However, there is one major issue with videos, which is the special temporal information that is needed to reconstruct videos. In very simple terms, videos are simply a combination of frames which are static images. However, there is a dependence of the current frame with the previous as well as the subsequent frame. And that is one of the main challenges when you're trying to reconstruct videos instead of just single frames. Now in order to do this, they're proposing a two module pipeline designed to bridge the gap between images and videos. Brain decoding. Using their approach, they're able to get state of the art results. But we're not going to go into a lot of technical details. Rather I want to show you some potential applications and their results. Now here are some of the results that they have shared. So just to explain it, this GT column is the ground truth and then ours is basically what the model has reconstructed. So imagine the model is only seeing brain activity that is coming through the fmri and based on the brain activity, it's able to reconstruct these videos. So the person is watching the video while the brain activity is being recorded. And then the model use that information to reconstruct the video. For example, the first one, there are a couple of people running and then just by reading your brain activity or literally reading your thoughts, the model is able to reconstruct a person running. Now, some of them are really good, but in some case, some of the cases you can't really make out what is going on. For example, the cat is pretty good, the water flowing is pretty awesome. Then here is a drone footage of a cityscape. Another now here's a video where people are actually talking and the reconstruction also shows people talking. Now most of these look pretty good, but there are cases in which I think the model has some issues. For example, this guy is playing guitar. Reconstruction, not sure what's going on over there. It's really hard to decipher. So according to them, this method can reconstruct various objects, so animals, motions and scenes. And the reconstructed visuals are of high quality and are consistent with the ground truth which we can see here. They have given some more examples. We're going to look at this in a bit. But the great thing is that the samples that we see here, they were generated with one RTX 3090. Now this is great because this means that you can actually run this model on a consumer grade GPU. Now, because of the memory limitation of this GPU, they're able to generate images with a resolution of 256 by 256 and the frames per second is just 3. Now before looking at some other examples in the Google Drive they have provided, let's look at a couple of things. So they're going to be providing both the data sets as well as the code pretty soon. However, the data set that they use for training is available and you can actually requested. So you can potentially use their training, the inference code once it's available and then reconstruct some of these results. Unless you have access to an FMRI machine and you want to capture your own data. Okay, let's look at some other examples because this is a really awesome work. So they have provided this Google Drive link where they're hosting some of the other videos. So right now there are three different subjects and we were going to look at some of the different examples they have provided. So for example, the person is dancing, right? And then you can see that the model, just by looking at the brain activity, is able to reconstruct dancing movements, although there are like multiple people. But still, keep in mind, it's just the first iteration of the model itself. Now this, in this specific case, I think it's really hard to make up of what's going on in the reconstructed video. Okay, let's look at a couple of other examples from this subject. So here's another one. There's a cat in it. The reconstruction actually shows a dog. So those are false positives. But still, I think it's a pretty significant work because this will help us to understand our own cognitive processes. Although science has progressed so much, we still do not understand how our brain works. And research projects like this are going to be a huge contribution to understand our own cognitive processes. And this can have very significant implications, especially in healthcare. I hope you liked the video. This was very different than my usual videos, but I think it is important to cover some of the latest advancements and some of the exciting research projects because they define the direction of where we are heading in future. As always, thanks for watching and see you in the next one.
