Stanford researchers decoded imagined handwriting directly from brain signals, achieving 90 characters per minute with 94% accuracy. A participant with microelectrodes in their motor cortex could spell out words just by thinking about writing them, with autocorrect pushing accuracy above 99%.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to read some minds. A few months ago our first video appeared on brain machine interfaces. It was about a paper from neuralink which promised a great deal. For instance, they proposed a brain machine interface that could read this pig's thoughts as it was running on the treadmill. And what's more, they not only read its thoughts, but they could also predict what the pig's brain is about to do. So this was about reading thoughts related to movement. And to be able to use these brain machine interfaces to the fullest, they should be able to enable some sort of communication for people who have lost their ability to move or speak. So what about writing or speaking? As impossible as it sounds, can we somehow restore that? Or is that still science fiction? Many of you told me that you would like to hear more about this topic. So due to popular request here it is a look beyond neuralink's project to see what else is out there. This is a collaboration between Stanford University and a selection of other institutions and it allows brain to text transcription where all the test subject has to do is imagine writing the letters and they magically appear on the screen. And now start holding onto your papers and just look at how quickly it goes. 90 characters per minute with over 94% of accuracy, which can be improved to over 99% accuracy with an additional autocorrect technique. That is absolutely amazing. A true miracle. 90 characters per minute means that the test subject here, who has a paralyzed hand, can almost think about writing these letters continuously. And most of them are decoded and put on the screen in less than a second. Also. Wait a second. 90 characters per minute. That is about 80% as fast as the average typing speed on a smartphone screen for an able bodied person of this age group. Whoa. It is quite remarkable that even years after paralysis, the motor cortex is still strong enough to be read by a brain computer interface well enough for such typing speed and accuracy. It truly feels like we are living in a science fiction world. Of course, not even this technique is perfect. It has its own limitations. For instance, we can't edit or delete text, have no access to capital letters, and the method has a calibration step that takes a long time, although it doesn't get significantly worse if we shorten this calibration time a bit. So how does this work? First, the participant starts thinking of writing one letter at a time. Here you see the recorded neural activity. This is subject to decoding. You can see the decoded signals Here. And we can't just give this to a computer to distinguish between them as is, we project these into a 2D latent space where it is easy to find which letter corresponds to which region. Look, they form relatively tight clusters. Therefore it is now easy to decide which of the squiggles corresponds to which letter. The decoding part is done using a recurrent neural network which is endowed with memory and can deal with sequences of data. So here in goes the brain activity and out comes the decision that says which character these activities correspond to. Of course, our Alphabet was not designed to be decoded with neural networks. So here is an almost science fiction like question. How do we reformulate the Alphabet to tailor it to maximize the efficiency of a neural network decoding our thoughts? Or simpler, what would the Alphabet look like if neural networks were in charge? Well, this paper has an answer to that too. So let's have a look. The squiggles indeed look like they came from another planet. So what do we gain from this? Well, look at the distance matrix for the English Alphabet. The diagonal is supposed to be very blue. But what is not supposed to be blue at all are the regions that surround it. Look, the blue color here means that in the English Alphabet the letters M and N can be relatively easily confused. Same with the letters O and C. And there are many more similarities. And now look, here is the same distance matrix for the optimized Alphabet. No dark blue in sight, outside the diagonal. Much easier to process and decode. If neural networks were in charge, this is what the Alphabet would look like. Glorious. Also, the fact that we are talking about squiggles is not a trivial insight at all. Traditional methods typically rely on movement in straight lines to select letters. And the other key thought in this paper is that modern neural network based methods can decode these thoughts of squiggles reliably. That is absolutely amazing. And wait a second. Note that there is only one participant in the user study. Why just one participant? Why not call in a bunch of people to test this? It is because this method uses a microelectrode array and this requires surgery to insert. And because of that these studies are difficult to perform and are usually done at times when the participant has brain surgery anyways for other reasons. Having more people in the study is usually prohibitively expensive if at all possible for this kind of brain implant. And note that research is a process and these papers are stepping stones. And now we are able to help people write 90 characters every minute to with a brain machine interface. And I can only imagine how good. These techniques will become two more papers down the line. And don't forget there are research works on non invasive devices too. So what do you think? Let me know in the comments below. What a time to be alive. This episode has been supported by Lamda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold onto your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus they are the only cloud service with 48gb RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances where workstations or servers make sure to go to lambdalabs.com papers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
