Brain Backups at NYAI 21: AI in Brain Imaging

Hello and good evening. It's a pleasure to be here. My name is Russell Hanson and I'm visiting from Mount Sinai just uptown. Our office is at 98th and Madison and we're currently working on applications of artificial intelligence in neuropathology and neuroscience. The title of the talk today is Artificial Intelligence and Brain Imaging: Brain Backups, What Are They Good For? A little bit about me in case you didn't do extensive research online before this evening, which is fine. I used to be really involved with hacker conferences like DEF CON and the CCC conference in Germany. Vice wrote about me doing brain imaging. I've been giving talks at hacker conferences around the world, like these excellent ones in Spain and Istanbul, and I also have a photo on the Mount Sinai webpage. Everybody likes demos, but, you know, I can't always bring all of my equipment around with me. We do a lot of work with MRI machines. And CT machines, so up here there's a large Siemens MRI machine. They're very expensive. You often have to remodel whole hospital suites just to install these. They're heavy. Here's another machine, which is a transcranial magnetic stimulator, and those are also difficult to carry on the subway, so I didn't bring them with me. But what I did bring were some videos, such as This video of a mouse navigating around inside a virtual world. So instead of demos, I have videos. It's the YouTube page. So I can talk at great length about very technical areas in science, but I think this audience is a little bit mixed between maybe more tech people or marketing people or AI people or Consider this a best of and not the deep dive into how to implement what you want in a PyTorch or a Keras or what have you. There's been a whole lot of really interesting developments in AI and neural, particularly in imaging, as you could probably expect. So a lot of the images we use are MR images, CT images, PET images. As well as a lot of microscopy images. Pathology and histology has a huge wealth of image data that we're starting to analyze using deep learning algorithms. In addition, there's a lot of work going on in brain-computer interfaces, connectome imaging, and to a certain extent, using knowledge of the brain to train the next generation of AIs. [Speaker] Here is a quick video about how artificial neural networks resemble human neural networks. So human neural networks are spiking neural networks, which means they have voltage potentials and gating potentials and usually consist of calcium ions and potassium ions gating function. But many people in this audience are probably familiar with things like perceptrons. So here you can see a handwriting task using different neural network architectures. It'll step through these earlier kind of computer science methods, and then it'll go on to spike neural networks like we have in our brain. Sorry for the dramatic music. This is the spiking neural network like we have in our brains. So the network topologies are very different, but they perform the same function. Same task. This is happening in your brain right now. Alright, enough of that. So, I don't know, I like that video. I think it shows like a lot of different technologies and artificial neural networks and also biological neural networks. What I do in my lab is I'm the scientist over here and I make artificial neural networks to analyze funky brains such as a healthy normal brain and a diseased Alzheimer's brain. So I use my brain to make artificial brains to analyze real brains. Your mileage may vary. And the way we do this is we take bunches of images of neurons. These are individual neurons. This is a neuron that has a tau stain on it. Tau is one of the two elements of Alzheimer's disease, the amyloid hypothesis and the tau hypothesis. There are Other hypotheses like vascular changes also affected, but these neurons are highly affected by this protein called tau, which is a scaffolding protein that, you know, if you get hit on the head a lot or if you're a football player or a boxer, tau may grow too much and it'll start engulfing the neuron to some extent and killing the neuron eventually. These are our actual data. Do not share them. This is the same piece of brain from a postmortem Alzheimer's patient, and this is a piece of human hippocampus. The brown areas are all tau-stained areas. This is a very, very large image. It's about 4 gigabytes on disk. It's about 30,000 pixels by 50,000 pixels, and it's— it has countless neurons on here and glia and many other features. And that's how big it is. When you zoom it up a little bit, we are currently in the process of labeling many different types of neuropathology, including mature tangles, which are labeled here, and also pre-tangles. The two classes we're currently labeling are intracellular tau, so the one to the upper left, and the pre-tangles, which are these much lighter kind of disease burden here. The disease progresses from pretangles to intercellular to these things called ghost tangles, which is where all the neurons have died. So the only thing that remains is this kind of scaffold of tau surrounding a dead neuron that doesn't have a nucleus anymore. What we're using those data for is to train a neural network. And this is one of— and so we have a patch classifier and we go from using the patch classifier, something like VGG16 or ImageNet or, you know, just a regular convolutional neural network and then to get individual pixels, we're adding on a fully convolutional layer. So for instance, you can go from these dog and cat to, you know, pixel-wise outputs for those images using the fully convolutional network. Another aspect that is really taking off in medical image analysis is essentially image-to-image translation, and the technical name for one of these methods is cycle-consistent generative adversarial networks, or CycleGANs. And what these allow is to transfer transform different pieces of data from, you know, in this case, a horse to a zebra and back. So you might be asking, why would we care about transforming horses into zebras and back in the context of like very real data that we have? And the thing is, is that a lot of imaging is very restricted to, you know, the setting where you took that data and, you know, the imaging method that you used. So for instance, if you image the patient on an MRI but you actually want the type of data you could get from a CT and the patient has left or what have you, you know, being able to transform data of one type into data of another type is profoundly valuable. So, you know, instead of having to bring the patient back in and, scan them in the radiology suite and, you know, charge them a lot of money and take a bunch of people's time, you could just transform the data using a neural network. In addition, one of the huge tasks in radiology and medical imaging is identifying different pathological features. You know, you want to find the tumor, you want to find the, the edge of the tumor, you want to spare the healthy tissue, you want to spare neurovascular bundles in prostate cancer, what have you. And it's very similar to a lot of the self-driving car applications. This particular one was from a paper from Apple where they named their network the VoxelNet, and they want to identify cars and pedestrians and cyclists. For the first half of this talk, we'll see a bunch of these kind of feature learning networks that take— that kind of break down the steps of how these networks are built and trained and analyzed. Here's yet another highly technical piece where these guys wanted to turn positron emission tomography images into MRs or add different pieces of PET or MR data onto it. I won't read through it. Another really cool thing that I thought was fascinating that's kind of taken off just in the last 6 months is adding additional information to images that you wouldn't— that essentially is labels like a fluorescent label or a stain or other types of like a chemical label. You can just run a computer program and add a chemical label to these images. Typically you'd have to take a piece of tissue and drop it in a little beaker of stain and then once that tissue has been stained it's done, you can't really restain it or reanalyze it. But using a neural network and using these kinds of GAN networks you can actually just take a piece of data and transform it into something else and then transform it into something else. And add these kind of synthetic data onto your original data. So here on the left side, you have the bright field image, which is— and then you have the true neuron label, and then you have the neural network predicted label, and you see the errors between them. So adding labels and adding labels, so what are you looking at, and adding features that allow labelers to also function is taking off dramatically. So in this case, there's a common stain called H&E staining and Malcolm staining, and these stains were added synthetically. And you can look at the full paper or, you know, if you are a scientist or you want to read these papers, feel free to reach out. I can look up the paper and send it to you. This one's called Deep Learning in Microscopy by Rivenzon. One of the things that we are currently doing a project on is basically taking low-resolution data and turning it into high-resolution data by using a super-resolution deep learning system. So you use high-resolution data that's more expensive to gather than the low-resolution data, but you train your network on the high-resolution data. And essentially, it's like a contrast enhancement scheme. You can turn existing low-resolution data into almost arbitrarily high-resolution data. In some cases you lose some information. If you have a pathological sample, you have a huge brain tumor, if you trained only on healthy brains you won't be able to predict that brain tumor, but if you train on two-thirds brain tumors you'll actually also be able to predict this huge brain tumor. In this case, the example is 3T MRI versus 7T MRI. So this is the field strength of the magnet in the MRI, 3 Tesla versus 7 Tesla. And you can see that the spatial resolution is 3 millimeters versus 0.75 millimeters. So using a synthetic neural network, you can go from, you know, 3 millimeters to 0.75 millimeters resolution just using a computer program. And here I walk through one of the cases where people did this. So on the far left you see the 3T. Here are a number of different computer programs, and here is the ground truth. And, you know, most of them look pretty close to the ground truth. They look at signal-to-noise ratio and other aspects of the neural network to to compare it. So part 2 of the talk is the whole brain backups thing. Who here has heard of a brain backup or would like a brain backup? Front row, do you want a brain backup? Could use one. Yeah. See what happens. It doesn't end up in some unsecured Amazon instance. Yeah, the security issue is always the big question. I'll just start by talking about brain-machine interfaces and telepathy. So who here thinks telepathy is not real? Good. Telepathy is very real. So— Why do you have to ask? You don't have enough gold in your hat. That's funny. So in this experiment, a guy— the receiver over here, he saw kind of a flashing light because of this machine next to him. And this guy had a lower resolution kind of a net on his head. And they were able to communicate telepathically. So the receiving guy was, I think, in a much less comfortable position. And he was just seeing flashes of light. Transmission. So I don't personally recommend this method, but it worked. They just communicated ones and zeros to, you know, from India to Ukraine or whatever. That's good. Another common question is, you know, isn't this impossible? Nobody's ever actually uploaded a whole human brain. Well, actually they have, and all you need to do is go to brainmap.org. And you can look at a whole human brain on the internet and find all the neurons and look at them for yourself. I'm not going to read this, but I, you know, there are applications in health and education, technology, entertainment, and business for brain mapping. Personally, I was kind of bored with genome sequencing. I just thought that, you know, you can sequence the genome for $1,000, the connectome is the next thing to work on, and so we're working on the $1,000 connectome. Again, I don't believe you. It sounds like science fiction. So the Telegraph reported that supercomputer models 1 second of the human brain activity. Let's say that that's a 2,000 times slowdown, so you just need computers to be 2,000 times faster or spend 2,000 times more money. Using existing supercomputers. In addition, people are skeptical that they've, you know, there are no connectomes. Here is a connectome. This is the simplest connectome, but it's still a connectome. So the C. elegans worm has 302 neurons, and this is the first 20 or so of its neurons. So you have an origin, a target, a type of connection, and what neurotransmitter is transmitted. These can be simulated. You can do the kind of artificial neural network spiking that we saw in the video before. It looks much like the spiking phenomenon I showed you. This is a video that's slow. A lot of— I think I'm going to run through this talk pretty fast, but if you want me to slow down or whatever, I don't know. It's only 7:30 and I'll get through this quickly. A lot of business has gone into next-generation neural interfaces. Bryan Johnson and Kernel invested $100 million to unlock the power of the human brain, and many other companies are being funded to do really great work in neuroscience and neuroengineering. Can I have any estimates as to how big you think the connectome is? I'll give you one number to start it off at. There are about 85 billion neurons in your brain, and on average there are 10,000 connections between them. You in the front row, how big in bytes do you think your connectome is? I have no idea. That's fine. 760 terabytes? That's a good number. I like that one. Where did you get that from? I don't know. I was calculating in my head. I don't know. Yeah, so one low-resolution estimate is 909 terabytes, which is very close to the 760 you suggested. Whatever the price of a terabyte hard drive is, it would be maybe $25,000 for that amount of storage. Is that compressed or uncompressed? That's uncompressed. And you can probably assume the brain has huge potential to be compressed. If we add a little bit of complexity to our model, so instead of just looking at the number of neurons the number of synapses, if we add how many inputs there are per neuron, what the adjacency list is, and also add the neuronal type. So let's say there are 1,000 subtypes of neurons and also add that there are a number of states of the synapse, so 1,000 states of the synapse, and add compression, you know, we get another number that's 384 terabytes. So something in the range of 300 to 1,000 terabytes I think is realistic. If you don't spend all day looking at brains, you may think that human brains are, you know, the only game in town, but they aren't. So here, human brain has 86 billion neurons and weighs about 1.5 kilograms. The macaque has, you know, 6 billion. More monkeys have smaller and smaller brains. The mouse brain up there has only 71 million neurons, and a rat has a brain that's about 3 times larger in terms of number of neurons. Rats are much, much smarter than mice, incidentally. I don't know about the color for the eastern mole. Why is his brain kind of blue? Technologies to measure the brain vary dramatically, and each of them has a different strength and weakness. This is a particular technology that we commonly call ECoG. And so ECoG is like 8 electrodes by 8 electrodes and sits on the surface of the brain, not on the surface of the skull or the scalp, on the surface of the brain. And it's very accurate, and I'll be playing you some videos where you can decode audition using an eCochG array. The disadvantage, of course, of eCochG is that you have to open up the skull. This is a more common system that people tend to play with a lot. This one on the left is OpenBCI, which is made in Brooklyn. The one on the right is Emotiv, made in California. The one on the bottom right is also an Emotiv headset, sort of designed in California. And we used to put these on people and run synthesizers. [Speaker:MAN] No, I'm like this. I was made for this. It's great. [Speaker:MAN] So you can forget about the screen. You're frustrated. Wouldn't it be cool if you had a Wi-Fi connected brain? You could just type ifconfig brain0 up and just use some commodity brain chip, also known as the you know, I'm brought a comm chip or what have you. The Defense Advanced Research Projects Agency has been very adamant in pursuing this and has, you know, tens of millions of dollars and asked in 2016 to make a device capable of recording from more than a million neurons and stimulating more than 100,000 neurons in approximately a cubic centimeter. This was an absurd request that nobody on the planet could even approach, but they still brought a bunch of us down to DARPA and asked us why we couldn't do it. I just wanted to show you quickly, so there's a— there's a Stentrode device that I'm going to show in a bit that interfaces with the premotor cortex and the primary motor cortex. And a little bit with the somatosensory cortex. So we're talking about this kind of area right in here. This is the stentrode. This is very new. This is actually an Australian company, and we're— I think it's super cool. It has many advantages over many other computing technologies to do brain-machine interfaces. ask the question, you know, why do you want to destroy the thing that you want to image, or why do you have to destroy the thing you want to image? And the answer to that question is what's been driving a lot of our research. Over here on the right side is something called knife-edge scanning microscopy. So you slice the tissue and image it at the same time, and that saves a lot of trouble. So you don't have to cut the tissue and then move it across the lab the other side to image it. The trouble is you're destroying the thing you want to image. Another alternative that has been proposed is to use deep sequencing or DNA sequencing to do the connectome kind of in one step. So you infect the brain with the virus and the virus cruises around and infects neurons, but it leaves a DNA barcode in all of those neurons. And then if you grind up the brain and sequence all those DNA barcodes, you'd get an approximation for how the brain is wired. It doesn't really work, but it might kind of work eventually. It— there are lots and lots of problems with this method. You still have to grind up the brain also. One issue is there's no real gold standard, so even if you did get all these like DNA barcodes, you wouldn't know what they corresponded to, for instance. One method that we've been working on uses contrast agents. So these are gold nanoparticles with ligands on them, and the ligands are specific to different subtypes of receptors and transporters in the brain. This was a famous result where you can reconstruct using a low-resolution MRI, so only a couple of millimeters voxel size, what people are seeing using something like a machine learning classifier. Again, this is very low resolution. And just uses existing MRIs. This one uses the eCochG interface that I described, and you'll be able to hear what eCochG on the surface of your brain can do towards reconstructing what people hear. Waldo. Waldo. Waldo. Structure. Structure. Town. Town. Town. Town. Copy. Copy. Answer. Answer. Answer. It's not perfect, but it was, you know, a graduate student working in a lab. It was not a Google or Facebook. The budgets were not extensive. What does the brain look like when you zoom in? It looks much like this. I mean, I showed you the neurons before, but those images don't show the densities of receptors and transmitters. Transporters on the brain. Generally speaking, you have to use electron microscopy for that. And so what this image is, is a bunch of nanoparticles or quantum dots with ligands for glycine receptors. So what you see is groups of quantum dots or nanoparticles that are, that are black, and you see groups of, you know, 20 to 15 of them around the sites that are being imaged. And this is very generally what gold nanoparticles look like on electron microscopy, just black dots, and they're localized based on the ligand on those quantum dots or nanoparticles. And they provide contrast in electron microscopy, and they also provide contrast in computed tomography or CT. Some of the sites that we're interested in One would want to image are, you know, subtypes of adrenergic receptors, subtypes of dopaminergic receptors, subtypes of GABAergic receptors, and these all have associated genes and gene product session numbers, which means that there are oftentimes drugs for these different sites. This was some data we took from a slice of mouse brain. On a very high-resolution CT machine. So a typical CT machine is about the same resolution as an MRI, a couple of millimeters voxel. This one is 10,000 times better, so 0.6 micron voxel size. And these are, you know, individual neurons and axons. Do you need a single trial for that or— What? I couldn't understand you. Do you need a single trial for that? Yeah, this one actually was the synchrotron. This was at the synchrotron in Chicago. And this is the same image zoomed up. So the advantage with CT is that you can do it in a volume. You don't have to slice the tissue. But the disadvantage is that you have ionizing radiation, which carries with it a risk of cancer, and you also might have to have a synchrotron, and so forth. Generally speaking, X-rays pass through soft tissues without interacting with it. These were some data we took of vasculature of a rat brain. This was part of an experiment to look at how rat vasculature and potentially human vasculatures change when these rats are exposed to blast exposure, like if a soldier was in a near a tank or next to a landmine. This is the healthy rat, and you see he has tons of vasculature and the blood is flowing very well in his brain. And this is the blast-exposed rat about 2 years after the blast, and you can see that his vasculature is really, really damaged. These were age-matched, litter-matched, essentially the same rat. And you can imagine that people who've been exposed to landmines or other kind of blast injury may have very damaged vasculatures. I mean, I'm not saying that it looks like this rat, but, you know, there are probably effects. Yet another approach is designing nanobots to do this kind of imaging for you. This is an artist's rendition of a nanobot imaging brains. Let's skip that. Alternatively, you can stack up electron microscopy images into a little cube like this. So this is an EM X-ray imaging of human brain in the somatosensory cortex using an electron microscope. And in this reconstruction, the goal is to image all the synapses, so all these little red pancakes are actually the synapses, so the connections between neurons. And you'll see just how many there are in this little volume of 9 microns by 7 by 4 microns. Quite a lot of synapses in this little line here. I will step away from my scientific hat and ask some philosophical questions. Why do you think you can simulate brains anyway? Well, this explanation says that essentially the brain is contained within a boundary such as this column and is therefore a finite amount of data, and that any finite amount of data can be simulated using a computer according to, you know, computability theory, which is just a theorem of computer science and something called model theory, which has— I don't think it's— it hasn't been disproven. So you'd have to have a pretty deep result in computer science to disprove this. In addition, oftentimes people ask, why would you even want to image the brain? And I guess I think it's a silly question, but still people ask. There are lots of pathologists and histologists and radiologists and so forth, and We just do this for a living, and the reason we do it is because we want to make people more healthy. And, you know, since we're at an AI talk, I guess I should discuss AIs. Now, as you go past through time, the hypothesis is that we're at a stage where computer performance is exceeding human performance, and I hypothesize that the Russell point as a point in time where human characteristics are preserved in the human performance curve. So, you know, we may be over here, but eventually we may be over here where human performance, or, you know, human characteristics actually exceed machine performance characteristics. What's next? Well, I think that machine learning to improve interfaces for electrophysiology arrays to sensorimotor cortex, like like the stentrode, has a ways to go and can be augmented using computer algorithms. The neural modem, like the DARPA project, is pretty fascinating. Using implanted CPUs with a database of neural codes is much like the Kernel project. And, you know, using AI and machine learning to trace neurons and axons and image stacks is, well underway, and, you know, our goal is to do connectomes for $1,000 and save us from the AIs. Sorry, New York AI. I'd like to thank you and the New York AI crew for setting this up, and I look forward to many discussions after the talk.