Scientist Stories: Keisuke Goda, Image-based cell sorting

All right, maybe we get started. Good morning everyone, or good afternoon, for another Terasaki Seminar Series. Today we have Professor Kisuke Goda. He is a professor in the Department of Chemistry at the University of Tokyo. A young professor in the Institute of Technological Sciences at Wuhan University, and a young professor in the Department of Bioengineering at UCLA. He received his Bachelor of Arts from UC Berkeley summa cum laude in 2001 and a PhD from MIT in 2007, both in physics. At MIT, he worked on the development of gravitational wave detectors in the LIGO group, which led to the 2017 Nobel Prize in Physics. After several years of research on high-speed imaging and microfluidics at Caltech and UCLA, he joined the University of Tokyo as a professor. His research group focuses on the development of serendipity-enabling technologies based on molecular imaging and spectroscopy together with microfluidics and computational analytics to push the frontier of science. The floor is yours, sir.

Thank you very much for the introduction, also the invitation. Let me share my screen. Okay, so thank you. So I want to talk about this new type of imaging technology called image-activated cell sorting combined with deep learning. So let me begin my talk by showing this picture. This was taken back in 2013 at Terasaki Institute. I think this was taken during an event organized by NDC, the Nichibei Doctors Club. Nichibei stands for Japanese American doctors. So basically this event was held for sort of supporting Japanese doctors who are visiting research institutes at Karasaki Institute, UCLA, and USC from Japan. And, you know, Paul Karasaki used to visit this event and then say hi to everyone. So I cherish my memory. You know, this has been renovated, this background. Yeah. Significantly recently. So my group, as Mehmet said, is focused on the development of serendipity-enabling technologies to realize Louis Pasteur's statement, "The chance of serendipity favors the prepared mind." So some of my group's previous accomplishments are shown here. And today I will talk about this, uh, intelligent image-activated cell sorter, publishing cell, and also the related technologies. Please feel free to ask me questions anytime by raising hand or directly asking me questions. So today's topic is cell analysis. So there are a lot of unmet needs in cell analysis. For example, how to identify unknown cells and their functions. How to isolate unique cells, how to increase microalgae biofuel production efficiency, how to produce many antibodies using cells, blah, blah, blah. And cell analysis is quite challenging. Because it's somewhat analogous to the word's wordy problem. Well, it's called wordy in the UK, and I think it's in It's called Wordle in the US. Basically, the theme of the game is to find this wordy very quickly, find the wordy very quickly with high accuracy. I think when you are a kid, you used to play. Can you find wordy here? And you should be good at it because you do a cell analysis. Under the microscope. And it's here. And this is easy. I think this is one of the lowest level games. And but this becomes more and more complicated. And this, I don't know what level, but it's a higher level, no, higher stage. And this situation is sort of analogous to biological cell analysis. You know, the easier cells are already found, and people are trying to find more difficult cells. And, and that's what's happening. And Wally's here. And the traditional way to find Wally or Wally-like cells is to use the optical microscope. And then once you find unique cells, you pick up cells under the microscope by pipetting. And here, this process is extremely time-consuming, labor-intensive, and inefficient. Even the automated microscope with the robotic arm is available nowadays, which is available nowadays, is still very slow. The throughput is roughly 1 cell per second. If you are very good at it, you can win a Nobel Prize. Perhaps this Nobel laureate Satoshi Omura is the best word refiner. There's a modern technology or automated technology that can partially overcome this problem. So this, I think I have seen the fax machine, similar machine when I visited the Terasaki Institute. This fax machine can counts, analyses, and sorts cell population based on light scattering and fluorescence measurements. Basically, you start with the cell population and you mix it with fluorescent-labeled antibodies, and you flow these cells, and then you excite these cells with a laser, and then you measure scattering signals and fluorescence signals, and then those signals are used to activate the cell sorter to sort cells. And it works quite well, but it does not have spatial resolution, so it cannot identify and isolate cells based on spatial matrix. So there's a trade-off between these two technologies. Here, the microscope with a cell picker, which provides high content but low throughput, cannot find what can find Wally, but it will take a long time. On the other hand, the FACS, or fluorescence activated cell sorter, which provides low content but high throughput, cannot find Wally. So there's a trade-off between these two technologies. And if you show these two technologies in this figure of merit, It's kind of obvious here. The x-axis is the throughput in units of cells per second to analyze and sort, and the y-axis is the cellular information content. And the microscope with the cell picker is here, the fluorescence-activated cell sorter is here. So there's a trade-off barrier here. So this blue region is sort of accessible area by conventional technologies. And if you place the specifications of these, uh, conventional technologies or commercial instruments, uh, they are basically located in the yellow regions here, as you can see, which is kind of expected. But the worry is here, right? Uh, this is a sort of unexpected area where unknown cells, rare cells, smart cells are. To find Wally or Wally-like cells, we need more information to make the invisible visible. And also we need to eliminate enormous time and effort by making this process high throughput. So we need this Wally-finding tech. This is our goal, because we need a technology that provides both high throughput and both both high throughput and high information content. So that's the technology that addresses this need. So intelligent image activated sensor. So let me introduce this technology. And let me begin the introduction by stating the history of the name. So basically, the fluorescence activated sensor, That was developed almost 50 years ago. And from fluorescence to intensity measurement, from fluorescence intensity measurement to imaging, that's a conversion. So the fluorescence is replaced by image. And now because imaging enables most sophisticated analysis, deep learning and other kind of sophisticated advanced computational methods can be used in real time on the image-activated sensor. So it becomes intelligent, highly intelligent image-activated sensor. So that's why we call this intelligent image-activated sensor. And as Mehmet said, I was working on the development of the LIGO detector. LIGO stands for Laser Interferometer Gravitational-Wave Observatory. Which received the Nobel Prize in 2017 for the detection of gravitational waves. And I used to work on the development of this huge, uh, Michelson interferometers with 4-kilometer-long arms, uh, in two states, Louisiana and Washington. And luckily, um, the LIGO actually shares many aspects of flow cytometry or the other way around, such as optical engineering, complex machinery, interdisplay integration, and precision measurement. So I brought the greatness of the LIGO technology to the field of flow cytometry to realize this intelligent image acquisition story. So this is a paper published in Cell in 2018. What I want to show you is this number of courses, the 51 courses, and also the number of affiliations such as physics, chemistry, biology, computer science, blah blah blah. So as you can see, this is a pure accomplishment of interdisciplinary integration. So this is the movie animated movie that shows how this technology works. So this is the box, and then this guy puts a sample that contains the solution, and this is the microfluidic chip. Cells flow through, flow into this chip through this yellow tube, and initially the cells Cells are randomized, but we use hydrodynamic focusing to create a single stream so that we can do imaging of single cells one by one. And here we do fluorescence imaging. I will get to the details of this image. And we use acoustic focuser to maintain the single stream. And the cells are flowing. To capture cells with high sensitivity, we use a polygon scanner to cancel out the motion of the flow by rotating it in the opposite direction so that the acquired images, acquired cell images, are placed or integrated on the, on the fosset of the image sensor continuously without motion blur. And then images acquired and sent to the image processor. Where we do image processing based on deep learning. And then using this convolutional neural network, we decide whether we sort or unsort cells. And then the decision signal is sent to the cell sorting module where we kick or unkick those cells when the target cells come. And then eventually collect two tubes that contain collected and uncollected cell populations. So here's the sort of recap of this, uh, the configuration, system configuration. So this is the microfluidic chip and the cells come in, it's a mixture of cells, and then we use the hydrodynamic cell focusing to create a single stream of cells. And then we use a high-speed imaging technique to capture every single cell by imaging. And then the acquired images are sent to this real-time intelligent image processor, which is composed of many CPUs, GPUs, and FPGAs. And then we do a lot of image analysis and image acquisition image construction, image analysis, and then finally we make decisions about whether we sort or unsort those cells. And the decision signal is sent back to the microfluidic chip, which contains this dual membrane push-pull cell sorter. And here the key is that this chip, this, the distance between the optical imaging part And so the sorter part is 33 millimeters, and the flow speed is 1 meter per second to maintain high throughput. So that means we only have like 32 or 33 milliseconds between imaging and sorting. So that's quite challenging. So this part corresponds to the recognition of the human brain, and this part corresponds to decision-making. And this part corresponds to commanding, and this part corresponds to actuation of the human action. So in a way, the whole thing is sort of close to what analogs to the microscope with a highly efficient mechanism. So this is the picture of the real setup. And this is a huge microfluidic and optical electronic instrumentation. And this is a zoom-in picture. And this is a microfluidic chip we use to sort, identify and sort cells. And cells come in here and flow from right to left. And this is where we do imaging. And this is where we do sorting. So this, the distance between the imaging and sorting is 32 micrometers. Millimeter. And it's again, the cell flow is at 1 millisecond per second. So we only have 32 milliseconds to decide. Let me tell you the details of each component. So here's the imaging technique we use called virtual freezing fluorescence imaging, or VFI. Here, the cells are flowing again, you know, it is actually challenging to capture image cells which are flowing at high speed. Again, the cells are flowing at 1 meter per second, and fluorescence measurements are kind of weak, okay, fluorescence signal is weak. So if you take pictures with a short integration time, you get low sensitivity, right, although you can get the clear pictures. But if you try to increase the integration time, you get motion blur because the cells are flowing, right? So we have to somehow overcome this trade-off between sensitivity and speed. And here we do a light sheet excitation, and then the cell images are formed without this polygon scanner on the CMOS camera. The cells are flowing like this. I mean, I would say the cell images are flowing, but by using this well-controlled polygon scanner, which is rotating in the opposite direction, cell images are formed on the CMOS sensor, but in a frozen manner, so that this technique virtually freezes the motion of flowing cells on the CMOS sensor to effectively achieve 1,000 times longer exposure time. For sensitive blood-free fluorescence image acquisition. This is critical for this intelligent image activated cell sorter. This is the brain of the intelligent image activated cell sorter. It's complicated. I'm not going to go into the details, but the key is this part. Basically, it's composed of many CPUs, GPUs, and controlled by the FPGA. Also, we use all-IP networking. IP stands for Internet Protocol. So we use the same language to communicate between different modules to speed up the communication so that we can meet that 32-millisecond decision-making requirement. And this is a microfluidics chip for cell focusing and imaging and sorting. So this is a 3-layer microfluidics chip. Glass, silicon glass sandwich structure. And this is where we do hydrodynamic focusing, where we generate a single stream of cells. And then this is where we do imaging. And then for about 32 millimeters, we need to maintain a single cell stream. So we use acoustic focuser to maintain it. And then finally, at the cell sorting part, we have these two reservoirs with glass membranes. And on top of the glass membrane, we are— we have piezoelectric actuators. So when the target cell comes, we push this membrane and pull the other membrane so that there's a 90-degree secondary flow. To kick the cell. And we can do this the other way around so that we can do this very quickly, left, right, left, left, right. So this, uh, the right images show the, the demonstration of sorting of cells flowing at very high speed. And this, uh, these images or these figures, uh, kind of show the performance with frozen beads to make sure that the computation can be done within 32 milliseconds. So under the classical image analysis algorithm, data transfer image analysis took all of the events took less than 32 milliseconds. So here the x-axis is rank order function, basically the event. Event means single cell, cell cluster or cell debris or something, whatever event we detect. Y-axis, the processing time. So all the events took less than 32 milliseconds. With the convolutional neural network or deep learning, which is computationally heavy, still 98.8% of the events took less than 32 milliseconds. As a result, we achieved the throughput of about 1,000 cells per second, plus 1,000 events per second. Sorry, so this is the movie that shows the demonstration of this technology. So this is the setup located at the University of Tokyo, and in the morning we discuss what kind of experiments to do. And we prepare cells. And then this guy loads the sample of suspended cells. And we push the run button and wait for, you know, 20-30 minutes depending on the experiment. And we monitor each process on a monitor during the sorting run. And finally, when the experiment is finished, we collect two tubes. Containing sorted and unsorted cells. And then afterwards, we put them under the microscope to verify whether we have sorted the right species of cells. And also we evaluate the sorting yield and purity. And as a result, we discussed the sorting outcome and also we discussed what to do with the sorted cells. Some people want to use them for further analysis like sequencing, you know, DNA and RNA-seq, and also electron microscopy, and also some of them want to use them for directed evolution. So this is a figure of merit. So back to this. And so for the first time, we have achieved this set of specifications by conquering this unknown area for the first time and by actually increasing or upgrading the specification of the microscope with a serial picker 1,000 times. And then recently we have even pushed this limit further, and so we are here basically, the couple thousand cells per second with decent image quality. So we want to go that way. So many biologists wonder how we can actually use this technology for applications. So let me show you some interesting applications. So these are some of the pictures or representative pictures of cells captured by this intelligent image optics. So we can use these images to to activate the cell sorter and sort these cells. Euglena gracilis cells, budding yeast cells, T cells, cancer cells, and in platelets. So Euglena gracilis cells is a model organism in algal research, and that consumes CO2 to produce lipids. Actually, microalgae actually consume 60% of CO2 on Earth. So it's very critical. And many companies like these, you know, in the US, Europe, and Japan are interested in using microalgae to produce lipids and eventually biofuel. And here, the number, size, and the localization of lipid droplets in these cells are important biomarkers for optimizing the efficiency of lipid production. And then they want to do this, they want to do image-activated sorting of uroinvasive aggressive cells with unique morphological lipid droplet distribution features, so that they can do direct evolution to eventually produce highly efficient, highly productive uroinvasive aggressive cells. So we are in collaboration with these companies. Another application is is to food science based on budding yeast cells. You know, yeast is extremely important. It's a model organism used in microbiology and food science, such as wine and beer and sake. And there are many strains, more than like 1,500 strains of budding yeast cells with unique morphological traits. And in fact, there's a strong correlation between the morphology of budding yeast cells and the taste of wine, beer, sake, and so on and so forth. And this is a kind of example. So wine, this wine yeast, this beer yeast, sake yeast, and they look kind of similar, but actually they are different. The differences are very minute. So it's very difficult to distinguish under the microscope. But we can do this, we can actually use these differences to isolate these, say, wine yeast cells from the mixture of the yeast and Saccharomyces cells. And, and so the size and shape of budding cells are important biomarkers for optimizing the efficiency of winemaking, and brewing, and baking. So we actually in collaboration with a beer company to produce sort of a new beer product. Another application is to cancer biology. And, you know, CTCs are kind of hot topic, maybe used to be a hot topic. So, you know, 90% of cancer deaths are related to cancer spread. So the primary tumor sheds so-called circulating tumor cells, and they circulate in the body and move to a new location. And so the ability to visualize multiple characteristics of cancer cells, or CTCs, is extremely important because CTCs are extremely rare. And to increase the accuracy and precision of CTC detection, we need multiple parameters. And conventional CTC detectors rely on surface antigen or the mass or only one parameter. But imaging actually provides a lot of information such as cytoplasm, surface antigen, nucleus, and so forth, and even the ability to cluster. So, and also the metastatic power of CTC clusters is about 1 100 times higher than that of single CTCs. So it's actually more important to identify CTC clusters from a mixture of CTCs, blast cells, and CTC clusters. So by using this technology, we can actually identify and sort specific types of CTC clusters from the large population. In fact, we have done this. So these are— we can identify these like chance CTC, white blood cell clusters, single CTC, CTC-CTC cluster, or CTC-CTC doublet, or CTC-CTC triplet. Another application is to cancer immunotherapy. So CAR-T cell therapy has been very hot nowadays. And basically, we reprogram T cells collected from the human blood, and then inject them back into the human body. And we need to evaluate the sort of the ability of the T cells, reprogrammed T cells, before injection back into the body. One way to do it is to check the immunological synapse like this. So, the localization and intensity of the immunological synapse are important biomarkers for understanding T cell activation, in particular the coordination and recognition and integration of adaptive and innate immune responses. And we actually use intelligent image-activated cells, SORA, to identify the power or the ability of T cells and for evaluating cancer immunotherapy. And we're doing this with Sysmex later. Another application is to COVID-19 research. And, you know, purple rash, leg swelling, COVID toes, brain infection— these are sort of typical symptoms of COVID-19, and, and they are related to blood clots caused by unusual platelet aggregation. So for better understanding of pathophysiology of COVID-19, it's critical to isolate these platelet aggregates from blood for detailed analysis. So these are the images captured by the Intentional Image Activation Solver, and we have actually done this research recently and discovered that there's a strong correlation between the concentration of saturated platelet aggregates, and the severity, mortality, gender, respiratory condition, and so on, of course. So basically, we identify and sort platelet aggregates from the blood of COVID-19 patients. And here we actually analyzed the blood of 110 hospitalized patients at the University Hospital, and then we identified very interesting correlations. Okay, what about the future? So after the publication of this paper, many people contacted me and told me that they are interested in using this technology. So we started to make this technology publicly available so that they can come to the University of Tokyo and use it for joint research. And this is a network of collaborations right now. It's not everyone, but currently more than 20 groups in the world use this system. I mean, at least before the pandemic, because the pandemic has slowed down this international collaboration a little bit, but now we are sort of resuming. And because we couldn't handle all the requests So we started a company called Cybo. Cybo stands for cell in Japanese. And Naomita is the CEO. He's one of the founders of Sony's fax business. And then he joined my group and then started this company. And we currently have two products, image activated sensor and ultrafast 3D microscope. So it will be commercially available. We currently have a beta version of the technology. And recently we have made it level-free. Level-free. So I'm not going to go into the details, but we have made this image activated cell sorting technology level-free of Raman image activated cell sorting. Here the source, optical source, is the Raman source for stimulated Raman scattering. So just like a laser, we excite the molecular vibration in the cells and we directly measure them without the need for fluorescent labelling. And this label-free Raman image-activated cell sorter is quite useful for certain applications such as stem cell research. Or hematopoietic stem cell transplantation, because we want to make them label-free or less toxic before injecting into the human body. And these are some of the pictures. Another direction is the so-called Rosetta stone single-cell biology. So, the biology is somewhat analogous to this Rosetta stone. Rosetta Stone is an Egyptian stone that describes something in 3 different languages, but actually 3 different languages describe the same thing. And this situation is somewhat analogous to this biology. In biology, you know, we have flow cytometry at population level, microscopy at cell level, and sequencing at gene level. So these regimes, or these different communities talk about biology, same biology, in 3 different languages. And, you know, I was a physicist, I'm still a physicist. I'm interested in unifying things, different areas of biology. And by introducing this intelligent image-activated cell sorting, I combined flow cytometry and microscopy. And eventually I want to combine all of them. To relate all those things so that we can make serendipitous discoveries coming from 3 different layers of information in biology. And machine learning is actually composed of 3 different techniques: supervised learning, unsupervised learning, and reinforcement learning. And today I talked about the supervised learning. Basically, I told intelligent image activist, sort of how Wally looks like, right? But we can do this by using unsupervised learning. So we don't have to tell Wally how— well, we can tell the intelligent image activist, sort of, we don't have to tell the intelligent image activist sort of how Wally looks like, right? But still, the the machine can identify the Wally or Wally-like cells. So this is interesting, then we can use this machine to identify unknown species of cells. And this is just a list of references related to this work. And finally, I'd like to acknowledge my group members and, and the funding agencies. So these people have made significant contributions to the development to this intelligent image processing. Thank you. Thank you very much for listening.

Thank you for this fantastic talk, Professor Goddard. We appreciate it. As I wait for questions, the cells in the body sometimes are floating, sometimes are suspended, sometimes attached to surface. Do you do preprocessing Let's say you want to analyze hepatocytes from liver, do you do different, you know, the dissolution of cells or just these suspended cells?

Sure, so the condition or the preparation is pretty much similar to FACS. So as long as you can use FACS machines for your particular application, of course the type of species really Depends on the application. And as long as you can use FACS machines, we can use this, or you can use this iHCS, imaging image activated cell sorting machine. So the same protocol, the same preparation, apply to those cells of interest.

Thank you. Thank you for your great talk. Last few years have seen a surge of interest in single cell analysis using omics data. Have you used or are you planning to use this technology for such single-cell analysis applications?

Yes, that's what we are currently doing. Yeah, many people are in fact interested in identifying the genotype-phenotype relation, especially morphological phenotype and genotype relation. So once we isolate unique cells with unique morphological features, and we do sequencing and then relate that information with the phenotype. Then eventually, once the correlation is found, we don't have to do sequencing anymore. We can do morphological analysis to identify the genes. That's what we're doing right now. Or we're actually looking forward to collaborating with new collaborators. So if you have great ideas, we'll be happy to consider a potential partnership.

I see. Thank you. Another question is, this CTC is similar to Wally, needle in a haystack, one in a billion or maybe less. Are there any ideas to— because they said you can detect CTC is on stage 4 or stage 3, which is too late. Do you have any ideas where we can detect CTCs on stage 1 or 2?

Yes and no. Well, that's something we are doing right now. And we think that the probability of finding CTC clusters is sort of proportional to the cancer stage. So yeah, that's why we are trying to find CTC clusters more, then we can use that information to sort of identify the different stages of cancer.

Thank you. Another question is you mentioned a lot of very high-tech technologies, electronics, optics, microfluidics, fluidics. Where do you see that we can— which aspect can be improved, and which aspect are you working on? Also machine learning, you mentioned.

Yes, that's a good question. So here again, it's a configuration. Imaging is good, microfluidics is also pretty much mature, but the digital technology is evolving very rapidly. So, which is good. And we actually not really developing the digital and software technology because Silicon Valley, or the smartphone technology is the strong driving force to improve it. So we are actually importing or adopting some of the latest technology to speed up the computation on the real-time intelligent image processor. So this technology keeps improving significantly, not just by us, but driven by the smartphone technology, which is good.

Thank you. And compared to the label-free versus the ones with labels, which one do you think would be the winner at the end?

Eventually, People want to get rid of labeling or labels because it's a time-consuming process, and also nowadays it's quite expensive, and you know, it's kind of hard to obtain reagents now because of this Ukraine war and stuff. So yeah, we, uh, eventually the label-free technology will win, yeah, but the label-free technology is still limited compared with the fluorescence-based techniques because the sensitivity is lower, but I think the technology will eventually overcome that difficulty.

Excellent, excellent. That's all the questions I have, Dr. Goda. Appreciate your time and a great seminar. Thank you so much.

Thank you very much, everyone. Hope to see you again.