CTRL-labs lets you control machines with your mind

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

Welcome to Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I've got Chris, my co host, on the line who's Chief AI strategist at Lockheed Martin. How you doing Chris?

Doing great. How's it going, Daniel?

It's going well. You just came back from O'Reilly AI, right?

Yep.

I spent last week in New York city at the O'Reilly AI conference. Had a lot of fantastic conversations, did a little bit of recording and so maybe some of that's to come down the road.

Awesome. I can't wait to. Can't wait to hear hear that. Any, any highlights?

I really was talking, I talked to quite a few people about TensorFlow 2. Everyone seems to love going from 1 to 2 and truly adopting Keras as the blessed way. And so I'm probably biased because I've been, I've been pushing into TensorFlow 2 myself, trying it out a bit. So yeah, I had a lot of great conversations, met a lot of great people there.

Awesome. I can't wait to hear more about it. Well, today on the show I think it's going to be a lot of fun because we're going to talk about something that might seem a little bit sci fi like to certain people, but I think we're going to try to bring it down and make it practical because after all this is practical AI. I'm really excited because we've got Adam Barenzweig from Control Labs with us. Welcome, Adam.

Thanks. I'm excited to be here.

Yeah, Adam's going to help us dig into a little bit of what Control Labs is doing specifically around neural interfaces or processing our brain signals to control machines. So I'm pretty excited about this. Adam, do you want to start by just giving us a little bit of your background and how you got into AI and ended up at Control Labs?

Yeah, absolutely. So academically I studied speech recognition. I have a PhD in Speech and signal processing, which is kind of, if you think about it, it's the branch of machine learning that's closest to a human computer interface application area. In general, if you think about machine learning as one big aspect of machine learning anyway is artificial percept, sight and sound and hearing. A big part of the world that we as humans need to perceive is communication from other humans. And so speech recognition is I guess the branch of machine learning that's always been focused on understanding the output of human brains in the form of speech. So maybe that's going to set some of the context for what we'll talk about later with what we're doing at Control labs. After my PhD, I actually sort of in the middle of my PhD, I went to Google as an intern and decided it was too amazing and I stayed. I did go back and finish my degree later, which was, I consider one of the most heroic things I've ever done is to actually leave Google for a bit and go back.

Yeah, bold.

And finish. And so at Google I was there for 10 years and I worked in general. I'd say my beat was kind of building products around machine learning. So I was no longer really doing hardcore deep research, but taking what was coming out of the lab, taking stuff that was being worked on in academia, of course, within Google and thinking about how to apply that to products. And that was across a number of domains. I worked on text with Google News and the recommendation algorithm there I worked on Google Music. My PhD was actually, even though I was talking about speech before, what I was particularly interested at the time was music and non speech audio and understanding just the world of sound around us. Computational Auditory Signal processing sometimes it's called and this was kind of post Napster, it was late 90s and so I was also A little bit idealistically, politically motivated to come up with technological solutions that would allow musicians to connect directly to their fans without these intermediary giant corporations controlling all the intellectual property in the middle. So I thought, as many people did at the time, that recommendation algorithms are going to be this great equalizer. We'd already had a great way to distribute music, and we had a great way to produce music cheaply. People could make little recording studios at home, but in order for people to find music, we had to solve that problem. So I was interested in doing recommendation, but from the audio signal. So actually analyzing the content of music from the audio and then using that to understand it and make recommendations to people. So then that turned into Google Music, and I helped build the music recommender for Google Play. And then eventually I started to work on images. And I worked on Goggles, which was one of the early image recognition mobile phone apps. You could point your camera at something in the world and we would tell you what it was. Pre deep Learning, actually, while I was working on that project. And so the project at the time I joined was the app had already been launched and it was kind of an amazingly cool toy, but it wasn't quite yet, like a really useful tool for people. So the obvious thing to do next was to put this into. To run this technology, run recognition on people's photos. At the time, Google didn't have a photos project. It had the Android camera. There was photo visual search on the web. You could upload photos. There was Picasa desktop app. So at that time, the photos project at Google was kind of in its nascent form, but this is clearly where the technology wanted to have its home. So I worked on images for a while with that. Then I left and I helped start a company called Clarify, which was an early deep learning image recognition company. So, yeah, in between the time when I joined Goggles and when I left Google, the deep learning, this was about 2012, 2013. So working on Goggles, I saw some of the early large convnets that were coming out of research. And there was an intern at Google, Wochek, who basically was the first deep convnet I'd ever seen. And it was doing things that we'd never seen possible from an image recognizer before. That it was really clear that, oh my God, this is really going to change everything. And it was technology that was ready for prime time.

Yeah. So for some of us, for me, I guess in particular, I feel like a lot of what I've been exposed to is kind of like post the deep learning revolution. So for someone that was involved in machine learning for quite a while before that, or how did you perceive all of these things coming in around deep learning and the acceleration that happened?

Oh, it was magic. It was incredible. I mean, the field had been making advances, sort of incremental advances for 15 years at that point. And SVMs were kind of still the hot things, statistical machine learning. And we could do a lot of the technologies were okay and pretty good for a narrow range of things and a narrow range of classes. You could get a binary class of FHIR that would work fairly well. But the whole idea of having a single network that could recognize a thousand things at once or eventually 10,000 and upward was enormously. It just changed a lot. And similarly, on the speech side, the results that were coming from the early deep learning based speech recognizers were just blowing everything away. It was pretty wild.

So at what point did you, I know that you were saying that you'd kind of gotten to clarify around 2012, 2013. Was there anything else you did prior to jumping into Control Labs?

Yeah, so I worked for a year on an idea for a startup, which I still hope somebody will go and do. It was actually quite different than what I'm doing now. But I was just thinking about the problem of teams collaborating together and how machine learning can wrangle all of the data that sits between people on a team. And probably you have this problem. You've got data scattered across Google Docs and Slack and Trello and who knows other places and where is all this stuff and who knows where things are?

You know me so well.

Yeah. So I worked on that for a bit. We had some good ideas. But eventually I got sucked into Control Labs because. So Reardon and crew reached out to me at some point while I was doing that, and it was really just up my alley. It touched on some ideas that I'd been thinking about for 20 years at that point.

So can you kind of tell us all about what controlabs is and how it came about and what you're doing there and kind of just give us an intro to it?

Absolutely. So Control Labs, we're building a neural interface. And maybe a bit surprisingly for what people think of when you hear that phrase, the device that we build is worn on the arm. And I'll get into what that means in a second. But essentially the goal of the company is to become the new interface technology that allows people to connect and control machines. And so you can think of this as eventually, but perhaps a Replacement for everything you use to control machines right now. Keyboards, mice, touchscreens, joysticks, all of that. Because when you boil it down, all of those physical mechanical controllers are just a machine that is designed to convert signals coming out of your brain that get transmitted into muscle movement that you move your hand to touch something, which eventually gets transduced a signal that goes into the computer to control something in software. And so we're just trying to cut out all of that middle stuff and go straight from the signal coming out of your brain through your muscle, decode it from your muscle and go from there. So Reardon and Patrick started the company about three years ago. @ this point, they met at the neuroscience department at Columbia. Reardon's background is quite interesting. He was at Microsoft back in the day and basically started the Internet Explorer project and saw that through and at some point got sick of tech, decided he was going to go back and study Latin and Greek and got a classics BA and then eventually started dabbling in neuroscience and got a neuroscience.

PhD. And so he met Patrick and now he's.

Back. Now he's back in tech. He's back and he's.

Yeah. And this is the CEO, is that.

Right?

Correct.

Okay. Yeah. So you mentioned that what most people might kind of think about when they first think about like a brain machine interface, the first thing that I think about is like a bunch of wires coming out of my brain or like, you know, someone drilling a hole into my skull and inserting something and then like connecting it to a big server or something like that. And then I, you know, take over the world. I, you know, I've seen too many sci fi movies.

Maybe. Yeah, bad imagery.

Man. This is, this is not the case with Control Labs.

Right? Yeah, we're not going for the plug in the back of the skull just yet. I mean, and it's. So there's actually some really interesting reasons why we believe that that's actually never going to be the right way to do this if what you're interested in is control. And I'll talk about what that means. But if what you're interested in is having some effect in the world, then the motor nervous system is the part of your brain that is specifically evolved to do that. So let's maybe just talk a little bit about the neurophysiology and the neuroanatomy to kind of back up what is behind that belief of ours. Not to mention the fact, of course, that it's not so user friendly to drill holes in your skull in order to Be able to play video games. But thank goodness besides that. Yeah. So maybe a good way to start is that one of our science advisors, Daniel Wolpert, who's just incredibly brilliant neuroscientist, and his expertise is in the motor nervous system. He's got a great TED Talk that you should totally. Everyone should go and watch. I think it's called the Real Reason for Brains. And he makes this super surprising but really cogent argument that from an evolutionary perspective, the only reason that brains exist, they're not there for thinking and feeling, even though that's obviously what we think of as our human experience of having a brain. But as far as animal evolution goes, brains are there for one purpose, and that's to move muscles to have some effect in the world. Because from a survival evolutionary perspective, nothing else that you do has any purpose in the world. With the one exception that he makes is sweating. It's the only thing you could do to affect the world besides move a muscle. Breathing, talking, everything involves your motor nervous system. And he also gives this really cute example of there's a sea squirt, which is this super primitive form of life, and it is an animal. It's born with a brain, and it swims around for a while at the early part of its life, and then at some point, it settles down, finds a nice little rock, and then it digests its own brain because it no longer needs it, because it's not moving anymore and it doesn't need this extra.

Baggage. Yep. My wife claimed I did that years.

Ago. Well, I have some evidence that you still have a brain, but I'm not going to argue with your.

Wife. Go ahead. I'm.

Sorry. If you think about it from that perspective, all of the cortical bmi, in other words, the brain machine interface that is focused on getting signals out of the cortex. If what you're trying to do is figure out. Okay, so let's talk about control. So our name of the company is Control Labs. And from our perspective, what we think of control is it's the process of turning intention in the mind into action in the world. So you've got some desire to have some change happen in the world. That's kind of our loose definition of intention. And then to turn that into action, actually make something happen in the world. The part of the brain that does that, you might have a lot of swirling unconscious thoughts and all the stuff that's responsible for regulating your internal organs and whatnot, but the moment you go to do something in the world and that intention Becomes action. That's your motor cortex. That's the part of your brain that is like what we call the output port of the brain. And in fact, even people who are doing invasive cortical BMI or opening up the skull and trying to plant electrodes in there, they're planting those electrodes, they're targeting the motor cortex because that's the part of the brain that is involved for this stuff. You specifically don't want information. All the rest of the processing that's happening in the brain, it's essentially background noise that you want to get rid of. So that's the first principle, principle, kind of from a neuroscience perspective, why we're interested in the motor nervous system. The second thing is just like a matter of scale. Like there's billions of neurons in the brain where we are focused on the arm and the forearm. And the reason why we're there is because it controls the hand. And the hand is super important for humans. And in fact, about a third of the motor cortex is dedicated to the hand. Another third goes towards the speech production system. And then the rest of your body, like all of your body, only gets like the remainder. So, you know, there's on the order of 10,000 neurons, motor neurons in the arm. And so we just have a much simpler problem in terms of trying to decode signal from noise being outside the body. And then the third thing is that there's just a signal to noise ratio question. And the really great thing about the neuroanatomy, when we talk about surface emg, so the technology we use to decode the signal, I should say, is surface electromyography. So that's basically taking the electrical voltage changes when your muscles contract and decoding that with electrodes that sit on the surface of your skin that are embedded in a bracelet or eventually a watch. And they're just reading voltage potential differences as your nerves make your muscles fire. And the critical thing to understand and why we call this a neural interface is that nerves communicate in this kind of quasi digital way with these spikes. So even though it's an analog signal in the sense that there's a voltage potential wave that goes down the axon of a neuron and also through the muscle, as we'll talk about the shape of that wave and in fact the size of it, the amplitude is extremely rigid for a given kind of neuron, it basically always looks the same. And so information is encoded in the nervous system through essentially the firing rate of these spikes. It's almost like a digital binary. Ones and zeros sort of information flow. So when the brain is trying to move a muscle, a spike train, a series of these very fixed shaped spikes, will be sent from the motor cortex in the brain. One neuron will carry that signal down the spinal cord. Second neuron will carry it from the spinal cord out to the muscle. So there's only two neuron hops from there, and then when it hits the muscle, it's the same exact shape of the signal is propagated through the muscle as the muscle contracts. So the signal that we get is essentially the exact same information that's coming out of the motor.

Cortex. That's.

Cool. So this was like the whole insight that Brierdon and Patrick, as neuroscientists understood is like, why go into the brain when the signal that you actually want, as far as control is available so much more easily on the surface of the skin in the muscle? And there's one more point to make, which is that the voltage signal as it's traveling, the size of the voltage in these signals as they're traveling in the nerves is like in the order of microvolts. But what happens in the muscle, a single nerve motor neuron will come and innervate, meaning attach to a nerve, a whole sort of bundle of of muscle cells. These muscle fibers and that whole thing together is called the motor unit. So one nerve and all the muscle fibers that it innervates is called the motor unit. And each of those muscle fibers in the motor unit only gets input from that one signal. So it's kind of like a one to many mapping. And that whole thing, when a spike comes down the nerve, it all fires at once. And it's basically an amplifier. So from an electrical perspective, we've got this amazingly nice built in amplifier that takes the tiny little microvolt signal that would be traveling in a neuron and pumps it up to millivolts that's readable through all the tissues of the arm and through the skin and not worry about hair and all that.

Stuff. All right, Adam. So maybe it's just because I don't want to have a hole drilled into my head, but you've convinced me that that's not the way to go. And you know, there's this really great signal that we can tap into through the surface emg. Maybe you could kind of switch directions a little bit here and kind of motivate. Why neural machine interaction or a neural interface? Why is it important that we pursue this sort of interaction and don't just stick with the kind of mechanical interfaces that we're familiar.

With. Yeah, thanks. That's a great question. So I guess maybe one way to get into this is think about the transition that seems to be happening in personal computing. So a lot of people who pay attention to technology believe that we're on the cusp of a giant paradigm shift. What is going to be the personal computing platform that is going to replace the phone in your pocket? So maybe let's talk a little bit of the history of HCI and the kind of paradigm shifts, as I see it, over the last 40 or 50 years. So in the beginning was the command line. By the way, that's a really great essay by. I think it's Neal Stephenson about just how cool it was when used to compute, just using terminals and text input. But it's also interesting. And then it's a perspective about the transition from that to GUI based Windows. What do they call it? WIMP Windows? Icons, menus and pointers. The desktop computing paradigm that we all know and love, maybe love, that came out of the research at Xerox PARC and Doug Engelbart's mother of all demos. And so that was a second phase. And then there was touchscreens and mobile computing. And so it seems like with VR, AR, XR, wearable computing and the whole idea that computing is getting cheaper and smaller and will start to become embedded in the things around us and maybe just embedded into things that we can wear and a multitude of sensors and a multitude of processors around us. And so for a long time, HCI researchers have been sort of anticipating this transition. And I guess ubiquitous computing is probably the best umbrella term for this. Some people now calling it spatial computing, wearable computing. So the vision is that once the display becomes mounted in your eyeglasses or your contact lenses, then you no longer have to be holding anything. So how do you do input? You're not walking around with your magic AR glasses in the future holding an Xbox controller so that you can navigate your email. And if you think about it, every one of these big HCI paradigm shifts really comes with a pair of technologies that play the input and the output role that together make up the new interface paradigm. So when the shift from console to computing to gui, the mouse is the input, and then a cheap rasterized display technology was the output. And with touchscreens, it's kind of interesting because the output and the input are fused together into the single device, but they have two very different roles that they play and they're actually completely separable. You could have a touchscreen that's only input if you wanted and if you go back and read the writing. So I just reread some of Jaron Lanier. He's one of the early VR pioneers and I think is credited as coining the term VR. And he was so focused in the beginning on the input as much as the output. They did all these experiments with data gloves and all kinds of other motion capture technologies because they recognized that if you want to have a truly immersive experience, then it's not enough just to have this visually visual immersion like the input side. You also have to understand what is the person trying to do and let them interact with the world. And so that was a big focus in the beginning. And I think what happened around the time when, when Palmer was making the hardware work on the display side, that raced ahead and cheap and good enough display technology with good head tracking happened before there was another big shift on the input side. So that kind of leapt ahead and that leads us to where we are right now where there's this kind of a big gap. And if you use the VR systems today, it's really beautiful. You feel visually immersed, you look around, it's amazing. And then you just have these sticks where your hands should be with these kind of game controllers stuck onto a positional tracker. And it's very kind of disappointing from interactions perspective. You don't have the naturalness, the expressivity, the incredible dexterity and agility and skillfulness that you have with your hands. So that's I think the big picture that we're thinking that this technology is going to be the thing that is in a device that you just are wearing all day long and is the way that you interact on a minute to minute basis with all the devices in your life. Trolling the volume in your car and typing an email while you're walking down the street or playing a video game, whatever it is, Hands are the best controllers. And so all you need is hands. We're just building a device that will just allow you to plug in directly to the signal that's controlling your.

Hands.

Gotcha. So one of the things I'm kind of wondering, and that was a great explanation by the way. But if you're in a very practical sense, I'm imagining that person that's starting to utilize this technology going forward. And I have seen the AI conference video of the presentation and so I have a sense of being able to utilize the technology where you're starting to remove the hardware between. But we also have other HCI which for listeners if you don't know is human computer interface forms like voice interfaces are so popular. I have kind of a two part question. How would you imagine what you're doing at Control Labs interfacing with these other types of hci, such as doing this chat, which is so popular now. And we're already kind of programming that in APIs. And the kind of. The second half to this is when you're looking at that combined world of utilizing all of these methods together. I know you guys have ControlKit that you've released and it has what is essentially, I guess you're describing as an API for the brain. I mean, how do you realize that in an API, considering that you have different modes of.

Communication? I think there will be a lot of things used at the same time together. I think that there's going to be a multimodal approach and maybe let's shake the two things of voice input and this idea of a neural interface that's getting the output from your hands and the way that those things can be used together is really fascinating. And they have different strengths and weaknesses. Obviously there's privacy concerns or just kind of social awkwardness. You don't always want to be speaking out loud in public to control your computer or to do input in terms of text input and communicate. You might be sitting in a cafe and you're writing a sensitive email, or you just don't want to bother your neighbors and so people still type. On the other hand, there might be circumstances where your hands are full and you can't use them and speaking might be the most natural thing. Although we've been exploring some really interesting ways that you can still do quite a lot while you're holding objects. Because in some sense we didn't really talk about how sensitive the device is, but tiny little twitches that are essentially invisible to the naked eye are enough to do control with this technology. So you could be holding your coffee cup and just doing little presses or wiggles with your fingers and be typing. And then of course there's a combined impact, right? If you have all channels available at once, if you're in your office and you can speak out loud, but you can also use your hands, maybe we'll just use that to boost up the bandwidth. If you think about this technology as just a way to increase the output bandwidth from the brain to the world, we have so much bandwidth coming in, high resolution video displays and great audio, and the bandwidth coming out from the brain has just lagged behind for 50 years. Typing on a keyboard is One way to kind of estimate what's the maximum just in terms of pure bits per second that you can get out of the brain, the fastest typist in the world doing 150 words a minute, that's something like 30 bits a second. And compare that to gigabytes of information coming in. So, yeah, I think this multimodal thing might be a way that it goes. And another point I was going to make is when I was working on speech recognition, it always seemed to me, especially when I was at Google and the first really great ASR systems were coming out and were put into Android in terms of speech input on Android, it always seemed to me a waste that I couldn't also use my hands while I was doing speech input. I'd be holding the phone and it would be making all these errors and you know, hopefully not too many, but still, still makes errors sometimes. And then you kind of have to slowly go back and correct things with your hands in this old school typing way. It just seemed to me that there was like had to be a better way that you could combine the input of your hands with beach. So that's one of the interesting areas that we're.

Working. This episode is brought to you by the O'Reilly Open Source Software Conference in Portland, Oregon, July 15th through the 18th OSCOM. The O'Reilly Open Source Software Conference is where you go to understand what's shaping software development from AI and cloud technology to distributed computing, and learn how to put it to work for you and your team. Whether you're looking to understand where software development is heading or where machine learning can make or break your code, OSCON is where you'll find your answers. Hear from industry leaders like Pete Scomaroach, holden Carew, Allison McCully and Sam Charrington. Passes start at 796 when you register with the code changelog20 before June 7th. Again before June 7th, use the code changelog20 and head to oscon.com changelog to learn more and.

Register. You know, because this is a podcast, it might be hard to imagine the sort of interface that you're talking about. And I know Chris mentioned that you've released this kind of control kit. Developer. I don't know what you. I guess kit is a good word. You already picked the best word. But could you just describe to our listeners what the interface or the device looks like and kind of how you wear it and then how it connects to another device, let's say a computer? You're wanting to kind of type with your muscles. I guess in how you're describing it. So could you kind of just describe for our listeners how that looks and how it.

Connects? Yeah, it looks kind of like a chunky sort of cyberpunk bracelet right now. We currently wear it kind of midway up the arm and the forearm. But the goal is to get this down at the wrist and have it be able to be integrated with a watch. We have devices that work on the wrist already. It's actually the biggest challenge to getting to the wrist is really just your tendons move a lot. If you kind of look at your wrist as you're kind of moving, look at the space where you would wear a watch. As you move your wrist around, there's usually all these kind of divots and the topography of your. Just the mechanical challenges of keeping the electrodes in contact with your wrist as you move around is the major thing keeping us from being at the wrist. But yeah, the electrodes that make contact with your skin are on the inside of the device, obviously against the skin. They're pretty cool. They're actually gold plated right now and they. It's got some bling, it's got some serious bling. And the whole effect is this sort of like cyberpunk meets like studded biker leather.

Look. Oh, I'm into this.

Now.

Yeah. Chris, do you think I could pull it.

Off? Absolutely.

Daniel. So that's where we are now. And of course, you know, this is, we're just beginning. This is the first device that we have that is not just like a total research prototype. Our previous generations of device were like literally we started with sweatbands with electrodes sewn into them and then kind of ratcheted up the ladder of hardware productionization. So it's Bluetooth. It communicates to a host computer over Bluetooth. And currently with the developer kit as we're shipping it right now, you need a fairly. You can't connect directly to a phone. You need a host computer that can run the other side of the pipeline that's actually doing the machine learning, inference and the processing. Signal processing mostly happens off board device right now. But as we go through the iterations of hardware, we're going to move more and more processing on board so that you can connect directly to a phone, let's.

Say. And can you, since you've kind of segued into it, can you tell us a little bit about what you're doing in terms of machine learning for this? Where is it being used, what kind of problems are you solving and what are some of the choices able to share that you're using to tackle.

Those. Yeah. So maybe one thing to say about the machine learning side of this from the beginning is I think the most exciting thing that we're doing kind of from a machine learning perspective is this is an entirely new domain, a signal domain to work on. If you think of actually, you had a guest on a couple of weeks ago from IBM Healthcare, Guillermo, and he said something about you guys are talking about processing speech as a way to diagnose psychiatric disorders and stuff like that. And again, I think the idea of some kind of brain waves came up and he said, speech is a brainwave and we very much feel the same way about the signal that we're working with. It is this bio signal, a signal generated by the brain that is extremely information rich, carrying all this content about what people want to do. And it's as rich as speech really. And, and it's new. It's like nobody has had access to this kind of signal before, not outside of very narrow clinical settings and neuroscience research. So that's just really cool. It's like as if we just invented a microphone and we can start doing speech recognition. Except instead of the microphone, the time distance between them, when the microphone was invented and when machine learning was good enough to do speech recognition was almost 100 years. And for us, it's like we're there already. So we've got this access to a new signal that's extremely interesting and already have great machine learning tools to work on it with. And the parallels to speech actually go pretty deeply because it's a multi channel, time varying continuous signal that has frequency content and some of the information is there, but it's got this particular structure in the way that it was generated. And then there's all these layers that you sort of start by parsing the kind of lower levels, understanding how to do the signal processing and get rid of the noise and figure out where the information is. And then as you go up the stack of processes, you get into meaning and semantics and what is the person trying to do and actually recognizing people's.

Intention. All right, so you kind of alluded to part of this a little bit ago where you were talking about where machine learning fits into this pipeline and there's kind of two sides of it. So is my understanding right, that kind of the device that you're wearing on your arm, this super cool looking cyberpunk bracelet. Armband. Yeah, cyberpunk armband. And it's sending these continuous neural signals to a computer or to a host computer, and then you're running a model on that host computer that's kind of making inferences based on those signals. Is that kind of the overall pipeline or did I get something wrong.

There? No, that's exactly right. And the kind of differences that we make are dependent on what the application is and what we're interested in doing. So the basic capabilities of ControlKit, the API as it is right now, include things like reconstructing the position of the hand. So sort of the canonical classic demo we have is essentially like you'd see something from like a leap motion. It's essentially a motion capture experience where you just see a virtual rendering rendition of your hand. And what we're doing is we're predicting all the joint angles, the positions of your fingers and your wrist as you move your hand around. And we can also detect things like how strongly your muscles are contracting. And this is one of the kind of primary advantages that this technology has compared to a camera based hand tracking system. So most of the effort and investment over the past five or 10 years in terms of hand tracking for the purposes of VR and XR have gone into camera based systems, inside out tracking and outside in tracking. And with any camera based system, you always have occlusion problems. What happens if your hands are not in view of the camera? So you can either try to solve that by just placing cameras everywhere, or you can have a technology that doesn't have any occlusion problems at all, which is what we're providing. So we sometimes have been talking about this as maybe like inside in tracking, where because you're sort of wearing this device that is looking inward at your own muscle and your own neural signal, there's no possibility of any occlusion problems. So the device works as well with your hands in your pockets, or in a glove, or in a spacesuit or whatever could be used in the future. So you don't have occlusion problems. And since we sense the strength of your muscles contracting is an enormous signal for us. And that's really another interesting difference between camera based tracking systems. If you kind of make a fist and now make a really strong fist, visually there's really not that much difference there. But in terms of the signal that we see, the firing rate of the neurons that are controlling these muscles, it's huge, it's like an enormous event. So the other part of our API is recognizing stuff like, like forces of pinches on each finger and your fist and other interesting ways that you can kind of tense or relax your muscles. And that I think is going to be a really important part of the design problem around this, which is how to use that signal. It feels extremely natural and expressive when you can kind of use muscle tension and how lighty or hard you're squeezing to be able to control software. But it's not how we're used to thinking about designing interfaces, so. So that's going to be a big challenge for us and for designers who are going to be working with this technology. Let me think if I covered everything in the API. So the hand skeleton, tracking poses and pinches, also recognizing just hand pose, different finger pointing and shapes. And then we're working on turning that stuff into usable, reliable controls. The equivalent of point and click. And we are working on the extremely interesting to me personally, project of typing or figuring out how to do text input with this.

Technology. Gotcha. So given the fairly unique type of signal that you're pulling off with the glove and passing over to the computer, what kinds of machine learning architectures are you trying to utilize and apply to these types of problems? How are you approaching. I know you mentioned machine vision, which is typically done with convolutional neural networks. Would this be like with a recurrent neural network of some sort or something like.

That? Yeah, we do use RNNs of certain flavors and we use convnets in other places. It's actually a kind of grab bag of approaches and different things work better for different kinds of inference. We're trying to do. One of the other interesting things is that. So our current device is 16 channels. So. So there are 16 pairs of electrodes that go around your arm to kind of COVID all of the area and pick up lots of different muscles and the way to combine those signals from neighboring electrodes and there's actually. It parallels a branch of signal processing that's think about beam forming and multi microphone array technology. This is popular in audio processing if you're trying to isolate one signal from a crowd of background noise or you've got a room of people talking and trying to extract one signal out of many. So there's some of that kind of work as well. But yeah, we use RNNs, we use convnets. We're writing the machine learning stuff in TensorFlow. Our research team works mostly in Python. And then we have kind of the production system which is in C. And we've actually got a really nice way of blending those things together. I think that's just on the practical side, one of the challenges that we've always in any Machine learning based team that I've always worked on is there's usually a big gulf between what researchers are comfortable doing and then getting code into production can be this big gulf. So we were very focused on making that hop very short and allowing researchers to work in an environment that's extremely close to what the production systems are like and having they natively can work in the environment that they prefer and have it be a quick hop to.

Production. So I'm curious. I think so I may be misremembering things, but I think you said that something about the signals that you're getting off the muscles are fairly consistent in how they're shaped. I was wondering, as far as like shipping the developer kit, is this a thing where you train kind of models on your end, on the control lab side with a ton of data that you've collected for different tasks and then you kind of ship the models with the control kit to just be used for inferencing? Or do you have to do any sort of transfer learning or fine tuning for a certain person's. Are my signals different enough that you need to do some fine tuning for my signals versus signals you've seen.

Before? I pinpointed one of the most probably critical problems from a research perspective that we're working on right now. So we've come at it from both ways, right? Everyone is different, everyone's anatomy is different. The signal does look quite a bit different across people, and not just across people, but across sessions. You take the device off, you put it back on the electrodes in a different place, they're going to see a slightly different view of the underlying neuroanatomy and the signals will be different. So you have kind of cross session generalization that's, you know, take the device off, put it back on, and you also have cross user generalization. I want to train something on people that we have available here in the office, or getting people to come in and do data collection and then have it work on someone in the world who we've never seen before and we'd never seen their signal before. So there's two basic approaches. We can try to collect a lot of data from a lot of people and build models that will just generalize out of the box. And we do that. And then we also can go the other way where we come up with ways that can learn very quickly on the least data possible to get good performance with data from that one specific user. So in that case you have an experience where there'd be some kind of onboarding, calibration, training period where you would do some movements or play a little game or something like that. And it would be learning about you and your own particular signal as you're doing it. And then there's hybrid approaches where you kind of get as far as you can with generalization and then have a transfer learning or domain adaptation, as you were mentioning approach where you try to be build systems that are designed to be customized or personalized. So they might not be. The goal of them isn't necessarily to get the best out of the box performance as possible, but the goal of them is to learn about the kind of space of signal that you could see across people and learn how to adapt very quickly to a new user signal when you do get a little bit of data from that.

User. So as we been talking, there's something in the back of my mind that I've kind of been wondering about and that is this is such a specialized domain that you're addressing. Is there a role in this for biologists or doctors in the development process to kind of capture that domain knowledge? Or is this something that you guys have been able to just to master on your own and move forward with the.

Technology? The company was founded by neuroscientists and it's pretty important. There is a lot of domain knowledge that goes in. And even just at the very basic level of understanding what the signal that we are looking for ought to be and ought to look like, the essential idea of trying to use EMG as a consumer device for controlling computers goes back a ways, actually. I remember when I was in grad school and we talked a little bit before about my interest in music, but I was also interested in new musical instruments and musical performance systems. And I remember reading about people who were hooking up electrodes to their body in various ways and then using that signal as a musical instrument. And so the ideas are floating around. I think it does require some deep understanding of the neurophysiology to. I think one of the key things was knowing when to keep iterating on the hardware and when to stop. So Reardon and Patrick knew that if they could get the hardware to be good enough to recognize individual spikes on single motor units, then that essentially was all the information that there was to be had. And they didn't have to keep iterating after that, but they shouldn't stop making the hardware better until that point was achieved. So I think that was key and that differentiated our attempt at this from earlier ones. We've got MDs we have MD PhDs on staff. And so some understanding of the anatomy is important. But there is some level at which machine learning is machine learning and the signal is what it is. And kind of, I mean, one of the cool things about working in machine learning, I'm sure you guys guys have appreciated this, is that it's just a chance to become a kind of mini expert on a particular domain that you happen to be working on a new problem. You might come into knowing nothing about phonemes and phonetics and the way that language is constructed. But you start working on speech recognition, you learn a lot about it really quickly. Or if you're analyzing stock market data, you suddenly learn a lot about financial signals. And so I think that's part of the fun of doing machine learning. And probably anybody could come in and work at this company and learn a lot about that stuff. It's important, but it's not critical.

Prerequisites. Yeah. So I appreciate you sharing that. It's definitely one of the things that I really appreciated about this industry and working in this area is all of the things that you constantly get to learn. Speaking of kind of people working in this space, I know I'm already kind of all of these ideas for utilizing the Control kit and this technology are already popping into my mind. Like you were mentioning text input and I'm thinking about like over 350 sign languages that are known in the world and typically people have not very good access to information that speak those sign languages and being able to have one of these devices on each hand and document those languages and get resources to these people. There's already all sorts of stuff popping into my mind. So if I want to pursue something like that with Control K, kind of what's the rollout plan and what's the Current access to ControlKit for.

Developers? Yeah, we really want to hear about all the amazing applications that people have in mind when they hear about this technology. And we want people to start building with it. That's absolutely where we are as a company. We've got some basic stuff proved out. We really believe that this is a technology that's going to be super important in the future. And we want people to start playing with it and help us figure out how it can be used and what's it good for. So the first step would just be go to our website, it's control labs, that's ctrl-labs.com and we've got a sign up there for the wait list on our dev kit. We're starting Slow. We're a startup, we're a small company, we don't have the bandwidth quite yet to support the hordes of people who may want to work with this stuff. But sign up and rolling out over the next year, starting slow with developers hand picked for a couple of Apple applications that we think are really interesting. And as we go, we're just going to pick up the pace and see what people want to build with it. The sign language thing, by the way, it's super interesting. That's something I've been studying a lot about recently. And sign language is. I mean, I don't want to go into a super deep dive on it, but the grammar of sign language, the phonology of sign language, a natural sign language, is extremely interesting. It's obviously as complicated as spoken languages, but it uses the articulators and that's the technical term for the part of your body that makes the signal go from the brain to the world. And so I think we have a lot to learn from it because it's the main proven use case right now where we have super high bandwidth output coming from people's brains in a communication channel using the muscles of a hand. So it's very.

Interesting. Yeah. Well, I know I'm going to be headed to the website after this to fill out the application and describe something about sign language and see if I can get one of these super, super dope bracelets on my.

Arm. Awesome. I know someone who works there. You might have an in if.

You. Okay, sounds.

Great.

Yeah. Well, I really appreciate you sharing so much of the great context around this technology and also kind of bringing it down to make it practical and help people understand that there is is a practical route towards kind of processing these signals and getting them into a computer to utilize them. So it's really, really been useful for me and I appreciate you taking the time to walk us through that. We'll definitely put the links to the website and the control kit and the TED Talk and all those things that we talked about in our show notes. But thank you so much, Adam. It's been great to.

Talk. Thanks. It was really.

Fun. It was super cool, man.

Thanks. All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it if you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robo.com change changelog and we're hosted on Linode Cloud servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in in. We'll see you next week. Because you've listened all the way to the end of the show. Got a little preview here for you of our upcoming podcast called Brain Science. This podcast is for the curious. It explores the inner workings of the human brain to understand behavior change, habit formation, mental health and the human condition. This show is hosted by myself, Adam Stakoviak, and my good friend Mariel Reese, a doctor in clinical psychology. It's brain science applied not just how does the brain work, but how do we apply what we know about the brain to better our lives? Here we.

Go. That applied brain science really stood out to me because I want, I don't want it to just be data. I want you to go, how can this fit? What can I take away now? How am I going to change? And that, that sort of is where you come in more and more. And even some of the questions like, so, like, I want to ask you, what are some of the most challenging things working in the tech world? When it comes to relationships, probably.

The most important one is isolation. More and more, the world and companies are being for good reasons. They're being okay with what they call distributed.

Teams.

Yeah. And that means that you and I, we work for the same company, but you work from your home office. I work from my home office. I might go into the office a couple times a week if I live local. But even if I live in San Francisco, I'm still probably a remote worker, even though I can hop in an Uber or hop on the train or whatever and go into the office and be there in a half hour. But why waste the time, you.

Know? And this is where I would revisit what I want to talk about with resonance. And that whenever we're learning, no matter what thing, it's really helpful when we get feedback that's both immediate and specific. And so when you're by yourself and you don't have any interaction with other people. How can you get any feedback? I mean, you're losing most of the non verbal communication and you also don't have all of the voice inflections or facial expression. Have you ever tried to, you know, be sad, feel sad and smile at the same.

Time? Try.

It. It's pretty hard.

Right? Because facial expression is exactly what's involved when it comes to empathy, which is relationships. I was reading a research article recently and it talked about, you know, how couples who are together a really long time end up sort of looking like each other. Yeah. And so what they've looked at is when we actually empathize with other people, facial expression is really key within.

That.

That. And so when you empathize with the partner you're with over and over and over again, your face begins to make the same creases and facial expression as it relates to where somebody else is.

Emotionally.

Wow.

Right. So that's, that's.

Creepy. Well, they've. Again, this is sort of the hotbed when it comes to neuroscience these days is mirror neurons. And these mirror neurons are what are involved with empathy. And so mirroring meaning I get another person's emotional world. And so one of the research studies looked at Botox and what they found is that Botox, because it actually assists in paralyzing facial muscles, but then you can't contort your face so you don't get wrinkles, but actually levels of empathy go down.

Right. Because your physical appearance can't reflect your inner.

Appearance. Yeah, you got it. And so when you're working in these remote locations, it might facilitate better work or more focus. And it allows people to be distributed and to capitalize on the talents across the country.

Right.

Yeah.

Wow. So see, that's like a treasure trove in my opinion. Talking about out in a scientific way, you know, not just like, hey, this is my.

Opinion.

Yeah. About all the cons of that. Because I think what we can do is still have remote work, but do it in more healthy ways. Because I'm, I'm fully, I mean, I've been self employed remote worker since 2006 now. I'm a unique animal. I know that, My wife knows.

That.

Right. I'm fine with it. I'm a good human being, but I've got some flaws and I'm willing to accept and share those to some degree. And I think the problem is we just lack maybe a more purposeful or intentional feedback loop, which I think is super important to being able to operate in this world in just Good ways, I don't know. Healthy ways is probably the best way to use in this show context is healthy.

Ways. One of the things that's fundamental, I would say, to being human is change, right? And so sometimes the people come in and are really key in our life for a period of time and then things change. Either we grow or they grow or they change in a different direction and then the relationship changes or that feedback loop gets modified in some way. That isn't always a bad thing, it's just going. My sense of choice actually is a critical component when it comes to feeling good about my life. If I feel like everything is sort of outside of me and I don't have any charge over it, like I didn't choose to work in a more remote location, or I didn't choose to go to school, or I didn't choose this person, then it feels far more oppressive as opposed to I actually participated in the outcome that I'm actually experiencing. So I then also have more charge over whether or not I want to change.

It. I think this feedback loop process that we're talking about here is super common to developers from people who write code, to people who plan and to engineer and to manage and lead. There's no one in the software process that doesn't understand the feedback loop. And the reason why is because in product development they have this concept of agile. And basically it means you produce something, you put it out there and you expect the feedback loop to happen in order to gain insights and course correction, to then release another version of it that that continually and iteratively becomes more and more improved. So this whole process in day to day work in software is normal. And I think it's interesting how it can apply to their lives and people's lives to take the same importance of a feedback loop, for example, and.

Apply it right well, so this is very much how it goes in relationship, which is why there is an important. When it comes to sort of things resonating, you ever walk into a room or an interaction with a couple other people and like something just feels wonky or off, you're like, I can't put my finger on it. But definitely been there.

Right?

Well. And so to be able to identify that in relationships and even go, wow, I need to, I'm experiencing this person in my world with the limited interactions that I have with them, it hasn't really resonated with me and so I don't get good feedback. So now I'm going to be more defensive because I feel as though There's a threat. It doesn't necessarily mean the person is threatening. However, my brain is going to tell me, hey, we need to be more protective. We need to do some strategies so that you're not fully.

Exposed. You know, one way I look at scenarios like this, I would say as of late, is because have you ever watched a TV show or a movie where the, you know, the narration, the storytelling part of it, they expose a character in a certain light. And you may dislike that they may be a villain or.

Villainess. Right.

Sure. But the moment they turn the story to their backstory and why they are the way they are or why they're acting the way they're.

Acting.

Yeah. You then kind of fall in love with them and you're almost rooting for.

Them.

Right. I feel like that's the same thing that happens day to day to our lives is that, you know, there are people who seem villainous or not for us, but we don't understand their backstory and why they are the way they are. For us to have and employ that empathy that's required to have this. This dance, as you say, this iteration of relationship, you know.

We.

Right. We just assume they are who they are, and we project our worst fears onto them and they become.

True.

Yes. You got it. This is why, in the absence of a face, I don't really get to engage with people in the same sort of humanness that we are all in. And so you're exactly right. I mean, over and over and over again, because you can identify and go, oh, that's why they're harsh, or. I recently had an interaction I had shared with someone that I was a competitive gymnastics coach for a number of years, and so somebody thought that my response to them when they were really struggling was kind of harsh, but they remembered that I had told them I was a coach for so long, and they're like, oh, this is just another side of her coming out. And I'm not sure I prefer it, but I get it. And then it switches for their reaction because then they're like, oh, wait, we're on the same team. She's not trying to, like, oppress me or fight back against me. She actually is helping me, trying to get me to where I want to.

Go. My wife and I, we've learned this. This concept of goodwill.

Right?

Yeah. I can take your feedback or your criticisms in a different light if I know that you have goodwill for me. Meaning that you're not trying to harm me. Me that you are for me, not against me. And sometimes change, as we all know, is painful and can be painful. So sometimes the necessary feedback and or criticism that can influence that change can also be painful. But I can accept it differently if I know that she or they or whomever is in the scenario with me has goodwill for me. Whereas if you know that they're not for you, then you obviously take it a whole different way. And at end, the that's an okay thing. But we often are in relationship with people that are giving us crucial feedback and we need to have that kind of, that lens. It was significant in our marriage to understand, hey, I know there are times when you give me feedback. I am not happy about it, but I know you have goodwill for me. So therefore I calm down, I listen, I take that in and I process it, whatever, but I take it it in a different way because I know that she's for me and not against.

Me. Yep. One of the key things when it comes to change is a sense of openness and even relationally like of going, I need to be able to see some how somebody else responds or how they're feeling as based on their perspective of what they're going through and not just my perspective of their perspective. And so this goodwill is like, I believe that we're on the same side and that you're not trying to make it harder for me. But so I can understand if I were sitting where you were sitting, had the background that you had, why you would have taken it in that way. And then I can provide an opportunity to clarify or create more connection even when it doesn't feel good. And I honestly think this is so much of what's missing in people's relationships. If I look at relational interactions through the notion of conditioning, wherein I get a sort of hit of dopamine, feel good feelings because I went to a person, I had a conversation that didn't necessarily feel good, but there was openness on both parties to hear one another's perspective that it actually then reinforces like, oh, when I go and I have this exchange with people, I feel better. So now I'm going to go and engage with other people and get the feedback, even if I might not like the feedback because now I'm buffered and I'm not alone in this and I somebody else sees my.

World. That's a preview of brain science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us@enortschangelo.com in the subject line. Put in all caps Brain Science with a couple bangs. If you're really excited, you can also subscribe to our Master Feed. To get all of our shows in one single feed, head to changelog.commaster or search in your podcast app for changelogmaster. You'll find it. Subscribe get all of our shows and even those that only hit the master feed again. Changelove.com.

Master.