#107 The Deep Learning Revolution in Space Science

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey. On datacamp.com. You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization. Join us for in depth discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in.

Hi everyone. Welcome to Data Framed. It's World Space Week, so today we're talking about using data in space research. I'm extra specially excited about today's episode because I've been interested in space since I was a kid. My grandma used to live near the Jodrell Bank Radio telescope in the UK and when I went to visit her, going to Jodrell bank was my favorite. Space is of course literally a huge topic to cover. So rather than trying to go over all of it in this episode, we're focusing on the work of the Space Systems Command at the US Space Force in using deep learning on telescope data to monitor satellites. My guest is Justin Fletcher, an artificial intelligence and autonomy subject matter expert at Space Systems Command. He has something of a tricky job since it requires expertise in physics, computer science and military applications on top the deep learning skills. How he manages to juggle these competing areas is unclear. So let's interview him to find out. Hi there Justin. Thanks for joining us today. To begin with, I'd like to find out a little bit about what you do. So you're working for Odyssey Systems. Can you just talk a little bit about what Odyssey Systems does?

Certainly. So Odyssey Systems Consulting is a business. We do advisory and assistance services on behalf of the U.S. government. So I work on a contract called SDA that stands for Space Domain Awareness, SDA Support Systems. So we work on the government's behalf to represent and their interest managing. It's a large portfolio, manages large scale acquisition activities all the way through to research and development. We do a lot of studies work and a lot of technical management. So we're helping and advising the government as they build out advanced technology portfolios for, in our case, what's called space domain awareness, really concerned primarily with what's happening in the space environment and by the US Government.

Which particular government departments are you working with?

For the contract that I referred to before, that's with Space Systems Command. So ssc, which is part of the United States Space Force. Okay. The field commands within the United States.

Space Force, all military stuff. Brilliant. So tell me just a little bit about what you do in your job.

So technically I'm a subject matter expert. That's what my job title is. But my role is to lead a small but focused team of multidisciplinary. So we have several astronomers, some computer vision specialists, a few data scientists, many software engineers to try to advance the state of the art in space domain awareness, primarily through the application of artificial intelligence and autonomy. Those are our two sort of major technology focus areas. And so we try to move the state of the art forward in space domain awareness. What that looks like in practice is a lot of applied computer vision work for scientific imagery as well as we automate telescopes, so closed loop autonomy for telescope control so that they're reactive to stuff that's happening in space. That's sort of my overall area. We have a sort of direct team of six, and then we have an extended team, including performance contractors on behalf of the government of about 40 now. And I'm responsible for that group.

All right, brilliant. So it's a, it's a pretty substantial team. And you mentioned that in addition to the data scientists, the robbers are actual scientists as well. So how does that sort of relationship work? How is, how is this team structured then?

Yeah, so that's been one of the most interesting components of this job over the past few years is, you know, we're out here in Maui. A lot of our team comes from the observatory at the summit up at Haleakala. And so we have this really interesting interplay between people like me. So I came in as a computer scientist having absolutely no background in space. I was at the time when I moved here, I was a military officer for the Air Force, and I moved out here as a computer scientist and was introduced to space. One of the things that we find that's really interesting is cross pollination of technical ideas. So we have, you know, this is, this is almost cliche at this point in sort of the maturity of data science as a field. But domain knowledge is really, we found is the essential ingredient in applying these techniques to advance the state of the art in new domains. So like, you can't just take a RESNET and apply it to an image classification problem. For it requires a variety of detailed transformations of that data and that interplay between the technical specialists in that domain. A lot of this is astronomy and optical physicists and things like that for us and what are more traditional computer science or data science roles has been very interesting. What's been really compelling is to watch those people grow towards one another. So what we see is a lot of computer scientists picking up instrumental astronomy skills and a lot of astronomers picking up software engineering and data science skills, building and annotating data sets, things like that. So that's been really fun.

So I'm always fascinated by the kind of domain knowledge that you need to do these things. So can you give me some examples of the sort of skills like you mentioned, optical astronomy and things like that? So what does that actually mean? What skills do you need there?

Yeah, I mean, so for astronomers, a lot of Those people have PhDs in astronomy. This is like instrumental astronomy. We have some people who specialize in their PhDs in solar physics. We have a gravitational physicist on our team. He did his entire PhD in gravitational physics and then switched skills. And what's really interesting is we have a lot of from the astronomy and optical physics community. Those skills tend to be very research oriented and very focused on the production of physical hardware in the real world. That has a really interesting synergy with what we tend to think of as entirely software oriented skills. You know, when we think about data science and computer science, we tend to think about mostly about the production of software to do things that we want it to do well in our domain we really don't have the luxury of just pretending that data is going to come in from somewhere and then we'll figure out how to get it, we'll figure out how to clean it and to get it ready for processing to train models. It doesn't work that way. We have to think about the physics of the instrument all the way through the trained model. And so that for us is we see a lot of the skills that are needed for that are instrumental astronomy, building the actual instruments. There are analogies in people building advanced camera concepts. Really cool stuff happening in metasurface optics right now with deep learning. And so it's really about having a broad base of skills that is relevant to the domain you're trying to apply these data science techniques to. And then of course everybody's got to know Python. We use primarily TensorFlow, but the team's moving towards Pytorch now as well. You've got to build those skills as well. So that's a skill base for us.

So you've got like these hardware challenges and you've got challenges with physics and you've got the data science and machine learning challenges as well. I can see why a lot of people have PhDs on your team then. So let's go a little bit more into these data science and machine learning skills. So you said you're doing a lot of work with TensorFlow and PyTorch, so perhaps you can talk a little bit about like what you're doing with. I presume it's a lot of deep learning in this case.

Most of our applied work, especially in computer vision is of course deep learning. We really run the whole gamut. So we have a variety of classification problems. Everything I'm going to talk about today is in the public literature. So you can find all this stuff. If you go to Google Scholar and look up, look us up, you'll find all this stuff. I'm not talking about anything.

No defense secrets today.

Yeah, no defense secrets today. That's right. Everything you can find here on Google Scholar. So I'll give you a few examples of the kinds of things that we're building, right? So one is treating the problem of classifying an object, identifying an object, right? Saying that that thing that I'm pointing at right now is this specific satellite. To be really concrete about what I'm talking about right now from a spectral sample, so you point a big telescope at the thing, you use a, use a spectrograph to split up the light, you get this sort of two dimensional blurry image that doesn't really look like anything. And then you apply a convolutional neural network with a classification head to classify that as the object's identity, right? So the classes correspond to like identity. There's not that many. So it's pretty easy to formulate the problem that way. So like traditional classification problems, there's an example of it, but it's got a twist, right? Because you have to actually care about the physics of that imaging process to make that model work. We got classification. One of our big, like our most widely proliferated families of models is called satnet. There's overlap with that name. Now there's other things called satnet, but we have one called satnav that does deep space object detection. So this is, this is basically in rate track Imagery satellites out in GEO or beyond, if you point a telescope at them, they have a sort of distinct signature on the focal point of the telescope. Right. And so our objective is to detect those. Well, so what we do there is originally we did this when we first started it, it was like faster r CNN. And then we moved on to YOLO v3 for a little while and now we've our most recent state of the arts deformable deter. And so we have a variety of models that we apply to these problems. That one is really interesting too because, because we have a. We have to sort of host and serve inferences for this thing at scale and also near edge devices where connectivity might be really poor. We also have to do a lot of ML OP stuff. So we have that in addition to the model development work, we also have to retrain. We have to do domain adaptation and retrain and, and of course everything. In defense, we have the additional twist of sometimes we work with classified data. I of course can't talk about any of that today, but what I can say is it presents a deployment challenge that is really unique to our domain. So that's, you know, the kind of techniques that we use are traditional computer vision. We actually have a really fun paper coming out soon and we have some. There's already some work published on this using reinforcement learning to basically like think about. This is like controlling the gain of an emerging kind of neuromorphic sensing modality so that we have some forays into deep reinforcement learning. Yeah, that pretty much covers our deep learning portfolio. And then on the autonomy side, which we haven't really talked a lot about so far, on the autonomy side, we have a whole portfolio dedicated to multi agent global distributed autonomy for in that case, telescope control. So that's a brief overview of the domain.

Okay. Wow. So there's a lot to cover there and I'd love to get into some of these in more detail. I hope I'm not butchering this to say too much, but it sounds like a lot of what you do is like telescope takes photo of the sky and then it's. Is, is that little speck there a satellite or a star or just like speck of dirt on land? And this is where like the classification comes in. Is that sort of the right angles?

No, that's, that's more or less right. I mean we have, we have these, we call them ground based optical systems or telescopes. Right. Variety of sizes, a variety of costs. They go down to 50k, all the way up to several tens of millions of dollars and they point at the night sky. They do have different instruments, so it doesn't always look like dots and streaks, but they do have different instruments and you control them different ways. But yeah, that description is more or less correct. We point these big ground based optical systems at stuff in the night sky and then we, we get the data back, which comes in as scientific imagery. So it, it looks, it's very analogous actually to medical imagery. If you've looked at like the computer vision literature for medical imagery, they look very similar in terms of how the data is actually presented. And then we, we then train models to do information extraction tasks from that data. Yeah, that's pretty much it.

So there's a lot of kind of exciting things you're working on. But before we get into the detail of like specific challenges, I'd like step back and just think about what the goals are of your team. So you mentioned that the field is called Space Domain Awareness. And I'd like to know a bit more about what does that mean and what is it you're trying to achieve.

So our objective in Space Domain Awareness is to discover what is present in space and to track those objects across time. It's a really reductive definition because there's all kinds of additional dimensions to this problem. You might want to characterize objects and think about what they're, if they're changing across time in some way that you can't directly observe. But the general objective of the field is to keep track of what is happening in space. In order to enable, for example, you know, space faring. Nations have to send people into space. You have to keep those people safe. Commercial entities want to do business in space. You have to ensure that those satellites can operate in that domain or at least provide the information to the world about what is happening. And so that all falls under the very broad application domain of Space Domain Awareness. So that's our overarching goal, is to provide the US government with knowledge about what is occurring in space.

Maybe I've watched too many like Bruce Willis movies, but imagine that it's, oh yeah, there's this new spec there and we got an asteroid hurtling towards the earth. But it sounds like it's more about checking the integrity of existing satellites just to make sure that they're still functioning. Is that sort of correct?

Yeah, it's like that. So that Bruce Willis reference was apt. I've actually met some people from NASA Planetary Defense. There is a group of people, they have the coolest job title as far as I have Determined in the United States government. They do planetary defense stuff that they do that, you know, tracking asteroids and stuff like that. We actually use asteroids occasionally as like reference targets when we're imaging things to try to understand the science of our data exploitation techniques. That happens occasionally. But yeah, we're mostly focused on what are called man made resident space objects, sometimes called rsos is the acronym for them. And these are, that's, you know, satellites, space debris used rocket bodies that aren't really doing anything up there. So when I was talking about characterization before, that's a little bit of jargon for what you described like health and status. How's the satellite doing? Right, Is it still up and running? Is it, if we're no longer hearing from it, is it still stable? Is it dangerous, has it exploded? Things like that. So that's the primary focus of space domain awareness. But we also have to care about debris. Lethal debris can, can destroy entire satellites. You have debris clouds that could make entire regions of the sky inoperable. And one dimension we haven't talked about yet is really the satellite owner operators have to have this sda, the Space Domain Awareness knowledge in order to decide what to do with their satellites. The most critical thing we do is do what's called conjunction analysis, which is where we make sure two satellites, oh, it looks like these two might come into contact with one another. You two should probably do something to make sure you don't because nobody wants to lose a satellite, right. And then debris warning. So you know, those are really important functions that are the data that we exploit. Inform.

I mean you sort of think of space as being kind of big and so the chance of collisions are fairly low. But I guess there's a lot of things in space now and so maybe the chance of collision is bigger than you think.

Yeah, it's a little counterintuitive because yeah, you would assume that there's plenty of space in space. There's no probability that it could collide. Well, it turns out because we only know the orbital parameters with a certain level of accuracy, which is actually a function of how well we exploit the very data I was talking about before. Because those detections inform our ability to do what's called correlation of the orbits to get the orbital parameters and predict where they're going to go in the future. There's a lot of uncertainty in that, especially for small objects that have like solar radiation pressure effects. And so while it's true that there's not a lot of stuff and there's plenty of space, the Problem is because you don't know where it's going, there's a sort of cone of uncertainty around where an object might be in the future. You have to take that into consideration and do risk reduction, potentially maneuver your satellite in the off chance that it will hit you. And so if you think about these objects existing in probability space, they spread out across time. Right. So they're really small and there's a lot of space. Except that because we don't know where they're going, they are in effect larger, which is a really interesting way to think about the problem. And if you have better information, that's why we do computer vision for these problems. Right. If you have better information, those objects become, in effect smaller. Which is interesting, I think.

Yeah, absolutely. So maybe let's get back to the data you're working with. She said you work with a lot of image data. So how big sort of data sets are we talking about here?

Sure, we have a lot of problem domains. So it's imagine these as like we have. Each problem domain has a different. Think about it as a benchmark data set that we build and it varies from problem to problem and maturity from level of maturity. Right. So we have R and D projects that have 500 frames. So we're just getting started and we got maybe a few gigabytes of data. Right.

So it's kind of small data problems rather than big data problems in that case, I guess.

In that case, yeah. And that's like R and D. Right. So we're just seeing, can we build a model that does the information transformation we care about? Right. Can we just map from the data shape to the data shape and not worry so much about performance. Just, let's just see if we can build something that works end to end that's on the small end and then on the large end we have one data set. It's the one for that detection problem I was talking about before. That now has well over a million annotated images in it. And these are fairly large images. They're 16 bit images and they're usually 512 by 512 to 1024 by 1024. They're very large format images. And so that data, I have to be honest with you, I don't even know how large it is. It's probably several hundred gigabytes at this point across, and that's across multiple different cameras. Matters a lot what camera the data comes from in our domain. It's not like they're not homogeneous like an iPhone camera or Natural imagery tends to be. So it matters a lot what the camera is. So we have those from a variety of different telescopes all over the world at different altitudes and with different quality of cameras.

It sounds like you've got lots of different teams working on different problems then. And so this sort of strikes me as a slightly academic environment. Is that sort of correct? Is there an academic feel at work?

Sometimes. So one of the things that we really pride ourselves in with how we have constructed this team is that we do really like full spectrum development. We do full spectrum development. So we go all the way from low technology readiness level things. And that is very much an academic environment. So this is like we are not quite basic research, so we're not doing like proof of principle stuff, but really early applied research. These things tend to terminate in a peer reviewed publication that tends to be like the target success criteria for those things. Things all the way through the fielding of operational solutions. So think about like deploying model to LiveOps to inform space domain awareness decision makers about problems in space. So we do that whole thing end to end. And the level of academicness, if you will, varies across the portfolio depending on what you're working on. And so one researcher might have, we've had this happen, might take a research project all the way from effectively ideation, so no basic research because we people already invented Comnets and people already invented spectral imagery, but all the way from ideation through a fielded concept operating on an operational telescope for the Space Force. So it depends on which part of the life cycle maturity you're on and what you're working on that day.

So in that case, I know a lot of people listen like trying to work out can they get a job in data. So first of all, how did you get into your team? And then, yeah, I'm curious as to more generally, like how do people go about becoming space data researchers?

Sure. So there's two answers in two different directions for in my case, I really had some incredible opportunities. I was a active duty officer in the Air Force for many years and I moved to space domain awareness as my third assignment. So I came out to Maui. Not a bad gig as far as assignments go. There's definitely worse places to be assigned. So I came out to Maui and I was the program manager for an autonomy program. Had some really incredible opportunities with the Air Force research lab there. That's how I got to this location. My background is in computer science. I was not a space person initially. So I have sort of on the job trained into that domain if you are interested going to the more general answer of you know, what paths can in practice can I follow to work in this kind of space? There's a variety so it depends on where you're physically located. Though less so than it used to. We live in a remote work world now but. And what stage of your career you're at. A great thing to do is to reach out to, to look at job recs for defense contractors working in this space. That's a really accessible way to approach working in this area. Every contractor on my team is hiring. So that's a great way to get into the field is go and look for job recs in defense. They're all over the Internet. Look at the major defense primes. I won't endorse any of them in particular, but go and look at the major defense primes and look for their space related job rec. That's a good way to get into the field. If you are a PhD student today, if you're. That's where you are in your journey, you could look at potentially doing a fellowship with potentially the Air Force Research Laboratory. So there's a great scholars program that we every year we host several scholars out at our group. These are usually PhD students who are interested in potentially going into this field. We have hired several people out of that program. Some have gone on to be civil servants in the Air Force Research Laboratory. And so there's just, there's a lot of paths. It really depends on where you are and what your skill set is. If you for example today are a budding web developer and you are interested in how can I apply my skills to do problems in this domain. You could look at one of the many software factories around the Department of Defense. Just Google defense software factories. A dozen will come up. If you want to work in space, there's one in Colorado Springs. So there's just a variety of ways to get in. And we're hiring, the business looks good for the foreseeable future. There are a lot of problems we have to solve in space. And you know, I got to tell you, as far as reward for the work goes, a lot of places are hiring right now. You go get a job in Silicon Valley, there are places that will hire people. But we offer, I think what working in the Department of Defense offers is really compelling problems and huge wide open frontiers. These are not problems that have been optimized over. These application areas have not been optimized over decades to maximize click through or something. Right. Like these are new problem domains and you can come and homestead here if you want to. It's a really, really inviting environment. So my contact information I think will be associated with this. I encourage people reach out to me.

Wonderful. Yeah. So it sounds like business is booming, but you said you came through a sort of computer science background into the military and then the sort of space stuff came later. But it's also possible to go in other directions. So maybe you come from space science first and then get the military application later. For people who are sort of interested in a data science background, it sounds like Python and then some deep learning skills are sort of the way forward. Is that sort of true of everyone in your team or are there other kind of data science skills that people use?

Yeah, the primary skills that you need are infrastructure as well as Python and at least one deep learning framework. So we're, we're moving towards framework agnosticity. That's not as easy as it sounds in plain English. It takes a lot of work to support multiple deployment frameworks, but for the most part we're not going to be picky about what kind of framework you bring to the table. I personally am an old school TensorFlow graph mode kind of guy, so if you're TensorFlow graph mode, that's fine. If you are Pytorch, that's fine. Obviously Python is the lingua franca. You've got to speak Python, so that's a requirement and then it's really expected at this point in technical maturity, if you're going to be working on one of these teams, unless you are literally in a pure research role, and that's for like people who are graduate students coming into the program. Unless you're in a pure research role, you have to speak containerization, you have to know how to construct a container image and produce them at runtime. You got to know at least the basics of Docker and Kubernetes, helm that technology stack. You also need to be generally aware of data intensive application systems. So you have to know about databases and stuff. No need to be an expert in it, but you got to know how to write a SQL query and yeah, stuff like that.

So I have to say I love the fact that you described yourself as being old school tensorflow since it's a fairly recent technology still. But you mentioned how your team is transitioning from TensorFlow to PyTorch. Can you tell me a bit about why you decide to do that?

Well, to be honest, it was a grassroots movement, so it wasn't really something like I decided and told the Team. Yeah, we're going to start moving to Pytorch. It was occasionally it really started. Remember before I talked about the spectrum of technical maturity, it really started on the low maturity end of that spectrum. People are doing research projects and wanted to prototype quickly because it's just, it's, it's much more approachable. The data management is a little bit easier if you don't have to have too rigorous or high performance data pipelines. So like dataset generator pipelines and stuff like that. That one of our researchers, it was that same PhD in gravitational physics I was talking about before, was new to the domain, very competent developer though, and just said, listen, it's going to take me a lot of work to do this in TensorFlow. Let me just do it in Pytorch. Is that all right and sure, yeah, go for it. And it worked out fine. Nothing bad happened. Right. So it's sort of a grassroots movement. There's another couple of developers who have moved over as well and we don't really see any need to enforce a particular framework. It doesn't seem to be that important to enforce a particular framework. The only thing that we have to enforce is the interfaces to inference time models. So whether you build in Pytorch or you build in TensorFlow, you have to present a standard API at runtime and it preferably needs to be externally legible. We like those represented with open APIs so that we have the ability to interface with those models at runtime. We find that we are very rarely inference time performance constrained, so we don't really even care that much about inference time performance. That's just almost never the problem. Our images take a long time to take. They can take 30 seconds to take an image sometimes. And so that's just not the problem. The problem is, is the software engineering dimension of fielding and deploying and sustaining and retraining a model.

So this is when other people try and use your model for predictions. That's when the standardization matters, not when you're actually trying to figure out what should the model contain when you're training it, that sort of thing.

Precisely. Yeah. It doesn't matter that much at training time. It matters a lot when you're running inference models.

All right, so we talked a little bit about the technical skills. Now, I imagine since you've got some pretty serious deep learning going on, you've got the physics, you've got all the other kind of space knowledge and things like that, that it must be fairly challenging to try and communicate what you're doing and your results to other teams, particularly to non technical people. So how do you see like communication outside your team?

Yeah, so I used to spend a lot of my days building stuff. Now I spend most of my days talking about stuff. And so I'm at the sort of pointy end of the spear for this communication problem and perhaps an unpopular opinion here, but certainly in the. Because it's unpopular opinion when I'm talking to non technical audiences, even general officers, ses, these are like senior civilian executives, et cetera. Yeah, I give the same talk that I do to technical audiences, exactly the same talk. One of my principles of communication is I never talk about artificial intelligence in the abstract because people will always fill in the gaps with what they think that means. And that is an enormous diversity of things. So I give exactly the same deck of charts whether I'm talking to a four star general or I'm talking to a new developer on our team to introduce the work. So for me communication about these problems is really about specificity and relevance. So you have to be specific enough that people know what you're talking about. You have to put it in terms that are physically comprehensible. I am detecting an object. When I detect that object, what I mean is that it is at this location relative to the celestial background. Right. Like you have to be able to communicate those things and you also have to be able to represent that in a way that is comprehensible to them. I do not believe in talking about these things in the abstract because it becomes very challenging to do that. So I give exactly the same deck with the same papers side at the bottom, same nerdy papers sided at the bottom to every audience. And that has worked fine for years. So so far nobody's made me stop. So I guess that's, you know, whether or not that's a good idea material. It's what I'm doing. So yeah, it seems to work out okay.

That's kind of interesting. I really like the idea of being specific about what you're talking about. Can you give maybe some examples of how that talking in the abstract versus talking specifically is going to work?

Sure. So if I say, hey, I've got an artificial intelligence model, which at that point I've already been a little bit. But, but if I were to say abstractly, I've got an artificial intelligence model and it's going to do satellite positive ID for you. So what I've done there is I've used like three different abstractions in that, in that very general sentence. About three different things all at the same time. And any given listener might fill in those gaps with three different levels of interpretation, right? And so that's not really a helpful way to formulate the problem, Even though, in general, make sure it's relatable to the people, it's not really a helpful way to formulate the problem. But if instead I say, bear with me, I'm going to take a minute to talk about this. I only need a minute of your time. If instead I say, we collect images from a large telescope, we take the light, it goes into the telescope, we break it up into its constituent wavelength components that produces this blurry image that kind of tells you what the relative contributions of different wavelengths are in the image from this object that we took. And we know that different materials absorb and reflect different wavelengths differently. So we think, we hypothesize that it should be possible to tell what the object is based on its material composition, which you can roughly infer from the image. But we don't actually know how to write down a set of rules to do that. So we learn that process via a large parameterized model. If I say that I've so far found no one who can't understand that, right? Like, they might know, what does a large parameterized model mean? Well, they might not know that, but they don't really care that much, right? Because I've given them something concrete enough that they can fit all of that in their brain. At the same time, I think that sometimes, especially for senior leaders, we expect a little bit too little of them. We think that, oh, you got to give them, like, you know, crayon level diagrams. That's not true for the most part. As long as you're brief but specific, they will understand what you mean, and they will walk away from that with a very specific understanding of what it is that we can do and what it is that we can't do. And where I've seen communication in the abstract about artificial intelligence failure multiple times in different organizations that don't have anything to do with one another is when senior leaders make the assumption that a technology is much more mature than it is, and then they go and make investment decisions based on that misunderstanding and maturity. So that's why I'm almost fanatical about talking about things in concrete terms, making.

Sure people understand what the limitations are before. Well, certainly before they start spending any money. That seems very helpful. So while we're talking about communication, one thing you've mentioned a few times is that your team publishes a Lot of scientific specific papers in peer reviewed journals. And I guess my perception of the military is that in general it's quite secretive. So it surprised me a little bit that you're actually publishing a lot of your results. Can you tell me a bit about how that came about?

Certainly. So that was a deliberate decision. And by the way, the military does actually publish a lot. If you look at for example the Air Force Office of Scientific Research, right. AFOSR it's called, they fund grants all around the country. Some of the most pioneering work in to this day in sequential decision making under uncertainty in area I study personally, you go to the bottom of those papers, you're almost always going to find an AFOSR grant number on those things. So it just sort of depends. The public doesn't in general know when they see that acronym at the bottom, they don't necessarily know that that means oh, that was the military that funded that grant. But in the Navy, of course, as a similar Department of Navy has a similar program as does the Army. There's actually a lot of defense publications out there. But you're right, it's somewhat unusual to see the level of publication that we do from an organization like where we are. Right. So in Space Systems Command at this particular location, this tends to be a lab oriented thing or an office scientific research oriented thing. As a general rule, it was a deliberate decision. So the first thing that we realized is we're working in emerging technology. So we are not doing deep applied technology that like constitutes trade secrets. We are taking things in the public literature and we are applying them to publicly available or data that we can easily make publicly available. That's not sensitive. Right. What we do is we take those two basis of justification for being able to share this information, we put them together and say we're using publicly available techniques. Comnets are not secrets. Right. And we are using data that is not intrinsically sensitive and putting them together. It should be okay to publish this stuff. The reason that we do that, and sometimes this is, you know, it is somewhat onerous to go through the publication process. You got to be careful not to over publish. It can become all you do. The reason that we make that choice is because we are really trying to incentivize a broad and self sustaining base of researchers in this applied technology area. So we want people doing their PhD students on this work. And the reason for that is in part selfish. Right. So what we really want our job is to act in the best interest of government. Right. And so, so it's in the government's best interest. If a bunch of PhD students dedicate some of the most productive years of their life to solving some of the hardest problems on behalf of the nation, that is in the government's interest. And then those students can go on and get jobs in this domain. And it's a virtuous cycle. So that's part of it is trying to frankly generate work for free so we don't have to enter into a contractual relationship to make that happen. That'll just happen naturally. However, there is also the dimension of it is very easy to fool yourself in this domain. It is really easy to fool yourself when you're doing applied data science, when you're training deep learning models, especially in domains where not a lot of people are working, it is very easy to fool yourself. And so that necessary step of peer review from in particular, because we don't just publish in artificial intelligence forums, right. For that peer review in particular from the astronomy community and then the defense computer vision community. Those peer review opportunities that come from publishing those areas are things we actually can't get inside the Department of Defense. There's really no one who could do that for us, which is sort of a paradox of being out towards the leading edge. I think it's going to happen in all kinds of domains all over the world for different industries and different disciplines, is if you're out at the cutting edge, there's not a lot of people who can peer review you. We were very worried about fooling ourselves in the early part of this process. And so that's what took us down this path. And then what we discovered after we did that, those were the two reasons we started the path. What we discovered is it makes it possible to do things like I'm doing right now. So this is kind of hard to do, going on a podcast and talking in public if you don't have a track record of publication. But because we do, it makes it possible for us to go and communicate about our problems. And I think especially given that we have a brand new branch of service here, the United States Space Force, which frankly, it's difficult for the public to comprehend exactly what the underlying technical problems the Space Force has to solve is because people are busy and they're not going to. Not everybody has time to go and study this stuff. Being able to talk about this in public has its own dimension of value to it. So that has sustained us now that we've got the cycle going. And once you get started, it's easier to keep going than it Is. So if you're out there and you're working in the Department of Defense in applied technology, my advice is publish.

All right. Brilliant. Yeah, that's a good sales pitch for publishing your work. It's brilliant. It sounds like because of this that your team's actually like very collaborative with other organizations, certainly around the US Maybe around the world. Are you able to talk about examples of where you've collaborated with other organizations?

So first of all, we fund PhD students sometimes and that has produced some really valuable collaborations with labs around the country. Country. So that tends to happen through contractual means. But these. At the end of the day, it ends up being one team. And so we funded some PhD students at. We have one at Stanford, did some really amazing work in basically small satellite stuff. I'm not going to go into the details right now. So that's one way that we've done partnerships in the past. We've had a variety of academic institutions, US and international academic institutions raise their hand through. We have these sort of. They're like public space war games is the way to describe it. They're like places where to everybody to get together and try out their stuff. Right. We participate in those very heavily and through those activities. One of the things we did one year was we just said, hey, could people just send us fits? That's the image format that for the telescopes take. Can you just send us your fits and see what happens? And we got like 700,000 images from that. Right. So. And there's five or six organizations involved there. So we collaborate with them, we'll annotate their data for them. We have a great annotation as a service company that's on sub to one of our prime contractors. They're called Enabled Intelligence. So they go out, they annotate our data for us. They can do multiple classification levels. And then we give that as a collaboration, right. We give that back to the people who gave us the fit. So now they have training, quality annotated data in exchange for letting us use it to train our models. That's a collaboration model that works really well. We are really just beginning to get into the allied partner. This is the international community, basically collaboration game. So in coming up in November, there's another one of these things. We've got a partner from Chile that has raised their hand and wants to participate with us. We were going to be providing autonomy for their telescope, I think, think. And then another partner in Australia who has an emerging sensing concept that we're very interested in. We'd like to Participate with. So that's how we tend to do collaboration. But we've also had, you know, I talked before about funded PhD students. We've also had students at universities just see the work and say, oh, this is relevant and interesting to my research. Would you like to collaborate with me? And we'll, the students reach out to us. Like we'll give them the data. We can give them most of the data. Some of it is sensitive, some of it is even classified. Obviously we can't give them that, but we give them all the data that we can. If they've got data, we'll annotate it for them and give it back. We'll train models and give them baseline performance. It's a really interesting collaborative community.

That seems pretty incredible that you are collaborating with a load of organizations around the world. I just want to pick up on something. You said that you provide an annotation service and I have images of cadets just having to go through labeling telescope images with this dot's this particular star and that's satellite. Can you just tell me a bit about what annotation involves?

Yeah, sure. So I mean there's a fun story here. The first SatNet data set, the one I was talking about before, wasn't annotated by a cadet, it was annotated by me. I hand annotated that data set many years ago. But no, the annotation is. We've matured a lot over the years. We have like tight instrumentation on dollar per frame, dollar per second, all that stuff on our. We have a very mature annotation pipeline now what it entails for. I'll just give you one problem. So every problem is different. I'll just talk about one. The most mature one is a satnet problem. We have a company, one of our prime contractors has a sub relationship with, with them. It's a contractual relationship with them. What they do is we have built a tool for them called silt, the Space Image labeling tool, which has a nice like pun on fertility as well. You know, all the data grows out of it. But SILT has been, was built by the prime contractor and has been fielded to them as their annotation platform. Right. And so what they do, it brings these images up for them and there's all kinds of domain specific stuff you got to do. Right. When you look at these 16 bit images, if you just try to display them on a computer screen, they don't look like anything. So. So all those knobs are tunable to the annotators. These annotators do this professionally, full time. All those knobs are tunable and they can play back and forth through the images. Extremely important. They're not true videos because they might be taken minutes apart, but they're not moved. The objects don't move a lot. So you can basically track them if you watch across time. And you can even impute the location of objects with the eye. You can impute the location of objects if they're not visible in the frame. Sometimes they get too dim because the sun's not shining on them just right, they disappear. But you know they're there. So you can annotate them that underlying source of truth and what it looks like is they're going through and basically connecting these spatiotemporal lines to produce the location of the objects across time. And then actually we just fielded a brand new feature. We have an excellent new developer who's just joined us recently. And so she has built an extension to SILT that makes it possible for us to also annotate the stars in the image. And to. This is a bit down in the weeds, but basically, if you haven't, if you know where enough stars are in an image, basically think about, you're looking at the nice sky, a little piece of it, there's some satellites in there and a bunch of stars in the background. If you know where enough stars are, you can do what's called an astrometric fit. This is where you basically, you can determine where that telescope was pointing based on what it sees in the background of what it's imaging. This is essential for that orbit characterization stuff I was talking about before. You have to do it or the observation is useless. We can now even go through and dynamically annotate those stars. And it will, it'll run basically an astrometric fit engine in the background. And if it doesn't get one, it'll tell the annotator, nope, that didn't work. We need more from you. They'll keep annotating stars until it satisfies that requirement, which is really powerful because right now that you have to both do the object detection and the star detection and fit successfully with high precision and recall in order to have a valid observation. It's not enough just to say that object is there, you have to say that object is there and there is here. Here is the location of the night sky that this telescope was looking at.

Ah, okay. That kind of makes sense. Just knowing that there's a satellite somewhere in the sky is less helpful than knowing where it is relative to the rest of space I guess. So it's really interesting hearing about the problems you've been working on. So can you tell us what's the project that you've been most proud of working on?

So I mean it's hard to pick one. It's sort of like choosing amongst your kids your favorite and whoever I choose, someone else will be hurt. There are a lot of projects that have been really fun. I think that it's a toss up between satnet, the problem I've talked about many times, it's detection problem because that one's really close to me because I started it personally. It was my nights and weekends project right before I, before I just did management stuff all the time. That one is really, really close. And also we had a great, in the early days of that program we had a great collaborative environment. Some of my colleagues who were out here at the time, we really, you know, it was very much a startup kind of atmosphere at the time. Nights and weekends on a shoestring building out this framework for the DoD. Anyway, that project is a, is a favorite of mine but it also has grown into a very mature, operational capable product. But really I have to say the one that I've invested the most in really haven't talked about almost at all today. And that I think has the most potential for disruption in the department and I think is sort of for that reason. The project of which I'm most proud is this project called Machina. That's an acronym. I'm not going to go through the torturous acronym right now. But basically this is a global autonomy program. It is designed to make it possible to get the right data from telescopes. So far we've only been talking about what do you do once you get the data? Well, in a, in a sensor poor target rich environment like space where there's too many things to look at and not enough sensors, it's not enough just to exploit the data correctly. You also have to collect the right data. And that itself is a really challenging sequential decision making under uncertainty problem. What does it mean to do a good job? Right. We basically have a program for that called Machina and this is a really, I think it's a novel programming paradigm. It's a very new way to approach the problem of multi agent autonomy. It's built entirely on kubernetes and we have a really interesting deployment construct for how the that system will be fielded and sustained across time, updated and managed. And it has, as part of it, it's a closed loop agent. So it has These computer vision approaches inside of it, effectively, it's a thing that we transition these technologies into. That project is enormous in scope, and if it succeeds, we're in the early stages now. In fact, this upcoming exercise I was talking about in November is going to be a major test for that program. It's going to be really its first major milestone. If we succeed at it, it could totally change the way we do sensor management in the Space Force. I think those are the two projects I'm most proud of. I wasn't able to pick just one, I guess. Sorry.

Your two favorite children. It's not bad. All right, so that last project seemed pretty interesting because it sounds like you've got the computer vision aspect, but you've also got this automation stuff. And just it sounds like it's getting to the point where it's artificial intelligence, where it's trying to just take the humans out of the loop completely. So just all the telescopes are run automatically. Am I understanding that correct?

Yeah, so they do run automatically already. All these telescopes are already robotic telescopes. So they're so the. They're automated in the sense that you tell them to point at a location in the sky and they'll basically do that for you. What that program provides in addition to that is what we might call orchestration. Think about this as we have to use these sensors in a collaborative way. If these sensors are very different. Imagine one has a really wide field of view, so we can see a lot of stuff, but not very dim, so stuff. Another one has a really narrow field of view, so it can't see very much area, but it can see really, really dim stuff in the little area you can see. You can put those two sensors together, and that's the hard part, right? But you can make those two sensors interoperate, collaborate with one another in such a way that the autonomous closed loop system they constitute together, working together is more than the sum of its parts. And there's some really interesting. You know, this really. I can't claim credit for that idea. That idea goes back to. There was a DARPA program about this in the 60s. The site here, Amos, the site that we're at out in Maui, has been researching this for years. That was actually that program that I was on when I first came over as active duty. I can't say a lot about it in this forum, but basically was doing similar work. That paradigm of extracting more than the sum of their parts value from a collection of heterogeneous telescopes working together has the potential to be game Changing for not just space domain awareness, but sensing in general. And of course these are not original ideas. Right. These ideas have been studied since the 80s and 90s and actually interesting connection back to AFOSR has funded a lot of that stuff. So again the military funding research and development for those kinds of problems.

That's really interesting. And so you mentioned some of these programs have gone back a really long way that sort of predates deep learning. So how would you say as deep learning affected what your team can do in terms of research?

Yeah, so this has been, this has been really fun over the past few years. So we're coming into a community, but we, we. When I say we, I'm really talking about basically me here. When this first started, it was really just me. And so I was coming into a community that was venerated, had been around for generations. People have been doing space domain awareness since decades before I was born doing space domain awareness, since they called it SSA at the time, but since my parents were children. Right. And so, so one sort of gets that what are these kids up to these days kind of attitude when you say, yeah, I'm going to do computer vision for this detection problem you've been doing doing for 50 years. Right. Why we've been working on it for 50 years. What do you have to add? Right. It's not necessarily that people are malicious or trying to prevent progress or anything. It's just that they know this domain. They've been working in this domain an entire career and they know it. Right. And so their expectations are very high. Here's what we discovered when we brought the deep learning revolution to space domain awareness by accident because that's what I did in my graduate research and that's what I wanted to do in my day job. So I did it as a nights and weekends project. What we found was when we actually took the time to measure the legacy systems in terms of the way we tend to measure in particular detection problems in terms of precision and recall. Right. At a particular iou, really we actually use centroid distance because our objects are point sources. But that's, that's down in the weeds when we think about formulating those detection problems in those information retrieval terms, as opposed to detection theory, which is much more common in the astronomy community. What we found was when we put it in those terms, which are really the terms we need for autonomous systems. Now explain why in a moment we find is that the legacy systems were not as performant as we thought they were. This wasn't because anybody did anything wrong. It's just because as anybody who worked in computer vision before the deep learning revolution will tell you, computer vision is really hard. So hand designed computer vision is harder yet. And so there were challenges and in particular there were edge cases that caused some truly egregious failure modes. And we uncovered those only because we systematically measured our performance so that we could do deep learning. You can't do deep learning. You can't be serious about doing deep learning if you don't have really carefully controlled and well understood metrics for your problems. Because otherwise, going back to earlier, it's really easy to fool yourself. And so if you don't do that, you can get in all kinds of trouble. And the trouble that the legacy community was getting into is that they were assuming that ground truth couldn't really be known. And so they were doing their best to calculate their detection performance based on case true. So when you actually knew an object was there, and that caused all kinds of downstream failure modes with precision. So we are getting really low precision on certain kinds of images, on really important kinds of images that you really don't want to have a lot of false positives on. And so that begat this whole program.

So does that mean like a false positive means you think that there's a satellite there, but there isn't?

Precisely. Yeah. So a false positive in this case would be hallucinating the existence of a satellite. And so that can happen if the model is trained to be really high recall, if the legacy system has been hand tuned to be really high recall, which means if I say an object is there, it's definitely there. Right. And so if you only care about case true, you don't care about case false. And this, this transcends domains. Everybody can have this problem, right? It's not just space domain awareness. If you only measure one dimension of the problem, you can get truly aberrant behavior in other dimensions. Dimensions. And that did happen. We have demonstrated that numerically, experimentally, I should say on sky in the real world many times over the past few years. But it was slow going initially because we were coming into a very well established community.

It sounds like there really has been a deep learning revolution then, or at least there's a deep learning revolution ongoing then throughout the field.

I would describe it as ongoing. So just to be clear, some of these techniques, these non learned techniques work exceptionally well. We to this day do not have, I talked about astrometric fits earlier, telling where you are in the night sky. We don't have a learned solution for that today the best solution is the one that was hand designed. So the revolution might not conquer every city. You know, like some technical disciplines might actually be done best in a hand designed way. That's very explainable and has really well instrumented performance. But there is an ongoing revolution this week, this very week that we're recording. This is AMOS conference out here in Maui. It is the conference conference in space Domain awareness. I don't think anybody would be offended to hear me say that. I think it is the biggest conference. It's gonna be like 1200 people here this year. The whole community will be here. And I've been coming to this conference since there were no deep learning talks. It was like one or two deep learning talks across the whole conference to now. I haven't quite counted this year, but last year there were dozens applying deep learning to these problems. And not all from our group. Some, many of them were. Almost all of them were independent of us. So it really does feel like we're getting that revolutionary wave that the rest of the world had in 13, 14, 15, now in 19, 20, 21, 22 in our application area.

Okay, so it sounds it's gradually becoming a lot more popular, but not quite there throughout all the teams yet, perhaps. All right, so you've talked a little bit about your wins. Can you tell me a bit about what's the biggest sort of challenge that you're facing at the moment?

There are a variety of dimensions of our application area that make it difficult. Some of them really aren't exciting. Our biggest challenges are often frankly, institutional and bureaucratic. No, you can't deploy that software to that host is a very common problem. So getting to yes on questions like that consumes easily 60% of our program bandwidth, which I think that's probably a fairly numerically accurate estimate. So big problems in the Department of Defense relate to digital infrastructure and the ability to field these things. Our solution to that is to build our own and certify and actually go through the process. Process that takes time and is laborious. But we're doing it now motivated by the larger scale of deployments that is being demanded by the modest successes that we have had so far in computer vision. And so, for example, the existing cybersecurity process, even with a continuous authority to operate what's called an ato, even under those circumstances, if I'm retraining a model every hour or so because I'm trying to keep up data as it's flowing in at a very fast rate, there is no viable path Path to do that deployment action today, that is a huge challenge. Context is a huge challenge. How, where do I put this model to run inference? How do I deploy this to this classified system? Those are always our primary roadblocks. We also have, you know, rearing its head again is sensor sparsity. Right? So we have a high demand, low density sensor network. We don't have a lot of these things. They're very expensive, especially the large aperture telescopes with these exquisite instruments on them. And they're expensive to maintain and to operate. We have access to all of them, which is an incredible benefit for our research. But the problem is that they can sometimes be very difficult to work with. You don't get, they get pulled away for higher priority missions. Necessary. But it does happen and it does disrupt research and development. And so we have a problem with data sparsity, especially in our more advanced domains. That setnet problem, we've got such a large number of collaborators now. We really don't have a data sparsity problem there. But in the rest of the domains we very often find ourselves bootstrapping with simulated data. And you can never fully eliminate the sim to real gap, even with domain adaptation. It's never the same as having the same number of images from the real domain. And so that proves a persistent challenge for us. So there's infrastructure and then there's access to data collecting devices which have a very different flavor than the kinds of problems that I think most people see when they think about doing computer vision. In industry, for example, most people are processing natural imagery to do information extraction tasks. They're not thinking about what if you only got 27 of those images a week? Right. So how would you approach that problem? Of course people are working on data efficiency, but at that scale, in this case small scale, that can be basically a performance limiting challenge. So those are the two I think of off the top of my head. Okay.

All right, thank you. So before we wrap up, I have a slightly silly question to ask. So just in the past, like I've worked with like biologists working in the lab and I've worked with engineers who get to blow stuff up. And it's always been me there as a data scientist, just like sat on my laptop analyzing data. So I have this kind of hardware envy. Now I know you're based in Maui in Hawaii, and so you have these like giant telescopes nearby. I have to ask like, do you get to play with them yourself? Do you get to play on telescopes?

This is one of the coolest, like I joked before about like, oh, you can maximize click through. That's something you could do with your life. Or you could operate one of the world's largest telescopes at the summit of Haleakala in Maui. Right. Like so I think we have a compelling case for people who have hardware envy. Yes. So unfortunately these days I've moved into a role now that is a lot of technical management. So I'm doing a lot of communicating a lot of, a lot of architecture level work. So if you're asking do I personally not recently have I in the past. Yeah. And it was a lot of fun on our direct team. So you don't have to go more than one step removed from me. There are people who work primarily at the summit. We have world class astronomers, my colleagues Ryan Swindle and Zach is Zach, both PhDs. They work primarily, they work a lot at the summit and they are, I mean Zach restored this sensor on the trunnion port of this three and a half meter telescope to do spectral imagery. We published about that. Now Ryan is really one of the most masterful operators of that world class optical instrument alive today. So they are operating these enormous like room sized systems, some of the most advanced technology the species has to offer to generate data which they then come back to sea level or, or you know, if they live in Coola halfway up the mountain, they go back to their home and they train models on that. Right. So it's, they're truly cross disciplinary. It is a really motivating part of our work. It is a non trivial component of the reason that we all work here. Not, not the hardware. It's not really the hardware itself though. That's cool. That wears off, right? What doesn't wear off is the fact that we are working out towards the edge of the species technological capabilities and occasionally, just occasionally bumping it forward a little bit. So anyway, yeah, that's hardware all right.

So yeah, playing with giant telescopes on a mountain and then deep learning on a tropical paradise then yeah, I've got to say I'm jealous. That does sound pretty amazing. So thank you Justin for your time. It's been really fascinating just hearing what you do with your team and your work. Just some really important, exciting stuff. So thank you once again, Richie.

It was my sincere pleasure. Thanks so much for your time today. You've been listening to Data Framed, a podcast by Data Camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment and share episodes you love. That helps us keep delivering insights into all things data. Thanks for listening. Until next time.