Will AI Enable us to Understand and Communicate with Animals?

Huge thanks, first of all, to World Government Summit for inviting me here for the opportunity to tell you a little bit more about our work to decode animal communication using artificial intelligence, or put more simply, to figure out what animals are saying. And I'm really excited, actually, in many ways, that we left this to the last day, because I think this is going to be one of the most fun sessions that you attend during this summit. I wanted to start by playing you a sound from nature, and I would like you to have a little think about what animal makes this sound. Okay, Any thoughts from the audience? Whale. Sorry. Sperm whale. Orca. You're close. Penguin. Sea lion. Very close. This is a bearded seal. It's a mating call of a bearded seal from the Arctic. He's a pretty cute little guy. And I think the reason I wanted to play this for you at the outset is because I think it's a great illustration of the foreignness of nature, of the other species that we inhabit this planet with. And I think what this really leads to is a sense of the fact that we as human beings are so limited in our abilities to actually perceive what is happening in the world around us. Whales communicate in frequencies too low for us to hear. Bats communicate in frequencies too high for us to hear. And the ultrasonic, the infrasonic. And there is just so much going on around us, and this really limits our ability to understand and connect to the world that we inhabit. There's some really great examples of this. This is an evening primrose flower, amazing study by University of Tel Aviv done in 2019, where they played a who suite of sounds to the flowers and then tested the sweetness of their nectar following those sounds. And only when the flowers were approached by a pollinator did they produce, almost instantly, more and sweeter nectar. So what does this tell us? It tells us that in some way, plants can hear. And there are other studies that show that plants can actually probably see. We don't know how. The latest research shows that when plants are distressed, they actually emit sounds, but at frequencies that we can't even begin to hear. I was attending a talk recently, and the speaker just blew me away with this analogy, which is that humans can hear at about 20 kilohertz. Your cat can hear at about 70 kilohertz. And when a plant is emitting a distress signal, it's at about 70 kilohertz. So if you think about it, you could be sitting at home drinking a cup of tea, and you could have forgotten to water your houseplants and your Cat might be able to hear them calling out for water. And you have no idea that this is going on around you. This next example, coral larvae, which we used to believe were just these little tiny organisms pushed out by the coral, and they just got floated around in the current and landed wherever they landed, some successfully, some not. The latest research shows that in actual fact, coral larvae very intentionally navigate to a place to settle that will be good. They can actually hear the sounds of a healthy reef versus an unhealthy coral reef. They're figuring out where they're going, and we actually have no idea how they do that. This is a great Amazonian river turtle. Again, up until about 10 years ago, we thought these creatures were entirely silent. New research by Camila Ferrara, biologist has shown that not only do they have more than 200 distinct vocalizations, but that the mother turtles actually talk to their babies in the eggs before they hatch. So, you know, what this goes to show is that the world is just awash in sound, which is completely beyond us. And it really allows us to connect to the fact that nature is rich and distinct. But so what? Why is this important? I think we all know that we are living through a time of existential biodiversity loss and the climate crisis. More than 69% of wildlife on the planet has been lost since 1970. That's according to the Living Planet Report of 2022. 60% of all mammals alive on the planet today are livestock, many of them existing in factory farms, and another 36% are human beings. So that only leaves about 4% of wild animals. And these animals, these other species, actually form the fabric of the ecosystems that provide the life support systems for us here on Earth. So there's a fundamental disconnect here. Somehow we have lost our connection to nature. Somehow we've forgotten that we're part of it. And that poses huge challenges for us as human beings. So how can technology actually bring us closer to nature? This seems really counterintuitive, but it's really kind of the founding question for Earth Species Project and very much inspired by the development of the large language models that you've all been hearing so much about over the course of this conference, and in particular work in 2017 that allowed us to translate between two human languages without the use of a dictionary. And so we think of AI in some ways, like the invention of modern optics. Just like as the telescope allowed us to look out at space, can AI actually open the aperture for us of our imagination and reconnect us to nature in some way? So who is Earth Species Project. We are a small nonprofit organization based out of the United States, but globally distributed. And we have an extraordinary team, larger than what is shown on this slide here, now primarily made up of AI research scientists. So we have a dedicated team of people who are working on this problem, coming from very, very diverse backgrounds, from neuroscience, from math, from physics, and they are working really, really closely with a whole suite of partners. There's no way that this work could be done unless it was in collaboration, deep collaboration with research institutes and people who've been out in the field studying animals for decades. And I also wanted to just point out, too, that I joined Earth Species Project about a year and a half ago, and at that time, even everyone I said, I'm going to do this work on decoding animal communication. They were like, oh, my God, that's so crazy. Nobody talks about that. But in the last year, we've seen several books published on this topic. We have seen huge momentum in the press. This is beginning to capture the popular imagination and really following the release of chat and people's recognition that AI has transformative properties. But as I said, we couldn't do this without dedicated research partners. And so the people that we work with are people who have studied other species for decades. People like Dr. Ari Friedlander, who is one of the world's leading marine mammal specialists working out of UC Santa Cruz, people like Dr. Joyce Poole, who literally has decades of data gathered from elephants in Kenya, and people like Dr. Valaria Vergara, who is an expert in beluga whales. With these partners, we have put together a roadmap. This was published recently in Science magazine, and you'll see that it basically lays out a number of stages on the road to decoding animal communication. Not surprisingly, it starts with data, as all artificial intelligence does, but thinking very specifically, not just about vocalizations, but also about behavioral data, multimodal data. How do we bring all this together? And also alongside context, the machine learning models, then, are the tools that allow us to analyze that data. And that's really where Earth Species Project is focused and helps us to get into the space of decoding by classifying vocalizations and movements, by starting to make associations across those things, and then by testing our hypotheses through a series of experiments. This video is of Ari Friedlander, who you saw on an earlier slide, placing a sensor, a remote sensor, onto a whale. And you can see that it's capturing. You're not hearing the vocalizations right now, vocal data, but also video and environmental context as well. So the exciting part is that there is actually reams of data being gathered today. The challenge is that it's actually probably a little bit too much to be analyzing easily. And I think this is something I wanted to bring home, that although we have tons of data and we have the power of AI, this is still a really, really hard problem to solve. So I just want to play you another sound and see if you can identify this. Anybody stumped? Bats? Bats? Anyone else? Dolphin. Again, close. That's actually a beluga whale. Sounds a little bit like an alien modem. And the crazy part is that that data was gathered by Valeria Vergara, who you saw earlier. And she has actually told us that despite the ability to put a hydrophone in the water and gather those vocalizations, she can't use more than 96% of the data that she gathers because it's too noisy. It's either too noisy or all the animals are speaking all at the same time. And it's really impossible to tell what's going on. So it just gives you a sense that there are these really, really basic challenges when you're trying to gather data from animals and make it usable for machine learning. So we are operating essentially in the space of trying to build AI for the rest of nature. We're trying to help researchers with some of these really basic problems of how do they denoise their data, how do they do source separation, how do you then, in 10,000 hours of recordings of orangutans as an example, how do you actually go through and detect where the animals are vocalizing and then classify those? So we're building tools that will allow basically automate tasks that human beings. Either it's going to take them too long or really, it's not the best use of their time to do. They don't even do it that well. But we're also going further then and using AI to kind of open up our understanding of how this problem might be solved in different ways. So, as an organization, this is a new field, and in a new field, you need to produce benchmarks in order to know whether or not you're making progress. So really, really exciting that in the last year, we have published the first ever benchmark of animal sounds. This is a data set of animal vocalizations. We've also published the first ever benchmark for animal movement. And we've gone further than that to publish the first ever foundation model for animal vocalizations. And so you could kind of think of this as like the GPT1 of animal communication. This is A model that is generalizable can be used for a whole suite of different species and allows biologists to both do detection. So figuring out where vocalizations are and then classification of the vocalizations, what type of vocalizations. And that model performs incredibly well against the benchmarks that have been developed. And then we're moving into, as we go forward over this year, multimodal foundation models as well, that compare vocalizations with movement data. So thinking about this in a slightly different way, we're basically building the foundations or the fundamentals in terms of the data and the benchmarks and foundation models that you can then build a whole suite of applications off that will deliver against a whole suite of tasks. And we're building the flywheel that essentially brings in reams of data that allow us to build the tools that power the work of our partners, that then generates more data so we can iterate on the model. So we are, as an organization, working to unlock understanding better, connect us to nature. And I just wanted to play you a super quick video to bring that to life. The oldest cultures are not human. They're from the ocean. 40 million years ago, before we walked upright, before we sparked fire, whales evolved to build relationships in the dark. I'm trying to start a conversation, is the most basic way you can say it. I'm going to put a speaker in the ocean and talk to a whale and hope it talks back. Starting playback. If this work is successful, it will be the first experiment where we have engaged in a dialogue with a humpback whale. Pretty incredible. So that's Dr. Michelle Fournet from her documentary Fathom. And essentially what she's doing is recording a humpback whale, saying hello and playing it back to the whale to see if they will respond. This is essentially a playback experiment, been used by scientists for decades to test their hypotheses about what an animal might be saying. But the question then is, can we go further with AI? Can we say something more complex to an animal and see what the response is? Can we, in fact, potentially generate novel vocalizations? And so that's the really exciting part of where we're at right now, getting into the space of generative AI. And so this is an example produced by one of our scientists, Jen Yu Liu. The first part of this vocalization is the animal, and the second part is generated by an AI indistinguishable. And this basically puts us in new territory. And so this year, for the first time ever, we are going to be conducting, conducting a series of experiments with zebra finches. In a captive laboratory setting. They're a model species, very well studied. We are actually going to have an AI in two way conversation with a zebra finch, which, you know, the potential that this opens up, the opportunities in terms of understanding the structure of an animal's communication is huge. But then, of course, there's all kinds of risks as well. Once we get into this territory, you know, we could have an AI communicating with another animal without us actually understanding what that communication is all about, which, you know, gives rise to the potential that we could be messing with other cultures. So this has got to be managed very, very carefully. I wanted to bring us back to what I raised at the beginning in terms of the biodiversity crisis, like, so what? So we can communicate with animals. How is this going to help? And I think it's really important to point out that there are all kinds of potential conservation benefits from this. I mean, you can imagine that if you were able to actually, in a very rudimentary way, communicate with a wolf or a bear or an elephant, you could actually avoid some of the human wildlife conflict that we're facing in many parts of the world today. Imagine if we were able to understand why dolphins or whales are stranding themselves on beaches in large numbers. Like, how would that help us to do something differently? But I think there's also something really, really powerful here too, that goes beyond those kinds of conservation benefits to help us think about a different way of interacting with the planet. In the same way that when we saw Earth for the first time from space, it changed our thinking about ourselves. I think we're at a place where this work has the potential to open totally new scientific frontiers. It has the potential to open new human perspectives. And so if you think about, again, the telescope and once we had the ability to look out into space, we actually recognized that Earth was not at the center. So we think about AI as a tool that can help us look at the patterns and the complexity of nature right here on Earth. And if we're able to do that, maybe, just maybe, we will be able to acknowledge that humanity is not at the center. Thank you so much, everyone. I think I made it in time. I just wanted to say we are raising friends and partners in this effort. So if you're interested in following our work, if you're interested in finding out more, this is a dedicated email address, so please, please reach out. Thank you.