AI for search at Etsy

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean. Guess what? DigitalOcean recently added MySQL and Redis to their list of managed databases. Their full managed databases lineup now includes the three most popular databases out there for developers, Postgres, MySQL and Red. Eliminate the complexity involved in managing, scaling and securing your database infrastructure and instead get back to focusing on building value for your users. Learn more and get started for free with a $50 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence pract, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM and now onto the show.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. How you doing this week, Chris?

Hey, I'm doing fine, Daniel.

What's up?

Not, not much. Busy day. It's submission day for ACL and I'm trying to get something ready. We'll see if I actually make it. By the time this goes live I will have either failed or not made the deadline.

And ACL is for the listeners.

Yeah, that's the computation. It's a large computational linguistics conference, but it's kind of one of the larger natural language processing community research conferences. And so there's like EMNLP and then there's ACL and there's larger, you know, like right now started maybe today when we're recording is the start of NeurIPS, which is another large AI research conference and hopefully I'll be live streaming some of that later and trying to keep up because I'm not there. But yeah, it's one of those sorts of conferences so we'll see if I. If I make it, we'll make it through.

I gotta say, I had one of the coolest weeks last week I've ever had. Started at Carnegie Mellon University. We had there's a big conference on future of AI and STEM in Society and got to Do a breakout on AI and ethics and such in STEM and what things that could be done. That was really a cool conversation. Solved all sorts of world problems right there, I bet. So, yeah, I got to sit on a panel called Protecting AI from Threats. And the guy beside me was General Cartwright, who used to be the Vice chairman of the Joint Chiefs of Staff. And he just had brilliant insights. He's not an AI person the way we are, but was just really impressed with what he had to say and then did an opening keynote in Philadelphia later on on ethics and AI. And finally we finished out the week. Listeners will probably recognize this. We had the championship for Alpha Pilot, which we had a previous episode on in Austin, Texas. And we handed out a $1 prize to Team MAV Lab. They're from Holland. And so it was a pretty cool week.

Yeah, that sounds extremely eventful. And I imagine that in the midst of all of that travel and logistics and all of those things, you were utilizing some form of search in some way to manage your life.

I might have been, yeah.

Today on the show we have Andrew Stanton with us who is a staff product manager of search ranking and platform at Etsy. And I'm excited to talk with Andrew about search, but also some other things. And also this is the first episode, I think, where I told my wife who was coming on the show, and she recognized, obviously Etsy and was pretty psyched that I was talking to someone from Etsy. So we're all excited to talk to you. Welcome, Andrew.

Thank you so much for having me.

Yeah, definitely. So maybe to start us out, if you could just give us a little bit of your background, how you got into AI, ML related things and search and eventually ended up at Etsy.

Yeah, great question. So I've been kind of blessed to be working with machine learning and search on and off for about 15 years at this point. And the irony is I actually never intended to go into either. When I was in school, I was much more interested in kind of distributed systems. And the funny thing was is that as our data grows, I kept running face first into places where we needed to have more sophisticated search. We needed to have better predictive performance than kind of standard heuristics. So when I was in undergrad, I was actually working full time for AOL at the same time. And big focus at that point was working for an online radio show of all things. But it ended up boiling down to this kind of predictive problem where we were trying to understand how basically our listeners would kind of tune in from around the world. And so that was my first kind of face plant into linear regression and kind of time series prediction. When I left, I ended up moving into something called entity recognition, which is this kind of process of trying to understand from unstructured data the different types of entities that might be represented in it. Could be people, could be companies, could be any type of entity that might be useful, and then building kind of this typo resistant search on top of it. And that was also probably my first real interaction with extremely big data. We were dealing with billions and billions of records. And so how you build kind of performance search on top of this entity recognition system, which is constantly ingesting hundreds of millions of records per day, turned out to be kind of this scratch and itch sweet spot for me. From there, went and worked on a bunch of different problems. Ended up in a startup called Blackbird Technologies where we were working on E commerce search and the B2B space. And our big kind of value add was being able to leverage multimodal deep learning to basically tease apart a lot of these products that companies had to provide a better search experience on top of it. We were acquired by etsy back in 2016, and I've been making my home ever since.

And that sort of multimodal side of search, when you mean that you're meaning like images and text sort of thing.

That's exactly right.

Okay, all right, so like products, if you're searching products on a website, Right. There's obviously product photography. Right. I guess that could factor in somehow.

Yeah. I would say a good example would be think about something like Craigslist or Facebook Marketplace. You have kind of an image and then you have maybe a sentence or two about what that item is. But somehow you have to understand that when a potential buyer comes in and they type in kind of this highly specific query such as colors and materials and other types of attributes, you have to take this very unstructured piece of information and convert it into something which is both relevant and searchable.

So kind of wondering, as we start to dive in and we're talking about search right off the bat, before we get fully into what Etsy is doing with search, can you kind of talk a little bit about what types of search problems are out there? We tend to use the word search in all sorts of different contexts. You know, and there's full text search, web page search, product search, you name it. Can you kind of give us an idea of kind of the overall landscape of what search problems look like and how they're related? If at all.

Yeah, that's, that's a really good question. There's really, I would say maybe three major areas of search. There's information search, which you know from like Google, you go and you type CNN 99% of the time you're intended to go to CNN.com maybe you find the Wikipedia page as number two. But largely you're searching to find pieces of information. The second type is probably E commerce search, which I'm most familiar with. Amazon, Walmart, Alibaba, et cetera, et cetera. How do you match these buyers who are oftentimes giving these very vague queries like jewelry and trying to understand what are the latent factors that are actually interesting to the buyer. The third, which has kind of grown up much more recently, is probably question and answering. So kind of the stack overflow problem where you are asking questions much more natural language, but you're trying to tease out this kind of community aspect of retrieval where the intent is not necessarily on finding a single piece of information, but perhaps finding a collection of pieces of information and the domain is a little bit more NLP heavy.

So in those different areas, I mean, obviously E commerce developed at a certain time and certain things like stack overflow are probably have been more popular recent times. Looking back over the history of search, you know, when did machine learning and AI start being applied to these types of search problems? Was it always applied or did it kind of start out as rule based algorithms and like, oh, you know, this thing includes this word this many more times than this other thing and so it's ranked higher. I'm assuming some of those things started earlier. When did AI start being applied to search?

It's always been somewhat applied. Right. So you can think about kind of the nexus as originally starting with catalogs. You had a bunch of records you wanted to retrieve across these kind of old library systems. And I don't know if you remember, but you used to have to give these kind of Boolean queries and then these kind of quasi rankers built into it. For machine learning, you know, we've always tried to use and try to understand relevancy. And so we've had things like TF, IDF and BM25 which have kind of existed for several decades at this point.

And those are like methods that are based on counting instances of tokens and working off of tokens that are in certain samples and not other samples and that sort of thing.

Correct. Oftentimes it's statistical based. The idea is that rarer tokens contain more information is kind of the primary motivator for it. And they were really the first attempts at kind of understanding relevancy inside of kind of free text search facets and filters. So there was a company that really started popularizing this in the e commerce space called Endeca that was I believe late 90s, early 2000s.

I used to work with them.

Oh wonderful. So you know them well. They kind of innovated on things such as filters and facets and really kind of scaled out the initial ecosystem for E commerce search. As it came to machine learning. I would say probably the biggest step functions in terms of improvements were kind of the learning to rank work in the early 2000s, late 90s, where we started to apply basic machine learning problems such as support vector machines such in the case of SEM Rank, to kind of improve relevancy across a whole bunch of different signals. And at that point it moved away from these kind of generative type models such as TFID up to these more discriminative type models.

Awesome. Yeah. And has I guess there been a lot of momentum in new like deep neural network unsupervised, all these sorts of like kind of hype things that are happening now? Has that impacted the search world a lot?

Oh, so much.

Okay, yeah, tell us, tell us more.

Oh goodness. Okay, where to start? So I would say search is really an interesting kind of problem space because it's really kind of a confluence of a bunch of different technologies. You can think of the most, a pretty standard stack as looking like something like a Solar or an elasticsearch where you index all your documents, you retrieve some type of candidate set based on the input query and other conditionals like filters, and then you re rank them a bunch of times and then spit out some output to the end user. We have innovated in the industry across every single one of those elements. Learning to rank models have improved. The recent hotness in deep learning has really started to have an effect in search in the form of things such as neural. So this idea that you can build these massive models, these massive neural nets which know how to translate from the query space to the document space and just replace the retrieval systems that were historically just text based matching.

So I'm kind of curious, you know, what types of data get involved and when you're building out a machine learning model these days and you know what data is relevant, where do you go for your data for search? It's just not something that I'm familiar with. And I was, I was rather curious.

Yeah, I was Thinking while Chris was saying that like, are there data sets like you're talking about learning to rank and like going from query to document, are there like existing data sets that you know are kind of standard in that, or is it still like a lot of people using, you have to kind of build up your own internal data sets and that sort of thing?

I would say it's a bit of both. There's definitely learning to rank data sets out there. So the Yahoo. Learning to rank challenges from the mid 2000s, you have Microsoft had a bunch of different learning to rank data sets over the years, such as the web 10k and the web 30k. You have TREC and you have a whole bunch of these kind of historical data sets. The problem with them is that they were all universally kind of overfitted to kind of the information piece. Right? So that kind of web search element. And the problem is that we've learned that search inside E commerce is actually quite a bit different. So when it comes to taking a lot of these kind of benchmark data sets and applying them, they don't necessarily translate well from one domain to another. So most companies will build up their own data sets internally and they will apply a variety of different methods, some of which might be state of the art in kind of the traditional information sense, some which might be bespoke to their own needs.

So I'm kind of curious, you know, can you kind of tell us a little bit about why search is relevant to Etsy? Just to cover a little bit about the tie in on why it is that you're doing that. What does Etsy need search for in that way?

So Etsy has over 60 million results and most of them, or a good portion of them are handmade, customized, one of a kind. We have a big vintage basis where you only have an example of one. And so unlike Google or Bing or Yandex or any of these kind of massive search engines where you're constantly returning a top result, we have this constant turnover in inventory and our inventory is just growing every day. The other problem with Etsy is that we don't have SKUs. So unlike Amazon, who's able to leverage a lot of the structured data that's provided to them by the manufacturer, we basically rely on our sellers to figure it out and our machine learning algorithms to try to tease apart the different pieces in there into the type of information which is actually useful to the buyer. To give you an idea of why search is needed, like before, back in the late 90s, inventories were so Small you could properly navigate through. You go through a drop down box at the top and click on jewelry and then you'd see all 200 items that were for sale. If you go on Etsy right now and you type in jewelry, you get 18 million results. There's no human out there that is going to go through 18 million results. And so that's where ranking and search and personalization, all these different elements kind of come together to try to hone down that 18 million total result set to something that's actually digestible by the buyer.

Yeah, and I was just thinking while you were describing things, it seems almost like there's so many outliers in Etsy in the sense that I just searched for for. For example, I search for R2D2 because you know, we've been watching Star wars stuff recently and I see like R2D2, like gift pack of like planting pots. And then I see a R2D2 like chalk bag for rock climbing, which if I'm just thinking like an R2D2 rock climbing chalk bag seems like extremely outlier to me in terms of products that you could create some rules around. It seems super challenging.

It's a really good point. Etsy, we have a lot of very niche niches and that's really spectacular for the buyer too because that means there is likely something out there for you that almost feels like it was made specifically for you for that one person, person that Star wars lover who you know, likes to get up on rock walls. That that's a very applicable gift. And understanding when to surface those versus when not to is a big part of the challenge. This episode is brought to you by Bray Brave. Big news from the Brave team. Version 1.0 is official. That means our favorite open source, privacy focused, blazing fast browser is ready for primetime. Their brand new iOS app landed just in time for the announcement. And the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to changelog.com hitting the triangle icon in the upper right hand corner and flipping us a tip.

So it was mentioned to us that, that Etsy is using neuroevolution for search and, and I guess if you could kind of tell us a little bit about what is neuroevolution and kind of what does that mean? It's a new term to me and so I'm kind of curious not only to understand what it is, but how it relates to search.

Yeah. So neuroevolution has been kind of a moving definition. It originally started, at least the first time I heard about it, been evolving. It is about evolving. It's combining kind of these evolutionary algorithms. You might have remembered them as things like genetic algorithms from back in the early 2000s, but really neuroevolution is kind of combining these kind of evolutionary algorithms to neural nets, effectively. I first came aware of it from a project called Neat, which I believe was mid-2000s. And the idea was, is that it could actually evolve both network structures as well as the weights associated with those neural nets to solve these kind of black box problems.

I know jargon is often a point of confusion, but we've talked before on the program about meta learning and sort of learning to learn and different things involved with that. Is that sort of how evolutionary algorithms are being applied to neural nets? You mentioned kind of learning architecture and weights and other things, or is evolutionary algorithms, is that a sort of different, different piece of the puzzle?

Meta learning is much more about, as you say, learning to learn. Right. So you can either figure out ways to learn optimizers to train models, or you can learn parameter weights which make fine tuning on those models a lot faster, such as the case of mammal and reptile. Neuroevolution is more of a competitor to things like stochastic gradient descent. I would say it's more of a way of learning models based on these kind of beliefs of populations of answers that can kind of compete with each other. And based on a very rough estimation of, say, Darwin, where the best survive, the candidates in the population which end up performing better end up persisting through multiple generations of work. So I'd say it's more common to think of it as kind of more of a learning paradigm. It became a little bit more popular recently back in, I think just 2017, OpenAI published this paper on how they applied this one particular technique from neuroevolution called evolutionary strategies to train agents in reinforcement learning. And they applied it to the standard Atari data sets and they found out that it was actually very competitive. And so this field, which was much more popular in the early 2000s, that kind of got back burnered when neural nets started really taking off In, I guess, 2013 when Alexnett showed that they were actually using useful for solving these big problems, we're starting to see a resurgence because much of the same reasons that neural nets have become successful, neuroevolution has become successful. The computation is finally there for some clarification.

Can you Talk a little bit about. You mentioned kind of as a replacement for stochastic gradient descent. Could you actually kind of talk about where you might use neuroevolution instead of that? Because obviously, as a lot of our listeners and certainly myself having come into this, we're very familiar with stochastic gradient descent. And can you kind of say where it would be productive to consider neuroevolution to replace it in kind of a use case?

Sure, absolutely. And I can speak specifically to etc's use case. So whenever you can compute a gradient, it's almost always better to use sgd. The problem that you have is that there are a number of domains where it's very difficult to compute the gradient of the actual objective function. And you can think of reinforcement learning. We have this environment where we send in some actions and we get some rewards and then we get an updated state. But there's no real closed form mathematical equation we can use to try to understand where to step the policy next. And so most of the policy gradient methods and Q learning and all those are really trying to do sort of credit assignment and figure out ways to kind of compute gradients which improve the model. Neuroevolution is really nice because it makes very little assumption about the underlying objective function. In fact, all it really needs is to be able to know what you're putting into your fitness function and get some type of fitness score out of it where the higher the fitness score is, the better the model is or the input space is that you pass into it. So anytime you have a situation where it's very difficult to compute a gradient and you need to do it based on sampling or some other form of estimation, it can be quite competitive in.

Those scenarios where it might be hard to compute a gradient. Is that typically when you have such. In the problem with Etsy, I'm trying to connect this to the search problem with Etsy, is that because there's so much diversity in your dataset between query and product, match or rank, where there's not recognized categories of things, but there's so much diversity, is that what produces that sort of scenario?

So in the case of Etsy, one of the challenges we have is that we're a two sided marketplace, which really means we have two customers, we have buyers and we have sellers. And one of the places where things like evolution algorithms have been very successful is in this particular subfield called multi objective optimization. And the idea is that relevancy is only one of the factors that go into a health marketplace. Let me Give you a hypothetical. So imagine that you want to make your seller successful. Now you have this kind of problem, right? You have sellers who've been on the site for a long time, and they're relying on a consistent form of income from the marketplace. But at the same time, you want to make sure that new sellers are also successful. And so you need to kind of expose them artificially higher in the rankings to make sure that they can do it. Now, those two needs are naturally in conflict with each other because there's only so much space for search.

Yeah, you can't optimize one because you would necessarily kill the other.

That's exactly right. And so that's called the rate between trade off between those two objectives is called kind of the Pareto frontier, the Pareto efficiency of the actual problem. And it turns out that it's really hard to do when you start combining a lot of these different objectives in there, and especially when you start boiling them down to things which require relevance to be considered as well, which is a very hard gradient to compute in the best of times. And so you have this kind of black box function which have all these different factors which have these trade offs between them. How can you, in a very principled way, train a model that's able to kind of adjust for those trade offs and kind of learn an optimal balance between. Between all of them?

So I'm kind of thinking, when you are implementing this, and I'm still kind of very focused on how neuroevolution can be implemented in a kind of a practical way, what kind of challenges did you find yourself facing in implementing that into your algorithm versus some of the more traditional approaches that might have been more obvious?

Yeah, it's a good question. I think there were a couple of different challenges. First, neuroevolution is many things, but it is not computationally efficient because you're basically relying on sampling kind of. Basically you have like a model, let's say, and you have parameters of that model. The way you can kind of estimate a gradient step is by sampling slight perturbations of the parameters around that model space and trying to intelligently combine them into kind of a gradient, which hopefully improves the model's performance. Unfortunately, we've learned from certain types of research in zeroth order optimization that as the dimensions of the model increases, you need about a square of that in terms of samples to accurately measure that. And so as your model gets bigger, you need to spend more and more time in that exploratory space. And that gets really Expensive. Now, where you can kind of mitigate some of that is by using more efficient languages, being smarter about the size of your space, being smarter about the type of algorithms that you're using to combine them. So one of these is something called evolutionary strategies and it's actually pretty good. You can combine it with these second order approximators like ADAM or rms, prop or Momentum to kind of take advantage of some of the work inside of the classic stochastic gradient sense base to speed up optimization. But it really becomes a question of how do you maintain the efficiency of the search at the same time get the results that you're hoping for.

So before we move on to some of those things that you just mentioned, like the language, which I particularly want to follow up on, I was wondering if you could just kind of give an update as far as, you know, how did this end up working? Did it improve things by leaps and bounds? Was it marginal? And what, what are your thoughts in terms of after doing these neuroevolution experiments? What's next in terms of upgrading search at Etsy? Or do you feel like there's other things maybe that are more important to focus on now?

Yeah, great question. So the way we decide to integrate it, I mentioned before kind of a rough topology of what a search stack looks like. You have an information retrieval system like Solar elasticsearch where you get some candidate sets back and then you go through this cascade where I think system where you're constantly re ranking and refining down the result set. And that means you can go from simple models which are very fast but are operating on a fairly large candidate set, down to expensive models which are slow but are operating on a much smaller one. We put it at the very end, we call it the kind of the business intelligence layer. And it allows us to kind of incorporate both beliefs or priors about what would be beneficial for the marketplace, but apply it at the end of the rankings. So we're always getting the best possible relevance we can out of our systems, but we're adjusting the ordering at the end to try to influence these other factors from online experiments. It worked about as well as we could have asked for. It is somewhat funny. There were trade offs as well as we find the metrics that we're optimizing for. It's one of those funny things where you almost have to be like a lawyer when you're writing the type of fitness functions for these things to evolve because it will follow the letter of the law, but it will do it in kind of weird ways. So, for example, we were optimizing relevancy at K. Right. So precision at K, where you want to get the item that was purchased in the top 10 results. And what we were finding is that it would oftentimes put that purchase at the 10th position, even though that's not what we actually wanted. We wanted to move it higher up because as far as it cared, all it needed to do was get into that top 10 position and it was able to make balances up there. So it'll do what you say, not necessarily what you want it to do.

Yeah. So it sounds like maybe part of the future is really now that you have some of these things implemented, really exploring the policies that you put in place and the strategies that you're using to kind of rein these things in. Is that right?

That's exactly right. A lot more work on metrics, a lot more work on kind of understanding what the trade offs in the marketplace are.

So I know that one of the things that we talked about earlier on in the conversation that I've been kind of waiting to get to because I'm pretty fascinated with it, is is that you guys are using Rust in your, in your line. You've mentioned it a couple of times and I know kind of Daniel and I are both interested in that. We're both actually gophers and you know, there's a friendly competition a little bit culturally between the Rust and the Go people. We certainly have a great respect for each other. I'd love to understand how you're using Rust in productizing your machine learning systems.

Oh, it's a wonderful question. So for folks who don't know, Rust is a language that was developed by Mozilla. And Mozilla, this was back In, I guess, 2010, they had a problem. Mozilla is most well known for its browser Firefox. And every week it felt like there was some type of security flaw that was being found that they had to release a patch for. And they started looking at kind of the core reasons behind and they realized that these kind of low level systems languages, which browsers like Chrome and Firefox, Internet Explorer and all those are written in, are really not optimized for solving these kind of common problems that you run into that can result in things like buffer overflows or use after free or null pointer dereferencing all these kind of problems that you might run into in practice when you're writing in a language like C or C. So they got together and they started looking around at kind of modern Programming language theory, and they kind of picked and choose some of the best pieces from languages in the ML space such as Haskell and OCaml, as well as practical pieces such as Algol based systems such as C and C, and try to combine it together with really strong static analysis to produce a language which was both extremely fast and a suitable replacement for C and C in systems language, but at the same time kind of had the static analysis you needed to write safe and efficient code.

So just as a quick follow up, could you describe a little bit about how one would apply Rust in that environment? Does it basically replace, replace the kind of the software architectures that are wrapping your machine learning pipeline or how does that work? Where does it fit into the overall architecture?

So I'll talk about more generally and then I'll talk about the Etsy specific case more generally. The kind of frameworks that most people are accustomed to using are actually written in Python. But the lie about Python is that none of the fast bits are actually actually written in Python. It's all indexing into C and C or Cython or in some cases fortran. And what Python really becomes is this kind of domain specific language. For gluing these together, the ones that are probably most familiar to everyone on the show are scikit, Learn, tensorflow, Pytorch, lightgbm, xgboost, all of those have the kind of core performance pieces written in C and C and they also aren't immune to these problems. And so you can actually look and find TensorFlow has had to release patches because they also by nature being in C and C, have these problems with safety and reliability. So the place where Rust tends to have the biggest benefit is by replacing those back end components with a safer, faster language. And we've had a lot of kind of work recently done in the space to make it a little bit easier to integrate with Python. So there's a project called Pi03 out there which makes it much simpler to interface between the back end and the front end as it applies to Etsy. So in the learning to rank space, we have to do a lot of feature engineering. The state of the art is still gradient boosting models for the most part. And that means that a lot of the benefits you get from neural nets, that feature engineering piece kind of being deferred to the algorithm you have to do manually. And we were running into this case where every night we were training hundreds of millions or billions of records and we were trying to plumb them through a whole bunch of different Features and it was taking an exorbitant amount of time. The second piece that's really challenging in the search space is that the machine learning algorithms are traditionally written in Python, but Solar and Elastic and these kind of core inverted indices based systems are actually written in languages like Java. And what we really didn't want to have to do is write feature engineering twice, so do something like the hashing trick in Python and then have to port that same implementation over to something like Java to get the models trained in Python and then deployed on Java. And so we were really looking for a language which would allow us to kind of embed it in both Python and Java at the same time. And that kind of put some restrictions. So you mentioned that you both are gophers. One of the problems with Go is that it actually has managed memory, and Java and Python don't necessarily work particularly well with managed memory while managing their own memory. And so those types of constraints made it kind of hone down the number of opportunities we had. And we were mostly focused on trying to find one where we felt we could be productive quickly, but at the same time didn't have to pay a performance penalty.

And with that, as you looked into Rust for those particular problems like you're talking about with VST feature engineering, but also considered maybe some of the neural evolution things that you were exploring, I'm assuming that some of those fundamental or foundational papers, like the OpenAI paper that you mentioned, if you went and looked at the implementation, maybe the model is implemented in Python, in PyTorch or TensorFlow or something like that. How does that piece fit in along with the sort of feature engineering pre processing stuff? Are you taking models kind of from the ONE frameworks and then doing a lot of the feature engineering and that sort of thing with Rust and how does that play together?

So we really have two main systems that are written in Rust. Both are powering hundreds of billions of predictions a day. So our first one was Buzzsaw, which we wrote a paper on back in 2018. And it really is kind of the backbone of how we do feature engineering at Etsy at this point. We pump a lot of data through it, it can scale across clusters, we can embed it inside Python, inside of Java. And it's really nice because when you're training models, especially in the search space, you want to make sure that what you're training against doesn't change. And so you can imagine that by adjusting the implementation of say, TF IDF just slightly, you can actually have these changes to the prediction space. But because we're able to ship the library to both our cluster compute which does the pre processing and run that exact same code in our Python based learning to rank prediction services, we don't have to worry about that gap in terms of implementation. As for the neuroevolution space, as I mentioned before, it's not super sample efficient. So there's been a lot of work around trying to figure out how to scale up these systems. And one of the ways we were very successful in doing that is by moving the neural evolution pieces down into Rust rather than from Python. And so when we originally prototyped this out in Python, it worked. It was slow, but it worked. But by moving it into Rust, the main core implementation, we were able to speed it up by some hundred x and reduce the memory footprint of it and just scale up both of our data as well as the size of the models.

Was that sort of reimplementation overhead was that high or did you find it going fairly smoothly? And I'm not sure about. Maybe you had experienced Rust people and that sort of thing, or was it more like you have experienced AI like Python people and they're kind of dipping into Rust?

I would say it's more the latter than the former. Rust is a new language. By the general timeline stuff of language, I mean, C came out in 1985, I believe, and Java was 98, Python was 1990. So there's been a lot of time to kind of bake engineers there. Rust only hit 1.0, I believe, back in 2015. So it's still new and there's still kind of this community building that's going on. So most of our developers are coming from. I know Python, I know a little bit C. You know, I had to deal with the Java installation school, but that's largely my experience in languages that I use in my day to day. From an implementation perspective, swapping from Python to Rust, there was some cognitive overhead in terms of learning how to work with static analysis, learning how to use these kind of more advanced features that come in the language to your benefit. And then there's just the nature that when you're writing code in Python, you're really gearing for kind of a prototyping space. You're not really necessarily thinking about performance, but when you move to a language like Rust and you're trying to do it to squeak out performance, you have to think about things such as memory allocation, using simd, potentially using Cuda, and how those all kind of play in to Building robust systems.

So I'm kind of curious you're recognizing that Rust is still a very new language and kind of scratching that itch of the language that it's replacing as you're looking further into using it in the kind of AIML space. Do you think that the community, though it be small today, is likely to grow and develop going forward? Do you think it's a substantial enough use case for Rust to really kind of blossom in that area?

I do. I don't think we're necessarily all the way there yet, but I think there's a bunch of indicators that are really positive for first, big companies are starting to depend on it. So Dropbox, for example, uses Rust in their storage layer because they need reliability. Facebook came out earlier this year with their cryptocurrency, which is written in Rust because they need the security. Microsoft just a few days ago published a result of a project where they were trying to replace core pieces of their Windows 10 code with Rust because they got tired of security flaws. So we're kind of seeing a lot of big companies starting to adopt it, at least in prototyping cases. And more seriously, in the case of companies like Facebook, I think that it is going to be really valuable because the one thing that I've noticed for us is that there's really kind of three payments that you have to make. Right. When it comes to building machine learning, there's the cost of development, there's the cost of running, and then there's the cost of maintenance. And Python does a great job at the cost of development. Right. We have this rich ecosystem filled with libraries out there. You can just pip install them, they just kind of work. You can prototype an idea very quickly, but it's not an efficient language. So that cost to run starts adding up, especially when you have to clone it over hundreds of machines to train these very large models. And the final piece, and this is the more insidious one, is the cost of maintenance. What is the cost of failure in your systems? And when we think about things like production, whenever our learning to rank models go down, we start losing money really quickly. We have 18 million results for jewelry. Right. Turns out that there are better answers in there than others. And so we need to be very cognizant of what that long term effect actually is when it comes to building out production.

Yeah, and I mean, that's something that definitely resonates with us. Like we mentioned, having worked a little bit in Google, I'm really curious to, you know, start dipping into Rust a Little bit and get my hands dirty. Are there good places, like, let's say, for people that have some experience doing Python stuff in machine learning and AI, what are some good ways for them to start getting into Rust a little bit? Are there kind of tutorials or machine learning parallels that they could go to out there in terms of. Of common machine learning problems and that sort of thing?

Yeah, it's a really good question. So I will say this about Rust, is that I feel like I'm marketing right now, but the truth is, is that it's the nicest community that I've ever run into. So the IRC channels, the Rust subreddit, just generally the way Mozilla has run the community is the most welcoming I've ever seen.

Yeah. With Mozilla behind it, I think one could expect some good things in that sense.

And they've done a really good job at kind of baking it and kind of. I heard a really interesting quote recently which I think does kind of resonate. Google built Go for Google. They didn't necessarily build it for the community, whereas it really does feel like Mozilla kind of built Rust for the community and it just so happened to also work extremely well for them. And that really is kind of permeated through all layers of it. And so you can find good books on Rust, on kind of adopting it from other languages. You can find a lot of GitHub, repos. I know there's not much value in popularity contests, but it's one, I believe, the most loved language on Stack overflow for like five years running now. So there's a lot of joy kind of involved in that space and there's lots of people very eager to help you with your problems.

Great. Well, as we come near to a close here, in our conversation, we've talked about search, we talked about everything. Evolutionary algorithms, we've talked about Rust. I'd be curious just to hear about, you know, what you're excited about in terms of the search space and how AI is influencing that space as we look forward, you know, what are the biggest open problems that you think, you know, are really interesting that people are working on or. Or what are you just excited about kind of over the next years as the computer community grows?

Yeah, that's a wonderful question. So we've never lived in a more exciting time for search. We have this kind of openness now in industry, in terms of publishing, state of the art. You have Alibaba, who's been making a lot of work recently, and then the classics like MSR and Google and other folks who are really publishing great research on how to both build production grade systems and how to kind of push the state of the art in terms of retrieval at KDD. This year, one of the big workshops was from LinkedIn and they published this fabulous deck on how they've kind of integrated machine learning at every level of their search platform. So using things like Gans and using BERT to kind of tease apart the NLP pieces. Then you have great papers talking about how folks like Amazon are embedding billion parameters neural nets inside of their information retrieval stacks to these real production problems that folks like Chinese e commerce companies deal with such as Singles Day. How do you handle the scale of those systems? One of the things that I've been starting to see as a trend over the last few years is that the blending between lines of where machine learning starts and distributed systems and systems engineering starts is starting to be a little bit fuzzier. And it turns out that the best search systems are really going to incorporate techniques from both worlds into the code that's being built rather than kind of having them segregated apart. So all the improvements that we get out of conferences like Neurips and ICLR and Sigir and all these wonderful conferences, we're finding they're making their way faster into search to actually, actually solve real problems.

Awesome. That's great to hear and I definitely resonate with a lot of what you said like I mentioned and I certainly hope that we do see some of those trends and of course we'll keep looking for great things coming out of Etsy and what you're contributing to search and really appreciate you releasing your findings and other things like you said with Buzzsaw and other things. So yeah, great work and thank you so much for taking time to talk with us.

I really appreciate being here. Thank you.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you got to do. Share the show with a friend if you enjoyed it and bandwidth for Changelog is provided by for fastly. Learn more@fastly.com and we catch our airs before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there popping or your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.