Reinforcement Learning for search

You can take reinforcement learning from an AI standpoint or you can take reinforcement from a statistics standpoint. So you know, people would probably be familiar with things like optimizely that use bandit algorithms for optimization. And so there that's one of our favorite techniques is actually a Bayesian bandit style reinforcement learning. And the reason for that is that it just works with very little overhead as opposed to building you know these big complex machine learning models with anything can happen. And yes you can probably do a lot better in a very fixed use case, but if have broad use cases and lots of different variables changing, it's a different story.

Been with her Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog whether you're working on a personal project or managing enterprise infrastructure, you deserve simple, affordable and accessible computing solutions so you can take your project to the next level. Simplify your life with Linode's Linux VMs to develop, deploy and scale your applications faster and easier. Get started on linode today with $100 in free credit for our listeners. You can find all the details@lino.com changelog or if you're not at your desk, just text changelog to 474747 and get instant access to that hundred bucks. Linode has 11 global data centers and provides 24, 7, 365 human support with no tiers or handoffs regardless of your plan size. In addition to shared and dedicated compute instances, you can use that $100 credit on S3 compatible object storage, manage Kubernetes and more. Visit linode.com changelog and click on the Create Free Account button to get started. Or just text changelog to 474-747. Get started today on Linode. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're Racticalai FM.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing Chris?

I am doing very well, Daniel. Enjoying this beautiful Fall weather. And how's it going with you?

Going good. It's a busy week. There's a conference paper deadline tomorrow that I'm trying to meet, so last night, and I'm guessing tonight will be a little bit restless for me. I'll bet.

I mean, has anyone out there ever actually submitted their conference paper before? Like, this, this final moment, actual day of.

Yeah, I mean, when you have to make, like, three latex tables that, like, each one, you know, the tweaking of those adds, like, each one adds, like, a day or something.

Yeah, I'm terrible about that. I'm like, yeah. Coming down to the final hours.

Yep. But it's a good time. I enjoy it. And we're collaborating with some interesting people, so it's. It's good to do that and good exercise to kind of organize all your thoughts into that paper format. It's. It's kind of a good. But, yeah. What's going on in. In your world?

I am actually trying to do as little AI as possible right now. Uh, the weather has been just too nice outside, and if I even look at my laptop, my wife gives me a scathing look like, you idiot.

What.

What are you doing? Are we not, like, hiking or.

Exactly. Look outside the window. Yeah, I spent most AI, so we're trying to get some. Some outdoor time in when we're not working right now. Awesome.

Awesome. Well, Chris, I don't know if you remember, actually, this is kind of a long story, but part of the reason why this podcast exists is because I met our editors. I met them at GopherCon, I believe it was in 2016. So I do a little bit of go programming, as our listeners probably know. And I met Adam Stacowiak, and actually, I think it was just Adam there, who is editor in chief at the Changelog. And that's actually what got us started talking about doing a. An AI podcast. Well, at that same conference, I met our guests today and have been following their company for some time and using some of their packages that they've open source, and they're just doing really cool stuff. And so today we're joined by Hamish Ogilvy, who is a founder at Sajari, talking to us all the way from Australia, I believe. How you doing, Hamish?

I'm good. I'm in San Francisco, though.

Oh, really?

Did you move?

I did, I did. I had to move across. Yeah. We ended up with most, so from.

Our perspective, it's earlier today rather than tomorrow. There you go.

That's right. Yep.

Awesome.

It's Not Tuesday yet.

So is the whole company now in San Francisco?

No, most of our guys are in Sydney still. We have a few people in Vietnam, New York, Ohio, and then there's a few people here in the Bay area as well.

Yeah. So were you all doing kind of remote workers before it was cool and necessary to do remote workers now in this world?

I'd like to say we were, but we weren't. We were actually our engineering, we always had in one location, which was in Sydney, and I was probably the exception to the rule, but I came across to build our team up in the US but we always had an in house, like no remote engineers. Because some of the stuff that we do is so complicated that we often fill whiteboards and argue for two hours at a time before we make a plan to do anything. And so Covid's been really interesting because that's kind of flipped everything and everybody's working from home. Although it's not too bad in Sydney at the moment. The guys still get into the office. But yeah, we've, we've talking about reducing office space and everyone's working remote. So it's, it's definitely a different world.

Yeah, I would say there's a lot of things about working remote that can be really efficient, but that sort of like trying to get a new effort underway and that like brainstorming and whiteboard time. It's pretty incredible when, when you're in person and I'm not sure that I've experienced anything of that sort of nature in a remote setting.

There's a business opportunity right there.

Yeah, I'm sure there's like virtual like.

Remote team whiteboarding that's actually really amazingly wonderful, which pardoning, you know, those that are already in space. I haven't experienced it either.

You know, I think I have to have some type of special surface to write on or something because otherwise drawing with my mouse, it just look like something a 4 year old would draw or something.

Even if you go past that, you've still got time changes and stuff for us as well.

Oh yeah, sure. That's rough.

Yeah, it's brutal whichever way you look at it. But it's definitely not solved, that's for sure.

Well, as interesting as that topic is, I think you and I have talked a bit about what you're doing at Sajari and some of your work there, but I actually don't know your kind of personal background and how you got into this space and you know, eventually ended up founding Sajari. So maybe you could give us a little bit of that story.

Yeah, it's. I'll try to give you the short version because it's probably a long story, but my background was actually originally in physics. I got a PhD in physics and.

Another one for the physicists.

No kidding. You guys are just all over the place.

Yeah, we have a few on here.

Occasionally we're broken out and physicists have gone to AI. What's going on with that?

I know it's a common path. It's funny, but I was one of the rare few when I first made that path into the space. And then now I think the university's got full programs set up around transitioning people down that line. But yeah, So I did PhD in physics, used to design lasers. So some of my lasers are still used for surgery and skin treatments and other things. But I shipped out into analytics, worked in analytics for probably four or five years, and then jumped out to start the business. So, yeah, different route.

Maybe you could just. Because I'm interested in kind of the backstory of Sajari a bit, but maybe before we get into that, maybe you could just say kind of in a short blurb what Sajari is, what it's trying to do.

Yeah. So we're basically trying to offer machine learning based search as a service, but we want it to be really, really fast. And so I guess one of the trade offs that people make is speed for accuracy. So we're trying to kind of mesh the two of those things together in a short way.

Cool. Yeah. And how did that idea, how did the sort of AI and search, but also the performance side of that, how did that kind of creep up as something that you thought was a space that you wanted to live in?

Yeah, so it's really interesting. But search kind of interested me even back during my PhD because I was doing something really obscure that was very difficult to find literature on. And you'd go to search for things and type in, you know, if you typed in two keywords, you get the phone book. If you type in three, you get very, very few results. You type in four words, you get nothing. And that's just the space because it was very small. And it always seemed really bizarre to me because I'd written a bunch of papers, I had enormous volumes of context in text that I'd actually written, and yet I couldn't use that to actually search and find something on context. And so that kind of sparked my brain, I guess you would say, because it always seemed like we should be able to search more conceptually and then the university actually spun out some of the research that I was doing during my PhD into a company. And one of the first things that they did was an IP literature search. And then the pattern emerged all over again. The IP search company was basically, they had a spreadsheet and they said you keep filling it up with queries until you keep getting the same results. And so just looking at that was kind of crazy. And so fast forward I kind of ticked along in the background and chatted to some friends about it and played around in the background. Couldn't really get any of the technology out there to do what we wanted to do and kind of saw that there was this convergence of, you know, machine learning was starting to come up. Search technology was basically mostly built from around the 90s. It hadn't changed too much in approach. It's changed a lot more recently. But you could kind of see that there was this convergence of smart machine learning and information retrieval. And so we kind of, we jumped on that train and been following it ever since. I think the fast side of things, when we were talking previously about making things really fast, I don't know if you guys saw, but there was a study a few years ago, Amazon, they intentionally added blocks of 100 milliseconds delay in their E commerce queries and saw a drop of 1% of revenue for every 100 milliseconds delay they introduced. I think Google did the same thing and saw a 20% drop in click through rates and things like that. So there's this huge correlation between speed and performance. But then a lot of the things that you do in AI can be quite hard to make performant. And so there's just these interesting challenges where you're trying to balanced.

That's definitely true. And it's like you'll read about this cool model and you want to make it work, but then you read about like oh, how do you make it performant? Oh, you get like this like GPU server.

Take the smarts out.

Yeah, that like just like brute force it, which not all of us have the capability to do. Maybe some fortunate people do.

Yeah.

So as we kind of get going here, and I know Daniel's known you for a while, but I'm kind of as I'm learning as a novice here, like can you differentiate a little bit more or talk about kind of the specific areas that you're addressing versus a Google? So like, you know, with obviously AI being a topic that's in search kind of broadly at this point, what areas are you guys really addressing and how does that differentiate you from the others that are out there?

Yeah, so interestingly, we jumped into site search around 2015 when we were first kicking things off. And the reason that we did that was because Google was talking about getting out of site search. And I don't mean searching the public web, but I mean searching your internal website, for instance. And so they retracted back and sunsetted the Google search appliances. Those bright yellow boxes that people used to put in their data centers. What a different world it is. But they also got rid of their product where they were doing that as a service and basically said you could use ad supported or find another product basically. And so we saw an opportunity there and it was a good test bed to start in because all of the data was public already. So you didn't have security implications and privacy. So it was kind of an easy area to get started. And I've done so much in the background in marketing analytics and site tagging and things like that that we saw that we could basically automate that entire workflow. And so that was the origins of the company. And then now as we move forward, we're kind of going where the transactional value of search is highest, which is E commerce search, where you're just seeing that you can, you know, you make small changes and they mean millions of dollars and you can measure it, which is very different to searching a website where people that they don't care as much, if you know what I mean.

I'm kind of curious on the site search side of things in terms of how big and how many like results a site needs to have to like really like until this really starts becoming a problem. And I was wondering if that's at all related to as well. Like you can put in a lot of design work in terms of making your site easy to navigate, but it seems like you're, if you're able to search something and have it come up sort of instantly and it was a good result, then it also maybe reduces the burden there. So could you talk to that a little bit in terms of the scale that we're talking about and when this type of thing becomes a really big problem.

Yeah, I think it's interesting because remember websites back in the day had these enormous navigational structures and there's 500 links on a page and then now you go to sites and there's basically nothing. So we've had this interesting move where everything's gone to mobile and people have an expectation that they can just search and find what they want. And so you find that you don't need that much content before it actually becomes useful. And the second aspect to this is that in user testing we've actually had customers that have come to us and they said in user testing the first thing people do if there's a search icon, they go to the search icon. They don't want to be bothered going through five different nested navigation structures to find what they want if there's a search box there. But it has to work.

I was just going to say I've noticed that especially like I will go through other, other avenues myself, but I do notice like my wife, she goes to the search box for absolutely everything. I mean like everything. And like almost doesn't use navigation other than that.

And that's user testing showing that more and more and more. It's just becoming a base level expectation which is, it's kind of interesting. But the other side to that is that you actually get people's intent. So we talk about it in terms of intent analytics. But like a cable TV company that's a customer. For instance, cancel is their biggest search. And when people search for cancel, they may happen to know their account id so they can automatically do things like feed retention programs. You can't do these things as easily as if people are navigating around because you don't know specifically what their intention is. But if they type it in, you got a pretty good idea that you, you know what it is. And so the other thing that we see is that if you have searches that are by far and away bigger than anything else, then they probably should be navigation components. And so there's a whole suite of sort of this. I guess it's a workflow where you can just continually improve user experience.

I was going to say it almost informs the UX of the site itself. So you can evolve it with the results from the search. That's great.

That's right.

Yeah, that's really interesting. I wasn't thinking along that vein. It's almost like you're treating the search box because I've worked a bit in chat and other things and it's like you're treating that query almost as like a chatbot input. Not that the user is wanting a chat response, but you're trying to detect potentially the T's and those sorts of things and track those over time. That's super interesting because users website data a lot of times is something that they hold near and dear and don't want to like they don't want to give their data out or you know, all of that sort of stuff. So especially like usage type things. So has that been an interesting challenge in terms of like bringing AI to someone's site or maybe their own infrastructure versus like them or maybe it's not. Maybe they reach out to Sajari service and like as an API or something. How does that balance work?

Yeah, so we do get a lot of companies that ask to host our stuff internally and we don't do it at the moment. We've just chosen to offer a service for now, which is tough because we know that we could do some really interesting things. And there's been banks and defense and investment companies that have treasure troves of data and they want to be able to actually start doing more interesting things with it, but there's no way they're going to ship it out to third party providers. And so we haven't gone down that route yet. But we do keep it in mind for future. For sure. Yeah.

As Daniel was asking you that and I was listening to you talking, it made me think. I suddenly realized that we all probably assume that we know search really, really well over the years, but I know it's evolving very rapidly. It just occurred to me that I probably don't know search nearly as well as I think I do. So actually if you would take just a moment and maybe kind of give us a landscape of how searches have evolved. What is search right now as we are at the end of 2020 and you know what, what is different from the way we might have thought of it a few years ago, you know, when we were originally thinking we were learning it so well.

Wow, where to begin?

I know it's wide open, you can go anywhere you want on it.

Yeah, it's really interesting because when you see things like AI and machine learning and where they can apply in search, it's not just one area, it's many, many different areas that they can apply. And so I think like historically we saw search engines split away from databases because you wanted to be able to run these really long tail language based queries that used to just absolutely crush databases. And so search engines enabled you to do basically write out indexes that were immutable that you could have very high query concurrency on. But then that came with trade offs where you couldn't update the data as fast. And then we fast forward through the years and machine learning became more and more involved in search. And so I guess we saw a transition to learn to rank models where people could re rank a set of results with machine learning. And that started becoming popular. We've got NLP, entity extraction and things like that. So if you go to an e commerce site and you search for size 14 black shoes, then you should get size 14 black shoes. But most often you don't, which is really interesting. And so there's this sort of language extraction aspect to it. And then you've got things like reinforcement learning, which we're big fans of, to if, say you don't want to build a model. And one of the challenges that we have is that we have hundreds of customers and so we can't go and build a custom model for somebody paying on a credit card. It's not economical. So how do you improve performance for them without the added overhead? And so reinforcement learning is one of the ways that we see that work really well. So there's a lot. I mean, it's a hard question to answer really. It's so broad.

Yeah, it's interesting to me. Of course, we've talked about reinforcement learning a few times on the podcast and I think particularly in relation to robotics or maybe automation, autonomous vehicles and that sort of thing. So I think maybe people might not be as familiar with the idea of reinforcement learning in search or also like marketing analytics type things, which I've also heard. So you said you were big fans of reinforcement learning. I know there's still some people out there that are like, you know, maybe think that it's not quite there in terms of like being practical yet or something like that. How has your experience been over the years? When did you start thinking that this is really practical for our use cases and has the tooling around it to make it useful? I guess.

And we may want to define it along the way, by the way, just for those who have joined who aren't familiar with it.

Yeah. And that's the interesting thing and defining it. I mean, you can take reinforcement learning from an AI standpoint or you can take reinforcement from a statistics standpoint. So, you know, people would probably be familiar with things like optimizely that use bandit algorithms for optimization. And so there, that's one of our favorite techniques is actually a Bayesian bandit style reinforcement learning. And the reason for that is that it just works with very little overhead as opposed to building, you know, these big complex machine learning models where things, you. Anything can happen. And yes, you can probably do a lot better in a very fixed use case. But if you have broad use cases and lots of different variables changing, it's a different story.

Yeah. So could you kind of describe, I guess, then from your perspective, where reinforcement learning is specifically applicable to search, like, what sort of tasks are. Because I think most people, when they think of search, they think of, oh, I put in a query and then I get a result. But of course, you're doing much more sophisticated analytics and intent mapping and routing. So where does it fit into that puzzle from your perspective in terms of tasks?

Yeah. So, I mean, I can give you an example, say with COVID 19 this year, if people went pre Covid and searched for face masks, you can imagine women sitting at home in front of the TV with the face masks on, and then Covid hits and Everybody's looking for N95 respirators. How do you evolve when that change started to happen? How do you know when someone's searching for. For face masks? Do they want the face masks of the parts or do they want an N95 respirator? And that's a. That's an example where.

It took me a second to get there. I was like, I don't think about that first type of face mask.

It took me a moment too.

You're talking about like the, like the goop you put on one's face, the green face.

Right. With the towel.

Yeah, yeah, sorry.

Mud mask or. Yeah, so I should have.

No, no, I had my own search difficulty in my mind context.

Right.

Daniel, We've really shown, you know, we're not thinking expansively enough. Shame on us.

It's a common problem. And I mean, I guess it alludes to the same problem that you have in search context. What is the context? And text in particular is highly ambiguous. You can use the situation to help you. You can use past queries so you can see if there's a pattern in what people are doing. But overall, it's ambiguous. And this is where keyword search kind of falls down. And another example that we saw quite a lot of is as seen on tv. So if you type tv, you could possibly match on things that say as seen on tv because they have TV in the text, but obviously there'd be poor results. But how do you know that a word is a word? And so one of the interesting things is that AI and word embeddings and other things have been able to transition that out into vectors. And so you can start to actually see the context that something's used in to understand what the meaning is. So when you search for TV, you get TVs, you don't get as seen on TV. There's a whole host of Interesting ways to try to embed that context into the way that the indexes work. But going back to reinforcement learning, the reinforcement learning is really interesting because it's trying to look for the maximum reward. And so in the case that we mentioned before, the N95 had a high purchase rate. And so that's rewarding the algorithm to move more towards that and optimize the results to the medical masks over the cosmetic style masks. And that happens in a relatively short time frame. And so then you have trade offs in time frame. If you put a six month time frame, it wouldn't have changed. You put a two week time frame on, it would change. But you've got less data, so you've got less certainty. So there's a whole host of different things that play into how that would work. But yeah, it works really well.

I feel like I'm almost asking it as a tangent, but I was just curious, as we're talking about reinforcement learning, just out of curiosity, are there other architectures that you're finding very common or is reinforcement learning really kind of the dominant architecture, given the use case that you're addressing?

It's really interesting. I mean I think learn to rank style stuff is probably more popular, I feel like, but reinforcement learning, I know a lot of people have picked up more reinforcement learning, particularly recently, but Microsoft were big pioneers of that. There's a whole host of research articles on Microsoft thing. They have a whole unit dedicated to reinforcement learning. Yeah, but it's for us that's always been one of our main things and it's probably getting more and more popular because it's easier than learn to rank models and such for sure.

So one of the things that even in my workshops that I teach and this topic comes up kind of a lot, I like searching around for tooling for reinforcement learning and all of that. And there's some things that have come onto the scene, they seem like they're sort of rapidly advancing. Have you found yourself having to sort of roll your own internal kind of infrastructure around reinforcement learning?

We rolled our own for performance benefits. And yeah, it's probably quite different, but we actually bake some of that information into the indexes themselves. That makes sense. Yeah, yeah it does. Except the implementation is very difficult because you actually have to be able to live edit the reverse indexes which historically in search is a. Is a. No, no, basically. I mean search indexes have been immutable, particularly the scoring. And I think it was a few years ago LinkedIn, they released a few different libraries around actually Being able to update the scores for intersections in reverse indexes in place. And part of the logic behind that was they don't have to rebuild the entire index every time they do it. And so it's not a common thing, though. It's one of those things that if you used to immutable index structures, you would probably not even try to do this because it would seem so foreign. But because we built our own index from the ground up, we were thinking about this from day one, basically. So, yeah, probably makes us a little bit different, I guess.

Yeah. And for those out there that may be a little bit more new to search when you're immutable indexes and reverse indexes, could you just explain sort of briefly how that fits into search and how it, you know, would interact with a model or something like that?

Yeah. So a reverse index is basically like when you read the back of a book and you have a list of terms that link to the pages that those terms actually exist on. And it's very similar, but the links are to actual document IDs. So for any given term, you can tell which documents that term appeared in, where it appeared in the document. And we also include things like the context of how it occurred in the document as well. That's what reverse indexes do. And what was the second part, Dan? I forgot.

Yeah, so just like why in general these would be immutable and like why maybe it's necessary that they not be in the case of applying these more advanced techniques.

Yeah. So the beautiful thing about immutability is that you can have as many readers as you want, basically because you know the data is not going to change. So as soon as you have mutable indexes, you have to manage concurrent reads and writes, which has a whole heap of locking and other implications that can affect performance and other things. So originally when search indexes were first done, they were immutable because of speed and performance reasons. And remember back a while ago, disk space was very expensive, CPUs were expensive. And so if you could compress the information down to as small as you could possibly get, that had huge advantages. And so historically, search indexes are very well compressed and they can do that because they know exactly what's in the data set before they write it out into an index. So I'll give you an example, delta encoding. If you know the maximum minimum number in a list of numbers, then you can massively cut down the size of the storage for those numbers because you know that you can basically add an offset to recreate the number. And so there's all these compression techniques that allow them to get the indexes very small. When you're working with mutable indexes, you can't do that the same way because you may change a number and then that would change the boundaries for how things are encoded. So you just have a different set of trade offs that you have to deal with. From an algorithmic standpoint, it works okay today with things like elastic and solar because people have built in these sort of differential. When you delete something or when you edit something, they don't actually make that deletion or edit straight away. They buffer them and then re merge them out to disk. So you kind of buffer up the changes and then you pay for them in one big hit as they're remerged out to disk. And so just different trade offs there. But from an algorithmic standpoint, that's why Learn to rank is typically a second third however many passes where you get an initial set of results and then you basically re rank them with a model externally. And so the model you can update more frequently. And it's totally decoupled from the way that the indexes work. So we kind of went a different route where we said we wanted to have the indexes more involved in that initial ranking process. We wanted to have a better initial ranking process. So it's just a different, different way of solving the problem.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the Changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com changelog.

It's better.

So I guess to get very practical. Given that we were practical AI, I wanted to ask if I am in charge of an organization's website and we are trying to figure out, you know, what, what to do that. And I decide, hey, I want to put your tool set into my website to be able to apply search. What does that workflow look like to me as that practitioner who has to actually get it up and running, you know, what am I experiencing? What do I need to know? What kind of prerequisites are there that I need to be thinking about? Kind of. Can you describe that whole process of how you go implement it?

Yeah. So there's basically two ways that we implement. The easiest way is just by a crawl. So you point it at a website, tell it what to touch, what not to touch, and then we go and crawl that information that's public. You can also link it into intranets and things like that. But basically it's a crawl process. The positive side of that is that it's very easy, very low overhead, no code basically to implement it. The downside is that crawling is a kind of a black art. It's amazingly hard to keep everything up to date and in sequence. You can't. You have to know when somebody changed a particular piece of content and then to go and recrawl it, for instance. And so that's difficult. So in applications like E Commerce you find that people want to have things more tightly coupled to their data. And so in those cases you have integrations. So we have integrations for things like Shopify for instance. You can use an API so you can custom load your data in, which gives you a lot more control. And so some of the E Commerce customers we have do over 500 product updates a second and they have, they're updating things like inventory in real time, the performance of things in real time and other such things. And even the other day I went to buy a jacket online and it wouldn't go through. It kept telling me I could buy it but it wouldn't go through. And then I just basically gave up. And then I went back in 20 minutes later or half hour later and the jacket was actually sold out. And so you could tell that the search and the front end was not aware that it was sold out. And so it was kind of letting me. But then when it went through the purchase it wasn't. And so there's this sort of the need for real time data integration is much more important with things like E Commerce.

So would it be fair to kind of given those integrations and given the fact that you know that's a whole bunch of different dependencies that you have to deal with and those integrations, I mean, would you almost think of an implementation sort of as a system of systems where you based on whatever that user is needing. And as you address that, obviously you've talked about the speed and all the things that you've done to make your service fantastic. How do you keep these integrations that are depending on APIs beyond your control from impacting how if something does go down them thinking that is your service as opposed to another, how do you manage that process?

It's a really good question. It's one that we think about all the time and at the moment we're kind of taking this approach of middleware where we kind of, we have a translation layer that does the connection with the external services, but then once it's inside, it's all kind of the same for us. So it's much easier for us to have a consistent view and see when something connecting to external is broken. But there are interesting implications as well, because say things like we're trying to generate a decent ranking algorithm for you out of the box. So if we see that you have something like a size or a color, then we're trying to add in the NLP automatically to be able to pick that out. So that if somebody does come along and search for size 14 black shoes, it will automatically map exactly to your categories and so you will get that exact result. And so there's interesting implications here. If you come along and then you get rid of sizes tomorrow, I mean, I don't see why you would do that. Then what's the implication there? So there's not only the connecting the data, but there's also how it's inferred in search is another problem there. And so one of the things that we're trying to do is actually run algorithms in the background and we do quite a bit of backtesting as well. So if you come up with a new algorithm, you can actually look behind the scenes and we can back test to see would that have given you a greater return on historical data. And what we're looking to do moving forward is actually be making changes in the background, back testing them for people automatically and then reporting back to them to say, hey, we found a new algorithm that is actually better for you. If you press this button, you can put it into an A B test and then move it into live. And eventually I would like to get to a point where we're actually surfacing and allowing people externally to come in and write better algorithms within our system. So it's quite complicated.

Yeah. And coming from a company that does manage all of these different models and config for all sorts of different clients, do you have any kind of practical advice and wisdom in terms of model management and sort of managing like, or maybe even automating some of the training and updating of models while not also losing the ability to debug when things go wrong. Any advice there?

Yeah, I would say version everything and make it part of your entire workflow so that that flows all the way through to the analytics. And so we record analytics for every single key press. We know exactly what that flowed through, where it hit how long it took everything. And so versioning in that is critically important to making things work. And yeah, that's probably the thing that stands out the most.

Do you have to do something like, very meta and like search your own search analytics or something to like, be able to like, find things?

Basically, yeah. Basically, yeah. Yeah.

That's funny.

I'm curious as you look forward, you know, we're kind of going through this time over the last couple years where voice has become a big mode of making queries instead of everything being through a keyboard or touch. And as we look forward and there are other modes of interaction that are on the horizon. How does that change how you think about the process of search on the front end? And maybe, you know, given the fact that not all modes of interaction yield the same information at the same place, does that change how you have to architect your solution to accommodate those? Any thoughts on kind of where you're going with that?

Definitely, and it raises a really important point because when people use voice queries, they add a lot more context. Google went through this period where the query length, the average query length got shorter and shorter and shorter. And then now we're in a period where it's getting longer and longer and longer. And that is definitely due to the technology, like being able to take longer queries, understand the context. And like when at the start of this, when we were talking about academic search, you know, three or four keywords and you get no results, that doesn't happen today. And that's because there's a better understanding of how to map that text through into the indexes. And so I think as voice gets more and more, you're going to see these sort of hybrid models and keywords aren't going away. If you search for very specific things, then you want to get very specific things. And people, if they don't see those specific things, they're less likely to click on the result. So it kind of feeds back into itself. But then you have the hybrid of turning things into vectors and understanding them with models. And so you'll probably see that different queries, some will optimize more to the old style and some will optimize more to the newer style. Yeah, if that makes sense. I don't know if I explained that well, but it does. No, you did.

That was good.

So you emphasized E commerce a couple times as you were discussing use cases. And I know that this year in particular has just been like insane in terms of the growth of, of E commerce. And I'm wondering If like, that's shifted, like some of the priorities within Sajari in terms of like having a lot more inbound requests for, for that type of search, what have you seen in terms of the impact that that kind of whole segment, you know, the explosion of that segment of the industry has. How has that affected your growth and the company in general?

Yeah, it's really interesting. I mean, I wish we went to E commerce earlier than what we did. In a way we always plan to. And it's a natural area for us because we have so much focus on improving the transaction rate. And so we always talk about the flywheel of improvement, how do you get continuous improvement in search that drives more revenue, etc. And I wish we switched into E commerce sooner, but this year's been incredible. I think it's about 350% growth in ecommerce queries from E Commerce customers. I mean, they're just growing at such a rapid rate. It's crazy. There's definitely demand there, but then there's also fatigue in E commerce because some of them have just been trying to keep the machine working. I don't know if you guys wouldn't have seen this, but in Australia, Kmart brand had a virtual queue because they got so busy. It would actually say you're fifth in the queue. You have to wait three minutes before you can shop online, which is. It seems kind of crazy for today. But you have to remember that a lot of these companies are still operating on some legacy systems that don't scale. And so they've had to deal with those. They've had to deal with, you know, increase in inventory and all sorts of other problems. And so it's top of mind. But they're also busy, if you know what I mean.

I do. I guess, you know, as you're looking forward, you know, as we're kind of winding up here, and you're kind of thinking about the future of your company, how you're implementing these algorithms and you're busy tracking the future of AI on one side and maybe the AI and kind of more specifically the future of search on the other, whether it includes AI or the other non AI components of it. That's a lot to track together. What are you expecting and hoping to see and how do you think that will, will impact your own service as you develop this over the next few years? What's your whole future outlook at this point?

Yeah, I think it's a really good question. And in terms of keeping up with AI, it's almost impossible. I mean, you guys do this blog regularly and you probably still struggle.

Part of the reason why we do.

This podcast, yeah, it's just it's moving so fast and I think like for us being able to be pluggable is really important because I think that the AI aspect has to be able to evolve on its own. And from the search side we're trying to get better at we have the mutable indexes like I mentioned before, and we're working on being able to distribute them globally so that wherever you're searching, it's right by where you are. But that means you also need to distribute models and other things. So there's a whole host of challenges from DevOps standpoint Systems engineering that you need to deal with as well. So I think pluggability is one of the big things. Don't try to keep up or stay ahead because you're going to struggle.

Awesome. Well, thank you so much for enlightening us on so many things and it's great to connect again. I know I've been wanting to make this conversation happen for quite some time, so really happy to connect and hope that someday we're able to see each other other again in person. But thanks for chatting with us virtually. It was a real pleasure and looking forward to seeing what Sajari does in the future.

Thanks Dan and thanks Chris. Great chat.

Thank you.

Great.

If you enjoy practical AI, we would enjoy a 5 star review on Apple Podcasts, a blog post in response to something set on the show and or or a recommendation to a friend or colleague. Those word of mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whitenack. It is produced by Jared Santo with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Shout out to Fastly Linode and Robar. That's all we have for you today. We'll talk to you again next week.

Sam.