Cresta and Abnormal Security | Practical Innovation

Hello, welcome to Gray Matter, the podcast from Greylock where we share stories from company builders and business leaders. Today we're talking about the key factors entrepreneurs must consider when building machine learning companies. Greylock partner Sam Motamidi sat down with Tim Shi, the CTO and co founder of Cresta and Jeshua Brattman, the head of machine learning at Abnormal Security. Jeshua and Tim share stories from the earliest days at their companies, from developing machine learning models with limited data and earning their first customers to practical business models, pricing and product innovation as their companies grow.

Tim Jeshua, welcome to Gray Matter. I'm excited to have you both here today and we've been talking about this conversation around how to build applied ML companies. Cresta and Abnormal Security are really good examples of this and so there's going to be a number of interesting topics to talk about and exchange points of view on. I'd love for the both of you to introduce yourselves and briefly introduce the two companies and the role that machine learning plays at Abnormal and at Cresta. So Tim, maybe we can start with you.

Thank you Sam. I'm excited to be on Gray Matter. I'm the co founder and CTO of Cresta. Before starting the company my co founded I worked were both PhD students at Stanford. I was in the NLP lab working on natural language understanding and applying reinforcement learning to automate repetitive workflows and Zed was working on automating udacity grading. So we all converged on the idea of how we best apply AI to make humans more effective and productive. And we met at the Entrepreneur Club where Viv Lee started. That's how we started the company. And over the past few years we've applied cutting edge NLP dialogue systems to augment human knowledge workers, in particular contact center agents where Cresta provides real time recommendation and suggestions in terms of what to say, how to say things in real time and that allows contact center agents, especially low performers, to behave as good as an expert.

Awesome. And Jeshua, can you tell us a little bit about yourself in Abnormal?

Yeah, absolutely, and thanks for having me. So I was on the founding team of this company called Abnormal Security and I run our machine learning. I sort of got into this. I did half a PhD in reinforcement learning and deep learning back in the day. I ended up dropping out of that and jumping through a few startups. One called Telepart which did predictive marketing that was acquired by Twitter where I worked on the machine learning platform as well as some abuse detection products where where we're Trying to stop abusive behavior, hate speech and harassment on the Twitter platform. I knew the two co founders of Abnormal Security from telepart, which is a previous startup. And we had this idea of trying to bring modern machine learning to the space of email security, which just in the last few years has been just riddled with much, much more sophisticated social engineering attacks, causing all sorts of damage across almost every industry that uses email, which is pretty much everybody. So we started this company and it's really ML first trying to identify these social engineering attempts through deeply understanding the behavior of people who are communicating, to try to identify abnormal behavior, which is where the name of the company came from, as well as deeply understanding the content, understanding what is in these emails, what is being linked to. And the approach is really machine learning. First, we're about three years into this company now, things are going well and we're expanding to other types of security problems like account takeover and data loss prevention.

Excellent. One of the things that I find really interesting about both Abnormal and Cresta is both companies are the rare example of sitting at the intersection of both cutting edge ML research innovation and new product development, but also doing that in a highly practical way. Right. And both companies in their respective markets, broadly deployed across the Fortune 500, serving machine learning, use cases in production with real customer impact. And I want to double click on that. Joshua, maybe we can start with you. As you think back to the early days at Abnormal, when you began the company, how do you maintain that level of customer obsession and centricity and focus on practicality and solving customer problems while also developing innovative ML solutions?

Yeah, it's a great question. And honestly, it was one of the most difficult parts of the beginning of Abnormal. You know, when you think of building a startup, you think about very, very quickly identifying problems and finding what customers need and building your product towards that. And when you think about ML, it's a little bit more like thinking about the data, right. How do you build algorithms that best represent the data? But you have to be very flexible, you know, when starting a company, you want to solve the problems that the customers have and not just go and do data science off and avoid. And this comes to reality when you're dealing with these email attacks that we were. We maybe had an idea for how we build an algorithm to detect some kind of phishing attack or some type of business email compromise attack. And we really wanted to focus to build generalizing ML algorithms to find similar types of attacks. But, you know, every once in a while or all the time, we would Identify types of attacks that we were missing from customers that didn't quite fit into our paradigm. And, you know, one approach would be, you know, well, that's kind of like something outside the scope of this ML model and it's falling through the cracks or whatever, and we'll just sort of ignore it. And that's something you can't do. Right? You've really got to listen to the customers, listen to the data they have, the problems they're having, and really treat those sort of false negatives and false positives with extreme high priority. And if you can't adapt your ML models to detect these attacks, you're going to end up building rules and heuristics to do it and eventually get those into your models as features. So this is a process that we really had to develop quickly to be able to have this pipeline of false negatives, false positives from customers, and just ideas from customers of types of attacks that they were worried about and having those get into the ML system as quickly as possible.

When the company got started and customer number one, you didn't have a large data set to go build your ML models on and train them on and get them to the level of efficacy you'd need to actually deliver know superior customer value. How'd you overcome that? Like, talk about the very first customer. How do you, how do you solve that cold start problem that so many ML entrepreneurs kind of puzzle with and talk about?

We're able to find a first customer that really wanted to be a partner with us. And I think that was really crucial in making this whole company work. Our first customer was really willing to try out this new idea. They had problems they wanted to solve and nobody was solving them. And so they were willing to have us experiment with our algorithms, identify what we were catching, what we were missing, and allow us to sort of get labels from that to start improving the models to begin with. And this partnership was really crucial there because this type of data that we were dealing with, this email data, wasn't something available out in the world. Right. It's somewhat sensitive data. And so having that partnership was really necessary.

Tim, I'd love to get your perspective on this as well. I know Cresta's first customer was a very large Fortune 100 enterprise where you were dealing with customer data. How do you think about building the first iterations of Cresta? And how do you guys kind of solve this cold start problem?

As Jasher mentioned, that tying the product, especially the ML to the value is really a key piece when we Think about how to be customers obsessed with the ML product. And I think there's a difference between building an product where ML is the core versus like building a product where ML is sort of like the optimization that came after. Right. And for Cresta, it's really like we are shipping a product with ML as the core. And the crucial part of that is you have this co start problem where you initially have to build the system to a reasonable accuracy so that the user can start getting benefit from it. Right. And at Cresta, we had the benefit of like those customers, especially Fortune 1000 customers, they already have some kind of system in place where the agents can start chatting with customers. And Cresta is more of a layer on top of it. Think of it like the existing system as the system record. And Cresta is like a system of intelligence on top of the existing data they have. And then we can ingest the historical data they have and start training the V0 model. Of course, later on when we deploy the model, we can building the feedback loop to continuously learn from user actions and improve the ML. But it's really important to have that initial data set and to train the model so that user can start using it.

Tim, one thing you touched on there was connecting kind of the product value to the business model. And I think one of the things that's so interesting about Cresta is the way your business works and the way you all price the platform and how that directly ties to the ROI that the ML platform is driving. Can you maybe share with our listeners how Cresta thinks about that and pricing and any lessons for other applied AI companies?

Yeah, when we think about like ML, we really view it as like a hammer to the nails. Right. And we were developing the product, Our number one priority is to make sure the product is delivering value. And as you mentioned, in enterprise, the value can come in many different dimensions. And especially for Cresta, our buyers are usually the C level executives or the VPs. And they buy a product because we can demonstrate ROI benefits. And in contact centers, it usually means, you know, increasing sales conversion or some type of customer satisfaction. And when we think about the product, we have to make sure that our AI is improving itself and always like driving value in those kind of concrete terms. But also we have to make sure that the product is delivering value for the user, actual user as well. Right? Like not just like the leadership getting benefit. The agents and the managers who are using the tool on a daily basis, they should also feel they are getting value out of the tool and get they are excited to adopt it because more usage means more roi. And that's what we're trying to do. Sort of balancing the different, you know, users in the system and make sure that the system is like delivering value in those different dimensions.

Hey Tim, I had a question for you. Going back to something you said a little bit earlier at Abnormal, I think one thing that was interesting, we built out a product before we had any customer, right?

Right.

We built out a email attack detection product with, you know, some ML models, although it's more like heuristics at the very beginning. And there were some things that really did hold up when we actually had a real customer with real data, but a lot of things that didn't. And I'm curious what your experience is with that and kind of talking about how you think about building a proof of value that is ML. Right. And then how do you vet it against real data? Did you have that problem too?

Yeah, I mean initially we did have to build out a proof of value with ML. Right. We wouldn't be successful if we hadn't liked the initial data set because we were able to train on millions of conversations that the enterprise customer already had. We were able to leverage that data and build in a system with enough accuracy and be able to demonstrate value in the pov. I think that's the scenario where essentially the ML is the core of the product and we can figure out a creative way to build bootstrap the system. But as you said, maybe there are other products where you can ship a V0, maybe without ML or maybe with rule based heuristics and ML will become like more of a phase two optimization. And we see that in many recommendation systems where like, for example, like TikTok, where you have this initial set of features that user love, but optimization become like second. Like you know, phase two technology where you can continuously optimize the user experience.

Yeah, yeah. What was really valuable is before we got that first customer that really was a partner with us, we at least had built out most of the framework. Right. We built out the kind of like data engineering framework, how we represent data, how we're going to turn that into features and train models. We had trained models on kind of synthetic data because we didn't have a lot of real labels. And so I think that was really valuable for us of sort of imagining how we are going to solve the problem and, and what are the pieces of that that are going to change when it hits the reality of a customer and which are the parts that we know aren't going to change. And that was very helpful for us because when we integrated at first it was like yeah, once we realized the real data there's a lot of different feature engineering we had to do and a lot of model development we had to do. But at least everything piped together very well at the beginning.

Yeah, I'm super interested to learn about how you started with synthetic data because as you said, companies have to figure out a creative way to co start the product and for using synthetic data. But I'm curious how when you actually deploy the product in production because the synthetic data will be very different in terms of distribution from the actual data. So how do you adapt the system and make it more realistic?

Yeah, so in our case we had synthetic data which was real. Some examples of real email attacks, some examples of lots of examples of legitimate emails that we could train things on. But then when we first integrated it with a client, we started getting from their security team examples and labels of the most sophisticated real attacks that they were dealing with. Right. And so once we started being able to collect that and train the models, then they started surfacing new attacks very quickly that their security team hadn't found and that became this kind of flywheel effect that there.

Yeah, that's super interesting. I know that a lot of enterprises have data but for Cresta like when we go to a customer that doesn't have a lot of historical data, we actually take a similar bootstrap approach where we have a system where we take a pre trained model with maybe some rules where we can deploy initial V0 where the system providing some value to the user we call the call flow. Basically the structure that contact center agent have to follow for typical sales or support conversation. But over time as we collect data, as we observe more and more interactions, the system will be able to learn more nuanced situations or different path of the conversation that could lead to different outcomes. So yeah, I think starting with synthetic data or rules seems like a good approach to code start for scenarios without enough data.

These are like really interesting tactical questions entrepreneurs kind of think about as they start apply to ML companies. Like one is the one you just commented on which is how do you cold start and you know, using heuristics, using synthetic data as part of that. Maybe a few more questions just related to this. So the other thing I'm curious about is I think both of you have had conversations with me where you talk about building AI native companies and building an architecture instead of processes where you know, you build compounding loops into the way you're delivering these AI products. Right? And that's both on the user side and also internally. So maybe let's start with the user side, like how do you architect products where the end user interaction actually compounds and goes back into improving the quality of your AI and ML?

I think that's actually interesting part of building the product because Cresta is more like AI company, but in reality we actually think a lot about design because that will affect how, as you mentioned, the feedback loop will help us collect the data. And we know without high quality data the ML model wouldn't be accurate. And Maybe I'll use TikTok as an example for these type of products. The design is very different. The TikTok design, instead of a scroll list, you have every piece of content become the whole page where you can capture the intent from the user and how long they spend on the video and how many times they repeat the video. And that's a much better signal compared to a scroll list because the user might just be like scrolling through a bunch of things without paying much attention. So using that, that's actually a much better feedback loop to improve the system and you can get higher quality data. And at Cresta we also think about a lot of these feedback loops from the agents where they can flag suggestions that they think they're not accurate or from the manager side where they can have like we call a suggestion studio where they can give high level feedback how we can improve those suggestions. So we think about these kind of feedback loop a lot and it turns out to be very valuable in improving the ML.

Yeah, it's interesting. So you know, I was previously at Twitter in which I was kind of spoiled, in which like labels were free in a way, right? Labels were things like engagements and likes and retweets. And that wasn't the case actually for the abuse detection side there in which there is a human computation team labeling abuse and harassment. But in the ML, for sort of enterprise in the security space, it's a little bit different because it's not human facing in a way, right. All the users who are benefiting from Abnormal's product, really the effect is they just won't see attacks, right? They won't see email attacks or the security team will get a notification or a password reset when account's been taken over. And so there isn't a lot of built in feedback mechanisms which does make it difficult from an ML point of view to get that flywheel going. There's a few interesting ways that we've been able to do it at Abnormal One is interacting with the security team. A lot of our clients have security researchers within their organization who are identifying attacks that are happening and maybe identify attacks that we may have missed. So we provide a way for them to submit those to us. We provide annotation, post remediation, which means we go back and try to find things that were like that that we may have missed. But then we also immediately have this process and this is more of a process behind the scenes. Although there's some automated pieces in which we go and dissect that attack that we missed, do an investigation into it, and then those get turned into features and models to improve the system. It does require more work in a way to get that feedback back. One of our products called the Abuse Mailbox is where employees at an organization may forward a tax themselves that they have seen to this mailbox and we will identify those messages as feedback to improve our models as well. And that's getting feedback directly from the users. But that has been a big challenge for us is how we get this source of false negatives, this source of feedback to improve the models.

Right? Yeah. I think you mentioned there are two types of feedback where one feedback is directly collecting from users. And you also mentioned like you have an internal process where you have teams looking at the model predictions and filing bugs and reports. I think that's really interesting. And I think building out tools around the process and sort of help automating over time is really important because as we scale, it's important to scale the quality as well. Right. And because we have a custom model for every customer, we actually need to make sure our QA team can sort of leveraging the tooling and infrastructure to be able to serve more customers at a much higher efficiency. And we leveraged a lot of tools like active learning where it's similar to how self driving cars improve the self, where if you have missed a specific type of stop sign, you want to actually query similar examples like create some kind of data augmentation and label them to fix the issue. So this becomes like a ongoing process where you can continuously monitor and find out these edge cases and improve them.

One thing I want to double click on in some of the comments you just made is this notion of Mrs. I think was what you said, Joshua. And maybe I just generalize that to how one handles mistakes and fails gracefully. And I'm curious about this both on the product side, but also on the customer side. Right. Like how do you manage customer expectations? They're Buying your product based on some promise around efficacy. Because of the nature of these ML systems, you are going to make mistakes. And so how do you set expectations and handle that with the customer?

I can kind of answer this first. I mean, for us, number one, the solution is a company values and culture solution. You know, one of our, like, company values is customer centricity. And this kind of flows into everything we do. And so then the first thing is whenever we do make a mistake, we take it very seriously. So we will make the customer feel heard and they are heard. If we missed an attack, obviously we realized this is bad for their business and it's abnormal failing to do something. And so one is just really listening to them. Not only listening to them, but making sure that once we go and do work to improve this, build new models or improvements to models that will catch this miss that, we share that back with them and we sort of really make them feel like a part of the process, that not only have we addressed their comments, but thanks to their comments, we've improved our product overall. Right. And so in that way, we kind of build up this partnership with our clients. This is a problem that is never gonna be solved. This email security problem is never gonna be totally solved because it's an adversarial problem. Attackers are constantly adapting. So I think there is some empathy for the fact that we will miss some attacks. And I think the best we can do is just make sure that we are taking it seriously. We're improving the product and working with our partners to solve this problem. Together, we're all trying to solve the same problem of stopping these attacks.

That's super interesting. I'm actually curious how you think about predicting scenarios you haven't seen before, because it sounds like it's adversarial scenario where attackers come up with new ways to hack into the scenario system. But we all know ML seems like it actually relies on the law of large numbers. You have to have repetition to figure out the patterns.

I mean, it's all about generalization, which really is the core of AI. It's like, yes, the attackers are changing their tactics, but they all fall within some type of dimensions. What we've kind of done to address this is we've broken down the concept of an attack into these different things that must be true from the attacker's point of view. We try to put ourselves in their shoes. What are they trying to achieve? How are they doing social engineering? Who are they impersonating? How are they building up trust? How are they delivering the Attack, so on and so forth. And so we break it down into these general concepts and then we do a lot of modeling around those general concepts. So at least we know maybe they're going to come up with a new thing they're trying to steal. They were trying to steal invoice payments, but now they've decided they can also try to steal purchase orders. Like that concept is pretty adjacent, right? And so at least all the other techniques are probably going to be the same about who they impersonate, etc. So those models are still going to go off. Maybe one of the models about the goal of the attack or the content will have been missed because they're talking about purchase orders instead of invoices. But we can now adapt that part and be like, okay, there's more types of financial communication and we can generalize that piece of it. And hopefully the other legs of the table still catch that attack. Maybe they don't. And that's when a miss is going to happen. So it is trying to break down the problem and do generalization. And that is one of the hardest parts of solving this social engineering detection problem.

I want to zoom out a little bit and talk about some things that are a little more general and relate more to the landscape. The first is just on the company landscape. So in both markets that Abnormal and Cresta operate in, there's a lot of noise, right? And there's a lot of other. I mean, security is littered with companies that talk about AI and ML. And if you look at the conversational customer experience, management space, again, lots of companies that kind of message around AI and ML, and a lot of that is fud. And so how do you overcome customer skepticism? And are there any kind of techniques you all have used or advice you'd have on how to drive, you know, the differentiated product value into the messaging and as early in the customer engagement as possible?

I think the main strategy has been transparency. I think that we know that we're doing, you know, we're doing the work. We're actually solving it with the best ML techniques out there, and we have the team to do it. Our strategy's really just been be transparent about what we're doing, talk about exactly what we're doing, and try not to have this sense of trying to, you know, say we're doing something we're not doing. That has been very helpful so far.

I agree with Sam. Like, there's a lot of noise in the space, especially for contact centers. And every company is talking about, oh, we have the Best AI. So from a marketing standpoint, it's really hard to differentiate yourself. I mean we have like the background in AI, we can demonstrate our technical expertise. But for companies who are a little bit detached from the technical world, it's a little bit hard to convince them how good your AI is. For us, our positioning is like our AI is driving the business outcomes where there are a lot of tools in the space where they apply ML and AI, but they're really just an analytics tool or something, a dashboard that managers will use. But we saw that there's a huge gap between the existing contact center platforms and analytics dashboard and the actual business outcome that the leadership is trying to drive. So we feel like we're sort of filling that gap.

One thing I'll add to my answer too is the transparency is really around trying to get around the marketing. You know, the fact that this noisy in marketing, but then the product itself, you want it to speak for itself, right? I think that's something we really focus on and I imagine it's. Maybe it's easier in our case than yours, but we can do almost a head to head. We do head to heads with competition and we almost always, nearly always win those. And so it is kind of the data speaks for itself in that sense. Without trying to, I think we sort of try to, try not to puff ourselves up and we just say, okay, well integrate with the product and see what happens, see what we catch, see how we compare to competitors. I think that is the best. If you can do that.

I totally agree. Like if we can get in the door and just do a head to head benchmark of your AI and show you driving much higher value than your competitors, then it's a really good game to play. I'm also actually curious in your earlier stages where the product's still shaping up, how do you convince your customers to or how do you be your competition in terms of demonstrating the AI is much superior?

Yeah, I can answer that for abnormal, but I'm also curious for Cresta, for us, I think we started right at this sweet spot of this problem in that there was this huge increase in these business email compromise attacks that nobody was able to stop. And there were several other, there were quite a few other companies appearing in the space around the same time. But at that point it was sort of this race to build the best product right from the beginning. So it was sort of with a lot of our clients we integrated with, it was no one's stopping these attacks right now. I need somebody to stop them. As fast as possible. And so that provided a good opportunity for us to jump in and say, hey, look, this is everything we're catching.

Yeah, I think for us it's more about how to make it easier for customers to get started because we are not trying to replace their existing contact center platform, but instead we're like a layer on top. So it makes us almost like something that they can easily install and get started and prove value.

So it's just like frictionless for them.

To try it, Right? Try to make it frictionless, but also like maybe take one slice of a product and make it really easy to insert into their existing workflow. I mean, of course in the long term you want to be like the platform, but it's easy to get started when you're just focusing on one use case and eventually you can upsell and you can expand into many different use cases, become the AI platform for the enterprise.

That's reminiscent of some of our strategy too, which is one, the frictionless integration. We sort of have this one click integration. We try to make it as simple as possible. That has helped a lot. And then also this idea of once we prove out we can do a good job with email security, right? We can easily get into, you know, account takeover and DLP and other things like that.

The other question around landscape is on the technology landscape. So if I think about the landscape for aiml, there might be three buckets. One is what's happening on the academic and research side, right? And every week there are new papers being published around new approaches, techniques, algorithms that companies like Abnormal and Cresta can leverage. So that's one piece. The second is on the open source side, right? Ranging from large foundational open source projects like TensorFlow to newer specialized tools that emerge and are developed every day. And then kind of the third bucket is the hyperscale cloud. So Google, Amazon, Microsoft, as both of you know, they have an extensive set of AI services down to the infrastructure layer, but also increasingly up the stack. And so sort of the question for both of you as you built these ML teams is how do you decide what do we have to own and be best in class at as a company and where can we actually leverage resources from the landscape? Whether it's on the cloud side, on the open source side, on the research side.

As you mentioned, ML is a very open community. Actually there's a quote like there are always more smart people outside your company than within it. And I think for ML that's especially true because the community just evolving so fast. Like we Saw that in the last three and a half years Transformers just took over the entire NLP field. And when we started Cresta we used like more like LSTM based sequence models. But now everything is based on large scale Transformers. And I think as a startup you don't really have much time and resource to invent new models or new frameworks. But what you can do is to create an architecture that makes like integrating cutting edge work really easy. And when we think about like open source, we really view it as an opportunity, right? Like we want to essentially outsource any kind of a non differentiating piece to the community and focus on really like what differentiates Cresta. And for Cresta it's the AI that could learn from lots of conversations and be able to drive business value. And we can use like you know, open source building blocks and, and NLP models to develop that. But we will think about the trade off very carefully. Whether we build it in house versus sort of outsourcing it.

That part really resonates with me about trying to create a platform within your own company that is not rigid. Right. That you can mix and match these new models, new MLOps platforms as they come out because we know they're going to. Everything is changing so fast in this space, right. We've really tried to do that as well. Like not lock in TensorFlow or PyTorch. Right. We can use either one. It depends like maybe we want to, maybe someone's going to come up with a great off the shelf pre trained model in something. We can just grab it and use it. So how do we make it like agile under the hood so that we can grab the open source project or what the cloud providers give us and use those to solve the problems we don't have the resources to go build. Right. And focus our time really, really on the core technology which is for us stopping email attacks. And so anything else that we use is just helpful. The question is usually about is this third party system or open source system, is it more friction and work to adopt that and are we going to get locked into that? Versus can we build something that'll achieve the percentage of that that we actually need? Versus oh no, it's very easy and worthwhile to integrate with. So let's just do it. It's going to save us so much time.

I want to wrap the conversation just spending a few minutes talking about career development. Both of you have kind of built careers at the intersection of applied ML and customer innovation and you're now building and leading teams of folks who are building careers. If someone out there is listening and they want to build a career in applied ML, how should they think about that? And should they go join an applied ML company? Should they go join a regular software company? Should they go join the research team at a Facebook or a Google? How do those lead to different paths and what's your guys perspective on it?

The traditional view was this idea that there's sort of engineers, software engineers and then like data scientists and ML researchers, right? And I think it's really becoming clear that especially for applied ML, especially in the startup space, the really right place to aim for is this machine learning engineer, someone who is kind of a jack of all trades, right, can do the machine learning but can also build the systems to actually get your things out to production. And I think that the way that you learn this the fastest and the best, and I can speak for my own career having worked at an ML startup as my first job after school was being thrown into the fire and working at a startup in which ML is the core of it and you're responsible for making things actually work, not just from training models and making them predict something well, but also how do you actually serve them in production, how do they make them efficient, how do you monitor and scale them and be responsible for all of those things? I think that's the way you learn the fastest and also you just build the skills that you need to actually do applied ML. If you're focused on either one of the sides, you will have to play a catch up game to actually be able to build something end to end. Especially if you're interested in starting a product on your own one day.

That totally resonates for us. The key is really this end to end ML ownership in and I feel like it requires you to have many different skills, not just modeling, but also understanding data and building tools and infrastructure around the model. I think in applied ML in particular, 90% of the work is not modeling. You're actually working to get the right data in the right place, setting up the infrastructure, building the labeling process. So if you're good at back front end, that doesn't really actually is a good skill to have to be successful in ML. And I think there's a common misconception you need some kind of research background or PhD to work in ML. But it's actually not the case because having those skills you can have the advantage in working in applied ML and shipping production quality models.

And I would also add to that there is a difference between a company that ML is the core of the product versus ML is something sort of added on to optimize the product. Because there is a difference in what you will learn and how fast you'll learn and your career will grow. If you are working in the thing that is crucial for the success of the business, you're going to be forced to move in faster ways, be more innovative than if it's sort of an afterthought and you're a side team on top of it. So that's another consideration to think about, especially when joining a startup.

Great. Well, Joshua, Tim, thanks for this fascinating conversation. We covered a number of interesting dynamics and considerations that go into building an applied ML company. I really enjoyed it and I'm sure our listeners did too. And thanks to you both for joining us on Gray Matter.

Thank you.

Thank you son. That concludes this episode of Gray Matter. If you enjoyed listening to this conversation.

And want to hear others like it.

Please please subscribe to gray matter on SoundCloud, Spotify, or wherever you get your podcasts. You can also find new episodes and blogs on our website greylock.com and of course you can follow us on Twitter reylockvc. Thanks for listening.