Selina Tobaccowala on Building a Global Business at SurveyMonkey | Blitzscaling 10

This is session 10 of technology enabled Blitzscaling, a Stanford University class taught by Reid Hoffman, John Lilly, Allen Blue and Chris Yeh. This class features Reid Hoffman interviewing Selena Tobaccowala, the president and CTO of SurveyMonkey and the CO founder of Evite. This podcast has been produced by Greylock Partners. For more podcasts, class notes, slides and videos, please visit greylock.com Selena was co.

Founder of a site called Evite, which those of us who are a little older probably perfectly remember. Yes, yes, yes, exactly. And then helped grow Ticketmaster City Search and has been a fundamental agent of SurveyMonkey, which is very well known within the Silicon Valley community as being one of the interesting unicorns. And not as much generally, other than most people taking a survey have exposed the survey or not. And so we thought there were a bunch of different things that were interesting about the both selena and the SurveyMonkey story. So we thought this would be a perfect time for kind of this village inflection set of choices. Because one of the things that happens is when you actually think about like these things that are articulated as rules in Silicon Valley, actually most of these things are patterns, most of these things are heuristics. And actually, in fact, sometimes that specific break of the pattern, that specific break of the heuristic is the thing that actually, in fact, makes an opportunity for a unicorn company. And so that was one of the things that we, when we were conspiring. And so why don't we kick it off with, tell us about how SurveyMonkey, this is before your time was founded, what the early story of SurveyMonkey is.

So SurveyMonkey has a pretty interesting and unique startup story in the sense that it was one guy, his name was Ryan Finley, and he worked at a radio station. And the radio station said to him, hey, we want to get feedback from our listeners. And he thought, oh, wait, I can do this. And he learned to code and he built out the survey for the listeners. And as he did that, he realized that there was this opportunity beyond just this single use case. And so he built SurveyMonkey on the side while he still worked at the radio station. And eventually it obviously started, but right away, sorry, one of the interesting things is right away it was a freemium based business before the word freemium was actually existed. And if you think about it, it was 1999. It was probably he was one of the earliest, if not the earliest models, build a product for free and then let people upgrade for paying features. And that is how he started from day one and eventually this product started making enough money, and he had what he called email bankruptcy, where all these customers were emailing him. And he looked at all the customer support, he couldn't answer everything. And he turned around and he deleted his inbox and called his brother and said, hey, can you come help me with my company? So that was his second hire, which was about four years into it. And then he decided that Wisconsin was cold. And so he at that point said, I want to move to somewhere warmer, sight unseen. Decided on Portland, Oregon. And so part of this interesting part of this story is what he actually did was put a message that said, my site is down, and this is 2005. So this is very unusual for a web company that's already pulling in, frankly, millions of dollars. And. And put the servers on a truck and flew himself. And then who physically chased down the truck because it was going to the wrong address on the other side in Portland, got the servers back, put them back up without realizing that he had, like, his whole business was essentially going to be on the line. And there in Portland hired two more developers and about 10 more customer support people. And so fast forward to end of 2008, and he looked at this business, which, in Reid's terminology here, was acting in the sense of size. Wise was a tribe, but revenue size was already a village. And he knew he had reached a point where he had no idea how to take the company to the next level. And for a founder to have that much prescience of himself is just phenomenal in my mind. And so he basically hired essentially a banker to help him, say, look, I want to essentially get some liquidity for myself, and most importantly, I want to bring in a professional CEO. And that is just such an unusual thing to do. And so in April 2009, two private equity firms. So it was too big for venture capital. It was already like, the amount of money needed to buy Ryan and his brother out or to buy a portion of them out was already too big. And so this Ryan is truly amazing and is one of the most. Is an extremely humble guy. And one of the things that he was looking for was a professional CEO that held that same culture and ethos. And so he, Spectrum Equity and Bain Capital invested and brought on a gentleman named Dave Goldberg, who I think, as many of you probably know, passed, unfortunately in May of this year, but just had that same culture and ethos of just being somebody who was extremely intelligent and extremely humble. And that is something that both Ryan and Dave shared. But he realized it was time to bring on a professional CEO and did that, which is pretty interesting for Silicon Valley.

And this is essentially where I started this, because obviously Goldie and I were talking about this, and Goldie was saying what he was going to do, and I was like, oh, really, Portland? How is this gonna work? And all these sorts of things. And so now let's get to how you got to this. So Goldie calls you or you connect. How does that and what does the company look like?

Right, So I was actually working in London running European product and technology for Ticketmaster. So we were in about 11 different markets. I was having, frankly, the time of my life. I had done Stanford and Berlin when I was here, and always wanted to go back to Europe. So I had this opportunity to move to London and work in all these different market. But I don't see that many women in the room. But for the women in the room, I decided I wanted to start a family. And I was traveling about 60 to 70% of the time. And so I knew I wanted to take a job where I wasn't traveling as much, and I was ready for the innovation of Silicon Valley again. And I ran into Xander, who's our chairman of our board, and he introduced me to Dave. And when Dave first called me, and this is true of every single person at SurveyMonkey, when Dave first called us, we thought surveys, like, are you kidding me? Surveys. But when you start understand the impact of collecting feedback, whether that's this classroom feedback, hopefully whether it's employee feedback, patient satisfaction, you start understanding the power of listening to voices and what you can do with that data. The thing that was amazing about Dave was he had that vision in 2009. He had that vision. And that's the same thing that Ryan saw, was how do you take a vision of a company that has been going for 10 years, but realize you have a business of essentially data collection? I mean, you're all computer scientists, right? You know, you can build a forum pretty quickly, but how do you take a business, though, that has tremendous scale, think about how to both make it global, but how to build additional value? So how do you build insights and not just data? And how do you think about providing value back to the customer on that feedback? And Dave had that vision right from 2009, and I just couldn't have been more excited about it.

How big was the company when you joined?

So it was 19. I finally joined in October of 2009. There were 19 people, there was three engineers, and that's it. The company already had millions of users Tens of millions. And we don't disclose the exact amount, but tens of millions of revenue.

Your revenue per employee wasn't bad.

Yeah. And the margin was unbelievable if you think about that. And there were three engineers, there was one. David hired one tech ops person when he realized there was literally no backup of the database. And yeah, and so it was very small. And Dave started bringing in essentially various executives. Tim Malley, who's our cfo, he brought him in coming from Google, he had brought in someone to help on international, he had brought me in. So he brought various sort of executives in knowing that the business was this unique combination of a small company in people size and a scaled company. A somewhat scaled company already in, I mean not LinkedIn scale, but somewhat scaled in terms of in revenues and traffic.

So what was the initial plan like? What was the. You get to a tribe with village revenue that you have to build to a village. What was your initial like. Oh my gosh, there's so many things to do. How did you triage?

So the first big thing was the platform. So I'm CS Stanford, so kind of came in on the technology side and it was this. Monolithic. NET platform with literally HTML from the database to the app layer to the front end. Like it was sort of sprinkled everywhere. And these guys had done though a fantastic job. If you think about the fact that two people had really, two to three people had really built this application that scaled, but it scaled from a traffic perspective, but not an adding developer perspective. So the big thing that I had to figure out is how were we going to start taking advantage of all this business opportunity. But at the same time we had a platform that we couldn't use from a scale perspective. So that was the first big decision. And the decision I made was to split. Rather than say we're going to take a bunch of developers, put them in Fiji and go replicate the site in a different language, we decided to take, you know, area of the site by area of the site and think about what are the most important things to do first. And the first thing we decided was that given the fact that the business is viral and it's not necessarily like directly viral like Facebook or LinkedIn, but it's viral in the sense of the fact that there had been no marketing, not a penny of search engine marketing had been spent. When we asked Ryan what marketing he did, he said, well, I put a gorilla suit on and I stood in the back Today show. He wasn't on the Today show. He was like in the crowd waving that was like what he told us was the marketing he done.

So there had been no, it was literally guerrilla marketing.

Literally, Literally. Right. So he had done like, basically. So there's, you know, you wanted to drive search, but you knew there was this viral component of the business. And so the earlier we could start that virality in more markets, the better long term we would be. So the first thing we decided we want to do is quickly globalize the platform. So we were able to use a third party proxy that did some of the translation to try to quickly, within six months, get out the door about five to 10 languages. And the main thing we actually needed to rewrite was the billing system and the billings first. The billing system was basically had, if you change the price, it would change for even existing customers. I mean, everything had been, I mean it was pretty hard coded. And so we rebuilt the billing system with 28 different currencies, a bunch of ability to add different packages, different payment methods internationally, which is all very important to let people, to just enable people to buy your product. So that was kind of our first thing. And then we've gone area by area, survey creation, survey analytics, before we go through all that.

So you had three engineers.

Yes.

That was the hard coded system.

Yes.

Right. That had proven product market fit and revenue.

Yes.

Okay. You say, oh, okay. What are the first three hires I make?

So that's a great question. The very first hire I actually made was a user experience designer. Because our product, when we did so I sat there and ran through usertesting.com and watched five videos. And the first thing I realized is there was so much quote unquote low hanging fruit of how people were getting through the experience and learning about the experience. And I thought if I could fix some of these things quickly on the current platform and already kind of take a tick up in the business growth, that would give me some air cover to start rebuilding the technology platform. So that was the very first hire I made. It was a gentleman from LinkedIn.

I remember that. On the other side. Yes. That wasn't a code to talk about Brad.

Yes. But he's a fantastic user experience designer. And then the second was head of engineering, who. Head of engineering who was also from LinkedIn.

I remember that too. This really wasn't code for that, but keep going. This is part of how Silicon Valley operates. Because people actually, the book that Chris and I offered on the alliance is actually people do tours of duty. They work one place and another place. That is actually, in fact the way it operates.

And you Take companies that are awesome and amazing and doing well and you try to say, I mean, those are the companies that as smaller companies you look up to and you're thinking about how do we take some of that talent. But I hired a director of engineering and her first job was to figure out what language we should use because we knew we needed to rebuild the platform. We were building the team in Silicon Valley and we knew that there was not a good community for. Net. We looked at Python, Ruby and Java and we made a decision on Python.

What made you make the decision on Python?

So I looked at three factors. I had this deep spreadsheet. The first was hiring was who can we hire and what are the communities available. The second was what is the scale? So we looked at other companies that were using different languages at the time and like how people had scaled with those languages. And then the third was around, was around the. Just comparables. So like what. Because again, and that really kind of goes back to hiring and scale, but it is really kind of kind of focused on those different things because those are going to end up being the factors that matter, both scale of the tech platform. But also how can you have a.

Scale development team and did you have a target for how many engineers you'd have within 12 months?

That I don't remember because I was also four and a half months pregnant when I started and so I was also going on maternity leave in March. But. But yeah, no, I don't remember the exact target, but we were just hiring as quickly as we could. I think we ended up with about 8 or 10 at the end of that first year from an engineering perspective. A couple products and a couple ux.

And then how much, I mean global obviously played a significant role for billing and currencies. How much was the thought of. Okay, now part of beginning to prep for really massive scale was kind of the global scope.

Yeah. So in terms of, you know, again, we were really focused globally from the, from the. Get the platform ready. And not necessarily, you know, we weren't putting people locally yet. We weren't putting any footprint locally yet. It was get the platform and start that viral engine. And so that was really our focus from a global perspective was launch the different languages. And you do see an uptick when you launch different languages and start and then the second piece of is giving them that ability to pay. So that was really our only initial global strategy.

Got it. And how about any pattern differences on virality globally?

Absolutely. There's a huge difference both in terms of virality Globally. And that is based on where that number of years into how much that brand has been in market. Right. Because when you're depending upon a viral funnel, you're very much depending upon the fact that that customer has awareness of the brand. And you start seeing impact when people have received their second, third, fourth, fifth surveys. And then you start getting more of the viral funnel going. And that was actually true at Evite as well. At Evite, you had an average of 18 invitees to one user, but you needed to receive a certain number of evites before you then created. And so when you're thinking about those viral models, you have to think about there's like the primary step viral models and then these secondary step viral models.

And how many different cohorts were you tracking? What's the rough size of a B testing?

Yeah, so that's a great point. That was the other thing we did right away was we launched a B testing. It was fascinating to realize that Ryan had such good, just product instinct, but he had grown all by instinct. He hadn't done a single AB test. And so we added in just right away an A B testing. Very simple, like a B testing homepage, the end page, which is the page when you finish taking the survey, start testing. You know, do you want to check out flow? That's three pages versus five pages. How do you want to show your pricing page and just your experience? And we just started heavily, heavily AB testing. And if you haven't a B tested before and you're at that scale, and there will be a scale you hit where you can start. Because, you know, it's easy to get statistical significance if a test takes more than like two or three months, it's just hard to really get any motion behind it. But once you get to that point, then there are so many easy wins you can have because you suddenly have that traffic to start seeing user behavior.

We did something. Our first hack on this within LinkedIn was we actually set up a simple AB testing infrastructure that essentially was on a mod 10. So you could do. Yeah, so you could do up to 10 different Sal variants. You could do two at Ebite, we.

Did odd or even user IDs.

Yes. Although you have to get a certain amount of volume to get. Evite was actually one of the very. That PayPal and 6 degrees and a few others. The very first virality, the actual ones really deploying virality. So then one of the other pieces of the thesis in terms of getting ready for blitzscaling is when you begin to do this kind of metricization, what did that mean for dashboards? What did that mean for analytics? What did that mean for data?

So there were no analytics before 2009. Like there was a daily cash report and that was it. And so as far as understanding and first establishing what were the key metrics that we actually wanted to establish, and I strongly believe you as a whole company can't get initially behind more than three to five, and then there's the drivers underneath it. So the very first thing we had to do is actually put the data into a data warehouse and actually start looking at the data. But then we started having to build all the data dashboards globally. So we have, you know, you have an activity dashboard, which is what people are doing, your metrics dashboard in terms of your subscriber counts, but we have different dashboards and everything sort of gets, you know, multiplied by the number of countries. So there's more to watch. But for a subscription based business, which is probably the same as LinkedIn, you know, the key metrics are the number of free users, your free users that become paying users. But for us, the other main key metric is engagement. So how many users actually deploy their survey, how many responses they get, and then of course, how many users stay with you. So what's your turn rate?

So let's turn a little bit back to hiring. So the first couple of hires were let's go raid another company. And then after that, what was the plan?

So on the engineering side, because we.

Had chosen Python, I did not promise to make you blush.

Because we were doing Python, we did a lot of, we went to meetups, we hired our first Stanford intern who now runs all of our mobile. So we came to Stanford and did a bunch of recruiting here and we did Pycon meetups and tried to get a lot of people in the Python community from the engineering side. But you hire also just a traditional recruiter who is out there on LinkedIn and trying to get talent. But that was also one of the first hires we made was a recruiter and the data analyst, like you said. Yep.

And on the recruiter, was there anything in particular that you were thinking is the right thing to do in order to hire effectively?

So it was funny because recruiter still tells this story, but I basically sat me down. I just come from Europe and he had come from Google and so he had all of this still when Google was hiring based on GPA and all this stuff. And he sat me down, he said, okay, so I have all this criteria. He said, what's Your criteria. And I said, I want no assholes who can solve problems. And that's it. And he's like, well, how am I gonna screen for that? I'm like, I can screen for that. I'm like, you find me good, talented people. You find me good people who have raw talent, and we can figure the rest out. They don't need to necessarily have known Python. They don't necessarily need to have, you know, they need to have raw, good problem solving and coding skills. And that has been kind of our philosophy throughout. And we've had pretty good luck hiring that way. And we have a great. I feel like that's helped build a very good culture. And that was Dave's philosophy as well. He wrote a nice LinkedIn post about it.

I remember. And how, as you scaled, how did you keep that culture and hiring philosophy?

So it's been important to us from the start, but we basically. It goes into our screens, which is. We don't ask people. People do code on a whiteboard, you know, like the engineers do code on a whiteboard, but they can choose the language of their choice as an example. And then we ask everybody. We try to do kind of. We've been trying to do the sort of unconscious bias, like making sure we're interviewing without, for. Without trying to just take people like yourselves by asking kind of the same set of questions. And it's better to do behavioral questioning, so it's much better to ask somebody, how would you solve this problem or what would you do in this situation, than necessarily asking someone how they did something in the past. So I think that's been a really important thing for us in the interviewing.

And how many engineers are you now, roughly?

We're about 150. But there have been two to three acquisitions that have significantly added to the engineering count.

And when did you start doing the acquisition?

Dave liked doing deals. We did a major deal, actually, in 2012, which was buying our largest competitor called Zoomerang. And that was a huge thing because we had to, within nine months, deprecate their entire technology platform and move all the customers over to our system. Major effort from the technology teams. But we do have a few of our strongest developers from that acquisition.

If you were to go back and call yourself, you know, be able to tell yourself before you went into that acquisition of the tech move, what would you tell yourself to do differently?

Once again, not go on maternity leave the second time in the middle of it. But besides that? Besides that, I would say the biggest thing was we were really set on Replicating all of the features that were on their platform because we wanted to make sure all of the customers had a fantastic experience when they move over because we were causing them to switch brands when they had made clearly an active choice between two competitors. Right. But in retrospect, you have to balance that versus opportunity cost of everything we could have been doing. So I maybe would have spent. There was a few features that I think that we probably didn't have to build on SurveyMonkey that weren't applicable enough to large enough for the audience. But I think in general, it was, you know, we did. I was very proud of the team and we did it within nine months.

Yep. By the way, just. You may not know this, but PayPal merged with its primary competitor, a company called X.com, very early in the time X.com was running NT. Whereas PayPal was actually mostly C. Not even Java and Solaris, some Java on that. And one of the things that when I think about what I essentially call my younger self, it's be more decisive about, like, okay, this is what the new universe is gonna look like right away. Like, don't, like, oh, we're gonna kind of talk about it and we're gonna figure out what's best and so forth. There was just tremendous brain damage from that as opposed to, like, nope, we're.

Doing it this way that we did like within the first week. I was like, we're moving all the customers and we're bringing everybody over. So that part was good. It was more than, okay, what do you do then? Which.

Yeah, yeah. I have a question on why is behavioral worse than asking about the past?

Why is behavioral better? It creates less bias, especially around things like women and people of color who tend to have. Basically, there's a lot of research on it, actually a lot of research from Stanford. But is because women tend to sell themselves short in what they're saying that they did. And men tend to say they did more than that they did as an example. And so if you give somebody a situation of the future, which is, how would you handle this situation? You tend to get actually better.

And then one other question is, what was the most productive recruiting channel? Which one was the one where you wasted the most time?

Most productive will always, always, always be employee referrals. If you get great people in, they're always going to refer great people if they like the place that you work. I assume you found that obviously when you're starting from scratch, you don't have employees. I mean, LinkedIn is. Is the best place to recruit. I'm not just saying that because I'm sitting next to Reid. As far as. Or feeling guilty that I. But no, I meant your application, not your company. But the second, I would say as far as probably the worst channel. That's a great question. Actually a couple of bad channels. We didn't tend to find a lot of the career fairs because a lot of times the people attending career fairs aren't the same people that like we didn't find career fairs generally very useful except Grace Hopper one was good at the women's conference, but in general we didn't find careers fair very helpful. I mean college career fairs are good like Stanford, but in terms of like, you know, other ones that are put on are generally not that good.

So let's talk a little bit about how you also evolve the product as you were growing because you had this. It starts with small number engineers, super hard coded tech platform that basically none of the serious companies in Silicon Valley use, with already a partially global scope, but a need to go more global. And so you have to do all that, you have to fix all that, you have to make it more modular, you have to bring in data, you have to essentially do the data also as it ties to virality and all the the rest of. But you also have to evolve the product. You have to say okay, these are in order to affect our scale curve. These are the kinds of things that we need to do with the product. And one of it was low hanging fruit that already came with the I'm going to hire user experience. There's so much basic stuff that we can increase conversion usage. What are the others?

So the biggest investment that we had or the biggest strong belief we had was that over time the sort of creation of the surveys may get commoditized, which is creating forms, creating surveys. And there's still a lot there. Like there's advanced logic, there's all these things that people want to do, but that actually trying to create insights out of the data was where that we wanted to innovate, wanting to help customers understand data. So we invested a lot and we continue to invest a lot in terms of our analytics, which was how do you let customers easily slice and dice the data, understand trends. And one of the big things we launched was comparative data, which takes a lot of work because you need people to use a similar question set. So use questions from a question bank, but then look at a comparative data. We might get feedback about how this class was and you guys might have said us it was okay. But we don't know if it was more okay or less okay than a different class. And so how do we compare across the spectrum of your other Stanford classes? And that's what you need to know to understand if your class was good or bad. And so we started building out a bunch. So these are the things where we were really focused on providing the analytics to the customers.

Yep. And what were the. When you began to think about, okay, we're going to be global, how did you begin to figure out what is the right prioritization of which key features would unlock the scale?

So in terms of the very. Like I said, the very first and basic thing was letting them pay you. So the payments, which is, you know, obviously like integrating PayPal, integrating bank transfer. In Germany, 60 of the users pay with bank transfer because since World War II, people didn't get credit cards as much. So like the payment market in all these markets is very different. So that's something just by nature that you have to just allow. So that's sort of like base building blocks. And then for us, and then obviously the translation is like base building building blocks. What we found is that very much in product. Like once you're in the product, you didn't need that much customization besides the basic things, date format, like all that stuff. But it was really about how you were talking about your product that needed to be localized. So what types of use cases are people doing in different markets differently? The concept of collecting employee feedback is very foreign in some markets. It's not something that's done in every market. And so it's more about how you're talking about your customers and also how you're representing in those pages. In Japan, as an example, number of images on a homepage, I think is about 3 to 4x that on average in a US homepage. And so those are the starts to things where we had to. Where we have to. We've done some and we still have more to do.

Did you actually make that, for example, that Japan translation?

So in Japan we're actually doing that only now.

Ah, got it. Yes. That struck me as a very specific memory. That was a. So when you begin to think about go to market, like for example, one of the things you discovered is, is different markets would actually use surveys for different things.

Yes.

Whether it's employee satisfaction or education. What were some of the lessons there? That unlocking those lessons actually really helped with scale.

So I think it's. Again, you kind of said it. It's really watching the metrics and understanding what people are doing. And so therefore you're talking to them more about what they're doing. Because your brand awareness in these markets is tiny. And so the difference in the US is we can say like, hey, create a survey, get answers. In other markets. We need to say like, oh, here's, here's all of the things you can do with a survey. Here's why you might want to create a survey. So you suddenly turn to a lot more educational content that has to be relevant for their market. So that, I mean, it's more about that lack of brand awareness.

Any particular countries that were surprises on the upside or downside?

I wouldn't say so like in terms of the English markets are easier. I mean the other thing obviously where you can, which also is we started doing a lot of search engine optimization and search engine marketing per market. And so you're starting to build out content PA per market. But it's obviously easier to do that in English markets. And so even there wasn't, you know, our business was almost all us when we started in 2009. Now it's about 60% US and 40% international. But a chunk of that is just international English. But no, it's very much based on market size and Internet penetration. So where you start looking at the next biggest markets of Germany and so forth.

Anything that was particularly interesting because one thing that most people know intuitively is how virality works. The product is used or an invitation sent, someone touches it, they may convert. There's a whole bunch of different analysis about getting the K factors and time series and cohort analyses and all this to do that. SEO and SEM, that was also, that was important. What were some of the key lessons there? Because by the way, LinkedIn very similarly one of the things we realized was that one of the things that members wanted was a public profile page. And so we said, okay, let's allow you to a public profile page. Let's allow that to be indexed in Google. And then that became another path by which LinkedIn was discovered. Were there similar things?

Yes, the most similar thing is one of the big ways you build link authority is by having links all over the web. Well, SurveyMonkey has links all over the web. So our link authority was great in the US So we needed to create localized domains and, and country level domains and start sending the surveys out on those domains so that you start building up link authority locally and so that you start ranking in those local search engines. That's really important for local SEO and similarly and start Putting all of your logged out site experience kind of on that to build up the local SEO. But starting to create those backlinks were very important for us.

Did you encourage customers? That's a stretch, not a question, right? Yeah. Okay, I'll come back to the customer question. How do you start when you go into a new market, new domain, do you like a marketing campaign? What do you do to get it going?

So for us, because we already had countries users, sorry, in those countries, we just started switching the survey traffic over to those domains. But in markets where we had very little traffic, then we were also buying the keywords to start off with. So we were starting to buy some of the search traffic. But you have to start figuring out by market which are the keywords that have traffic and start building out content pages on those keywords. And so we also did that from a search engine optimization perspective.

In terms of building processes from a product perspective that percolate across different geographies. How did we manage that?

So we didn't have, we still don't have product and engineering globally. All of our product and engineering and we have a large development center in Ottawa which was also through an acquisition, but we've grown that quite significantly. But all the product and engineering right now we're doing is done kind of in a centralized way across the platform. We do. So our development methodologies don't necessarily have to change for global. Everybody does need to be able to have sort of a pig Latin version of the application on their site, on their, on their code base so that you can start understanding. Because you know, on average it's like I think 1.3 length of language versus English. So you have to start just making sure you're developing and designing for international from the start. But we do it centrally. I think you guys have LinkedIn as people.

We're pretty central on R and D. We have offices in 30 plus countries relative to sales, account management, customer service, these sorts of things. Okay, so customers, okay, customers. Did you actually try to get your customers to publish the results of their surveys as part of SEO?

So we have the ability for customers to say, I allow you to publish this publicly. The problem is that because most of these, a lot of our usage is for people to try to find that feedback around either like employees or patients or so forth is not necessarily something that people want to share. But what we did do is we launched a business again using our scale. We took the end of the survey page of people who had seen it a few times and we built out from that page. So basically we asked people if they wanted to take a survey, and if they took a survey in turn, we would give 50 cents to charity. So we used the scale of this page, which was basically a large marketing page, and we built out a secondary business on top of it. And in that business, we basically we run studies and ask people questions and publish that data out. So it's like higher quality data. And that does help with SEO and keyword terms.

Yep. Yeah, that makes sense. Actually just be the kind of the classic thing where you get actually some super high quality results would be very interesting from an SEO perspective, but difficult to get. So one of the things I think is endemic to kind of part of the thread that is web 2.0 is data as a platform where you start thinking about what kinds of new applications you can build. Given that you have essentially a pile of data, how do you guys think about that?

Yeah, so I mean we absolutely think about it. I mean it's one of the most important things that we think about. And as I was talking about where you start building these comparative data sets, I mean this market to build benchmarks is on average tens of thousands of dollars to even hundreds of thousand dollars. To get a benchmark. We offer it now for 600 bucks or 800 bucks because we have this platform and we have this scale of all these users and we're able to aggregate the data and actually anonymize it, of course, and then create an awesome benchmar.

And also students might not understand why benchmarks are so important. So say that.

Yeah, so I'm sorry. So right now, employee satisfaction, we have, we have Net Promoter score, which is how you might do customer satisfaction. We have a K through 12 parent education survey. So again, you know, it's. And we have an event feedback template that can get benchmarked. So it's basically where you start building enough data that's statistically significant, where you can slice it and dice it by. So if you think about classroom feedback, you would. Stanford probably would want to look at other universities similar to them in order to see if Stanford was comparing it or within campus. You probably would maybe want to like a CS professor may want to compare it against other CS classes. We don't have enough data significance on.

Classroom feedback though, so let's shift to management. So, and this is not so much of a code about the difficulties with Goldie, but as much as you're going to build a company for scale, Goldie hires you, Goldie hires Tim. How do you build out the Rest of the management, what is the principles of that? So we talked about hiring already some but how does putting in a management layer into the company?

And this has been one of the most fun but also hardest challenges we have. So we've had. So we had a very specific thing that both Dave and I believed in from a management perspective. A couple things. One is that we really like to find people who had had some startup experience but also some scale experience. If you look at Dave, he had started a company called Launch but had also been running all Yahoo Music. I started Evite but I had also been running, you know, 250 people at Ticketmaster. So it was like finding that people who had done both a mix of small and scale. And so our head of product similarly he had kind of been at Yahoo, which was more scaled company and then been in a startup called TripIt. So we were in general trying to find people who had a little bit of that balance because we liked the culture of the small company and we have the culture, had the culture still somewhat have the culture of a small company.

How big are you now?

How many people? I'm about 690. So we're right at that. Sort of.

Heading towards the city.

Is heading towards the city. I know I was like I need to sit on the other side of this class. But basically. But finding people who can scale up as you go is hard and some people won't like you will. You know, we had a very fantastic guy who kind of was responsible for various things and there came to a point where he started having to manage managers and he didn't enjoy like it wasn't fun for him, he didn't like it. And he was great at the hands on like getting stuff done, you know, managing a couple people, leading a small team. So when we were in that sort of, you know, tribe stage, he was fantastic. But then, you know, he wasn't. He didn't necessarily get to a point where he scaled to that next level. And then there's some people that have. And like our general view is we really liked hiring from within. It was all of our engineering managers that they've almost all except one been hired from within. You know, it's been something that's been important to us culturally but there does come to places where you actually need to bring an expert in who's done it before. And so it's finding that right balance of kind of keeping people who and helping them grow and keeping that, you know, they understand the culture of the company, they have the right motivations along with people that are bringing that experience to the table.

And how do you. This is one of the things we covered a little bit. We introduced the whole kind of what are the challenges of beginning to scale and multi thread your company? Basically, Tuesday. And one of the things is this whole question about how much do you hire externally and how much do you come up within and how do you identify those people and how are you navigating it? How do you keep the people that you actually really want to keep, even though you're not promoting them into the role? What were some of the considerations?

Yeah, I'd say that last thing is one of the hardest, if not impossible, which is when you have to layer people and then keeping them motivated and excited because it's very hard for somebody who's been responsible for a function or responsible for things. And even though it makes sense that as the company grows, your role may get smaller, and it doesn't mean your role has actually gotten smaller. Your role has actually gotten bigger, but the scope of it has gotten smaller, but on a bigger company set. But that's very hard for people in general to say.

Although what you also just revealed is that is actually, by the way, way the classic language that most of the people are trying to do this, do this. Like actually, in fact, your job is bigger, right? Isn't the. Yes, there's one person running this particular group now and you're not that person, but your job is actually still bigger than it was before.

Exactly. And so it is. I mean, that is difficult. And there will be some people who really only enjoy that tribe stage of companies or really enjoy sort of the tribe to the beginning of the village. And then it's like. And they're serial people who like going through that. And that's great. You know, if you know that that's what you love, that's great. Like, you know, there's a woman who did international at LinkedIn and did it at SurveyMonkey, and you're going to hear from Nirav and she's done international at Nextdoor. That's funny.

Yes, mina was at LinkedIn, then survey.

Mon Surve Monkey, and then Nextdoor. But she is absolutely fantastic at driving, you know, setting the platform for international when you stop. But then there's a point where she's like, hey, I've done it. I'm gonna go to the next one. And so there will be people like that who really enjoy certain scale or size of the company. And that's fantastic. It's like figuring out what you enjoy. But the hardest piece is when people were running a function and then you need to bring people over them. And that is tough.

The other part actually that highlights with Mina in specific is part of the arc that we mentioned. As you start, start with you always are hiring generalists, but how you're deploying them generalists early is doing everything. Then you're covering the new ground.

The.

In between the different areas of specialization. You know, the new entrepreneurial project, you're doing panels, you're doing, you know, benchmarking, you're doing these kinds of things. You can say, okay, do those. But you're also hiring much more specialists. Right. And part of what Mina is, is a internationalization specialist is like, I know this game cold and this is how you play it. And this is you do. And this is how you come from a company that hasn't thought about it and move into a company that does. Which is precisely what she does.

And there will be some people though that scale extremely well. The woman who I mentioned who started as that BI analyst, like when I said one of my, the first five hires, she now, she now is our VP of growth and analytics like she is. So there are those people who can scale and grow and then there's those people who can. And that is the hard thing is because you obviously, and there are people you've worked with for a couple years, two, three years, like you want everybody to scale and grow.

And as you were scaling, did your sense of what competition looked like or what the opportunity looked like changed?

Absolutely. So one of the things is that there's many factors that go on around you in terms of what the large players are doing. So even in our case case we.

Have.

Google has a forms product and so it has some similarities. And so you see larger players making changes and then you also see true upstarts that start really specializing. So there is somebody that has built a really nice product specifically around just employee engagement and they're going after just that HR space. So you constantly see competition when you're a large general platform. It's just a question of how significant is it and how much is it affecting your business and where do you want to think about M and A like we did with Zoomerang and Fluid Surveys.

Yeah. Did you do any M and A that wasn't actually within your space?

We did like an adjacent like Wufoo, which was a forms based product, but not. No, not really.

And no, you haven't set up an acquihires department yet for.

So we did Do a few. Sorry. We did do some of the Y Combinator or didn't quite make it because.

Actually that's another pattern that I'm actually seeing is there's major acquisitions, which is strategic, and then there's a number of small acquisitions, which is part of how you get entrepreneurial people.

Absolutely. And those are great. The Aqui hires are great in terms of finding because those turn out to be. We did this one Aqui hire and the two guys are still there five years later and they're basically running that entire audience business from product and engineering perspective. Just amazing to have.

And how did you feel about. You mentioned Google forms. How did the large players figure in, whether it's Google or Microsoft or. Obviously some of what you must do is social graph optimization as well, given Twitter, Facebook, LinkedIn.

Yeah, I think that in general, it's a question of where is the right place to partner as well. And so thinking about, you know, how do you create the right partnerships and work within their ecosystem as well.

And was there anything. Sorry. With the question.

Go ahead.

Yeah, some of the aqua highs. How did you identify them? Because they're small companies.

Like no one's really heard of them in general. Yeah, they're really small ones in general. It is a pretty small valley. And so when you have like good talent where they're not making it, whether it's one of the Y Combinator companies or. Or just other ones, you'll get a call into your BD M&A. Most of them came through Dave. Dave had a very wide, vast network.

The usual pattern I'll get to in a second. The usual pattern for that is you probably have a corp dev person now. Yes. Right. But the usual pattern pre having a corp dev person who. This is their daily job is. That's part of the reason why you're intensely focused on being connected into the Silicon Valley network. Some of that's through investors, some of that's through key people in the company. Founders, executives, et cetera. That almost all comes from some version of that. So it's like, oh, Dave knows Sam Altman. Sam calls Dave, says, hey, there's this group that actually really likes you and maybe you should consider.

Yeah, they're doing something slightly similar. They're kind of passionate about surveys like the one I mentioned. They were doing phone polling and we did another one which was again, it was like somebody I'd worked with in 1999, like their company. So as you start building your network, you'll hear about. People will reach out if they enjoyed Working with you. Yeah. Or you identifying it. Yeah. Not sure.

Yeah, we unfortunately can ask all day at the moment. It's not super expensive. I mean, I kind of have no idea for the pay scope these things. But you kind of hear like, you mean Aqua hires is expensive, a million.

Per engineer or something.

That is actually the usual benchmark for that, which is a million per engineer actually. In fact. Well, it's expensive if you said this is the only way I'm hiring. Right, absolutely. But by the way, it's not any five engineers. It's the right five engine engineers. And part of what you're usually doing in that case is it's five engineers who work really, really well together, who have demonstrated entrepreneurial ability, who can be part of the new blood of really driving something pretty serious in what you're doing. Right. And that is actually in fact worth it every single doc.

And remember, it's going to be a mix of cash and equity. So it's also equity in your current company that they are hopefully making a big impact on.

Some time frame as well. Yeah, almost always minimum two years, sometimes four. Right. You know, because the whole question is, I mean, the alignment of incentives question when you get to compensation is actually extremely important in circumstances. Like it's important all the time, but extremely important here. Did you have a question? Okay. All right. And by the way, I don't always. Okay. I'm curious whether survey bumping has a tool, has some tool where you help the users to remove certain biases that they may have survey. And also when you actually do it, you know, you have this product where you do across industry analysis. How do you remove some of these biases? Because a similar question could be asked in many different ways.

That's a great question.

And by the way, why don't you restate it just to make sure we.

So the question was, does SurveyMonkey have anything to try to reduce bias in the question? One of the most interesting things you'll see is if you actually ask people if they agree or disagree in the US they're gonna mostly agree. Just by nature. Americans like to agree with things. I disagree. But this research is also from Stanford and actually the amount of that actually changes by market, which is what's really interesting.

Have you ever published what that percentage is?

We have it on a blog post, but I don't remember.

I guess you go find that blog post.

So what we did is we actually built a couple things. One is we have this question bank which helps you create, write the questions or use the Questions that are worded in the right way with the right answer options because the difference in terms of how the answer options are also worded need to be equally spaced. And so we work to make sure that the questions themselves kind of help reduce bias. The other thing is we have the amount of functionality. So you can randomize, right? You can randomize options, you can randomize things that you're actually not because people often choose the first thing as well. But there's definitely things that you can do on a technological level. Also post the survey to help people with respondent quality and survey quality to like make sure that things are good. And we are making some investments there now as well.

Go ahead. Just a quick question. Aside from Mindshare, is there anything more powerful Network effects of conservatives? I mean what about if that certain smaller companies that try specific verticals have found success? So let me slightly generalize the question because there's a couple of different. What are the competitive barriers to SurveyMonkey as it succeeds at scale? Is NetworkFrex one of them and if so, how so and what are the other ones?

Great. So in terms of just in general, one of the main sort of, you know, competitive barriers is going to be both your data. So if you've been using the product, you start wanting to see your data over time. And then the second is as we build, as we provide you the comparative data sets, that also is a competitive differentiator that most people don't have. But part of the reason why we've done acquisitions is because of of similar companies is because once you choose a survey product and your data starts going in there and you have that data over time, your willingness to move products is very low. So when people churn out a SurveyMonkey, the main thing that they tell us why they've churned is because they don't have a need for it right now. But I'm gonna be back. And so that is a huge competitive differentiator that you actually have. But it is also again said is the more you can do to provide people insights of the data, the better you are whether it's the open ended feedback. So trying to do natural language processing. We have a basic text analysis today, but that's another place that we're investing.

Feel free to punt on this question if it's too sensitive. But what have you found in particular about the re engagement? Right, so like how do you model the when reengagement? Yeah, to get the customer who's apparently churned the back end.

So we spend a Lot of time on re engagement both through, you know, emailing the customer and also through retargeting on both LinkedIn and Facebook. So trying to find that customer where they are and remind them that they should be re engaging to do feedback. So we have very specific re engagement curves which we don't share.

That's what I figured. Okay. I realized that this would be a very sensitive area which is the reason I framed it as only what you can say.

But it is a very significant portion of our business is getting people to essentially come back and reuse you. So existing customers kind of paying you.

Again, do you use any machine learning on that?

So we do use propensity model which is using based on users behavior, which is looking at and what types of surveys they've done and using a bunch of variables to decide when to recontact them and similarly with what we should show them at the end of the survey.

Yep, yep, makes sense. Oh, question.

So you mentioned that you rejoined but the reason why you stayed the revenue.

So far ahead of, you know, what.

You expected for it. So I'm kind of curious. Yeah, I went. Yeah, sorry, go ahead. Sorry. Just that, you know, we asked for.

Flying cars and I got. And surveys.

No. How do you think about what's worth working on? Is it something that a lot of people are using? Yeah. So for me there's three things. Oh, sorry. The question was is that, you know, when I first heard surveys it didn't sound that interesting, but yet it was like a scaled company. That was one of the things that attracted me to it. So what are the important things to think about? From my mind there's three important things to think about when you're thinking about your company. A company, whether you're starting it from scratch or you're going to a company. And the first is the product. So I said that when I first heard surveys. I didn't think it was that interesting. But what excited me was how all of the people were actually using the data and listening to their whether it was their customers, their students, their parents, their patients. Like the product is the most important thing in terms of you need to be excited about, especially as an engineer. We're going to spend all your time developing and building it and that is very important to me. The second is always going to be the people. I mentioned Dave to you. I've mentioned him many times. He was a fantastic leader, the best mentor and manager I've ever had in my career. But just meeting, but having people. And then for me it was also the opportunity to build my team so I could choose the people I wanted and help build that culture. That's always the second thing. And the third thing is what you are going to learn. And for me, the part that I was going to learn was could we actually take this company that was at scale from a size perspective so we could a B test and learn, but the team was small and scale it up. And that was the part that I hadn't done before that I was super excited about. I'd had this small company, Evite, that we grew to 30 people when we sold it, and then I'd been at Ticketmaster, which was humongous. But actually taking that company from that small size to large was what I was super excited to learn. It's been six years and a fantastic learning experience.

The other thing I would add, which I'm certain is part of the survey monkey thing, but if you begin to say, let's see. So two critiques of Peter's statement, right. One of them is essentially that actually, in fact, software is transforming a number of different industries. And so actually, in fact, the fact that you're getting a revolution in bits actually means that you're also getting a revolution in atoms. The second part of it is that, yes, in the 50s, they were expecting flying cars, but they weren't expecting mobile phones as a way of doing this. Now, the key thing is that intelligence in how you operate actually massively increases productivity, whether it's an individual or organization. And so part of your actual mission is trying to whatever individual or mission, how do you help them operate much more intelligently?

Yeah. How do you help people make better decisions with data like that is an exciting mission and you have to be super excited about the mission of your company if you're going to stay there for a while.

Have you ever had Tufte come talk? Edward Tufte, visual designer?

No, but I read the books.

Yeah, you should have him come talk at ceremony. That would actually be fun. Anyway. Oh, in the back, I just had a quick question about re engagement. Could you talk more about how much of the reengage work you do is.

In house and how much is like you rely on tooling to do that?

Propensity, modeling, support. Is that all kind of outsourced?

No, that's all in house. All in house. How much of reengagement is as a business, as like a percentage of all of your customers? That's the piece that we don't share.

Yeah. And by the way, one generalization, more or less one of the truths of any business isn't so much of a scale up or blitzscale. If you thing is, anything that's mission critical is in house. Anything that's like this is the key thing by which we win. If that's out of house, that's a major risk factor. And intelligent management is trying to figure out how to bring it in house. The major sources of re engagement are Facebook.

Oh, you meant in terms of how much of it are you re engaging that customer through recontacting them through email and re reaching them through other channels? Well, there's like multiple pieces of infrastructure. I was kind of curious about the whole thing.

So that would be one part and.

Then the actual analysis, I mean that's just literally buying advertising on Facebook or LinkedIn where you have the ability to retarget users X days or x weeks or months after they've churned and try and convince them to come back based on specific advertising. I mean obviously you see it right now with E commerce where like you go in overstock and then three minutes later you're in another site and you're like that's the dresser I just looked at. So that's like how retargeting essentially works.

So it seems like SurveyMonkey hasn't really been aggressive in penetrating the mobile space.

Sure, we can talk about mobile. So as far as the question was, has SurveyMonkey doesn't seem like SurveyMonkey's necessarily been aggressive about the mobile space. So about people taking surveys today, domestically it's about 35%. Internationally it's about 50% are taking it on a mobile phone. Phone. So it is extremely important that our experience is optimized for mobile web. Those users who are taking surveys will not be downloading our mobile app. We have a mobile app that's on Both Android and iOS that's been pretty successful. That is mostly for creating surveys. But what most people are doing on the mobile app is they're really managing their results. They want to check their results in real time and they want to make quick fixes and edits on typos because we've targeted our apps at existing users who are trying to do stuff as more of a complement to them to the desktop. The other thing that we also have is essentially a mobile SDK which allows other app developers to integrate SurveyMonkey into their apps very easily. And so if people want to integrate, essentially we use it as an example for asking you if you click feedback. If you say you had a very good experience, then we might suggest you go to the app store and if you had a bad experience we might try to collect more information as an example. But it's using surveys in app. So that's been our mobile strategy. But it is absolutely vital internationally that you're thinking especially on that taking experience for mobile. But you're not going to get a user who is doing a non recurring activity to download an app. So you need to ensure that your experience is fantastic on the mobile web. And I don't know if you've seen the recent data, but mobile web is growing about 2x as fast as mobile apps. So there is no question that has to be an important part of anything that you would build today.

Actually a follow on to that which I actually hadn't realized would be a good question for you and you may or may not be able to answer given I just don't know which of this stuff is confidential. Obviously in a macro sense there's this interesting app versus mobile web war between Apple and Google because iOS is going let's run really make it more app, right Android and Google's making it, that's more mobile web. Does that play out in SurveyMonkey at all?

That plays out in the different markets. So as you can see, you see like certain markets where iOS is much stronger, like in US is about 50 50, but Japan iOS is much stronger. So where you see more iOS traction you'll see more app downloads. And where Android is stronger you'll see like we don't see as much Android app usage but we see, we do see a ton of mobile web coming from Android. And that also again it's like markets that are more Android heavy. We've had less success in downloading the app.

All right, next question. By the way, we're now at a point where feel free to just interrupt with questions. I have a bunch more but if you have them and the class has been entrepreneurial and asking anyway, that's cool. But just a prompt, you're more. I only have one more critical question. Everything else is bonus round. So if we get to it, we get to it and if we don't, we don't. So part of your scale up was getting the multi threading, being able to work on more than one project at a time. You have the developer tools, you have the global billing, you have going to panels. What specifically did you do to start trying to enable multi threading on decisioning, multi threading on different groups working separately but also together, which is one of the really key things that you begin to set up as the foundation.

So it was very much tied to the technical architecture decision we made to move to a service oriented architecture. And then I believe very strongly that it's important that the teams that are working on any set of the services have an embedded team of a product leader, a user experience design leader, QA automation, obviously the engineering leader, and then a BI analyst who's actually looking at the data for that part. You guys, I mean if you build something you want to understand how well it did and how well it's doing. And so it's important that those metrics get fed back. So we've basically structured the team so that you have these sort of teams that are working on different services or different parts of the app. And then there's standardization of APIs. And so how you're talking to the services tier is consistent. So and so that becomes important as you start wanting to add new features or new products on top is that figuring out what are the guardrails you want to give each team and what is the freedom you want to give each team? And so in our case the guardrails we gave was 100% code review on the services tier. 100% or I think it was 80 to 100% unit test coverage and then API standards. But we didn't give a basic and that you had to and that we wanted you to release at least once a week. But you can release more or you can release less. You can release more if you wanted to. But basically the thought process was I didn't actually care if people wanted to do three week sprints or two week sprints or they wanted to do kind of semi waterfall or agile, whatever it was like, commit to your quarterly roadmap, figure out what you actually want to get done and give as much freedoms to the team as as possible. But make sure that they're getting the metrics back of how things actually did when it launches. So I think it's important to do a 24 hour, a 7 day and a 28 day readout of everything that goes live. But I feel like that creating that embedded team model, like if you ask someone at Survey Monkey Engineering, they will more identify what regardless of their product or QA or whatever, they'll more identify as a gorilla, which is our analyzed team versus like I'm an engineer and I think that that's actually good.

So it seems like SurveyMonkey is pretty similar to Eventbrite in terms of product for organizers to then distribute their own way. How do you guys think about product development, given that there's two sorts of.

User bases, like the people, creators and the respondents. Yeah. So again, whenever you're thinking about Eventbrite, it's the same, which is whenever you're thinking about the fact that that respondent might become your creator, like, you have to make sure that that respondent experience is just like fantastic, that they can take the survey really well, that it's optimized for mobile, that then when they land, when they finish is a good experience. But if you think about, and that's, you know, 80% of our traffic is on the taking side. But obviously our customer, the one who's paying us, the one who's putting their faith in us to actually send their survey out to the customers, is the creator. So in terms of from a resourcing perspective, we spend most of our resources on the survey creation side because you need to get that person through the hurdle of having the confidence to put their name out there with a bunch of questions to a bunch of other people. And that is the customer who essentially, in the end we're trying to upgrade to get into a paying customer base. So I don't know if that answers, but you have to balance your resources, but you have to think about, you know, kind of where that entry and starting point is of that life cycle.

Do you guys spend a lot of effort on getting the respondents to become part of like, not just like the ecosystem, but also take other people's surveys or your own?

So when we launch the panel business, then you have the ability to sign up to take surveys and every time you take a survey, you get 50 cents to charity and then a creator can buy response. So that was a new business that we launched in 2013 using kind of leveraging the fact that we have all these users who want to not only necessarily send surveys to people they know, but to potential customers to do brand research or market research. And so we leveraged the fact that we have kind of this large traffic to build out this panel and we decided to do 50 cents for charity of choice versus money because we wanted to make sure people weren't B.S. like, we wanted to make sure people were answering and to make sure they're answering with very high quality. It was really important that, you know, no one's going to like, BS a survey to give 50 cents to, you know, the Teach for America. So that's why we built it out that way. So I don't know if that completely.

Answers your question, but let's go down the road. So none of you first, so you mentioned at some point that as you scale the organization to identify people who scale and drone to the those, which is a very observable viewpoint.

Like if I think of what do.

I need to do to become a person?

So that's a great question. I would. Oh, sorry. The question was, is that, you know, as a manager, you're trying to figure out, you know, who can essentially scale and grow. But as a person in the chair, how do I make sure that I'm scaling and growing? And in, I would say again, a lot of it is figuring out if you actually enjoy people management as it scales and grows. It's not for everybody. We put all of our engineers through this engineering management screen and we have a very nice tech track for people too. And so you can either scale through a tech track way and, and scaling the techtact way and growing that is like, is about gaining knowledge, taking more ownership of more services, showing, mentoring other people on the team and kind of being able to take tech leadership. But then there's also the people management track. And you have to really decide if it's something that you enjoy. But if you're in that role, like you brought somebody in to lead, then there's a question. If you think about it, if you're managing five people, then you're having a 20% impact on like five different things. Right. When you're suddenly managing like 50 people, you're having a 2% impact on 50 different things, like whether. And so you have to be excited and feel excited about. Not necessarily like, get like there's something so satisfying to like coding and pushing it to production and having it work. Like you have to feel satisfaction over influencing other people. And so part of it is like making sure that that's something you're excited about. And being a good people manager is about removing roadblocks for your team, making sure that they're getting what they need to get their job done and then making sure that you're communicating to them about what's like the most important things that you're working on. And so like that's not enjoyable to everybody, you know. So I think that that also becomes a question. Is also starting to be honest with yourself of like what you enjoy. Joy. And then depending upon that, like working on those skills. And I'm sure you can add to that.

Yeah. Although we'll optimize the questions.

And for targeting, do you have any.

Way to collect all these people?

If I ask you to target demographic.

In a specific country that has certain.

Characteristics yeah, do that. And if you can. So the question was with the panel, can you target based on specific demographic data? So the answer is yes, and we collect it through a survey. So we asked them which is actually the best way to collect the data. And then as a buyer you can choose different demographics. So if you're trying to launch a product for young women, you can target young women and, and ask them here for a second.

With the 2016 election coming up, do you have any plans with entering like a polling space surveying? Very similar.

I don't have you. Did you see me? Sorry. His question was if we have anything in entering the polling space. But did you watch Meet the Press? Okay. Oh, well, we are polling. Yeah. So we have launched. We just. Mark Blumenthal signed Yesterday, joined our SurveyMonkey. Yesterday was running Pollster.com, we have John Cohen, who was the head of the Washington Post polling. We were the only company to correctly call the UK election. So we basically used the end of the survey page and as well as our panel and we asked a bunch of questions. And if you think about it, our sample size is so much larger than calling people on the phone. So yeah. And so now we're. We have a partnership with NBC and to poll on certain topics. And so it's something that again, somebody asked the question around how do you think about global with. In terms of. One of the things is the brand awareness side. So if you start thinking about election and polling, if we can show that the data is accurate and that our sample is good, it helps obviously with that audience panel. But it also vastly helps with building that brand awareness and that confidence that SurveyMonkey is not just this tool, but actually is a data business over time. And building that confidence as you know what is your kind of next step from an innovation perspective. And the polling is an important piece for that for us. It's really cool though. You should look at the UK article. It's pretty exciting.

Chris, you were saying about seven day, 28 day readouts.

Oh, sorry. I said there's more to do. Seven day and 28 day readouts. With every feature or everything that gets built, it's important to say how it did. And there's some features you built that didn't work and that's okay. Like then you have to get rid of it because just adding clutter for the customer is like not a good idea.

Taking a step back to when you first joined Survey Monkey, it seems like.

There was a lot that needed to.

Get done on all aspects of the Company. One thing, something that Reed talked about earlier on in this class, figuring out which fires to let burn and which.

Ones to put out. How did you do that? So the question was, there seemed to be a lot to get done. How do we decide which fires to burn, which ones to put out? I'd say we were in a very advantageous position in that the business was doing really well. So we could take an approach to say, let's break up the problems. And I think that is, from my perspective, the most important thing is like, what are the problems that are the most important, which is problems can you let go for longer? And so a lot of it was collecting customer feedback and understanding which parts of the application, whether it was through a combination of user testing, a combination of looking at the quantitative data of what people were doing on the site as well as all of the customer feedback and surveys of our own customers, is understanding what were the pieces that the customers were telling us were the most broken. And so, again, globally, we saw customers couldn't pay. That was a huge problem because people wanted to use the store service. Then we saw that people couldn't understand the data. So focusing on analytics was the most important. People actually loved the way that you could create the survey. Like, the biggest feedback was, oh, this is so easy to use. It's so easy to use. To get my survey started, that was the last thing we attacked. So it was very much looking at both qualitative and quantitative from our customers to drive. What were those fires? I have a question about, really. Most people, when they're starting out, are advised to stay in this area, especially because the network is very important. But you said you spent a couple years abroad and it was an amazing experience. Yes, sure. So the question was that, you know, a lot of people recommend staying here locally. And then I made the decision to move to London and how did that affect my career? So I did start here. We started Evite in kind of our. Our dorm, actually, here at Stanford and then and stayed here for a number of years. Evite sold to a media conglomerate called iac, which owned Ticketmaster City Search Match Expedia at the time. And so that was down in la. So I was in LA for a couple years. But for me it was that, you know, I really wanted to gain that international experience. Experience and felt and also just on a personal level, really wanted to live abroad. And so, you know, maybe from a career decision, it may have been better to come back to the valley. It was 2005 and, you know, join a startup Here. But you know, you have to balance, I think what you really enjoy personally and professionally. And you know, on a personal level, it was the best decision I ever made. I met my husband. So.

You mentioned the first hire you had was user experience. Most people don't find surveys extremely fun or exciting to do.

Right.

So what kind of changes did you have to do to make it so that it's as pleasant as possible?

So the question was surveys. People don't necessarily find surveys fun. So how did you make it as pleasant possible given that we hired the user experience designer. So again, a lot of it was around the creator was that how do we get that person right from that logged out experience, whether they were coming in through search or they were coming in through having taken a search and drive that person into the funnel. So how do we even onboard them? Like, what are you trying to do? And trying to help that person along that experience. But as I said, the biggest, biggest thing we focused on with the user experience design perspective is helping people understand their data. The more people used our product to understand the data, that lowers our churn, so improves our engagement and retention. So that is a very hard problem.

Given that you like converted the team, the technical team, from like two, three engineers to what it is now. What were some of the highest leverage organizational strategies that you used at the beginning when you got there?

So the question was, is, given that we moved the team from 2 to 150, what were some of the organizational strategies we used to scale basically that.

Were very high leverage. So something that took me minimal effort.

For like a very big.

I was like, goldberg, hiring you?

Yeah, I mean, there's nothing with hiring that is a silver bullet. I mean, I, I just like you have to slog through hiring each person, getting them onboarded. I would say one of the things that had the biggest bang for the buck was onboarding is like getting people onboarded correctly so they understand the mission of the company and understood what we were trying to accomplish, where we were going from a vision. Perspective is one of the most important things that you can do to get people kick started when they're there structurally. And then I would say the other thing that we really like to do is that we always like to have. There has to be some cadence that everybody's agreeing to that the business should operate. In our case, we decided, decided on quarterly. And so every quarter it was, what are we going to get done this quarter? And no matter Whether we were two people or 150 people, we still keep that cadence. Where Everybody says this is what we're going to get done in the quarter. And I think that starting to build that in from the beginning is actually pretty important because then people get used to that sort of rhythm of the business and that helps the organization sort of know what to expect in some time period. And what you can do goes from this to this, but it's still the fact that your communication is. So every quarter, I get the entire team together and we quickly go through key things we did last quarter and like, great things coming up.

I'm going to ask the last question since we're actually over time, but it's still worth asking. You have a phone line to call your younger self six years ago.

Oh. At the beginning of survey Mikey, what's.

The one thing or two things you would tell yourself to do differently?

Okay, so I know I saved a real fun. Yeah, I know. So what would I tell myself to do differently? So I would say, and this is a very technical thing, I would say the biggest thing we didn't do fast enough was improve our deployment systems. And I just. And I survey my team every single quarter. And it just. As the team started to scale and grow, it was the thing that was coming back in the survey the most, which was make sure that you that like, for us to get our code from keyboard to production was just too long. And so we should have just automated all of the deployment systems right from the start. So I would say that's one of the things that caused way too much engineer frustration that I should have dealt with at the beginning. The only thing I also have to say is obviously, I wish. I think I did a pretty good job, but cherish Dave even more than I do.

Of course. All right, well, let's thank Selena for spending a friend.