What is Hadoop? Other big data terms like MapReduce? Cloudera's CEO talks us through big data trends

So who are you?

My name is Mike Olson. I'm the CEO of Cloudera.

And you got your company right behind you. Yeah.

The team is hard at work today.

But I'm really happy to talk to you because the whole startup industry is seeing a change in the kind of databases that we have to deal with. I mean, Twitter and Facebook and Google Buzz and all these things are spitting out huge databases that if, even if you wanted to study it, let's say you worked at Procter and Gamble and you wanted to study, or I have three or four months worth of Twitter data to see what consumers are talking about and thinking about. You need to have a literacy of big databases. People keep hearing these terms like Hadoop and MapReduce. So I just wanted to start there. What is Hadoop? What are some of the terms and some of the literacies that we need around big data?

Yeah, no, I'll be glad to talk about it. I think it's probably most useful to start with where this technology came from, Right? Yeah. Back in the early 2000s, Google had this bad problem. They were ingesting the entire Internet frequently, every couple of days, and they needed to process that data. And there was no way to basically build an index over the entire Internet in a reasonable amount of time using any commercially available tool. They had documents, I.e. web pages, and then also their own logs that they were generating. They wanted to chew up and mine for value, right? And they couldn't go buy a document management system that would solve a problem for them. So they designed and built this new infrastructure that they called MapReduce. And it's a pretty simple idea if you're a computer geek, right? I mean, you buy a bunch of commodity rack mounted servers, right? Every one of them has some local disk attached, everyone's got a reasonable amount of cpu. You spread the data out among all of those, even some replication, right? So that in case you lose a server, you've got another copy of the data somewhere, you chunk it up into blocks, you spread it across all the servers. And so now you stored it very cheaply and pretty reliably, right? But the best thing is that you've got this processing infrastructure, you've got CPUs attached to all that disk. So if you want to do some indexing, if you want to do some transformation on each of the servers, you can use the local CPU to chew over the local copy of the data. That is, you get this massive data parallelism. You don't have to read all the data and funnel it through a Single processor. Right. So that was the basic idea.

And you used to work at Oracle, so you saw the old world where everything was centralized, right?

Yeah, that's right. I mean, back in the day, right, if you had a big database, what you did was you wrote the biggest check that Sun Microsystems would take from you. Right. And you got the largest sunbox that money could buy. Right. And if you had any cash left, you gave it to Oracle. Right. And you put that software on that server and everybody went to that centralized resource for all of their operations. Right. And that was fine while data was at sort of human scale when most records came from, from people. Right. But what's transformed, what's happening these days is there's a lot more machine generated data, logs, sensor data, imagery, stuff streaming out of devices. Right. That scales up with Moore's Law in two ways. Right. The more of those devices there are in the environment, the more data is getting generated, right? Yeah. And the faster and cheaper they get, the faster and cheaper data gets. Right. So you can't keep up with that kind of large scale distributed data generation unless you have large scale distributed data storage.

Right?

Right. So yeah, you're right. I was.

So we start with MySQL on a box and then we start going, our data is scaling or we already know it's got scale. Right. We're trying to build another Twitter or another Facebook and we need to plan for a huge amount of scale. How do we approach, how do people approach that kind of problem? So the big web, the old one was go get venture funding and buy a big Oracle box. Right. Like you said, the new way is.

Well, people still use relational datab for lots and lots of workloads. They're really good if you've got predictable queries that run over pretty structured data, tabular data. Right. So there's a lot of workloads that look like that. All of the bi, all of the enterprise data warehousing stuff that happens in the market, that's a long term value, it's not going to go away. Oracle, IBM, those guys have long term markets they're going to sell into. What's happening now is the data that people want to work with is getting more complicated. So you said before, gee, I'm P and G and I want to know what people are saying about my brand. I'm going to go get a Twitter feed that talks about me. I'm going to ingest that stuff. Right. That kind of free text, unstructured data, complex data doesn't fit nicely into Tables. And even if it did, the kind of questions you want to ask are give me a sentiment analysis of the natural language that's captured in all of those tweets. SQL doesn't have words for that. You need a different kind of data platform that's able to look at very large amounts of information and to do new kinds of processing over. You mentioned before that I used to be at Oracle and that's exactly right. When I began my career in 1986, I was a relational database guy. Right. I worked at database companies for my entire career. Sleepy Cat sold to Oracle in 2006 till 2008. In 2004, Google published a paper about MapReduce, that technology I mentioned before. Right. I read that paper in 2004 and like almost all the rest of the database industry, I thought it was a joke. I just didn't understand what this platform could possibly be about. Didn't do transactions, didn't have any schema, didn't support SQL. I didn't know why people would use that. After I left Google and was looking around for the next thing I wanted to work on, I talked to Jeff Hammerbacher, one of my co founders at Twitter, at Facebook Rather, and Christoph from Google and Amer from Yahoo. I understood suddenly how those companies were doing this kind of large scale data processing and analysis. Not transactional processing, but data processing on this scalable platform. And the lights came on. Right. So the four of us decided that we would band together and start this company. We think that this problem, complex data at volume, is coming not just to the big web properties where it was born, but to banks, insurance companies, health care.

It's a work group problem now. Right. If you're working on a small team inside Procter and Gamble, you're probably working with 10 people. But you still need to study huge amounts of data.

Yeah.

Back in the day just to come out with a new brand, you know?

Yeah, that's right. I mean, you remember when terabytes were hard to get.

Right.

Terabytes are hard not to get these days. Right. I mean, you actually have to actively throw data away if your storage system is limited to that size.

So tell me a little bit more about the technologies that you're seeing that the bleeding edge is using and how does Cloudera fit in that? But so like what is Hadoop, for instance? I keep hearing Hadoop. Hadoop, Hadoop.

What is so Hadoop is the open source implementation of the technology that Google invented, the MapReduce stuff that I talked about before. It consists of two things. It's got a large scale distributed file system called the Hadoop Distributed File System or hdfs. And that's a disk on every server with the software infrastructure to spread your data out among all of them. And then the second part is MapReduce. That is, once your data is spread out like that, you need plumbing to push the code down to the data so that you can run your analyses in parallel across all of those servers. Those two components are really the backbone of Hadoop. Now there are a bunch. And by the way, Hadoop is an Apache software project. The really important early contributions came out of Yahoo. Yahoo invested early and significantly in the platform and did a great job at basically creating momentum. Yahoo, Facebook and and the other big web properties run this thing at scale in production every single day. Yahoo's got 25,000 servers. Facebook has two petabytes. Google runs 100,000 jobs for 10,000 applications. And map.

We use Rackspace on our mail product to study our logs.

Yeah, yeah, indeed.

Because we have hundreds of thousands of mail accounts and they spit out a lot of logs that you guys study.

Yeah, that's right. And as the volume increases, you can just lob another couple servers into the cluster and scale up with it. Right. So it's nice, flexible, elastic and kind of the modern cloud computing way. Right. So Hadoop is that open source implementation I mentioned a while ago. I'm an old line database guy. Right. What's interesting to me right now is that if you asked me in the 1990s or the 1980s, any question I would have answered relational database. That was the technology used for data management these days. The answer is different. That is it guys are buying Hadoop. They're looking at sort of NoSQL products, you know, distributed hash tables, other kind of storage engines. Columnar stores are getting very big. The rdbms, conventional table stores. They remain a really important, highly valuable market. But you're starting to see data centers choose technology that's aimed at problems rather than just this old mature technology and trying to shoehorn every single problem into it because there just are problems that don't fit.

I've heard memcached for a long time. Mark Zuckerberg was talking at Davos as we walked around about how he solved some of his early scaling problems by using memcache. D Tell me a little bit about what that is and how that fits into this world.

Yeah, you've been.

Is that still a bleeding edge technology or is that old hat now?

No, it is absolutely the ubiquitous invisible infrastructure that makes the web run. So with 80% certainty you go to visit some website, you're seeing data delivered out of a MySQL database because that's what everybody uses for sort of relational data storage behind their websites. But when your infrastructure needs to scale, having to query a disk every time you want records doesn't work. So some years ago an open source team basically invented something called memcache. And it's a memory layer between a relational database and your web server that's able to basically service all your reads, fetch data on demand, push writes through incrementally, but it's basically the memory layer that glues together a website with its relational backing. Now it works against a lot of other stuff, but it's absolutely ubiquitous in the web world. There are a few companies commercializing it. One of them actually is another Excel portfolio company called northscale. And it's got a bunch of the key project members. But yeah, it's another piece of new data management and data delivery infrastructure that everybody's adopting.

Yeah, what else are people doing? What other technologies are they using? I saw on your website Django, which I guess is a programming language and a framework. Right? Tell me a little bit about that, am I right?

So yeah, you're right. I think the really interesting thing that's happening in most data centers these days is first open source technologies generally are just taking over the planet. If you're building any kind of developer focused infrastructure, if you're building a service, you want a program, then you just go get open source tools. Right. And Django is a good example. But you know, there are a bunch of languages, php, Perl, Ruby, Python that people like to use to build these systems. There is storage of processing infrastructure, things like Hadoop that are very popular. Of course, MySQL is absolutely ubiquitous. Right. And I think that that's going to stay true for the long term. So the data management delivery platform that everybody's using is absolutely coming out of the open source world these days. Companies like ours need to make a living, are looking for ways to take advantage of that open source infrastructure and build a valuable business. But I do think that the really interesting, the best infrastructure software, especially for web facing properties, is all open source.

These keep going because this is interesting stuff. What other tool sets are you seeing people use in bleeding edge data studying and data management? So I keep hearing about, for instance MySQL drizzle. Is that something that's on your radar screen?

I know Brian Aker, who's one of the key contributors to drizzle. Right.

Tell me what drizzle is, first of all, because people don't know what that means.

It's a reimplementation of MySQL is the fair thing to say. So the team believes that they learned a bunch Of Lessons Building MySQL in the first place, and they want to apply those lessons to a modern modular architecture and basically roll out a new MySQL even better targeted at web applications. I know of those guys, I don't actually follow their work that much. It's sort of orthogonal to the stuff we're doing. There are actually a lot of other projects getting started that are kind of interesting. There's this NoSQL space space that is. Think of it as distributed hash tables.

Tell me about that. What's going on in that space?

So the idea is, instead of typing a SQL query and fetching records from a database, what I want to do is I want to go look up Robert Scoble, right? And I'm going to. I know your name, so I just want to go pull your record by name out of a store, right? That's a way that my program would like to behave. So there are tools like MongoDB and CouchDB aimed at that new kind of access. Actually, go back 10, 15 years, directory servers, LDAP was a protocol that was invented to do that. And you're now seeing a lot of open source implementations. Some other stuff that we track very closely is companion projects to Apache, Hadoop. So, for example, Hive is now a SQL implementation that lets anybody who knows how to write a SQL query talk to a Hadoop cluster and kick off MapReduce jobs in parallel to answer questions. There are storage layers that complement Hadoop. So there's something called HBase that sits on top of HDFS and gives you that kind of.

What's hdfs?

I'm sorry? That's the Hadoop distributed file system. So that's one of the two pieces of Hadoop. Hbase gives you the kind of key value storage. I just want to know about Robert Scoble. Give me his record on top of HDFs. So it provides that abstraction. What's really happening? And you know, from my point of view, as you know, as a guy who spent decades, programmers and systems people are thinking about storing using data in new ways, using new tools. It's a pretty exciting time to be starting a company because I can go sell this now. I don't have the resistance that I would have had 10 years ago because it's not an arcana it seems like.

By the way, that this seems be a competitive advantage for Silicon Valley because the only way you get experience on any of these tools is to work at a company that has the infrastructure to have dozens of servers, right. Or dozens of machines and big data sets. That doesn't happen too many places in the world.

Yeah, I want to be careful. So, you know, the things I want to say is first I have to acknowledge the Hadoop community, right? And that's a global distributed community community, you know, 185 developers scattered all over the globe. So you don't have to live in Silicon Valley to do interesting work. And that's.

But I don't meet guys like in your company that worked at Yahoo, worked at Oracle, worked at, you know, Facebook.

There is still Facebook and Google. No, there's still, there's still some geographic advantages to being here. You see clusters of innovation in New York and in Boston. But you know, even here in the US we've long bewailed the fact that you don't have this kind of startup energy in Chicago or in Kansas City or in Denver some, but not like the Valley. So for sure there are still advantages to being in the Valley, right? But as the network gets faster and as cell phones get ubiquitous, it matters less and less where you are to do a particular job now, bringing everybody into one room and actually getting the advantage of physical presence in addition to.

But where I was going with that is, you know, I used to be in the Microsoft world for the last 15 years and a kid could get a copy of SQL Server, put it on a single machine and learn pretty much everything he needs to know about doing databases. Now this new world, you need, you need a data set and you need multiple machines to understand how to scale something up like a Twitter or like a Facebook or like that.

Sure, that's true. Right. But you know, these days I'd argue that kid has an even bigger advantage. He doesn't even need to go buy a computer, right. I mean you can buy actually from Rackspace, from Amazon hosted infrastructure, very cheap to spin up a little cluster and try it out. We've got virtual machines that you can download from our website that will run a three node cluster on your MacBook. Right? I mean, uses virtualization and all that stuff. So experimenting with these new technologies gets easier and easier as cloudy kind of technologies start happening.

Are you seeing more people doing that? You know, starting up a Rackspace cloud, you know, six or seven machines shoving Twitter data into it? I know. Jesse, stay in Salt Lake City is doing that?

Yeah. No, it's unbelievably easy these days to spin up little clusters with that kind of hosted infrastructure. And I'm really glad to see it, because if I'm going to be a software and services company around this Hadoop platform, all I care about is that it'd be very easy for people to get and install. Right. If you really had to go buy a hundred boxes from Dell. Right. That'd be a pretty big impediment. Right? The fact that you can do your experiment by whipping out your credit card and doing a transaction over the Internet and instantly provisioning a cluster for yourself, that's a big deal, Right? That's important to us.

How do you guys fit into this? What do you guys do for this new data world?

Well, so a couple things. First, three of the four of us who founded the company were substantial Hadoop users. So Jeff and Amer and Christoph came from Facebook and Yahoo and Google and ran big clusters. So got religion early. Right. We hired actually Doug Cutting, who was the guy who created the Hadoop project. He works for us as well. We're doing a few things. We contribute to the Open Source project. We believe in the value of it. We want to make it better. We view ourselves as members of a great global community doing really innovative work, and we're pleased to be in that community, but we're just a member. This isn't like when I ran sleepy cat berkeleydb, where the company wrote the software like MySQL wrote MySQL. This really is a distributed. No single organization owns the copyrights. Right. It's an Apache project. And all the IP goes to the Software Foundation.

Yeah.

So we contribute to the Open Source Core, we package it up in an easily consumable way so that it's easy for an enterprise that doesn't employ a lot of PhDs from Stanford to install. So it was early adopted at the big web properties, where there's lots of IT muscle. Right. But if you're a bank or an insurance company, you might not have that many sophisticated people in your data center you want to be able to install. In addition, we build proprietary products that complement the Open Source Core. So Cloudera Desktop is a great UI for exploring your cluster and interacting with it. We'll be rolling out additional products aimed at enterprise problems over the course of the next year, and those products are what we're going to build the company on. In addition to a combined bundle, the Open Source Core, plus our proprietary ip, of course. We offer training Consulting, technical support for enterprises using Hadoop. And that's really the business that we've been in since we opened the doors. We help people, we help ordinary enterprises apply some of this Google magic to problems that they've got.

Yeah. Do you have any competitors?

No one is doing what we are doing with Hadoop. Right. But of course others have noticed that data is getting more complicated, volumes are going up and valuable problems are there to be solved. So you know, the big established vendors are absolutely paying attention to the emergence of this new kind of data and these new kind of problems. We think that this new platform, which was really designed for, for these problems, has a bunch of advantages, but absolutely others see this as a valuable market in which they're going to be active.

What kind of mistakes do you see people making as they move from the Oracle or the SQL Server world, or even just the old MySQL world into this new big data world?

You know, the key thing is that Hadoop is different in quality from those older products. So Hadoop is really good at exhaustive batch analysis, deep processing of data at scale. Right. If you've got a data warehouse, you're used to flying through your cubes interactively and very quickly getting answers to questions that are sort of pre computed. Right. Those are two very important things. You want to do both, but they really are different. If you've got an existing enterprise data warehouse problem, it's going to map poorly to Hadoop. Right. Where Hadoop will really shine is as these new data sources come online, as you want to understand them, digest them, and maybe even load summaries about them into your cube. Hadoop is the way you do that. Right. So you don't want to say, I've got a problem that I've been solving with IBM DB2 for the last 20 years and now I'm going to port it to Hadoop. What you want to say instead is, oh my gosh, suddenly we've got all of these status updates and user generated content messages that we need to understand and we've got to digest that stuff in new ways.

Yeah. If you were working at Twitter, what would you be doing differently than they are doing today?

The Twitter team is pretty smart, so they're big consumers of Hadoop, as I'm sure you know. I like those guys a bunch. I am impressed by the analytics that they are running and you know, they're rolling out not just Hadoop, but other projects. I think they've even recently announced that they're interested in another Facebook project called Cassandra, which is yet another one of these NoSQL layers. Right. So we remain close to those guys, very interested in what they're doing, and we talk frequently. I can't think of a thing that I would be doing differently from what I am now. I might have picked more than 140 characters for tweets.

They swear that they're going to stick with it.

I actually was talking to one of the seniors guys at Twitter recently and they just had a party in the company because they'd hired their 141st employee, their 141st character. So, you know, they no longer fit in a tweet.

Yeah, that's funny. Where else do I want to go with this? Do I? You know, when I talk to big data guys who work at Google and whatnot, they actually study how the data gets put onto drives or onto storage media and really try to understand that. Do we need to understand that world at that level or is that your job? And we're going to use tools that you've built that you understand how it gets written on the hard drive and gets pulled off in the most efficient manner. And I don't have to worry about that?

Yeah, I think you want to let the specialists, you want to let the platform guys worry about that stuff. But then I'll say something else. It's still true that companies like Google and Facebook and Yahoo are several standard deviations out on the curve. Those guys are in the future. Right. We'll all get to those data volumes. But you don't have to be Google to want to use this technology. Today. A lot of people use this term, big data. I don't actually like it very much because if I go talk to companies about it, they actually say, I just have medium sized data. I've got merely terabytes.

Is that funny? I remember the days when those two.

Words next to each other when Microsoft.

Research was playing with the first terabyte drive, you know, terabyte data set, and it was so cool.

That's right. Jim Gray used to call them terabytes because they were so unimaginably large. Right?

Yeah, yeah.

No. Now that kind of data is easy to get. You don't need petabytes. What's nice about this platform is, hey, don't worry. If you ever get there, it's known to scale to petabytes. There are production clusters in use that get that big. But it solves real and important problems. It does valuable processing at much smaller levels.

Is there anything that a newbie should read up on before they architect a system. Because understanding how hard drives work and how servers work and stuff is probably important stuff to know when you're building, when you're architecting a system.

Right? I think that's right. And I don't know if you're lobbing me a softball here or not, but I'll take a swing at it. Actually, I would recommend, if your audience is curious, if you out there in the Internet are curious about how to think about Hadoop and how to build a system, go to Cloudera's website. We've got a bunch of online video that is just viewable, right? No charge, none of that stuff that trains you on Hadoop. And some of the very first episodes that you watch are really valuable. One in particular is called thinking at scale. We see this happen at almost all of the customers that we work with. Right. It is actually a new idea that you can ask a question in reasonable time that touches every single byte in terabytes of data. In fact, not just touches it, but sometimes manhandles it. Right. We as old computer scientists, you know, I got my bachelor's degree in the 80s, don't think about problems like that. We've never been able to solve problems like that, so we don't actually conceive of questions that could take advantage of them. There's a time when people are working with Hadoop when you can kind of see the light go on, they'll suddenly say, ah, I'm going to use a thousand servers, because I can. And thinking in that way is a new skill. So there's a bunch of training material, including a VM you can download that lets you play with a cluster very easily. As I think I said a while ago, cloudera.com, the obvious URL.

Are we going to see some breakthrough new ideas from CERN again in this industry? Because they're dealing with, you know, that machine over in Switzerland that's kicking off I don't know how many terabytes a day of data spreading around the world, you know, to different data labs and.

Yeah, no, it's. It's an enormous amount of raw data that gets filtered down in stages. In fact, one of the data centers that catches that data is the University of Nebraska Lincoln is using Hadoop to capture and store it. Right. There is a whole bunch of legacy scientific data processing infrastructure that matters to the physicists that are sucking data off of the Large Hadron Collider. And you're going to see that stay true forever. Right? I bet there are still critical fortran programs that are in use every single day in that space. But as the data volumes get big. Absolutely. You'll see new architectures emerge. And we think Hadoop is actually a nice one. It's a scalable storage system all on its own. The fact that it's got this built in processing, totally bonus.

Of the people behind you, who's the smartest and why?

Oh yeah, that's a nice thing for you to ask on camera. They're all tied for smartest now. We've got an outstanding team I should introduce. Amr Awadalla is right here. Amer joins us from Yahoo. He's one of my co founders. Christoph Beschillia sits diagonal from him. Christoph is the founder out of Google. And then just past Christoph with the headset on is Jeff Hammerbacher. Jeff is out of Facebook. And the four of us banded together last year to found Cloudera. We have hired a world class engineering team. University graduates, people out of VMware, Cisco, Google, engineers from companies that know how to build largest distributed systems. Because of course that matters to us. And over the past several months we've begun to build a business team that we think is every bit as good as the world class engineering team that we've built.

What's the hardest thing that they're tackling?

The hardest thing that the engineers are tackling?

Yeah. What keeps them working until 2 in the morning or 3 in the morning? Because all developers, one of my friends said all good code gets written after midnight.

And a lot of code gets written here after midnight. We're making key contributions, we think, to HDFs, improving reliability, improving scalability, contributing to the open source project.

So there are guys in this building who understand how that data gets written. Hard drives.

Oh yeah, no question. Yeah, yeah, yeah. And helping to design that for the future. In addition, we're building new infrastructure that'll make it easier than ever to get data into Hadoop. So you can imagine those web logs out there, data streaming in from sensors and analysis. So we're concentrating on that and all of our engineers spend time working on real customer problems. So we've got a good support business. Questions come in from the field, we answer them. We've got really interesting customers solving really interesting problems in network security and finance and biotech and pharma and understanding what those guys are doing with their clusters and helping them scale them up to solve the critical business problems of the domains. Take some of our time. And it's actually pretty exciting to work on such interesting domains.

Data centers are starting to Switch over from hard drives to flash memory. Is that something that changes how you guys do your work? You know, because hard drives was a little, a little needle that sprayed data, you know, as it went around. Now a flash drive works completely differently.

So a flash drive doesn't need to spin and it doesn't need to move the actuator arm to write data around. Right. But whether you have flash storage or spinning disk storage, having that storage spread across a bunch of different servers, each of them with local storage is still valuable. That is, Hadoop still lets you be data parallel and to do large scale computations over data, whether it's stored on disk or in Flash or sdram. Right. So for sure it changes the performance characteristics of any single node, but the value of large scale storage at low cost and data parallel computation is still there even if you're using more exotic storage technology, newer storage technology like Flash.

Who outside of Cloudera is doing the most interesting work? Is it Facebook?

All of Facebook, Yahoo and Google continue to innovate in the way that they use this platform to store data. They're all making important contributions and advancing the state of the art. I think there's actually a bunch of really interesting academic research going on these days. And it's not just how do you build a great big cluster, but how do you reduce the latency among machines connected on a network. You want to grow these networks to very big clusters of machines. It's important for the networks to be faster, for switching times to reduce and so on. The pace of innovation in it is actually fast again. For a few years it was just sort of incremental increases in disk speed, CPUs getting a little bit faster. But with the transition to this new commodity server rack mounted infrastructure, the kind of data center that has everybody worried about, they're deploying so many servers, they got to worry about power consumption and heat and everything else.

Right.

Building that infrastructure at scale is driving new innovation and network speed and memory speed and the way that individual servers are architected.

We're struggling to keep up with demand. At Rackspace, we're working on several data centers.

Yeah, that's great. And the more successful you guys get, the more acreage you need, the more electricity you need. I mean, that's a business that gets expensive to scale, especially when your data centers are centralized.

Well, thanks so much for spending a little bit of time with me.

It was great, man.

I enjoyed it, explaining this world because it's a really interesting world and I think it's going to get more interesting.

Thanks for the chance.

Thanks.