#273 Reviewing Our Data Trends & Predictions of 2024 with DataCamp's CEO & COO, Jonathan Cornelissen & Martijn Theuwissen

Hi, I'm Maggie, VP of Curriculum at datacamp and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead. Covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey.

On datacamp.com there's a growing number of investors that worry that AI will not deliver the vast profits. There's this PR narrative, WordPress narrative that the hype is over. At the same time, I like data, so I did some research in preparation for this yesterday and if you look at just at Google trends, the number of searches for AI, the number of searches for ChatGPT, Gemini and those types of models, they actually tell a different story. There's a continued increase in demand and so I would more lean towards. The hype is definitely not over. The actual usage seems to be very much increasing.

Welcome to Data Framed. This is Richie. One of the brilliant things about New Year is that you get reflecting about the previous year and thinking ahead to the coming 12 months. That means it's time for the annual dataframe trends and predictions. This time we've decided to split it into two parts. So today we're going to review last year's predictions in an attempt to keep us honest. Our guests are datacamp's founders. Joe Cornelisson is Datacamp's CEO and Martin Thurwison is DataCamp's COO. Since worrying about what's trending in data and AI is something of an existential preoccupation at DataCamp, naturally, Joe and Mata M spend a lot of time thinking about this. I'll be judging how successful they were in predicting trends. It's this kind of power that makes me really love my job. I just need to decide how mean to be to them. Alright, let's see how they did. Hey there Joe and Martin. Welcome back. Glad to chat with you again for another round of data trends and predictions.

Hey Richie, excited to be here.

Hey Richie.

Cool.

So we're going to talk about your new predictions, but before we get to that, I want to talk about your predictions from last year and see how many you got. Right. So your first prediction was that generative AI is going to hit the mainstream. So how well do you Think you did on that, Joe, do you want to talk us through it?

Yeah, I think this one was a little bit of a layup. It's not the most risky prediction that generative AI was going to get mainstream a year ago, but I think we can confidently say that that has happened in the last year. I think some research indicated at the beginning of the year 20 to 30% of people were using ChatGPT in the US. The latest research I've seen puts it at about half of Americans using ChatGPT or one of the other models. And that's in line with what we're hearing from clients that monitor AI gen AI usage on their networks. So yeah, I think we can confidently say that we got this one right. The one thing I would notice is that if you look at daily usage though, like we're still at the beginning of this, I think estimates range from anywhere between 5, 10% of Americans using Genai daily. And it very much skews towards students, data folks, people who are part of the AI space, of course, people in marketing, customer support. So there's a whole group of people, people that doesn't use this at all or doesn't use this on a daily basis. So we still have a long way to go for kind of mass adoption, but it's definitely gone mainstream. So yeah, I would give us a point on this one.

Yeah, that's an interesting point about there's a difference between you having heard of it and used it like once and being a daily user for sure. Martin, was there anything you wanted to add to that?

Yeah, maybe like. Yeah, I think we got the point here. Maybe it was a bit too easy. I think one thing I would add is that there's a lot of like explicit adoption. A lot of the stats are about explicit adoption, as in like you're actually interacting with ChatGPT now. There's also like this implicit adoption happening which is growing very quickly. Like the best example to me is like if you do a Google search today, it's more and more serving these like AI generated responses which obviously help as well with making generative AI more and more mainstream. And I think we're going to see much more of that as well in the future. Like that keeps going.

Yeah, that's true. So even if you didn't intend to use generative AI in some way, just like you have to search for things on the Internet sometimes, so you're going to be at least passively consuming those AI generated results.

Indeed.

Okay, cool.

All right.

Yeah, so I think I Think you got one, but we'll give you the point for that prediction. So next prediction was that AI literacy becomes a universal skill. So talk me through how well you think you've done on this. Martin, do you want to go first this time?

I think if you look at it from, like, datacamp's perspective, like, it's definitely the part of the curriculum where we're seeing the most growth. I think we've been growing over, like, 60% year over year there. So that's really like, literally like hundreds of thousands of learners starting courses. We also see, like, quite a lot of, like, users within, like, large enterprise clients. So, like, more and more folks are taking up AI literacy, building it as a skill now. At the same time, I think there is, like, well, still a long road to go. I also think that there's definitely in the eu and I'll get to that, like, for my future predictions, like, there's still, like, some hurdles to take there. So I would say, like, we're pretty good on this one as well, but it's definitely not yet a universal skill.

Yeah, I agree with that. I'd give us maybe half a point or 60% of a point or something, because on the one hand, AI literacy is the fastest growing skills area. On the other hand, is it a universal skill, as in most people are AI literate? I think the answer is clearly no. So we're on the way towards that being true, but I think it's not yet quite as true as we expected.

Maybe. Yeah, absolutely. I suppose it depends on how you word it. A bit like there's millions, maybe tens of millions, hundreds of millions of people interested in it. It's just there's billions of people in the world and. Yeah, so the majority is still not illiterate yet. All right, so lots of work to do there. I guess we got our work cut out at datacamp. Try and teach the rest of the people. Okay, so prediction number three was that the data space and the software space are going to overlap more. Do you want to talk us through what this means and how you're going to judge yourself on this? Joe, do you want to go first this time?

I think this definitely happened. It's hard to quantify. It's hard for me to prove that, but I think there's definitely been a convergence of the data and the software engineering space in a way that hasn't happened really before. There's a lot of software engineers building AI products. There's the role of the AI engineer. It's one of the New roles in the marketplace, one of the fastest growing roles, and we've seen this at datacamp as well. So historically datacamp has been very popular with data analysts, data scientists, and especially for more technical learners, they aspire to become active members of data, the data community. More recently we've seen a lot of software engineers coming to the platform looking to build skills, for example, build on top of llama, build and tweak their own LLMs, or just understand what it's all about. And so I think we've clearly entered a new area where software engineering will be much less deterministic going forward. And the skill set for software engineers is shifting towards understanding AI. And logical reasoning skills will still be important, but I think creativity and AI skills will have a huge premium for software engineers in the next few years to come. So I think it's hard to quantify this, but it's definitely happening. So I would give us definitely a full point for this one.

Okay. Yeah, it is interesting how just the cool factor or the hype from AI has brought in a lot of software engineers into this space. And so, yeah, there is a lot more overlap in that way. Martin, was there anything you wanted to add to that?

Yeah, maybe like on another perspective here is that I think AI is causing that the data space and the software space is like converging more and more towards like a common language, like having more this universal tool that both groups know how to interact with, which naturally brings them closer together. You look at for example, like Byton and like Byton was doing like Python's most heavily used like in software engineering as well as in data science. But they still have their own libraries, their own frameworks and so on with. But now it's something like prompting. Both groups use prompting. Like you have something that's much more universal. It's often based on the same natural language that people use. So I think that's also like playing a role there and like putting these groups closer together.

Nice. Yeah, I like the idea of having a common language just to bring different groups closer together. Nice. Okay, I give you the point.

So Richie, question for you, because I totally agree with Martin. Python is kind of the, the universal language that brings software engineers and data folks together. But JavaScript is still very popular amongst software engineers. And is it going to become more popular amongst the data community then you think?

Yeah, so I think people have been trying to make JavaScript a language for data science for years just because it is a popular language and you can sort of do some things with data There, I think it's going to become more popular. I don't think it's going to assert Python anytime soon. Like a lot of the stuff isn't very natural for working with data. You've got no objects that are a bit like a data frame. So yeah, it just makes things harder. So I think we're at two and a half out of three so far. And the next prediction that you made was that generative AI is going to move from prototype to production. So I guess in 2023 we saw a lot of companies experiment with generative AI and the idea is that this year things are going to move into production. So do you think that's happened?

Yeah, I think so. I think if you look at the last year, every self respecting company has launched AI features. At a minimum they've added a chatbot. So that also points to how much potential there really is for truly integrating AI into the product. And then at the same time, if you look at like startup companies in yc, in techstars, the vast majority of them are AI first and they're building AI first alternative to some of the existing software companies. So I think we're kind of in the classic can the startups get distribution before the incumbents get their act together and innovate type of question. So it very much feels like that definitely shifted in the last 12 months where 18 months ago you saw a lot of demos and prototypes and now everybody's actually using it and implementing it. So I would give us a thumbs up for this one.

Voting for your own points? Yeah, yeah, so I agree there's definitely been a lot of AI features, AI products been released in the last year. So it just seemed like, yeah, there's a lot going on in the space, like stuff actually hitting production. Martin, you look like you wanted to say something.

Well, I was just going to say like, I think my mailbox would agree. If I look at my mailbox in the last 12 months I probably got like every week like some company announcing like some type of AI implementation into their product. So yeah, I definitely think we went from the pilot phase to like, hey, it's now in production.

Yeah, certainly I get a lot of emails too about, yeah, companies trying to advertise all these AI features. Actually, I guess my, my big query on this is bigger companies tend to move slower. So are these big companies enterprises, are they managing to get things into production yet or do you think it's just smaller companies that are a bit more nimble?

I actually think the surprising thing about this Kind of innovation wave is how fast the incumbents are innovating in almost every sector. Google used to be the exception to that rule, but it seems like they've gotten their act together. But if you look at some of the most innovative implementations in production, they actually come from the largest companies out there. There's certainly some startups that are really taking off. But I think compared to other waves, it's shocking to me how fast incumbents are moving and there's obvious reasons for that. But I think it's worth pointing out it's a bit different than other kind of platform shifts.

Yeah. And what even says, like it's outside of the typical technology players, like I'm in the market for a car and like the amount of AI advertisement I get when I walk into like a car shop, like is unbelievable. Like every car seems to be equipped with AI. Let's not go into the details what it means, but it's actually pretty fascinating that like from a, for a non tech, not traditionally tech company.

Yeah, certainly like a car sales room, you don't think of it as being like a sort of tech hub necessarily. But yeah, the fact that they're pushing AI as well means. Yeah, it's gone everywhere. Okay, nice. All right, so we give you that point and the next one was that it was two parts of this. So you said that image generation AI is going to mature, go mainstream, but video generation AI will, won't hit the mainstream until 2025. So I guess, yeah. Martin, do you want to take the image part of this then talk us through that?

Probably if he would have scored this like two months ago, we would have done a little bit worse. But in the last two months like you had both OpenAI with Sora and Google with Veo, which is, which both are actually pretty impressive. But at the same time, like, okay, I haven't seen a lot of video generation being like productized. So like the tools are nice, they're not fully accessible yet to everyone, what they can do. And like if you look around the corner, like it looks all very exciting, it looks very promising. People can directly see the applications if you think on social media, if you think in the marketing realm. But at the same time, like I haven't yet encountered it actually being used like marketing teams having their hands on it. So I think the way that we, that we made the prediction of okay, will it mature? It looks like it's like what you can do with lightning, what you can do with the angle points, like is very cool. Today at the same Time, it's not yet in production, it's not there for the masters yet the same way like a chatgpt is.

Yeah, interesting. So with video is actually, I think back in January when we did the 2024 predictions, it was like a week or two later that OpenAI announced Sora and it was like, ooh, maybe this is coming sooner than we thought. And it sort of didn't really hit until December. So yeah, I think the tools are kind of there. They're not really been used yet. So yeah, I think certainly with video, it hasn't hit in 2024, really. So yeah, I think that part of the prediction is right. So I think I'm going to give it. I'm going to give you the point again for this one. Nice. So next, one next prediction was that AI hype is going to fade and AI tools will just be seen as software.

Yeah, I think I could go either way on this one. I think if you go back to the summer, there were a bunch of articles claiming this and saying, hey, the hype is over. The profits are not there. There was this article in the Economist a couple of months ago and I quote, they were saying the Silicon Valley tech bros are having difficult weeks. There's a growing number of investors that worry the that AI will not deliver the vast profits. So there's this PR narrative, a WordPress narrative, that the hype is over. At the same time, I like data, so I did some research in preparation for this yesterday and if you look at just at Google Trends, the number of searches for AI, the number of searches for ChatGPT, Gemini and those types of models, they actually tell a different story. There's a continued increase in demand. So I would more lean towards the hype is definitely not over. The actual usage seems to be very much increasing and most of these companies think about perplexity and so on, are still on an exponential kind of adoption curve. I'm obviously biased and I'm kind of a tech optimist, but I would lean towards the hype is not over yet. What do you think, Martin?

The same, actually. Like I'm referring to my mailbox again, but I don't think the hype is over at this point. I don't think we're there yet where it's just seen as software. It still has that magical feel to it, like Microsoft Word is Microsoft Word, Microsoft Excel is Microsoft Excel. But at the same time, like when you talk with people about AI, when you talk about ChatGPT, like they talk about it in a Different way. Like clearly it's not like in the software category for them. Like it's in this other undefined, maybe yet tool category.

Yeah. So I think that this is a bit of magic, particularly for people who have not been like using this constantly for the last few years. For things like the AI generated, like Google search results that you mentioned before, does that still count as, as magical or do you think that's just going to become, oh well, that's just what a Google search shows you.

I think it'll become like that's just what a search will show you. I probably argue that people are probably already used to it by now. They don't think of it as like, okay, this is like some kind of AI wizardry. It's like, okay, this is now how I get my search results.

Yeah. So I think some parts are getting normalized even with the broader public. But at the same time, if you look at the announcement of Google's new video generation model, I think that's still blowing a lot of people away in terms of its power. So there's continuous innovation that still generates additional hype in my mind. So I would actually say we got this one wrong. The AI hype is still here.

It's cool that AI is outrunning its own hype, I guess and still saying cool or out running on we anyway. But yeah, okay, I think you've talked yourself out of a point on this. So okay, failure there. I think our first failure of the day, AI is still hyped and cool. Nice. All right, so next prediction is that narrow AI agents are going to become popular. Actually, do you want to talk us through what is a narrow AI agent first? Yeah, Martin.

Well, in short, I would say a narrow AI agent is like an AI agent that's used for a very specific task. So let's say for example, that I would like to run some performance marketing on a new course by Data Camp and I want everything to be set up in the well known platforms like Facebook, Google and so on. Like I give it to this AI agent specialized in performance marketing on Google. If you want to go very specific and like they do the task for me as an example and then to come back to the prediction, like I, I don't have such an agent yet, so I think we have a miss on this one as well. Do think that talk about AI agents is more popular than ever and again it definitely to me picked up in the last couple of months. It's one of those use cases that entices people. It makes you excited this idea of, oh, I have all these boring tasks and I'm going to be able to outsource them to this magic agent that's going to do it all for me. So it's definitely a fun thing to talk about. And with the advances that keep coming up, people, this hype is building, like, are we close yet? Like, are we there yet of having such an agent? So yeah, maybe also like some of the biggest stories this year involved like these semi agents, like if you think of Devin, like the AI software engineer. So we're not there yet, not even like in the narrow point, but like the hype keeps building. It's like fun subject to talk about. So. Well, maybe next year.

Yeah, certainly AI agents have been a recurring theme on data framed over the last few months. A lot of people getting very excited about it is not quite mainstream popularity yet. So yeah, I think this one's a miss again, I'm afraid, because. Yeah, yeah, like you said, the agents don't exist yet. We haven't got our own personal AI Butler. All right, so next prediction was around. Investments in data governance have a higher roi. I'm not sure how you even go about measuring this, but Joe, do you want to take a stab at this?

Yeah, this is a tricky one. We should think about how we're going to measure things for next year's predictions, because this is probably true, but it's hard to know and I haven't seen really good research on it, at least not recently. So the general theme here seems to be data governance. Data quality has always been important, but it's even more important as you have Genai and the ability to do more with that data. So I think it just logically has to be true, but it's hard to measure. But if you look at some, some implementations within larger organizations and how they're starting to leverage the existing data they have, it has to be true, right?

Absolutely. No one seems to want to publish numbers on, hey, I invested in this data quality initiative and this is how much money I got back from it. It does seem to vary a lot from one project to another. But I agree that because there are a load of generative AI applications and features out there, those wouldn't work unless you had decent data quality. There must be some useful stuff happening there.

One anecdotal thing I can share from the datacamp slide is that for sure, in the last year I had more conversations around our data governance curriculum and everything around that compared to in the years before that. So it definitely is A team. And it became more of a team in the last 12 months at least. If you think about upskilling people in the wise of data governance, that's very.

Interesting that people are actually taking this seriously now, because I know back years ago when I was creating courses for datacamp, it was always very difficult to get people excited about things like data quality because it's not the cool part of data science, but it's just incredibly useful. Okay, so I think I'm going to give you the point. Just even though we can't measure it, it does feel like data governance, data quality, these things are incredibly important and they're only getting more so. All right, we have got another prediction. So we talked a little bit about this before, but R is going to decline in popularity as a language and then JavaScript's going to challenge Python for AI supremacy. So we talked about JavaScript and Python before, but yeah, what's happening with R? Is it really declining as fast as we think?

Maybe this is a good one to like. Again, go back to you, Richie. Like, you're a very R evangelist. What do you say? Like, do you feel like we got this right or wrong?

I've had so few conversations about R recently and it does sadden me because it's still my favorite programming language. I did take a look at some of the programming language indexes to see how the popularity is changing. So the TIOBE index is maybe the most popular index for how popular languages are. Is actually, it's fairly flat on this, so it's not at its sort of peak. I think it peaked somewhere around like sort of 2018, 2019. It's been a slow decline. It's not quite as slow as you'd think. There's another index from the ieee, the Institute of Something Something Engineering. I can't remember what the other A stand for, but anyway, so that index weights job adverts and community values a bit more strongly and R has been declining faster on that. So the number of people who are getting excited about our or basically careers, that's dropping. So yeah, it's a slow decline, but it's going to be hanging around for a good decade or so, I think in corporate environments.

Yeah. So, okay, so it's that part of the prediction that we had. Right. I think the JavaScript one, like JavaScript challenge is Python that's like harder to judge. Like, if you look at documentation, like it's probably like a 5050 at the same time. Like if you look at the education element, I'm like, okay, Courses. Do they use Python or do they use JavaScript? I think I see more Python than JavaScript, so but that one I actually find like harder to judge where we stand there. I don't think it's like a clear win for JavaScript.

No, but I think it actually it links to what we were talking about earlier where JavaScript is extremely popular amongst software engineers in the web dev space. So as these two communities merge, you are starting to see more JavaScript interfaces to data and AI models and use cases. So it does seem like that there is a trend whether whether or not JavaScript is already challenging Python. I think that's debatable, but it's on its way to do that for sure.

Yeah, I suppose it's different parts of the stack. So if you're creating an application that uses AI, then you're probably going to be accessing things via an API and then JavaScript's the natural thing to do. But you wouldn't necessarily want to create your neural network in JavaScript. Like there's PyTorch but there's no JTorch or JStorch or whatever. So yeah, that sort of stuff sounds a bit crazy, a bit far fetched, but yeah, different parts of the stack, they're complementary rather than competing in many cases.

I think one thing that shocked me that you just mentioned, Richie and I looked this up over the weekends is that indeed the peak of people learning R was around beginning of 2018 and that actually shocked me that it's that long ago because R is still around, it's still fairly popular, but the decline started way back longer ago than I expected.

Yeah, it's hanging in there and I think with that there's a big difference between individual learners and corporate learners. On Data camp, I don't how closely you look at these statistics, but certainly for individuals they're generally motivated to try and find a job and so they go for whichever has the most jobs, which is Python. In a corporation you use whatever your colleagues use and their ass hanging on a lot longer.

Yep, yep.

So we'll give you that point and the next one was the next prediction was that data science notebooks become AI enabled. Do you want to talk through has this happened?

Yes, I think this is a clear win on the prediction side. Every data science notebook is now becoming AI enabled or already is AI enabled. Our own data lab very much has a lot of AI features. Google Colab has a bunch of basic features and I think it's a trend that will continue. Typically you actually see most of the innovation for happening for software engineers and the tools they use and for software engineers and kind of the IDE they live in. There's Cursor, there's the Replit agent, there's Devin, There's a bunch of other companies that are trying to get software engineers to use really AI first IDEs. I haven't seen that happen for data folks yet, but typically we follow a similar path for the data folks. That's what you're seeing for software engineers. It's just like one to two years later. And it very much seems that that's the next step here.

Absolutely. I guess where Data Lab is going, all the other competitors, the other creators of data science notebooks, they're kind of. They're joining in on it. All right, yes, we'll give you the point for that. Okay, so two more predictions to go. So the next One was that MLOps becomes more important. Martin, do you want to take this one?

Yeah. I think we're like a little bit the same situation as a prediction on data governance, like, okay, how do we measure this one? And it also suffers a little bit the same issues, like, how do you make someone excited about it? So I think to a degree it became more important. But at the same time, like, okay, I think given how important it is, like, the world hasn't caught up with the idea yet. So I definitely wouldn't give us a full point on this one.

Yeah, it's another one where it's very tricky to measure. But yeah, getting people excited about MLOps is another tricky one. It's a bit like data quality, where people are like, well, that sounds very important, but it's not there. The exciting bit in some cases. All right, so we'll put that one as a. Maybe it would give you half a point for that one. All right, Final prediction from last year is that prompt engineering won't become a real job. So are the prompt engineers out there?

I think generally this is true in the sense that prompt engineering is a skill. I think there are actually prompt engineers out there, but it's like a very, very niche role. I listen to a podcast of somebody who does that at Anthropic recently. So, like, there are people who, for most of what they do, are prompting a model, but it's very rare. I think generally prompt engineering is, or AI literacy, you could call it, is similar to computer literacy 40 years ago. And to us or people from our generation, it comes naturally because we grew up with computers and so we didn't quite have to learn it. But I think if you go back 40 years ago, most people who were part of the workforce actually had to actively invest in learning how to work with a computer. I think it's similar for AI and AI literacy, where it comes natural to young people, it will come natural to our kids. For the existing workforce, there's quite an investment needed in ensuring you get the most out of these models. So I think that prediction came true, even though there's maybe a few exceptions where it's not entirely true. What do you think?

Yeah, so I agree with that. And I think maybe when ChatGPT first came out, you had to write a lot of context in your prompts just to make sure it gave the right answer. Like, if you want to do anything mathematical, there's all these tips about, oh, you have to tell it to think step by step, and you've got to give it context on, like, oh, you, like, don't make things up and things like that. And now the models are smart enough to just do that automatically, so you don't need to provide all these, like, convoluted phrases just to make sure you get the right answer. So, yeah, all these sort of skills about, like, writing good prompts is like, well, you just write normal English and usually it does the right thing now. So, yeah, I agree. Prompt engineering hasn't become a job unless you're working at one of these foundation model companies or one of these sort of niche AI companies. Okay, nice. All right, so I'll give you the point for that. I'm not sure what the total is. I think you got two wrong and there were a couple of maybes. So we're hovering around something like a 7 or 8 out of 11. Nice. A decent score. We're going to try and beat that this year. That wraps up our recap of last year's predictions. Overall, I think Joan Martin did pretty well. Of course, I had to dock him a couple of points so they don't get cocky. That prediction on AI agents was a tricky one. It seems like we've been talking about them forever, but the delivery hasn't quite happened yet. In our next episode, we'll be sharing our predictions for 2025, so please do tune in for that. And also, I don't get to chat to all you dear listeners often enough, so I'd love to hear your predictions, too. Please post your ideas on your favorite social platform. Tag in at datacamp. I'm also richyrock's Most Places on the Internet. And yeah, let's have a conversation about what you think is going to happen. All right, thanks for listening.