#176 Data Trends & Predictions 2024 with DataCamp's CEO & COO, Jo Cornelissen & Martijn Theuwissen

Hi, I'm Maggie, VP of Curriculum at datacamp. And I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

I think what's happening now is a very fundamental shift. Just like 40 years ago, people had to learn how to work with computers, and 20 years ago, we had to learn how to work with the Internet and software. And I think data and AI skills in the next five to 10 years are at a similar level.

What I think is going to happen is that most people will need to learn how to do prompt engineering, and some will become really good at it, some will be average at it, but everyone will be interacting with an AI in some capacity. So everyone will need to learn some prompt engineering tools.

Welcome to Data Framed. This is Richie. 2023 was a tremendous year for data and AI, but we're well into January and that all seems like a long time ago. It's time to think about what happens next, so let's get some predictions about what will happen in 2024. Our guests are two of Datacamp's founders. Joe Cornelisson is Datacamp's CEO, and Martin Thurwissen is Datacamp's COO. They've both spent more than a decade trying to democratize access to data and AI skills. And without wanting to soak up to the bosses too much, I'm often impressed with how deeply they think about how this space is evolving. Hopefully, that's going to result in some accurate predictions for you. Let's hear what 2024 has in store for us. Joe and Martin, welcome back to Data Framed. Great to have you on the show again.

Awesome to be here.

Thank you, Richie.

Excellent. So I'm really looking forward to hearing your predictions for 2024. Before we get to that, I think we should have a look back at 2023, and I think we have to lead with the biggest story of the year. That's generative AI. So, to begin with, Martin, what do you feel has been the biggest impact of generative AI so far?

To me, it's actually the fact that it became mainstream. I think AI was already circulating in the technical realm. But what really happened in 2023 was that, well, everybody started talking about it. And if you compare what we've been doing with datacamp for, like, so many years, like, focusing on data and data is becoming everywhere, and everyone started to use data. Like, what you see now is that stepped into mainstream AI, stepped into mainstream, and that, for me, was definitely the biggest trend of 2023. And then what's nice is that it's really going after these customer experiences. Like, it's really breaking out of that technical part and moving into, like, okay, how are we? Like, if you think about datacamp and our students, like, okay, how can they use AI to write better code, to do better data analysis? And it doesn't matter if you're a beginner or an expert. So AI entering into the broader public for me was definitely the trend of 2023.

Absolutely. I have to say, usually when I tell people that I work in data and AI, historically, people just try and change the subject as quickly as possible. In this last year, there's been this glimmer of recognition. So I think we are making progress with these things coming into the mainstream. Joe, do you want to tell me what you think the most popular use cases have been of generative AI so far?

Totally. So I think what's super exciting is the explosion of use cases we've seen across the board. You've seen use cases in almost industry and almost every company within every industry. And then you've seen adoption of generative AI in every department of every organization. But to make that a little bit more specific, let's maybe talk about datacamp and how have we adopted AI internally? And then let's pull it broader and talk about what the general trends are. So if you think about education and what we do, our vision always was to bring the learning experience as close as possible to what people do in their actual job and make it really user friendly, really interactive. And ultimately, for education, the best and most effective experience is having your personal tutor. And I think what generative AI enables us is to get much closer to that much faster. Not just datacamp, but really any education company, I would say, is building better educational experiences. So to get more specific, what does that mean as a student? Now, if you're practicing on Data Camp, if you're solving challenges will help you understand the errors you make, will help you understand the solution of that challenge. You have a personal tutor that will help you navigate through our curriculum, will help what the best content is for you to do. Next. And so that means our engineering and our product teams now need to have a lot of skills around AI. And that's a general trend in the industry. Right. Like any engineering and product team in the last 12 to 18 months have been focused on building better chatbots, better search, more personalization, new types of experiences, and we're no different. But I do think within education, the ROI of those investments is going to be really high. But it goes broader than that. I talked about what's happening in terms of our product and engineering teams. I think the most fascinating thing that we've seen in the last year and a half is adoption in different departments. So to give you a specific example from our marketing team, we have people using generative AI and LLMs to analyze email data to help write copy, and we've had people in sales use generative AI. And that's a general trend as well that we hear from our clients. There's in every department, there are now early adopters, and it's enabling people to do their job much better, much faster, much more effective. And that's obviously a huge opportunity, opportunity for every organization. And it's a huge opportunity for datacamp to help enable and drive up that data fluency for everyone.

That's brilliant. I love that. It's basically every department has found some kind of use for this new technology. But with new technology, it means that people need new skills. And I feel there's a big difference between sort of general users and people who are developers. So, Martin, do you want to talk to me about what new skills the general users need in order to make use of generative AI?

So maybe to make the comparison again with data. I think in the past couple of years, given everyone got access to data, people needed to start developing data literacy skills. And they needed to understand, like, okay, what kind of graph do I need to use? How should I interpret this type of table? And I think the same thing is going to happen now with AI, meaning, like, okay, everyone is going to need to develop some kind of AI literacy skills. They need to understand some of the jargon. They need to understand, like, how to use some of the tools. And again, just looking at our own company, like, you can already see that. Like, we recently did some analysis on our emails, like open rate, click through rates, what kind of content really resonates with our learner base. And we used AI for that. We interacted through some prompt engineering and we discovered a lot of interesting insights. And people who are doing that are not necessarily, like, technical by nature or know how to code. And so that to me is AI literacy. And so when I think from like what skills do people need to develop, does the general population or the general population will develop? My point is like, it's going to be AI literacy, like understanding how to interact with AI understanding, like, okay, the jargon so that you can interact with an AI engineer if you need a certain feature or you want to build something. So I think things are heading that way.

That does seem so important, just being able to have a conversation that's intelligent about AI, even if you can't do everything yourself. And so going beyond the sort of the skills for everyone, the AI literacy, if you're a developer, if you're working in the field, what sort of skills do you need?

Maybe starting super high level. I think there's three areas if you want to get started as a developer. There's your large language model, there's your vector database, and there's your AI application development framework. So again, to stay super specific, what does that mean? What's the quickest, easiest way to get started? It's probably learn how to use the OpenAI API for language model, learn how to use Spinecone and then learn LangChain for the development framework. And that's how a lot of people are getting started today. Now, if you think about scaling that, if you think about putting that in production, what's going to happen is the costs are going to increase. You'll get questions around, lock in control. And so at the same time, we're seeing the emergence of open source, especially for the large language model components. And we've seen kind of hugging face community and ecosystem really take off. And so that's the second wave. As you get deeper as a developer, you probably should understand what open source models are taking off. You should become part of the hugging face ecosystem. And the advice so far was very kind of practical, but I think that the underlying trend is really interesting here too. If you think about software developers overall, their job has been pretty deterministic and software development timelines were relatively predictable, tend to be two times longer than what people think, but they were fairly predictable nevertheless. And what's happening now is a lot of what software developers are going to do in the future is more stochastic and is less predictable, at least in the next few years. And so for the first time, we're, we're seeing developers come to datacamp now realizing their skillset is getting outdated as well, and they have to kind of learn how to navigate these new worlds where they're building not just deterministic models, but they're building stochastic models and experiences and they might not know what the timeline look likes for getting things in production. I think there's a lot of companies that got really excited about generative AI, but then have really struggled to hit the timelines they were hoping for about a year ago. So, yeah, I think it's really fascinating what's happening in the developer space.

Yeah, a lot of very interesting trends there. The fact that the data space and the software space overlapping more, that just seemed like a very important trend to watch this year and.

Absolutely. I totally agree. I think kind of historically what datagame would focus on is data analysts, data scientists, and the citizen data scientists. And what's happening now is if you think about the concept of an AI engineer, some people come at that from a developer background picking up the data and AI skills, but some people come at it from a developer background. And actually probably the largest group of people are going to have a development background and they're learning kind of data and AI skills. So. Totally agree.

Okay.

That's definitely something to watch. Now, I have to say, with all these kind of new technologies and new skills required, I found there are a lot of people who are panicking a bit. They're saying the space is moving too fast, they don't know how to keep up. Do you have any advice for dealing with that?

Keep learning. Keep learning. Because I think what's happening now is a very fundamental shift. Just like 40 years ago, people had to learn how to work with computers, and 20 years ago we had to learn how to work with the Internet and the software. And I think data and AI skills in the next five to 10 years are at a similar level. Everyone is going to have to kind of upskill and reskill and keep learning. It's going to be fun.

I'm looking forward to it. Having a few more people learning some skills. Now, Randy Bing came on Data Framed a few weeks ago and he was talking about how most companies, they're still at the testing and experimentation stage with generative AI. So this alludes to what you were saying, that getting things in production is taking longer than people think. How do you think these are going to change in 2024?

Yeah, so I think if you zoom out, 2023 for a lot of larger companies was about experimentation. Large companies are like oil tankers and takes a while for them to change course, but once they do, they're going to continue ahead. I think 2023 was about experimentation. 2024 is going to be about getting a lot of what's been tested and developed in production at scale. So I do think will see a trend there. And one of the biggest bottlenecks has been, well, internal skill building around a lot of this stuff and making sure things work as expected. But there's also still a cost question. If you scale up GPT4 in an application, it's still pretty slow and it's still pretty expensive. So I think this year we'll see a drop in price. And especially as people move towards open source, I think we'll see an explosion in actual production use cases.

Yeah, I haven't realized just how expensive it is to even run these models. I mean, it's not trivial, the cost once you start serving thousands or tens of thousands of users. Okay, so I'd like to move on, but. So the other story of 2023 is really around text generation. So these large language models have been incredibly popular, but there's sort of talk about how well multimodal AI is becoming a bit more popular. Some of these image generation tools are getting better. Martin, do you want to talk me through what you think is going to happen with multimodal AI in 2024?

Yeah. So, as you said, 2023 was really about text generation. Like, you pose a question to ChatGPT and you get some text back and seemed like that's what everyone was doing all the time. I think for 2024, image generation will definitely become a thing and reach the type of maturity that text generation has today. Again, just looking internally at datacam, like, okay, how we already started using it, you can immediately think of, like, different applications in the image generation. And if you look at Dall? E, if you look at midjourney, their latest versions are really good and you can see production value in there. I think for the video generation, the story, my guess, the story is going to be a little bit different. Like maybe when we sit here next year, it's going to be the big trend for 2025, then it's just much more difficult. It's continuous. So, like, the creative possibilities are more endless. There's a huge difference between creating a video of 30 seconds a minute or half, an hour or an hour and a half. So I think that will remain still in its experimentation phase, infancy for this year, for 2024.

Okay, absolutely agreed. And I like that we're storing up predictions for next year's episode already. That's good.

I'm inviting myself again.

Excellent.

Yeah.

So I can certainly see how there's a big difference between creating a TikTok using generative air and creating a Hollywood movie. So before we move on from AI, have you got any more trends, Martin, do you want to go again?

Maybe picking another one is like, I think the attention span of the general public is not going to be two or three years. I think AI had its focus point, but I actually think it's a good thing. I think what we'll see is that AI is going to get integrated into lots of tools, applications, like our daily life and just like software, our mobile phone, our laptop, like, it just going to be part of our technical toolkit. And we don't really think, like, it's not that laptops are a hype today, but we probably still use them like multiple hours on the day. And so my other prediction would be that I think AI will start behaving more like laptops. So we're just going to become part of who we are, what we do, how we use technology. And so it's going to leave the collective mainstream thought, the active thought.

Okay, that does seem like very useful that AI is still present, but maybe it just works, it's invisible. I am slightly disappointed, though. It's been quite nice just this year saying, okay, I work in AI and it's cool. I'll go back to being uncool again next year. All right, sorry. Joe, do you have any more AI trends?

Yeah. So one trend that a lot of people predicted a year ago was the emergence of AI agents. I think that's one area that hasn't really materialized yet. I think if you look at the current AI agents, they're not really that good yet. But are we going to see a trend towards that starting to work for specific use cases? I can see that happening this year where for kind of narrow areas, the AI agents get good enough to really go from kind of start to completion and start performing specific tasks for people.

Okay, that seems plausible. I mean, certainly, like some of the demos have been like, oh, you can set a calendar invite by using ChatGPT. And it's like, well, that's okay, but it's not like a difficult problem that's being solved there.

Yeah.

All right, so let's move on from generative AI to talking about data. So do you have any predictions for the world of data, Joe? I know it's a big. A big question there. Yeah.

And I think the data predictions I have, they're not new, but they're things that are under a higher degree of pressure because of generative AI. So if you think about data quality, data culture, data availability, those were themes in the data space for like the last decade. But I think there is much more pressure in some of these areas and a much higher ROI in some of these areas as a result of generative AI. So let's quickly talk through that. So I think if you want to train custom AI models, data quality is really important. And so I think a lot of organizations are going to be waking up to the fact that their data quality is not at the level where they can actually train good customized models. So I expect even more focus, more investment going into that area. The second one is data culture. Right. I think a lot of organizations have realized the importance of data culture. There's been kind of an increasing level of awareness around it. But thanks to generative AI, more and more people are going to be able to do useful things with data. And so the biggest bottleneck is going to be what's the data culture of the organization? What's the data fluency level of a lot of people? Right. Because there is now a much higher ROI on building a healthy data culture. So I'm very excited about that one. And then the last one is data availability links back to data quality. Even if you have high quality data in a lot of organizations, that it's very siloed. And we will continue to see huge investments and a lot of effort to bring all of that data together. And again, I think generative AI is putting more pressure on that trend.

Okay, so this seems pretty positive that maybe executives are starting to think a lot more about data quality, finally taking this sort of thing seriously. And think about how do you get data driven decision making more widespread throughout your organization. So let's talk about programming languages. Martin, are you seeing any trends here with programming languages for data?

Yeah, so I think I gave the same answer last year. Like Python, still growing, still going very strong. They're doing really, the language is doing really well also in the AI space. So it seems like industry is settling quite a lot on the combination Python, SQL. Looking at another big language on the Datacam platform, R, you do see R leveling off. I think they missed the AI wave and that's definitely hurting them. If you look at the IB index, you also see R dropping in popularity there. And then, well, new technologies give rise to new languages. So we're keeping an eye on Mojo programming language mainly aimed at AI developers. Not expecting a big breakthrough in 2024, but it has some future potential and maybe finally like JavaScript. So it's going to be really interesting to see what's going to happen with JavaScript. You already see AI applications being built with JavaScript. I think it comes back to some of the things that Joe mentioned earlier. Like, okay, you have this whole developer community, software developers, like, they start to need to build AI applications like that overlap with the data space is becoming bigger and bigger. They bring their own frameworks, their own languages with them, of which JavaScript is really popular one. So, yeah, really interesting to see, like, okay, what JavaScript is going to do this year.

Absolutely. And I have to say, with my background in the R community, I am sad that R is no longer the sort of one of the leaders in the data space. And yeah, it's definitely missed the boat a bit on generative AI. But yeah, it is going to be very interesting to see will data people start learning JavaScript or not? That's definitely one to watch this year. So moving on, programming languages into sort of general tools for working with data. Joe, have you seen the trends here?

Yeah. So I think the workhorse for a lot of data scientists is still the Jupyter notebook. It's still kind of the number one tool of choice for a lot of data scientists out there. In a sense, that hasn't changed and that might not change. I do think there's a trend where Jupyter notebooks have key disadvantages and you've seen a bunch of companies step in to solve the challenges. With Jupyter notebooks, we ourselves have a horse in the race here. We built Datacam workspace that makes it much easier to collaborate. It's a great playground for learners. It's a great way to do hackathons within organizations. And I think as these notebooks in general move to the cloud and become more AI enabled, that's going to unlock a lot of potential because, let's face it, Jupyter notebooks are the workhorse for data scientists. But if you show that, if you show a jupyter notebook to the average person, their reaction is going to be, oh, this is not for me, thank you very much. And I think what's changing with generative AI is that you can build a layer on top of that and make it much easier to get started. So OpenAI has their code interpreter, we are building an AI layer on top of datacamp workspace as well. And we've seen people start to adopt that from all types of backgrounds to do things they didn't imagine they would be able to do. So that's, I think, super exciting for the next few years. Is that kind of the total addressable market for data exploration is really going to explode thanks to that kind of UI simplification and the much more user friendly interfaces that AI is enabling now?

Absolutely. I do find it fascinating that here at Datacamp and then at OpenAI, we started from completely opposite places. Like we didn't be doing sort of tooling for productivity for data scientists. OpenAI start with like a chat tool and then we're just converging in the same place. It's like a bit of chat plus some data analysis tools and yeah, both of those coming together. All right, so last year we talked about The Rise of MLOps and Putting Machine learning models into production has become a much more popular thing. Back then, the tooling for that was very heavily in flux. Is it starting to stabilize?

Mlops is a really important curriculum area for datacamp and I think we've been holding back a little bit historically because the space was so fragmented. The space is still fairly fragmented. We've decided to step in and build out a great MLOps curriculum anyway. And there's the larger ML platforms, databricks and snowflake there. We have build curriculum now. So I would invite everyone to check out those courses. It's an area of upskilling for a lot of teams and organizations out there right now.

Okay. Yeah, certainly like putting your data warehouse and your machine learning together in the cloud does seem to be a huge trend and we are seeing to see some sort of standard tools there. Excellent. Now, in terms of new roles that are coming up, there's been a lot of talk about prompt engineering and is prompt Engineer a real job on that? So, Martin, do you have an opinion on this?

Yes. I don't think prompt engineering will become a role in itself. I think it's going to fit more into the AI literacy story I was telling before. What I think is going to happen is that most people will need to learn how to do prompt engineering and some will become really good at it, some will be average at it, but everyone will be interacting with an AI in some capacity. So everyone will need to learn some prompt engineering tool. So I don't think it will become a separate thing. I do think there are other things like communicating data insights that will remain very important. But also there is going to be a universal skill. A universal professional skill rather than a dedicated job role would be my take.

Okay, so everyone needs to do a little bit of prompt engineering, but probably it's not going to be a full time job for most people.

Yeah. Okay.

All Right. So beyond this, Joe, have you seen any new data or AI roles appearing?

What's fascinating right now is the space is changing really fast. In general, there's the emergence of the AI engineer. I think in general, the trend I'm very excited about is the shift in terms of the skill set for a lot of software developers. I think every software development developer is going to need varying degrees of kind of data and AI skills in the future. And the ones that are at the intersection maybe right in between a data scientist and a backend developer, that's a new role that's emerged, I would say called AI Engineer. But I think ultimately every developer will be an AI engineer to varying degrees. So I think that's a really interesting trend we're seeing right now.

Okay, so it just seemed like the biggest new roles are really in that AI space then.

Yeah.

Okay.

All right. So in terms of jobs, I think there was a huge sort of hiring rush during COVID particularly in the data space. And then in the last year, things have softened a bit. And the most common question I think I've had from interacting with users in our webinars has been just, how do I stand out? How do I get that data roll? Martin, do you have any opinions on how people can stand out when they're trying to get a data job?

Yeah, I think it's two things. It's credentials, and it is a portfolio. So if you're thinking about, okay, how can you get credentials? You have the vendors like Microsoft, like Power Bi from Microsoft, but also like Tableau, Azure, and so on that have their own certificates. And so you can go out there, learn the necessary stuff to get that certificate, acquire the certificate, and you showcase them like, okay, I've done it. There are also more general certifications. Like, for example, in datacamp, you can find certifications for data set scientist, for data engineer, for data analyst. So showing that you have the necessary skills through this combination of credentials, as well as, like, portfolio building. Because when you're learning, you're most likely doing projects, whether as part of the curriculum or on your own. So if you collect that and you use that to showcase to your current employer or future employer, then I think that's the way to stand out today and to give some credibility to your application.

Absolutely. So a lot of it's just about reducing risk for the hiring managers just to prove they do have some skills. Okay, so I guess on the flip side of that, Joe, maybe do you want to take this, like, if you are a hiring manager, how do you Go about finding the best candidates for your data roles.

The good news for hiring managers, I guess, is that it's gotten slightly easier. The market was really hot two years ago. I think it's softened a little bit, so they will get a bunch of candidates. And I think what's important to understand for both the hiring manager as well as for people who are trying to break into this industry or trying to reinvent themselves in their career is that there's just not enough people that have the traditional credentials in this space.

Right.

There's been an emergence of data science masters in the traditional educational system in the last five to 10 years, but a lot of what's taught in most of these programs is already outdated. So I think the main thing to realize for both sides is, like, you will need to continue to learn, you will need to continue to update your portfolio, update the credentials. You're getting to be successful in this market. And that's true for both sides. It also means that hiring managers have to stay up to date and understand kind of what credentials are out there and exactly, kind of which ones can you trust.

This is very much a field where you can't just say, okay, I've got a degree and I'm done. You do need to continue learning throughout your career.

Yeah, exactly. And I think the speed at which the relevance of skills is decaying has increased in this space. I think that was already true for a lot of data and software development roles in the last few decades, but I think the speed at which you have to learn has only increased in the last few years.

Absolutely. I have to say, with doing the dataframe podcast and then with doing data campus webinars, every week I'm finding some instructor comes on or some guest, and they're always showing something interesting that I know nothing about. And I'm like, oh, I could spend years learning about this. So, yeah, always something interesting to learn. All right, so before we wrap up, can you tell me what you're most excited about in 2024? So, yeah. Final thoughts, Martin?

So 2023 was really about experimentation with all kinds of, like, the companies were trying to figure out, like, okay, how am I use AI in the services, the products that I offer? And I think 2024 will really be about bringing those experimentations, those thought experiments into production and making them available to U.S. consumers. So, yeah, really interesting to see. Like, okay, if I open Netflix six months from now, like, okay, how will they have incorporated AI? If I open Spotify, like, how will they have. Like, am I going to be making my own music six months from now. So what really excites me is seeing like, okay, how this whole hype of 2023 will materialize in 2024. I think that's what I'm looking forward to.

Yeah, yeah. Things actually hitting production. I do especially like that idea of being able to make your own music. I have a sort of long standing bucket list item to become a rock star. I'm horribly tone deaf and not musical at all. I've made no progress towards this, so it'd be nice if AI can take that on for me. Joe, what are you most excited about?

Yeah, I had a similar one from a slightly different angle. I think I'm excited about what individuals and startups are going to be building. And datacamp is ultimately in the business of enabling people, teams, organizations. And what I'm excited about at a high level is generative AI is enabling two new groups of people to really build totally new experiences and get to insights in new ways. So you have the citizen data scientist, someone who doesn't have a technical background in any business role. They're going to be able to get to insights much easier, much faster, thanks to a lot of new tools that are emerging. So what are they going to come up with? And then second of all, you have the software developers who are always building and shipping products, but they will be able to build experiences that we couldn't imagine years ago. So, yeah, a bit similar to what Martin was saying. Kind of like seeing that go in production is going to be really exciting.

Absolutely. Just bringing data and AI to the masses. It's exciting times. Excellent. All right, thank you both for your time. Yeah, thank you, Martin and Joe for coming on the show.

Perfect. Thanks for having us. See you next year.

See you next year, Richie. Thanks so much.