#337 DataFramed, Distilled. The Best Moments of 2025 with Richie Cotton

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At datacamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the Most out of ChatGPT and Copilot, or developers and data scientists looking to fine tune models, you can learn the entire AI skills spectrum on datacamp. Power your AI transformation today. Start learning@data camp.com.

It's been a wild year in AI. I can't scroll social media for more than 10 seconds without someone appearing my feed talking about it. One big change this year is that AI went from being okay at solving simple data analysis tasks to being very competent at solving more substantial problems. That raises big questions about whether the data analyst role is going to be automated away. I asked Mo Chen, an analytics manager at NatWest, if the data analyst role is dying out.

My answer will be no, because I clearly still have a job. I actually progressed in my job as well, so I'm making more money. I am more senior than I used to be. So I would say no. I think AI is changing the skills needed in the industry right now. So rather than remembering everything, I think it's about learning the foundations well and then using AI to produce even better, better work, even more complex work. And there are many things that at the beginning of my career, took me quite a while to do, maybe even a day or five days. Now AI can do it in probably 30 minutes, maybe an hour. You have to know how to prompt, you have to know what to look for, you have to know how to combine various AI workflows and tools so that you can actually do the thing you want to do.

That's a relieving answer that data analysts aren't going to disappear anytime soon. Many jobs are certainly going to change though. And one of the most intriguing ideas is that of having AI colleagues. I spoke to Karen Ng, the head of product at HubSpot, about the idea of humans and AIs working together.

I do think a lot of the ways that we work together as a team is going to be really important in our agent world. We believe a lot in hybrid teams, and I believe that's the way that things are going to grow. And hybrid teams are both humans supercharged with AIs and agents that need to work together with humans and other agents.

I love working with people, but being a boss of bots does seem like an easier path to management. One role that is Absolutely scorching hot right now is that of the data engineer. Without the right data and the right format being delivered to the right people at the right time, all your AI efforts are useless. Our next guest is Deepak Goyal, the CEO and founder of Azure Lib Academy. He's taught over 100,000 professionals how to be data engineers. In his episode, Deepak highlighted a lot of the technical skills you need to become a data engineer. But here I'd like to highlight one of the soft skills he talked about.

The first and foremost is that you should be able to communicate right compared to the software engineer. Data engineer has to move to communicate because probably they have to talk to the lot of stakeholders to understand what kind of output or the results they are looking for.

It feels like whatever role I discuss, whenever I ask an expert what the most important soft skill is, communication always comes out top. So your homework for the holidays is to chat to people. Remember to listen to them too. Zooming out. I have a nebulous sense that the shape of careers are changing. I asked Abhijit Bhadori about this. He's one of the leading voices in learning and development and was previously GM of Learning and Development at Microsoft.

You can either choose to do specialization in different aspects of data, you know, or you can choose the kind of field that you can work in, whether.

It'S healthcare or analytics or retail.

Or you can then choose whether you want to work solo and become a contractor or do you want to get.

Employed, do you want to do part.

Time, do you want to go into an office? Do you like to work remote? Each option that you create, you know, creates a set of possibilities and takes.

Away a set of possibilities. It's both a benefit and a challenge that careers are becoming more fluid. You have more choices about your career, but that also means you've got to make good decisions. Career challenges also exist for managers. So I chatted to Bill Alzia, the head of data and analytics at Duolingo. Now, Duolingo is currently a highly successful, very data driven company. But when Bilal joined, the data team was having problems. They weren't trusted by management and the team's generally underperforming. Bilal has a great story about how he turned around the team. So here's a highlight.

The biggest success I had was investing in the people, not in ideas. Because I think that order is really important. Ideas don't become successful if you don't have good people in the team.

That wraps up our career highlights. The next question is who? How do I track what's happening in my business. And well, the way you do that is business intelligence. Howard Dresner is often known as the godfather of business intelligence, having coined the term back in the 1980s. These days he runs Dresdner Advisory Services and he publishes some of the most influential reports on the state of the industry. Here's Howard's take on BI in 2025.

When I talk about business intelligence, a term that I coined back in 1989, it was always intended to be an umbrella term so it not tied to a specific technology. It' about fact based analysis which is driven by data. We've seen that evolve a lot over the past few years. And now of course, you know, you look at things like agentic analytics and conversational analytics. Well, that's all part of business intelligence because the intent is to deliver the same capabilities. How do we deliver data driven fact based analysis? We have so many external forces operating our organizations. Data is your friend. You want to know when things are changing and how they're changing as quickly as possible so that you can react. I mean, at the end of the day, that's the whole point of business intelligence. How do we respond to change and execute as quickly as possible?

One quirk of BI is that the tooling has become bifurcated. There's one set of tools for data analysts and another set for business users. This issue annoyed Colin Ziemer so much that he decided to create a new BI platform. Colin is CEO at Omni and his Data Framed episode discusses what the next generation of BI tools needs to deliver.

The only reason that Omni exists is because I sort of experienced this frustration as a data person over like the last 20 years where I had touched a bunch of different tools and sort of felt the good and the bad of them and I wanted them all to exist in one place. So the very simple story is I started in finance, grew up in Excel, sort of learned everything. Data from an Excel context, ended up later in bi, got exposed to looker data modeling, like very heavy SQL concepts. And I was always frustrated that tools couldn't do both of those things at the same time. There's a time and a place to cut corners and go really fast and be a little messy. And there's also a time and a place to operate like a software engineer and have governance and centralization. We just kind of never felt like it was getting executed.

Of course, the secret to Analytics Impact isn't creating the perfect dashboard.

It's.

It's communicating the results of your analysis effectively. Data Storytelling has long been one of the most important forms of artistry for data practitioners, and so I'm a sucker for any new ideas on how to tell data stories better. I chatted with Kat Greenbrook, the author of the Data Storytellers Handbook and founder at Rogue Penguin, about her framework for data storytelling. She has some great advice on using data storytelling to achieve influence and having.

This understanding of the problem, the goal, the action, the impact, the hierarchy in which that we are existing. It can help when we share the work that we do so we are able to influence better. We are able to share more of the context around the work that we do. So we're not just sharing, hey, I'm building this model, we're sharing. This is the problem that the business have that this model is going to solve and the flow on effect from that is that I'm going to create this positive impact that's going to be good for the business.

Data storytelling is such a broadly applicable skill that it can be considered a part of data literacy. That is core skills about data that everybody needs. Data literacy has been an essential life skill for many years now, but in recent years you also need AI literacy. I spoke to Jordan Morrow, he's often known as the godfather of data literacy, about how data literacy and AI literacy are related. Jordan is Senior Vice President of Data and AI Transformation at Agile One. He's also writing his fifth book on data data literacy. So very well versed in the topic. He had a great take on the challenges of behavioral change for making people data and AI literate.

Some of the first steps of a change management within data AI, any of that is number one, understanding what the true objective is. And the objective is not to teach data literacy course. The objective is to change people's behavior, right? That is very hard. So the key besides the right objective is what is your communication strategy around this? You can't just send people an email that says you have mandatory training. The moment you do that you've lost. There needs to be messaging around it. They need to understand what's in it for them. They need to have ensure that leaders truly buy in. They need to understand how much time this is going to take. All these things add up.

I was rather pleased that Jordan's message sounds suspiciously similar to what datacamp's customer success managers tell our enterprise customers when they're running training initiatives. Now, I've been incredibly restrained so far in not talking about the biggest technical topic of the year. I don't think I can hold back any longer. It is of course, AI agents. During his time at Meta, our next guest, De Kehler helped invent the popular AI knowledge lookup technique retrieval augmented generation, AKA rag. He's now CEO at Contextual AI where he's building an agent creation platform. His DE on what an AI agent actually is.

I think that a much more useful definition of an agent is just something that actively reasons. So something that thinks about what it's doing, formulates a plan, executes on the plan and then can revise that plan based on the information that came in. So that's active reasoning and so the really exciting technology that has enabled all of this is just test time reasoning and the insight that shifting the compute from the training side to the test time inference side actually has very, very nice properties.

The last point about generative AI being able to reason at inference time, that is after you give it a task, is key to enabling AI agents. One of the big challenges of enterprise AI systems is keeping them fed with up to date data sources that is giving them proper memory. I spoke to Jun Kian, the VP of Generative AI Services at Oracle, and his take was that the key to success is to combine your existing data infrastructure with RAG systems.

If you already have open search based knowledge base, you can easily add another layer of the RAG on top of it, right? Build a decent enough RAG system and however your data refreshing data management system can be as the same as they have today. You don't have to redo everything. So this is one possible way, right, to move RAG system forward quickly.

Perhaps the biggest problem of all for enterprise AI is AI governance. Manasi Vartak, the Chief AI Officer at Cloudera, had a great take on the relationship between data governance and AI governance.

I think data is a part of AI governance, but AI governance is a much bigger sort of umbrella. It does start with data. Where did the data come from? So if you think about the lawsuits that currently philanthropic and OpenAI are facing, they have to do with what data did you use to train the model. That's a data governance and lineage issue. Like if you could trace it down and say this model was trained on these 10 data sources, it's an easy question. So the challenge is the data goes through so many steps of pre processing and post processing that by the time the model gets to it, it's hard to figure out where it came from. For AI governance, we need to solve those kinds of fundamental issues around data governance.

More generally, one of the biggest questions around AI is what should you build with it? I asked Jerry Yu, the CEO at Llama Index.

Literally everybody is using coding assistance these days. You know, if you ask any engineer at a tech company, they're all using like cursor, Windsurf, chatgpt, et cetera. And then of course, like, you know, you're seeing AI agents deliver real AR ROI in certain enterprise verticals. So this includes like customer service, this includes like IT help desk stuff. This also includes, you know, areas that we're actually very specifically excited about, which is document workflow, automation, you know, whether you're a finance team or procurement, legal. Being able to sort through massive volumes of unstructured data and getting insights from.

It, that's a great high level answer. But within each industry there are also specific use cases. Let's consider healthcare. Aldo Faisal is a professor of AI and neuroscience at Imperial College London. He also runs the Nightingale AI Project, which is designed to give doctors advice. He also runs the Nightingale AI Project which is about medical decision support, that is AI giving advice to doctors.

The real big wave that's going to wash over the whole field is now we call ambient intelligence and the whole space of AI for operational improvements. So that's not medical activity per se, but for example, transcribing the conversation between a doctor and the patient.

Over in the world of finance, Andrew Rieskin, the chief Data officer at mastercard, discussed what his teams are building. He also had a great take on why you need to worry about data quality before you go all in on AI.

An interesting exercise we went through last.

Year was where our priorities are. Our priorities are still very much about data quality. The AI, because it consumes that much.

More data, that much faster, less human intervention, the quality of the data, the understandability of the data has become that much more important. So it's just this feedback loop that we're getting that is just strengthening the.

Need of just getting the basics right.

To feed the AI so that we.

Know what the quality metrics are relative to completeness to accuracy, that that is mission critical. Once you have your infrastructure in place, the two challenging things are people and processes. I talk Steve Lucas, the CEO at boomi, about running an AI transformation program. Here's Steve. On the human side of AI change.

The entire world is going to pivot.

From deterministic processes to non deterministic agentic process.

That's what's going to happen. I'd say probably 7 out of 10 processes that have ever been built will be augmented, rewritten, enhanced with AI agents.

To be more non deterministic or agenti. Our next guests, IWO Schapar and Aaron Peters are the founders of AI Maturity Index. Iwo had useful advice on how organizations can build momentum through low code experimentation and internal hackathons.

One of the best tactics that really works for organizations is to organize hackathons. Hackathons can be just a few hours session where you're using no code, low code tools or the platforms like Cursor replit, etc, where you build some prototypes as you go without the need to understand how to code or even to work with the data. You just experiment with this. What is really cool here is that you're building solutions for your own problems, for your own challenges, right? So you can immediately see potential value that it then creates, which then immediately creates a buying for such tools.

While generative AI is getting all the headlines, predictive AI is still responsible for most of the value creation. So let's talk data science. Jean Francois Puget and Chris Diot are both Kaggle run masters and distinguished engineers at Nvidia. They spend their days pushing the boundaries of modern performance and GPU accelerated workflows. Here's their take on what's changing in data science.

I've worked at Nvidia six years and I think one of the biggest things I've seen in these years is accelerating all these algorithms on gpu. And the significance of that is, besides just obvious speed, is that we're actually this can allow us to build new hybrid models. So I've actually recently in some competition seen people fusing deep learning models together with a machine learning model. An example is maybe you're doing an image regression problem, but you use an image model as a backbone to extract embeddings. Then you regress the embeddings using a support vector machine. But previously a poor vector machine running on CPU kind of wasn't fast enough to keep up with the iterative cycle of deep learning. So because of the speed, now people can incorporate knn, you know, to compare embeddings to find similar images or similar text. So you're seeing a lot of more uses of machine learning. So that's exciting, these new new ways to approach problems.

Mary Nova is an AI engineer and the founder at Break Into Data. We had a great discussion of how generative AI is completely overhauled natural language processing, I.e. machine learning for text. Mary also highlighted one of the greatest achievements of generative AI. It's fundamentally changed how we interact with computers.

There's a lot of opportunity when it comes to AI agents and AGI in general, just because now that we understood how to communicate with computers with English language, which is super exciting. It can do things for us just because the things that we need to manage are in English as well. And before we didn't have the capacity for the computer to navigate these spaces. So I think this is very exciting just to see how the field is evolving.

Some of the unsung heroes of AI are the people creating data sets that feed into the foundation models. Our final guests are Wendy Gonzalez, the CEO at data annotation company Sama, and Duncan Curtis, the SVP of Generative AI. Wendy had a surprisingly fun take on when you want to use synthetic data that is made up data in your AI model models. Get a nice example for self driving cars.

Synthetic data has good uses or great uses in edge cases. So imagine like nobody wants to real time capture, you know, a stroller in the street or an empty stroller or stroller with a baby in it or I heard one actually from a, from an auto company client where they're like, you wouldn't think of this, but sometimes hogs drop out of the sky and get in front of vehicles. That's probably not something you're ever gonna catch. It has happened before. Probably not something you're gonna catch, you know, in any regular real life. So it's a great example. Some edge cases, it's like, okay, synthetic data makes total sense.

And that's our 2025 roundup done. It's been a year of huge changes in careers, BI AI agents, all the rest of it. Wherever 2026 takes us, I'm sure there'll be more exciting changes to come. See you all next time.