#356 The Forecast for Time Series Forecasts with Rami Krispin, Senior Manager of Data Science at Apple

Generative AI is transforming industries at an unprecedented pace.

But as AI changes how you or your team work, one thing is clear: your skills also need to evolve. At DataCamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the most out of ChatGPT and Copilot, or developers and data scientists looking to fine-tune models, you can learn the entire AI skill spectrum on DataCamp. Power your AI transformation today. Start learning at datacamp.com. In a year from now, my kids could go and do a forecast without knowing anything. Now we can really handle more datasets with less effort. Think about like Walmart. They have thousands or maybe hundreds of thousands of products that they are selling on their stores.

A staple task of data scientists is forecasting, that is trying to predict the future. The tools of this haven't changed in almost a decade until now. A new crop of time series foundation models is emerging, and I want to know how they might help data scientists. And since the real world is messy, a persistent problem of time series analysis has been getting models into production. So I want some advice on that too.

I'm not sure if people would feel comfortable yet to give the power of those decision-making for autonomous agent. You could end up without ability to produce electricity. And then calls for a lot of unhappy people.

Our guest is Ramy Crispin, a senior manager of data science and engineering at Apple. Ramy's team creates data products for Apple finances, services, and infrastructure. He's the author of Hands-On Time Series Analysis with R and the forthcoming Applied Time Series Analysis and Forecasting. Ramy also created the DataCamp course Designing Forecasting Pipelines for Production.

When it's doing great, it's great. But you need to understand that when you're using auto models, auto algorithms, that sometimes you get it wrong.

Let's make some forecasts about forecasting. Hi, Ramy, welcome to the show.

Thank you. I'm happy to be here. Thank you for the invitation.

Yeah, great to have you here. So the hot new thing with time series is time series foundation models. So similar to we have foundation models for text, now got them for time series. But time series modeling has been around for decades now. Why do we need foundation models for it?

I think the short answer is scaling. When I started to work on time series data back like 10 years ago, I used R and then the common time series object was the ts. For those who remember, this is a very notorious one that you can, you know, it was built for a time that people use the monthly. Or quarterly datasets. And since then, like, uh, we started to collect data in an exponential manner. And think about it like your phone or any electronic device collecting logs. And so now when we are working with time series data, we have— this is probably the most common format of data, and we have more than we can actually process. And so trying to handle it with the traditional methods that are not very scalable it's very challenging. And I think that was in recent years, we moved from the statistical models that are really great to what it was designed, very structural time series, and to more like machine learning approach. And now we are seeing the models that are based on foundation models that are taking it to the next level where they are being trained on large amount of datasets and different use cases. To handle broader use cases at scale. And I think scale is the critical point here that now we can really handle more datasets with less effort. Okay.

I mean, you mentioned the TS objects in R for dealing with time series. I had, yeah, a lot of, I spent a lot of time like painfully wrangling those things. But yeah, I love the idea that this enables time series analysis at scale. So you need better forecasting, you need a lot more forecasting. Maybe talk me through like what are the business benefits? Like, have you seen any success stories where you can do things now that you couldn't before?

Think about like the, you know, the general use case of retail, right? You have, and also usually the competition, like the M5 competition, I believe it was M5 or M4 that they took, I think Walmart's datasets, like SKUs. You have thousands of those. And the challenge in time series that, like if you think about the traditional time series process, you need to go analyze, to understand the behavior, to understand what type of models you want to use. And it's literally impossible to do it at that scale. Like if you have thousands of SKUs, think about like Walmart, they want to know for the capacity planning, how many bread they're going to sell or to ketchup or any other products and do the planning. There are thousands or maybe hundreds of thousands of of products that they're selling on their stores. And this is a great use case where, uh, you can get a decent forecast. Now again, like there is a trade-off when you're going to scale your trade-off on some level of accuracy, but you know, there is that you need to choose your battles, right? Like, uh, you are getting instead of, you know, accuracy of the level of like a very low accuracy, you may miss some, but generally your overall picture, you understand what is your so a demand signal, and then you can plan better your storage capacity, all the process to get the product. So I think those are the use cases that people would like to use it. And I think this is where we are going. Oh, wow.

Yeah. I hadn't really thought about having to have a time series to forecast the demand for every single piece of stock because you want to know, just have exactly the right amount of that stock that you're going to sell. And once you've got hundreds of thousands or products that really is a scaling nightmare. I suppose, yeah, we've had what, just-in-time delivery for sort of decades now, at least in theory, but having the data to like make it right and like really optimize it. Yeah, I guess that's a real challenge.

You see some of the development is coming from companies like Amazon. And the reason that they also have the same problem, right? They managing inventory.

Yeah, so certainly once you do that better stock management, you can save on warehousing costs, you can save on shipping costs, you can save on like, I guess, spoilage costs if you've got like food or things that are gonna go rotten. So yeah, a lot of business benefits there. I'd love to talk a bit about how these new models work, but I guess to figure out that, we've gotta take a step back and talk about how time series work traditionally. Can you talk me through like, what does traditional time series involve? What's the process?

Yeah, so when we think about traditional time series, usually like the first name that's coming is the ARIMA or the Holt-Winter model. Models that are models that have been here for close to 100 years and they're working like magic when you have a very structural time series and with a low frequency, like a monthly data or quarterly or daily. And those models are based on trying to like breaking the series into the components, the trend, the seasonality, and apply some different modeling. It's a combination of models that you are stacking together to model those components. So for example, the RIMA has like 3 layers, right? The autoregressor, which is like the foundation of all time series models. Like time series is very correlated with itself, right? That's the magic in time series. The hour now is very correlated with hour an hour ago. So if you know what was the hour ago temperature, it's easy to know what will be the next hour and you will take this seasonality you have across the day, you put it together, you get a forecast that this could be very accurate, at least short term. So those models are taking advantage of those kind of seasonality that occur and the trend and altogether. So when you adapt, you are adding those all together, you get a decent forecast model. The challenge with those traditional, that they are working really well when you have a structural time series, like the demand signal is very clear. For example, the demand for electricity, usually demand for electricity or natural gas or those type is very driven by the seasonality across the year. If you are looking at the hourly demand for electricity, we are using more when we wake up toward the end of the day and then there is a drop, right? So those things are very repetitive. Where they will struggle is where the real world data, like if you're going to the business world where things change because policy change or there are some new products or something else that shift, COVID came, right? And this is where those models are less successful. And then where you want to either use some regression model, machine learning model, This is where the feature engineering coming into the picture where you can tell the model about how to handle events, how to treat them, how to project them in the future in your forecast. And I think that's the magic feature engineering. I think time series forecasting in the business world is mainly feature engineering.

Yeah. So it's interesting that, I mean, I do remember from like my sort of earlier days as a data scientist where you think, okay, well, we've got this model and then you're trying to figure out, well, you need to have a feature to say this is a weekend or this is like a public holiday. And then, yeah, there's all this kind of like subtle things you need to understand, things that are going on in your business, and then turn that into some kind of like code representation in the data. And yeah, you spend a lot of time just messing about, well, do we believe this has an effect on the data? Yeah, so it's a very manual process. Are these foundation models solving that? How do they differ from traditional time series analysis?

Yeah, I think they are, and that's still the challenge that I don't think anyone solved. They're good on understanding that, hey, there is some events here, it's outlier. Do they cannot tell if this event will occur in the future and put it in a forecast? There is no way today, to my best knowledge, that And that's always a challenge in time series. You see a spike, does this spike is just a one time or is it going to, you know, related to some events that could occur in the future? And if you know the date that is going to occur, you put it and then you are adjusting your forecast, you're learning from this event and apply for the future. And I think that's still like a challenge that, uh, I don't think we have a clear solution what they're doing. They're good at identifying those events programmatically. They cannot tell about the forecast. They can, like, maybe you can let LLM to kind of like estimate if this is a one-time or not, but it's very challenging for a model to understand, to make assumptions about the future of those outliers. Is it going to occur? When is it going to occur? 'Cause if it's not seasonal event, so if we don't have indication, it's outlier, right? So if it's recurring, it's not outlier, but if it's just the one time, why did we see this event, right? Is it just some new trend that we are going to see recurring or, right? So this is a challenge and I don't think those models still have answer for this challenge.

Okay. Yeah. So I remember, Was it last year or the year before? So Taylor Swift went on this huge tour and whenever she would go to like play a gig, it just like broke all the hotel booking models because everything just got booked out. And I guess the same is true, like, you know, you have, we've got the World Cup, the FIFA World Cup happening later this summer, that's gonna like break everything in all the cities there. So I guess there's a challenge in terms of like, how do you feed the models with the right information about these rare events?

Yeah, and I think what could augment, and I'm sure someone out there is doing it, or if not, this is idea for people that are listening, is take an LLM that go and research. Like if you see outlier, create agent that go to the internet and trying to understand what caused this spike and then conclude about is it the outlier or going to be in the future. That's something that I think at some point we will see it if it's not already someone build it. So I think we are marching toward have like a really autonomous, uh, forecasting process. But you need, depending on what you're doing, focus. If you are doing focus on some sensitive data for your company, I'm not sure if people would feel comfortable yet to give the power of those decision-making for autonomous agent. In some cases, it's okay to be wrong. Like the repercussion of getting a wrong forecast is, you know, maybe you, uh, going back to the Walmart, you overestimate the demand for ketchup and maybe you got stuck with some, some units. Nobody get, I mean, there are going to be maybe some money involved, it's not life or right, or it's not like a huge, uh, loss for the company and you get it in other places that you probably overset this. But there are some cases when you need to make a decision that are very critical. And I think in those cases you want to validate. And this is usually like when I'm doing a forecast at scale, I'm looking at the You know, like maybe you have a, let's say 100 series that you forecast or 1,000, but you will always have the top 10 forecast, 10 series that are covered more and they have more weight and there you will spend more time. So there is some kind of like a, when you're doing a scale, you also need to sometimes think about where you be want to be more strategic and where you are okay to let the model do its stuff under the risk that the model will get it wrong. That's the risk. Like, that's the trade-off when you're doing forecast at scale. And you need to also think about risk management when you are building your strategy about how to go for the, your, how to forecast at scale.

I agree that having too much ketchup, it maybe like it's costing money, but it's not a real disaster. You mentioned there are some more serious situations. Have you got an example of that? What, what's the worst thing that could happen by getting your forecast wrong?

Think about that. You're doing a capacity plan for energy, right? And if you don't plan, let's say that your power plant requires some, I don't know, they're consuming natural gas or you are doing, uh, some other natural resource or you, you are depending on wind or solar power and you get it really wrong. And so the, you could end up without, ability to produce electricity and then cause for a lot of unhappy people. And, and so that's where you want to have control. And there is examples, I know, remember there is a good example that it's not necessarily involved of AI, but if you remember a few years ago, Zillow, the website, they, uh, is the house, um, finding place. Yeah. They were relying on a forecast to, estimate the growth of the, if I remember correctly, the growth of the market or the prices, and they got it wrong and that's really damaged the stock price. The company was really affected by this wrong estimate and part of it where, at least for, I was not there and I don't know from the company inside, but from what I read and I remember, they were relying on profit where the model is sort of auto-forecaster. And, you know, when it's doing great, it's great. But you need to understand that when you're using auto models, like auto algorithms, that sometimes you get it wrong. And this is where you need to think about how to mitigate those risks to make sure that you cannot— I think it's like blaming algorithm. It's not fair to blame the algorithm, right? And it's a risk management rather than Just, I'll go with him.

Absolutely. Oh, but this is Prophet, the Python package for time series. We should probably clarify rather than like—

Yeah, Prophet. Yeah. People are new to time series. Yes. Prophet is a library that was released a long time ago by a team in, at the time, Facebook, now Meta. It was a really great library. It introduced new features that did not exist at the time and ideas in time series. But it was like a side project of a team in Meta that never went to a production level. And so the code architecture of it was not production ready. It was like 3 languages layers, right? If you think about, I use the R version, well, the Python version is the same. It's calling Stan and Stan calls C or C++. And so when you're trying to deploy it in production and mess up with the all the dependencies, it's very painful. And this team at some point left, like the people that created left Meta and also at least one of them that I know. And so it was not under maintenance or, uh, not really received the care and love it should get from as an open source. And I think that's where the concept remains, but, uh, it was not a production ready. And, and I also think about a lot of, um, people took literally like the words that they were describing in the library description of auto, auto forecaster that can do automation for forecasting and without understanding, I think, uh, there were great features and if you use this feature, uh, in profit, you can really get a good forecast. And when you're always, when you're using some auto algorithm, you just need to be aware that sometimes it gets it wrong.

I think that's true of like a lot of machine learning, a lot of AI is like, it's very easy to just believe, oh, bot says this and then you don't question it. And that's always been a problem, I think. But so we did a bit of profit now. The latest generation models. So these are the foundation models for time series we talked about. So Amazon's got Kronos, Salesforce has got uh, Moirai, I mean, there's one from Google, there's a few startups doing these things. Do you have a sense of like, what are the good ones? Like, uh, which ones should you be looking at? Like, how do you make a decision about which model to use?

There are a couple of things that you need to consider when you are choosing a model. One is the, you want to benchmark it. It's not always you need a foundation model to do a forecast that maybe Arima and, and, uh, AltWinter will do a great job. With a cost of a fraction of CPU instead of starting to try to run it on GPUs or stuff like this. So I think the first thing is to identify the needs. Then the second thing is you need to experiment and understand which type of models will work on your data. And it could be a combination, right? You can use different, and that's what I, I love to do generally when I'm doing forecasting, not necessarily foundation models, but traditional forecasting, I like to run all tracing between different models using backtesting. And the idea is that you are using backtesting to just to get like, it's like cross-validation in machine learning. You test each model on multiple partitions. You want to make sure that model is consistent in the results and you're splitting your data into multiple training and testing. The difference between backtesting and cross-validation. In time series, it's sequential, right? You have a window that you move it and you, each time you are having a training partition and you predict the next data points on, as, and you raise your testing and then score it and get some score. And so this is give you, you can understand if the model is stable and can reproduce, uh, good results over time. And then when you have a lot of time series, it enables you to, you know, not every time series looks like the same, right? The distribution, the underlying patterns might change between time series and some models will be performing great with linear regression, others will XGBoost and some ARIMA, right? And so it enables you to find what is the best fit on each one. And I think this should be the same when you are using foundation models because even within the If you go to Krono or Salesforce, they have a different size of models, right? In terms of, it's like a foundation, like an LLM, right? You have 3 billion parameters and you have a, I don't know, like a larger number of parameters that the model was trained. You don't necessarily need the big models to get a good result, right? And I think that's where you need to identify what is working best for your data and based on this, make a decision. And it's going to be a combination. You don't need to lock yourself to only one model. You can try different and then choose the one that works best for you.

Okay. Yeah. I love the idea that you want to try a few different models and then do some proper testing on them. And the idea of backtesting is incredibly important because it's one of those things that is quite peculiar to time series forecasting. You don't see it in other fields of machine learning. So I love the idea of like test them up till 2023, see if the model correctly predicts 2024, 2025, and if so, then it's okay to use it for your forecasting. Okay, so we talked about choosing a model, we talked about doing some testing. I feel like if you want to get a time series forecast into production, there are a lot of other different steps as well that we need to discuss. So yeah, talk me through like what's your workflow from I'm going to build a time series, I'm going to use this for like like real stock price for, sorry, real, I guess, stock forecasting in your warehouse or whatever, or you wanna do some real predictions?

Yeah, so it start, all start with the data, right? You need to get, and I think people don't realize that, I think in most, like every data science project, you spend most of your time to get the right data and in the right format. And if you think about production, You need to think about how often you want to refresh the forecast, how often you want to refresh your data, and start with the data, building a pipeline for the data and to get the right KPIs that you want to forecast. And once you have the data, it's— you start with the experimentation. Uh, you want to identify what models you want to use or what is the appropriate models to use, and then at scale, you may have different approaches and there are different methods to kind of like analyze your data at scale, right? Cluster analysis and so on. And you can find different clusters and based on the patterns, fit the right type of models. And experimentation is where the goal is to, you spend a lot of resources on identify different, you're doing like different experimentation of identify what is the best. Like we talked about the foundation model that you can try different. So that's the idea. You run backtesting, you iterate, and you know, you look at the results and you see if there is any room for improvement. And if there is, you keep iterating until you get to a point that you think that your marginal improvement is low, that it doesn't make sense to continue or feeding more features or training more models. And once you get to the point, this is where you move your models to production. You just select the one that you want to deploy instead of running all the things all the time. And you, when you register your models to production, then comes the iterative process. Let's say that you are predicting demand for electricity and you do it every 24 hours, you predict the next 3 days. So you need to build a scheduler that you can go and get the data. And fetch it with the model and generate forecast. And so that's something you need to think. And then the last and very important component in production is that you want to monitor your forecast, right? And so while you were getting great results in your training, that could change in your training and testing could change in production because you may have a drift, drift in the data, pattern change. There are some, something that might cause the models to go out of tune. And so monitoring is very critical in production and you want to, every time that you refresh, to go score your previous forecast and see if you align with your expectation, what you result you got on the backtesting and set some threshold where you want to set alert that you, hey, your model start to lose its accuracy. Let's take it back to the experimentation, fix it, put it back to production.

That is interesting that like the idea that once you've launched it, once it's in production, you're not done. You do need to keep monitoring. I think it's the same, a lot of like machine learning and AI stuff as well, is that like the underlying situation of the world changes and then maybe your model breaks and So you do need to keep track of things that way. All right. So I was thinking about a lot of business intelligence and like every company is just filled with all these kind of time series dashboards of like, oh, this metric is going up and down over the last few months. So it's often backward looking. And I guess, how do you go from that situation to now we're starting to think about the future and building in forecasts into your business thinking?

You mean like referring to Specific events?

Like in general, I was just thinking like, I mean, just because I spent a lot of the morning staring at a load of data cup dashboards and I'm like, oh yeah, a lot of these actually, they're just like, this is what happened in the last year. And most of the stuff on dashboards doesn't ever look forward in time. And I guess, is there like, is there a mindset shift there? Or like, how do you go about changing your analytics to start being forward-looking?

That's a good question. It's a big one. Yeah, I think, you know, again, like, Part of a, and again, like this is what I like to do and it may not represent the broader, but when you are building a forecast and delivering your forecast to stakeholders, you want also to be able to explain. This is where there is a gap between data scientists and business stakeholders that are not necessarily technical. And so dashboards are a great way to explain some of your decision that you made about the forecast and ability to also communicate your forecast, to provide your results. So usually I like to explain some of like the process we talked about, the backtesting, to show how backtesting looks like. And then show the forecast. Because sometimes I get asked like, why did you use a linear regression or use those features? So, and so usually when I'm providing a dashboard with the forecast, I also highlight the feature that were used. I highlight what their impact and enable the stakeholders also understand how the forecast is playing in those decisions. So that's one way to, I'm not sure if it's answering your question, but it's enabled to understand the forecast better and it's give like a ability to how to communicate it. I also like to, like, I think visualization is very critical in communicating or explaining forecast. For example, at scale, if you're going back to the scale, uh, thing in time series, when you're using some like all stressing between different models. One way is like you can start by, you have some knowledge and you can start to apply this knowledge and then score and test, or you're coming and you don't have knowledge, but you, you know, you don't think that there is something special in your time series that are very organic growing. And, uh, so you are just applying a set of models and then you get the heatmap of your score and you start to look for the one that are not performing well and then go and tune those as opposed to try to overthink about the problem. This give you ability to move faster in some cases. Not always we have the time to go and think about like when you are doing forecasting and you are like a, you have like a production line that you are a team coming with a problem to give you serious and you need to give them results fast. And, and so that's something I found that is very helpful, like, is to build those pipelines or, like, templates that you are just on board to understand what's going on and generate those forecasts and then figure out what is, what is working and what is not, and then go fix. There are some cases that you are seeing from the get-go that it's not going to work, and then you really need to be more strategic. But for, like, nice time series that is, you know, looks like an academic that is like in nature, that is like you can take a pen and continue the seasonal pattern or, you know, a line. This is work really well.

Yeah, certainly I can see if you want to make this a sort of an enterprise-wide standard that you are going to be trying to forecast as much as possible, then that communication aspect is going to be incredibly important. Making sure that like everyone, not just the data team, can understand what's going on. Like you need all your business teams to be able to look at the forecast and also believe them. Where do you start? Like, are there any metrics that you think are like good first use cases for if you want to start introducing forecasting into your organization?

Yeah, I would, I would think about the stuff that people resonate and understand really fast, like seasonality and growth, right? So those things that if you're working in the financial world, this is like the bread and butter. Like, uh, you want to show like year over year as your series are growing. And, uh, uh, that's a good one that people understand, like, uh, hey, I don't have seasonal patterns. Oh yes, I do have seasonal patterns. So we are using a model that handles seasonality. The second one that I really like to use is, uh, decomposition. Specifically STL decomposition is my favorite. And if you're familiar with this, what it does, it breaks down the series to its components, the trend, its estimated trend, the seasonality. And then the remainder, essentially the remainder is all the patterns that left that are not represented by the seasonal or the trend components. And what I like to do is to enrich this by calculating the standard deviations of the remainder and set like ranges between 2 to 3 standard deviation as orange. And above 3 in absolute value as red, and then point those points on the original series. And then you can start to have the conversation about why do we see a spike, right? There is some anomaly and you can have a better conversation with your business about identifying events as opposed to just stare at a series and see some spike and start to talk. 'Cause this is give you the ability to, and again, like it's algorithm, so it's, it's could also wrong, but generally it's doing a great job on explaining where you have points that are not normal, there are anomalies. And so it's easy to facilitate, facilitate the conversation with the business stakeholder when you're coming with this type of visualization and try to understand, again, going back to our previous conversation about Is it just a one time or we expect it to be in the future and then annotate it accordingly? The other component, the trend, I think that in forecasting, um, the most important component that you want to spend the time is getting the trend correctly because the seasonality, it's easy to capture, but if you are, if you're wrong on the seasonality, you can get away But if you are wrong in the trend, you might get penalized very hard as they, you start to get like this delta and they're separating your series with the forecast, the actuals and the forecast over time. And this is where like STL give you kind of like a estimate, a great estimate of your trend and augment it with another algorithm to detect change points in the trend using piecewise regression. So when you have this and you can automate it, you get a lot of information that could help you when you are feeding it to the model. So again, like we, before you are going to like use foundation, all the stuff. You, I think when you're understanding those concepts, it's easy to understand how the foundation models are working on the backend because they're doing more or less the same. They're learning from many series and maybe they are more like following the reinforcement learning approach, but the idea is the same, right? You can build those algorithm that help you to extract those insight automatically and feed it to the models.

Okay, so you mentioned the idea of divergence between reality and what the model is showing. And actually, we had Bilal Zia, who runs the data team at Duolingo. So he came on the show a few months ago and he said when he joined Duolingo, the data team, they were creating all these kind of forecasts that were completely wrong and no one trusted them. So I think we've talked about a few of these things here, but how do you go about making sure that any forecasts you produce are trustworthy? Um, because you're never gonna get perfect forecasts all the time. So how do you make sure that they're like, the people have the, I guess, the appropriate level of trust in, in this forecast?

I think first and foremost is communication with your stakeholders, uh, and knowing your limitation. Forecast is good if your past reflects the future. There are some cases that you see there, like there are some recent change that, uh, it's not something that your data, uh, can explain. For example, the COVID right? When COVID started, nobody knew where it's going to go and it's impact a lot of serious if you think about it, right? Uh, and so I think telling your stakeholder, look, the historical cannot represent the future. We need to go different approach, like let's do Scenarios, back your business model back to the forecast. Let's say apply some different scenario, what if, and build it in using features. And I think be transparent with your stakeholders where you know that you cannot create forecast that is accurate because of the limitation of your data is important. Okay. Sometimes people think it's like a magic. It's not magic. It's science and science has limitations like every field of science, right? And knowing those limitations and communicating those limitations with transparency, I think it's the first and foremost to build this trust in your forecast.

Yeah, I mean, I do think the idea of communicating with whoever is going to be using your forecast is incredibly important. There's some ideas that are quite tricky to communicate. So I moved to the US in 2016. It was the week Donald Trump first got into power and there were a lot of people like complaining at the election forecasters 'cause if you cast your mind back a decade, Hillary Clinton was sort of the favorite to win and then she didn't win, Donald Trump won instead. And there are a lot of people go, oh well, election science is absolute nonsense. But there was just a lot of uncertainty in the models and that wasn't communicated very well, I think, to the public at large. So talk me through how do you communicate uncertainty around models very well, because otherwise people tend to get a shock.

Yeah, I, so first, the first thing that I'm trying to do, and I'm, you know, I will first admit that I'm not always successful, is to advocate for prediction intervals. Use any type of methods that you want, but at least the goal is when you are, sometimes stakeholders, they just want to know the point estimate because they need to give one number and it's very hard to understand why there is a range because they need to deliver one number. But I think it's important to explain the range and expand that it's not just like, you know, we are the point estimate is what we think that most likely will be the kind of like the middle point. But if you think about 95% prediction intervals, there is a range that the true value could be between This range, expanding it, I think it's very critical. I had a professor in my master's. I took a statistics course at the University of Michigan at the business school, and he was a startup professor. It was really funny. Uh, and he had a sentence that resonated with me since then, and I understand it today more than I used to when he told me, They sent us like, statisticians are very sad people. They know that they're wrong from the get-go and they go and measure it. And so I think that when you're coming with this state of mind that you know that you're going to be wrong, the question is how, how much you're going to be wrong. I think that's a, you know, a very good start that you know your limitation. Uh, And yeah, so I think knowing the risk is of your forecast or how wrong you're going to be is fundamental. And that's part of me, for example, doing backtesting. I love backtesting because it enables you to measure how wrong you're going to be, or at least estimate how wrong you're going to be.

Absolutely. Yeah, I do like the idea that knowing that you're going to be wrong some of the time is an incredibly important mindset. If you're going to be a statistician. I guess communicating that to your boss is often a bit tricky of like, yeah, well, we might be wrong here. It's like, no, get it right. But I guess on that note, are there any other skills that you think are important if you want to get involved in time series forecasting or you want a career in this?

We are going to see more and more AI, right? The foundation models that kind of like self-drivers and But I think that you still need, if you want to be successful, you know, like there's no right or wrong, but I always think that people that understand the statistics behind of it could go farther than people that don't have this background. And again, it's going back to like models like Prophet. People that use Prophet out of the box without understanding how it works or how they can modify, I think they got less from the algorithm than people that actually knew how to use the features inside the models and tweak it and use it. And so knowing statistics, knowing how regression is working, knowing how feature engineering is working, I think will give you more regardless if you're using machine learning, statistics, or AI, right? Foundation models. 'Cause also foundation models, when you use like any model, they're wrong. And then you need to, when you look at the series, you, if you understand how those models are working, you will have sense of why they're wrong. Maybe there is a feature. Okay, let's build this feature. Let's add, fit this features and so on. And I think, uh, That's still requirements that I think that I see it in the near future. Maybe like we get in, things are moving so fast and maybe in, in, in a year from now, my kids could go and do a forecast without them knowing anything. But I think there's still going to be some requirements for some statistical knowledge.

I guess a problem in a lot of areas of data. I mean, I guess with machine learning as we're, I'm DataRobot, we've been trying to figure out automated feature engineering for more than a decade now. And then I guess more recently in the field of generative AI, it's sort of reached the point where a lot of, at least exploratory data analysis can be automated. You just need to give it a bit of business context and it comes up with some quite good analyses. It sounds like with time series forecasting, we're not quite at that state yet and humans are still going to be needed at least for the near future?

I think we are getting close to, I mean, we are improving. We are, you know, from where I started 10 years ago and where we are today, it's like completely different world. And, you know, I was very skeptical about the foundation models when it all started, but I think that's where we are, like where we are going. Definitely they are going to be very dominant, I believe, in the future. And so things are moving really fast. But I feel like still like, uh, we are becoming, and generally in data science, it's not just like a forecasting, we are becoming more architecture or designers rather than writing a code and building, you know, models from scratch. And I think in this process, when you understand as architecture, like think about like someone that doing building, like, uh, architecture to look at the creating buildings, right? They not necessarily familiar with, like they don't really are building it from scratch, they have the knowledge, but they understand the, the engineering, they understand like, uh, the materials that need to be to build the buildings and make it stable or, and so on. So this is the same. We are transitioning to this phrase that we are vibe coding and we're becoming more designers rather than hard coding that we used to do for many years until recently. And in this transition, I think when you understand what's happening in below the hood, you are, I think, can get better results.

I'm totally there with you that there's this sort of layer of abstraction now, like you're not going so much into low level. I can write this specific line of R code or Python code, and now it's more like Well, I'm doing something higher level in trying to get a project sold. Do you want to spell it out a bit more? Like, what does this new data scientist or machine learning scientist time series forecaster role look like to you? Like, what is this architect?

It all starts with like some blueprint of what you want to build. At least that's how I like to do it. Like, think about, hey, I want to build a pipeline to generate a forecast. Okay, what do I need? Like writing a requirement documents. And then start the brainstorming. Or maybe you already have the idea what you want to build and be direct with your edge, with your AI tool that you're using, like Claude or Codex, and say, this is what I want to build, let's create a, a plan. And they're really good. They're really good on give you documentation. And then my workflow is to, instead of like ask it to build it from A to Z from scratch, is like go step by step, make sure that I understand like the component or I modularize it that I can then, if there are something going to break, that I can understand where it's break. I cannot read, like go, like in the past we used to do like a PR that we code review and we go, you know, bigger chunk of code and understand now the amount of code I'm creating in one day is exponential. We cannot review it as we used to do it in the past. So I think part of it is building it in a way that we can then review it with AI tools is critical, but also that understand not let it just write stuff like a black box, but we are building it as a, building blocks that we can then, if something went wrong or we need, we need to maintain, it's easy to understand where we want to go. The other thing that's really the opportunities that I'm really excited about is that today when, you know, we have a, we all have time limitation, that's our big, big constraint, right? And in the past where we used to be like a, like a library, supporting library for forecasting or any application, usually, at least that's what I like to do, is like, you know, get me your MVP. What give you the minimum requirements that you can start to build something? And then you iterate over time, you add more features, and this is where you play with your time you want to allocate for development and to deliver. and you need to have the balance between the two, uh, 'cause your expectation from, you know, normal working place, they want you to deliver and not always aware about the time required for development. And so you are put most of your effort on the critical features. Now, uh, if there is a feature that I always want to have, but I didn't prioritize it, I just go and go ask and in 5 minutes and, you know, it's cost few dollars. Or tokens and you have a feature. And that's what is amazing. Like you can really go faster. The amount of stuff that I enjoy building now is all the list of features that I always have on my backlog that I just didn't have time. I can now build them and really use them. It's amazing.

Yeah. I think that seems to be the case of like anyone working in data is like, You're never short of things to do. It's like, well, things are being automated, but actually, I think there's just been a lack of capability to get through all the things you could possibly do in a data team, like by several orders of magnitude. And so now we're getting to the point where, oh yeah, if I automate most of my job, then finally I'm going to be able to catch up and we can do all the things with data we wanted to do across the organization. So maybe like it's finally where we're reaching a golden age point. Perfect. I love it. Okay. So just to finish, I always want more people to learn from. So tell me whose work are you most interested in right now?

So there is like the, I think the one that I really like the work or this is where I learn all a lot of the stuff that I learn is, you know, Professor Rob Inman from Maniche University. That is, I think all of us that started in our started with his forecast library and then he has his book about forecasting in practice, Principle and Practice, I think that's the full name. And now there is a Python version. So that's something I really recommend for people to start to go and check his blog and his book. I like really the work that Nixla are doing in the domain of time series. They did a great job on building this like, uh, en, like, you know, modules that are, uh, kind of like if you're coming from R to Python, they give you like the stuff, the good stuff that you had in R. And I think part of it, they work with Professor Robinman on bringing all the statistical models and they have like 4 blocks, uh, 4 modules. One is like core statistics. Models like Arima and Holt-Winter. And then there is the machine learning, there is the neural network models, and the fourth one is the foundation models. So they have a really good library and functionality that I like. There is another one, SKForecast. It's two folks that are working on it, and this is like a, you know, I think they're doing it as a They have a full-time job. I believe they're working in IKEA or a similar company, but in their free time they're developing this library and it's a great project and I really enjoy to see, uh, the work they're doing. This is where you see that some of the stuff that I, I see that is okay, they, they probably, this is people that are working in industry. Because you see that the features they're building, it's, you can understand that you need them only if you are working with real life problems. And so there, this is also, they have a great documentation and I highly recommend. So I think those are the one that I, I usually like to work with. There are other, many others that I'm, I'm sorry if I'm probably missing that I just cannot use all the, uh, all the, the one, the, the one that are available, but there is a great community both in R and Python for time series, uh, beyond just Nix and, uh, SKForecast, uh, or the Feeble in R. So yeah, I think they're a good place to start with mainly because they have a great documentation and easy to get started. There are other companies or other libraries like Dart. That also providing similar functionality and are also good option in Python. Okay. Yeah.

I mean, so many options. I mean, such a big community because I guess every organization needs to have this capacity in some form. Actually, so Rob Hindman, he was a very early DataCamp instructor. So he taught a time series course like way back in the day. Every now and then I think I should get him on the podcast, but he's based in Australia. I'm like, Time zones, very tricky. But maybe one day we'll get Rob on the show.

Back to, sorry to cutting you back on Rob Heinemann. I just, when I created my course on DataCamp and I went to see, you know, the portal. So it was really a closure because it showed me that I took 10 years ago when I started time series, I took 10 years ago, uh, the Rob Heinemann courses on DataCamp. That's how I started.

Oh, that's amazing. And it's come full circle. Now you've made your own course. Yeah. You're the new Rob. Yeah.

So that was really a closure. Really, that's a lot of stuff that I learned is from the courses that he made on DataCamp, his books, his blogs. Yeah.

Oh, wow. That's a fantastic story. I guess. Yeah. Congratulations. You've become your heroes. Wonderful. Thank you so much for your time, Ramy.

Thank you so much for having me here.