#35 Data Science in Finance (with Yves Hilpisch)

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Eve Hilpish about how data science is disrupting finance. Eve's name is synonymous with Python for finance and he is the founder and managing partner of the Python Quants, a group focusing on the use of open source technologies for financial data science, artificial intelligence, algorithmic trading and computational finance. Why are banks such as bank of America and JP Morgan adopting the open source data science ecosystem? What are the major sub disciplines of finance that data science is and can have a large impact in? How has the rise of data science changed the financial world and how the work is done and thought about? Stick around to find out. I'm Hugo Bauen Andersen, a Data Scientist at DataCamp and this is Dataframe. Welcome to Data Framed, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on TwitterBown and datacamp@datacamp. You can find all our episodes and show notes@datacamp.com community podcast for those interested, we've also got a special offer this week for Data Frame listeners. The opportunity to try datacamp yourself. All you need to do is email salesatacamp.com that's salesdatacamp.com with the subject line podcast and redeem your free two week trial. Hi there Eve, and welcome to Data Framed.

Hi there and thanks for having me.

It's a real pleasure to have you on the show and I'm really excited to have you here to talk about your work in finance, how you think about the use of Python in finance and the implications of all of this with respect to data science in general. But before we get into that, just want to get a bit of context and learn a bit about you. So maybe you can start by telling us what you're most known for in the data community.

Yeah, I think this is a, this is an easy one. So I'm known for Python for finance, so I started using python more than 10 years ago for finance. Back then we started with computational finance and people said, well, you can't do that, it's too slow, and all these kind of prejudices and reasons why you can't do something. And today the biggest institutions around the world use Python for exactly that, in particular for algo trading. So it has moved a little bit from the comp finance side to the algo trading side. And I think people now know me for this as well. So whenever somebody thinks of coding up their strategy and trying to deploy it automatically for trading. It's yeah, when they use Python and many people book our online courses or listen to talks or reading the books.

So fantastic. And how did you get there? How did you get into Python? Finance now data science. What type of journey led you to where you are now?

Well, after our German Abitur, which is how we finish school, high school so to say, on German Gymnasium, I studied business administration and already focused on the finance side of things. Everything that I did during my diploma in business administration was centered around finance, financial markets, banking and all these topics that interest me from the beginning. I later on started immediately after my start started working on my PhD thesis which was a mathematical finance. So it got more formal over time. Actually I had a topic about dynamic hatching and feedback in markets due to hedging of options. And so really involved in terms of math, what you need. But I really enjoyed it, the analytical challenges and writing some theoretical stuff and using latex and all these things. But in the meantime I started working as a management consultant because I thought, well, math finance is not nice and challenging, but if you later want to work in any kind of institution, any kind of like organization, corporation or whatnot, to have seen a little bit more might be good. So I started as a general management consultant working for different companies on different types of projects. But again the focus was on finance. Didn't leave the financial industry that much. Yeah, after that I finished my PhD. I moved for the first time within Germany to Hamburg. Back then it was around the hype of the Internet startups and the web was then the hot thing. And so we did some with the company I joined, we did some consulting around web topics and so forth. But after the birth of the bubble, this wasn't a walk anymore. So we needed to look for something else. And there's something else. We or what we did, we started our own company. So I founded my first company in 2001. Actually pretty nice because last Friday I was the company for their Summerfest. So I left the company due to personal reasons, moved back to my hometown in the Saarland area due to the family. But we're still on good speaking terms and the company is going to pretty, pretty well. I must say that must have been.

An interesting experience 17 years later being.

Back, actually the company has gone, I know to more than 70 consultants these days, has been taken over by a big large company. So I hardly knew all the people around there. But of course the founders and kind of the older ones, this was Pretty nice for me. Had a nice evening there. And when I moved back, this was actually the starting point where I used for the first time Python for finance, because I discovered it before moving back and I founded my own company, which I still own, which is now the Python Quant. And immediately I got started with some side projects which I couldn't pursue in the other contexts with other co founders that didn't want to do something like that. So now I got the freedom to pursue my own stuff. And this was, among others, this was Python. And we started back then where even Numpy wasn't around. So we used NumArray and Numeric and all these.

Well, that was going to be one of my questions, like this is before Numpy, this is before Pandas, this is before so many of the technologies that people equate with the data science Python stack now. So it must have been a wild landscape.

Exactly, exactly. So some things that are now kind of like standard and that you might teach I had, you know, in the third hour of training or what with Pandas, we needed to code up on our own, like time series management, we wanted to do something in computational finance and all these things. But I'm more than happy that, for example, Wes McKinney started the pandas project and happy about the project having Chrome that much and providing us with all the nice capabilities that we today use in finance. But for sure, we. It was a different landscape. Many people can't imagine how it looked back then, but I was convinced due to the beauty of the language and the whole approach, that this might be the future. And now? Well, this time I think I was proven right, given the success of Python in our industry. And yeah, today we don't do anything else. It's all about Python and finance and algo trading.

Yeah. And particularly with, you know, the emergence of IPython as well, and Jupyter notebooks, which, you know, back in 2001, these weren't around either. So that's very exciting that all these developments have converged in this landscape that allow us to do what we do.

Yeah, that's fantastic. I think from scientific point of view, from a developer's point of view, data science point of view, or financial data science, if you want to put it that way. Yeah, for sure.

So what happened then in your trajectory.

When I was talking about Python being a side project? Indeed it was a side project, so we couldn't make any money out of that. But we did regular other consulting work that I was doing before and I think it Was maybe like six years ago, probably when we started getting real traction, maybe seven years. I think it was 2011 where we got the first big client in Germany, actually Deutsche Bourza, the derivatives exchange Eurex approached us. They actually one of the executives has seen on YouTube a talk of myself at Europython in Florence and this is how things work. And I said well do you do training in this regard and can you support us? And I said well for sure, this is what we were waiting for. And so this was more, less than the formal starting point. And yeah, it has been growing. After working with many banks, big hedge funds around the world and having hundreds of training students, thousands of people on our quant platform that use Python and yeah, so today we don't do anything else. If people look up Python Finance or whatever, I think on Google or wherever they, they usually find us and get to our trainings. To my talks I've given I don't know how many talks, more than, well more than 100 over the last five years and conferences around the world and for sure.

So you write books, you consult, you develop training, you host meetups. Are there other things? I mean not that these aren't enough, but there are. These are the main things that. Where your focus lies currently.

Yeah, actually.

So I basically see us these days as more or less content oriented company. So this is what I think our core is indeed. So writing books of course is about content creation but also designing and delivering online trainings. Also I put in the same category like events where of course you also need some content. We used to do also more conference but these days we focus for example with our partner FitchLearning on the bootcamp side. So this is also more about training and what we need to deliver the content. So content of course is at the core and also I think skills and know how which is then used in addition to the content where we consult clients around the world like working with a big broker in New York, Manhattan or working with a hedge fund in London. This of course requires a certain skill set now how so? But this is more or less where I'm coming from. The content side has become more and more important over recent years, in particular with our online training, which is growing tremendously, which I'm really happy about. And you mentioned the meetups. I'm running a few meetups actually in London, Berlin, Frankfurt, Paris, also one in New York. Whenever I'm there I try to do there something as well. So it's keeping me busy.

Definitely sounds like it. So you're known as the Python quant. You work with a team of people and you call yourselves the Python quants. And I think myself and our listeners know Python is. But I don't know whether everyone has a lot of context around what a quant is. I definitely don't feel I have enough. So maybe you can kind of unpack this term for us. Quantum. Yeah.

Now, if you want to get a little bit academic, you would say, well, we have different types of quants, but I think these days it has narrowed down a little bit. So back in the days when I started with the quant side of things in finance, what people typically understood under quant was kind of the model quant. Somebody who is sitting down and comes up with a specific model, for example, to price a specific type of financial derivative. So a complex financial instrument that might depend on a couple of risk factors like interest rates, or let's say a stock index or maybe a basket of stocks, for example. So people sitting down and really doing research on their blackboard, or these days, probably they're more whiteboards or pen and paper. Back in the days, a little bit, maybe on the computer, but more or less to document. But these were really kind of more mathematicians.

Yeah, and physicists as well. Right. A lot of physicists started.

This is what people called back then the rocket scientists, because of the fact that many physicists came to. Well, it's because of the mathematics that you need to price financial derivatives, pretty similar to what physicists use on a daily basis in many of the areas. So engineers and so forth. So this was the origin, so to say, of the quants, the rocket scientists and so forth. But these days, I think it's much more data driven. So I would say these days a quant is more like a financial data scientist because they need to crunch huge numbers of data. And I think it might differ to some extent to the area you have a look at. For example, if there is, let's say, an equity research analyst who crunches numbers of certain companies, let's say Apple, you typically are not phased with that amount of data. But if you have others that do kind of like more systematic analysis, for example, for systematic trading strategies, they might call themselves even data engineers. So one of the biggest hedge funds out there, most successful ones to Sigma, headquartered in New York, they called these guys indeed data engineers. And because their approach is more or less independent of what the financial professors, people coming up with financial theories say, how the market should behave, they have kind of like a pretty neutral approach and saying well, let's apply whatever technique to the data that we can get our hands on and see how we can profit from that. So this is what I call in all my talks these days, data driven finance. Instead of kind of this equation driven and brain driven finance where people sit down and think of how the financial world should behave, they rather have a look at the data and try to figure out something. Maybe not coming up with kind of this fantastic nice single equation which might award you a Nobel Prize in economics someday in the future. No, but with things that might work, a similar approach that every big data company out there, like all the social based companies like Twitter or Spotify Fan, how they come up with the recommendations? They simply have a look at the data, if there are, recommend their engines, use machine learning approaches and recommend a song number three when people usually hear three in combination with one and two. And this is what these data engineers, these quants these days do as well. So having a data driven mindset and say, well, let's have a look at the data and apply whatever technique might bring us something in this regard. So people working with quantitative things, usually numerical data and more and more unstructured text based data, this is what the quants do these days. So we might have still a few handful of of model quants that I started with, but it's just a handful compared to thousand others that work with financial data in this area.

Okay, and so to reframe that slightly, one of the ideas there is that in the large data limit theory laden models may be unnecessary. Essentially.

Essentially, yes, because when you have a look at the history, people have been pretty successful with coming up with nice models when they made, let's say, appropriate assumptions in the form typically of normally distributed returns and linear relationships. But this is already where all these theories are doomed to fail because we don't have normally distributed returns in markets across all asset classes. You can analyze the returns in the markets, they are not normally distributed. And in general the relationships that you face are nonlinear and not linear. And therefore having a different look at the nonlinear changing world with different algorithms might prove more fruitful than to rely on the old theories from the 50s, 60s, 70s, 80s that are still in use today.

Yeah, absolutely. So in general, what are the major sub disciplines of finance that you think data science is and can have a large impact in?

So I must say I'm in general only on the investment and corporate banking side, so to say I hardly have any point of contact with the retail side. So most people think everybody who will listen to what we are talking about do of course, their financial stuff, maybe on a daily basis with their apps on the phones or online online banking on the web. But this is not the part I'm involved in. Typically there I think there are tremendous success stories of kind of like data science, for example, and credit lending, which is mostly automated these days using machine learning algorithms like scoring there and so forth. But again, this is not my field of expertise. So what I'm mostly concerned with is the corporate and investment banking side and there in particular the derivatives and the trading side. And in general. And this is how I think of this little world, smaller than the world of others, but it's financial data science. So wherever you need to crunch the petabytes of data that are available these days, it is the financial data science. And you might find, we discussed this with regard to the quants. You might find people in different areas of bank, for example, or hedge fund or another buy side company, like an asset management company that are concerned with financial data science smaller or in general these days on a larger scale. So crunching any type of financial data, market data, unstructured data like news data and so forth. Then we have the trading side of things where people try to come up with algorithms and there are different types of algorithms. And more and more we see people trying to apply AI based algorithms, contrary, let's say for example, to some deterministic algorithms that you need to execute larger trades, they try to come up with some machine learning AI based algorithms that predict markets and if they are well enough in predicting markets, they might benefit out of that. So we recently started such an endeavor ourselves as well. Then of course computational finance, which includes areas such as derivatives and options pricing. This also includes risk management, for example. Risk management is still a big, big topic. So when you have a big investment bank that is sitting on, let's say a million plus derivatives positions overnight, one of the major tasks overnight is to come up some risk numbers for the complete portfolio that they face. So maybe people have heard about value at risk or credit value at risk or the XVAs. We have a few more these days in our field and it's really computationally demanding. And such jobs are running on huge clusters with thousands of computers overnight to crunch the numbers in an appropriate way for big bank to get, let's say a decent view on the risk position. So computational finance usually is kind of the most demanding in this field. And this is I mentioned before we got started on a small scale. But these days, for example, the biggest banks in the world like bank of America Merrill lynch for their trading and risk management platform. They mainly use Python for example, as the implementation language. Although the hardcore calculations are still done in C, not so much since Python is too slow. But they started developing their pricing libraries like decades ago. And back then there was no way around C in this computationally demanding area. So financial data science, algo, trading, computation and finance are at least our areas where we focus on and apply data science techniques in the financial field.

Fantastic. And could you just slightly unpack the difference between financial data science and computational finance a bit more?

Typically what you do in data science is that you have a look at the data that is there, meaning historical data, be it on a simple level. End of day data for Apple stock over 10 years, then you have probably some 252 data points per year. After 10 years you have 2,500 data points. So this is not really challenging these days as we know, but this is basically where every financial theory is based on. But rather when you have a look at Apple tick data, which is submitted and provided by Nasdaq via data providers such as Bloomberg and Tom Reuters, with which we work, then you might get some 2,000 tick points per quarter of an hour or 8,000 per hour. So this is then where we get to bigger data. And people need some different techniques than say an Excel spreadsheet for example, to work with such data. But no matter what, it's typically historical data. And you might try to come up with some predictions, some forward looking numbers or whatever based on the historical data. Computational finance, it's more or less with regard to the areas that I've been describing, like risk management over time is based typically on Monte Carlo simulation, which is by definition a forward looking technique. So while I might have a look backwards 10 years in Apple stock, in computational finance, when I want to price a derivative, I have a look forward, let's say over three months, or 12 months, or two years, three years and try to simulate the markets and model correlations between different risk factors to come up with a somehow good understanding of what the future might look like in terms of market prices and other relevant quantities. So my thinking is in that sense that data scientifically looks at the existing data and tries to come up with certain points to predict. But computational finance in and of itself has a forward looking element that is dominant there. Trying to better understand what the future might look like, not coming up with a single, let's say, forecast for the Apple stock price in 12 months, no rather with the distribution of possible Apple stock prices in 12 months based on 100,000 or 500,000 simulations of the Apple stock price.

Fantastic. Now, there are two terms that you've used that I just wanted you to explain briefly for me and the listeners. Derivatives and options. Because none of us necessarily know much about finance.

Yeah, sure, sure, sure. These are actually, they are involved and typically as a, let's say, general investor, or when you say you want to save for retirement, you typically don't get in touch with these instruments, but they are used in many different areas. They are first used to do some risk management. You can use derivatives in general. Options is a subclass of derivatives to do risk management. You can also use to speculate and so forth. But basically what they typically are is this is where the name comes from. Their price is derived from another financial instrument. So, for example, the Apple stock is traded on NASDAQ and you can buy it and price might go up or down. This is a straightforward thing. But there are options traded on things like Apple stocks or on the S&P 500, or on other instruments to this end that derive the price directly from what the underlying. This is how it's called in this case, the Apple stock is doing. And the option, for example, a call option, for example, would represent the right to buy the Apple stock at some certain point in the future at a predetermined price. So options in that sense typically represent some rights which you can exercise, but typically you are not required to exercise them. So this is where the optionality comes from. There are other derivatives like futures, that are unconditional, so you buy them. The price is also derived from something else, but this is more or less live or die. Once in, you're in. You can only sell this thing. But with the option, you have the right to buy something at a predetermined price at the future predetermined date or over a certain period of time, or to sell it. Then we speak of a put option, put options to sell, call options to call. And the pricing of these instruments might get really tricky and involved, and you need advanced financial mathematics in order to come up with a proper price. The pricing of Apple stock is actually pretty simple for a single trader. You simply open your browser and you look up the price and it's there. So, but to price derivatives, it's not straightforward. So this is where many, many people and many, many books have been written in this regard that cover this topic for sure.

Great. That makes perfect sense. We'll jump right back into our interview with Eve Hilpish after a Short segment. Now it's time for a segment called Statistical Lesson of the Week. I'm here with Emily Robinson, a data scientist on the growth team here at Datacamp. Hi there, Emily.

Hi, Hugo. Thanks for having me on. I want to discuss an important, yet often misunderstood topic in statistics. The P value.

I love it. First, for our listeners who may not have a strong statistical background, what is a P value?

Well, let's start by talking about the statistical hypothesis testing process. If you have a hypothesis about differences between groups or the rate in a population, you can test that. For example, maybe your hypothesis is that making the sign up button on your website bigger will increase the percentage of visitors signing up. That would be your alternative hypothesis. Your null hypothesis, which you're trying to disprove, is that there will be no difference in the rate of signups. Let's say you run an A B test, sending half your people to the old version and half to the new version, and measure the percentage of people signing up in each group. After a week, maybe you get 45% signed up in the old version and 50% in the new.

Sounds like it was a success. Let's launch it.

Not quite yet. Hugo, if you flipped a coin twice and it came up heads both times, would you immediately say the coin was unfair? No. You understand that even fair coins don't always turn up tails 50% of the time. You need to take into account how often the results you see would happen if your null hypothesis that there is no difference was true. In hypothesis testing, we do this with a P value. The P value is a probability that if the null hypothesis is true, you get a result as extreme as the one you did. To calculate your P value, you find the appropriate statistical test for your data. In this case, that would be a proportion test. We're interested if the proportion of people signing up in the old versus the new version is different. Maybe we enter our numbers and find that the p value is 0.27. We generally use p equals.05 as a threshold for rejecting the null hypothesis. So in this case, we'd fail to reject the null hypothesis and we wouldn't be able to say that there's a difference between the old and the new version.

Oh, shucks. I guess you can't win them all. So what are some of the most common misconceptions that people have about P values?

Well, what is that? It's the probability of your result being a false positive. So if you run a test and you get a p value of 0.04, people think that that means that there's a 4% chance of the null hypothesis being true. But imagine you had a thousand people all running the same test to see if people are happier when they see the color red than the color blue. Sadly unknown to them, there's actually no difference and the null hypothesis is true. But we know that about 5% of them will have a P value of less than.05 and that they'll reject the null hypothesis. But all of those are false positives. There wasn't a less than 5% chance of a false positive. There was 100%. That's because the probability of your result being a false positive depends on the ground truth of whether the null hypothesis is or isn't true.

So are there any other misconceptions that people generally have?

Yes, sometimes people think that the smaller the p value, the bigger and thus more important the effect. In actuality, the P value can vary just as a function of the sample size. For example, let's go back to our IB test one where 45% of people signed up in the old version and 50% in the new. If you had 1,000 people in each group, the P value of the difference is 0.03. But if you had 10,000 people in each, that P value is now less than 0.0001. The percentage difference between the groups is the same in both cases, but we get different P values just because of the sample size difference. So what most people should really be interested in is the effect size. The P value just tells you whether you can look at it. For example, let's say you have a website with lots of traffic, you run an A B test on a new feature and you increase click through rate P less than.05. But maybe you only increased it by 0.1% and you have to weigh that incremental increase against the engineering maintenance costs of your new feature.

Thanks Emily. I think this will be very helpful to our listeners.

Thank you for having me on and looking forward to doing more statistical lessons of the week.

Time to get straight back into our chat with Eve Hilpish. Now the other thing you mentioned a couple of times when talking about, you know, the major sub disciplines of finance that data science is having an impact in. You talked about machine learning and artificial intelligence. So I was wondering what you see the role of, you know, these two, you know, coupled technologies and ways of thinking about modeling the world, what impact they're actually having or whether, you know, we have a healthy skepticism as well concerning things that are buzz terms as well as things that provide a lot of value. Right. So how does this apply in finance?

Well, actually getting to the trading side of things. So just today a book arrived which is called Pattern Trading. So I discovered this in a magazine on the weekend. I said, well, let's have a look at this book. Looks pretty nice. And already the name suggests pattern trading means trading based on some patterns or price formations that you see with regard to financial instruments. So be it, for example the Apple stock, or this can be, let's say the good price, or it can be the euro, USD exchange rate. And the theory goes that when you see certain patterns in the prices, this might in one case signal a further upwards movement or in another case a downwards movement, or that the market is most probably to move sidewards. But again, it's all based on patterns. And I think most of the listeners are completely aware of the fact that machine learning techniques are pretty good in learning about, let's say, the value of patterns, first of first of all and recognizing patterns, and second of all coming up with predictions based on patterns. My example from before Spotify, when people listen to song A, B, C, D, E, F, G, then the fantastic new song might be something for you because you have also been listening to song abcdefg. And this is the same with patterns. If you see patterns in markets and you might say, well this goes, let's say up, down, up, down, then the machine might learn that with a higher likelihood, it's more probable that the market goes down afterwards or up afterwards. So this is then what the machines of course should do. They should learn what is happening. And when I give talks, I typically have a couple of pictures showing certain patterns and people starting to nod and say, yeah, I know, I know this one, I know this one and I've been trading on that one. But my argument is, and you have been asking me about how machine learning, deep learning, all these is my influencer market. I'm saying usually I'm not saying that there is nothing in these patterns nor that there is something in these patterns. What I'm saying to people is that if there is something in these patterns, then for sure machines are better at recognizing these patterns, at learning these patterns, and then at executing trades based on these patterns. Because I mean, we all know maybe a human being might learn over the course of his trading lifetime, I don't know, 20, 40 patterns, maybe machine doesn't have any issue in learning patterns which are pretty complex, let's say 100 based on 100 features, for example, and maybe 150,000 relevant patterns that it immediately recognizes. And of course when we are about trading, it's about seizing opportunities. The faster you can trade on what you see, the better it usually is and the less emotional you are, the better it usually is. And this is I think where the advance, yeah the advantages of the machines are compared to human beings. I think we are not yet there that in every single area the machines and the algorithms will replace the traders. But there are good examples like Goldman Sachs. You always read this quote. I'm using it also from the Economist for example, that in 2000 Goldman Sachs had 600 equities traders on the single trading floor. And of the 600 there are just two left these days, just two people and the rest is kind of replaced by technology. And of course technology doesn't build itself. So let's say the human resources have been replaced from the trading skill to the technological skill. I mean you need people of course who build the systems and so forth. So it's when people say well this job is about to be replaced by machines, but there must be people who built the machines and who built to write the software that will replace the people. And this is what we see in the, in the financial markets I think in spectacular fashion that they are looking for more and more technologists, data scientists, programmers that are able to build the machines that you acquire these days in order to be successful in markets. But other stuff that has been done manually in the recent past is then not en vogue anymore and even high paying jobs are suffering in this regard as the equities traders that I mentioned here in the Goldman Sachs example for sure.

And actually this, this reminds me, you've got a great two page piece on your website which we'll link to which you've written called Computational Finance why Python is taking over. And you actually you quote Robin Wigglesworth of the Financial Times there. And what Robin wrote is traders used to be first class citizens of the financial world, which is exactly what you were just saying about them. So many thousands being on the floor. But that's not true anymore. Robin. Robin continues, he writes technologists are the priority now and this was three years ago. So I wonder what you've seen now in terms of technologists being the priority. And I presume by technologists you actually include working data scientists in that. So I'm wondering how data science and technology is continuing to disrupt finance.

Yeah, I mean this is exactly, I mean this describes it and it's already quite a while ago when this statement was made actually and what we see now these, I mean this is usually when you have something new, then people try to rush in one particular direction. But I think now we are getting back to a point where we say, well, we might need the market savvy people still. You know, when you hire somebody who is pretty good at programming but has no experience in markets, what would you expect from these people to program into the trading applications in terms of like risk and safety measures and so forth? You know, a little bit of understanding is simply required in that sense. And what we see and think this will be kind of the near future at least is that these days they try to merge the worlds in the sense of that people doing, let's say simple, or rather from a data point of view, simple equity research. They start using our fantastic technologies like Pandas for data crunching and visualization with let's say, I don't know, plotly cufflinks and all these nice things that we use on a daily basis to become better at their jobs and maybe at the same time accomplish more or being able to crunch the ever increasing amount of data. And also the traders, I mean history was kind of like in a way that there was one trader and maybe two people on the left and the right hand side or traders, they were in real time programming their Excel spreadsheet applications if the trader had a new idea. So these days the trader and people were responsible there are probably quiet to come up with their own solutions and use different techniques than Excel spreadsheet where some people are sitting on the left and the right and doing kind of like real time tweaks while the trader is trading. So this is one example that I recently retreated from the fortune where people say, well I forget Wall street lingo, the language Citigroup wants its incoming investment bank analysts to know as Python. So even in a field like investment banking, or let's say people working for consulting companies, they are these days required to have some programming knowledge. This hasn't been the case before. Which M and a banker used Python like 10 years ago? Nobody there. But these days people are instead of expected to know a little bit about Excel, they are, because in every field they are facing these huge amounts of data and people now know that it's much more efficient to crunch these numbers and data by technology such as Python. They are expected to know a little bit about programming as well. And I wouldn't say that everybody these days should become a software developer, architect or engineer. But you know, kind of like we with a little bit of training you can Accomplish quite a lot compared to the traditional approaches in this regard. So more like emerging, like hybrids. I think hybrid is kind of a trendy word anyway. So like the hybrid skill set that is required market knowledge, your background, maybe more banking side, market side or wherever which department you belong to. But nevertheless, to know coding, after all, it's a little bit like math. You know, math never hurts and I think to know about programming and data science doesn't hurt either these days. So English, math and Python, these are I think the three most important languages that anybody should master before getting to a job.

Exactly. And that actually reminds me that I've heard you say, when asked why Python for finance so much, you've spoken to the fact that you consider Python to be the English of the data and financial world. And I'm wondering why that's the case.

Sure. I mean, many people say, well, you know, this language has this fantastic feature, or Julia is faster or this and that. I think most people, and just having talked about the investment bank analysts that are supposed to learn Python, how many languages should such a person in their expected capacity should learn? I mean, it's hard to master a single language properly. So, and I'm coming more from a time constrained resource constraint point of view where I say, well, if you only have time to learn a single programming language, then it should be Python. If you only have time to learn one foreign language, spoken language, for almost everybody around the world, it is these days English. And this is where I see kind of the parallels to say, well, not too many people easily learn three, four languages, neither spoken language nor programming language.

But why Python? What, what is it about Python?

Why Python? I mean, of course it's a proper one. When I started back in the days, this was for me the first proper contact, I must say, with the scripting language, which on a high level allowed fast interactivity. Even back then, even without IPython, you could do on the shell, amazing things and so forth. So I wasn't used to that. So when I grew up, I started actually coding assembler and BASIC on a Commodore C64, you know, this is where I came from. Then I did C at university, compile cycles and so forth. And then for the first of this fantastic scripting language, and this was on top of the interactivity was so close to mathematical language. So when you have a financial theorem, equation or whatever, you have your equations there, it's pretty straightforward and you without that much of a training, you can translate what the math, the finance says to Python. This is what got Me hooked there. I think this is not the major argument these days anymore. From my point of view, Python is kind of the orchestrating language for all the technologies that you need. I think it's the best language to use for data science. It's a first class citizen in the AI world. So when people these days speak of TensorFlow, they use Python as their interface. We have the fantastic scikit Learn package and many, many others. I can't even get to somewhat comprehensive list in this regard. And I think this is what actually differentiates Python compared to all the other competitors like let's start with the compiled ones and the established ones in our field, C Java or with C or with Julia, which is a typical competitor, not in terms of numbers, but in terms of being pretty close with regard to the syntax and their approach and so forth. But the ecosystem is missing and I think the ecosystem is what makes Python unique. And today everybody who wants to enter the financial field, I think on top of all my arguments, which might be subjective, is simply almost every institution has chosen Python as their core language. So if you have some kind of career at the back of your head in our field, Python is simply a good thing to have on your cv because you might prefer some other, more exotic, maybe faster or whatever language. But if your potential employer doesn't use this language, probably you won't get too many plus points on your evaluation afterwards. So this is more the career aspect that if you know Python, you can work in many, many fields and many companies these days.

So Eve, exactly, having Python on your CV and resume is incredibly important. But also now having Python in your GitHub repository is also a huge step in, you know, the interview process as well. Right. When applying for jobs in finance.

Sure. I mean this is one of the fantastic aspects of living in an open source age that you really can showcase what you've been. Of course most people having kind of a professional job, they are usually not allowed even to talk about what they are doing. The financial industry is really selective in this regards and this is why the industry loses many fantastic people, we must say to the more open companies. But due to regulation, legal and all these things, people are hardly ever allowed to really talk about what they are doing. But of course you can do stuff on the side and you can build your, let's say programming or data science CV easily on platforms such as GitHub. And if you want to build a reputation beyond what you do at work, there's of course a fantastic platform. On the other hand, of course these days These are the side effects. There are some companies specialized crawling pages like GitHub or platforms like GitHub and looking for people, so replacing maybe the search on LinkedIn or via other means for talent in this regard. So every once in a while I get approached there as well if I would be interested in changing jobs. Maybe they should improve their research when they would have seen that I'm running my own company, maybe they wouldn't even ask. But for other people this might indeed represent an opportunity to present themselves, to showcase what they can do and also to learn to get feedback from the community. If they're working on something of interest to others, they might even win over contributors or get feedback and yeah, get maybe some fame even in the community for sure.

And something we've really been talking around in the emergence of Python as such an important tool in finance and data science in the use of GitHub and Jupyter notebooks, we've actually seen a huge shift in the past several decades from the use across the board, not only in finance, but academic research and all types of quantitative computational disciplines. A move from proprietary tools to open source tools, right?

Yeah, sure. And I mean this is also one of the benefits if you have a huge community behind it. I think hardly any commercial company can keep up because such community through an effort. If you have millions of users of Jupyter notebooks and not only for Python, of course we have multiple other kernels that can use like R kernels or Julia kernels and use the same environment. This is fantastic and the project has been growing tremendously. I'm really happy about it. Also for us providing content as that we are more or less a content creation company. It's a fantastic means of creating content and of sharing it. For example, my I'm currently writing on a second edition of my Python for Finance book. It's basically all Jupyter notebook based all the codes of Jupyter notebook and I have one of my books, list of volatility and variance derivatives was actually written to 100% in a Jupyter notebook and then I programmed a little workflow behind it, translated it into latex and finally Wiley published what I got all started and written down in Jupyter notebook. So many, many things that you can do. Also with regard to what I'm a big fan of reproducible research. So when I was growing up doing my research, you just had back then mostly printed research papers and people presented just their results, but you never could somehow get to the point where say, well, what Data have they were working with? How did they crunch the data? Are these results that they present reliable? And so forth. And today with access to many open data sources and providing for example a jupyter notebook not only with text but with the code that crunches the data and presents results. It's fantastic when you're after reproducible research again, which I'm a big fan of.

Exactly. So for someone who wanted to get started in financial data science and these types of things we've been talking about, what are the main types of skills and technologies that they would need to know to work at the intersection of finance and data science?

Yeah, I mean Python for sure. I mean there are many, if you believe in kind of like polls and overviews and surveys and so forth. People typically say is that that the most common combination is Python R&SQL. So you need somehow some databases. I think we have multiple other options these days like HDF5, we don't need SQL, but Python for sure. Maybe some R here and there if you don't find let's say statistical package in the Python universe. But this has hardly ever been the case for what we have been doing for ourselves and also for for clients and of course machine learning, AI in general, to know about the basics of statistics, to know about the algorithms, what unsupervised learning is supervised learning. All the techniques there, like support vector machines, logistic regression, Gaussian, naive, Bayes, whatever jumps to mind. I'm pretty sure you don't need to be an expert in every single area of this field. So knowing all the theories, more like from an applied perspective, to know what is there and to know what to apply when given certain data sets, then actually the points which I usually summarize on the basics. So we have for example in our certificate program, which is the largest, the longest running online training program that we have a complete area, tools and Skills, where we teach basic tools and skills that people from our point of view to know at least a little bit about, like using editors, processes, kind of setting up environments, deploying stuff in the cloud, working a little bit with Docker containers of course alone. The topics that I've mentioned right now, they probably require complete study over years, but to know the basics typically help you pretty much like basics of Linux, a few command line tools, some dev tools, Python, packaging, publishing, testing, all of these topics, software engineering and basics in general. Not being an expert maybe after that, But I think first 10% help you already with the 70% of your problems. Then about data storage, working with Data is important, of course, particularly in the financial field. Thinking of trading, if you implement back testing programs, you need to be able to fast work with huge amounts of data, to crunch the data, to store it correctly, to store your results and all these things, but also for example, to work with streaming data, which is actually pretty important in our case. Not in every area you have the need to process data in real time. But in finance generally it is the case that, for example, in trading, when you do algo trading, you need to be able to digest, tick data, streaming data in real time, to crunch it, to maybe resample it, to come up with your signals in real time based on your trained models and to act in real time. So this is from experience, something that people at least having a hard time getting started with. But it's simply required because otherwise how do you want to automate things with regard to algo trading if you don't know how to work with sockets and streaming and maybe also even streaming visualization for example, to implement your trading strategies? To keep with this example, but I think these are kind of like from my point of view, the skills and technologies that people should know about a bit.

Great, that's very useful. And I think it isn't as though you just need to learn all of these things straight away. I mean essentially you can learn them on the go in a project based way.

Yeah, sure. This is what we usually tell people as well, that we say, well, we provide you with a basic overview, maybe skip some details in the beginning. So with regard to our program, we have, let's say a 12 week structured schedule, so to say where we try to cover the bases, but afterwards it's more about practical things, practical modules and people are supposed for example to do a final project and there they can select a topic, but they are then expected to apply the different things. And this is from experience where really learn about the stuff. Maybe you know about it after the formal education part, but you'll learn about it once you get started applying it. And this is what people write me back. They say, well, I couldn't even imagine that one day I would sitting here spinning up cloud instances and deploying Python code in the cloud and doing remote monitoring based on socket streaming and so forth. Now it feels like second nature. But I'm pretty sure when these people for the first time saw what this is all about is, oh, I will never master that. So this is, I think this is kind of natural, this reaction. But of course applying it to something that interests you when you have A purpose. This is when you get started learning about stuff and learning to master it.

Exactly. And you've half answered this, but my next question really was where would you suggest beginners who want to work in financial data science?

Look, yeah, of course, I mean this is. I can hardly say anything else because this is at the core of our business that we provide online trainings, also live trainings, events in the form of bootcamps in London and New York usually. But if they have a look at our pages, we have kind of a broad offering which has been growing over years and is of course influence or what we have been done over more than 10 years. I've been working for the biggest hedge funds for big banks and other financial institutions in this regard. But we know what is kind of expected. And this is on one of our sub pages, training tpq IO and the certificate that I mentioned before, this is really our flagship offering where we have a 16 week program after which we hope that you're able to use Python for algo trading or for other financial things. But I got Trading is the focus. Indeed. And we even are able to award a German university certificate because we are cooperating here with our local university. And if you are doing a master program within Europe, this is even good for the so called 5 cts points European credit transfer system. Might be interesting then only of course for like current students or future students in this regard. But it's a formal certificate from university for Python on in Aga trading and or finance depending on your.

Fantastic. And we'll make sure to link to those in the show notes as well.

Sure. Great.

So in general, what does the future of data science in finance look like to you?

Yeah, I mean this is from my point of view an easy one. Data science will become a core discipline of finance. I said it before, we come from an area where the brains and math equations have been driving finance in this regard, but data driven finance is what will replace it. So we might lose quite a bit of beauty in financial equations and modeling and so forth, but on the other hand we might get back what I like to call the scientific method in this context that we start with the data, we have a deep look at the data in any area, every area that we have in the financial industry and apply the new algorithms. And in that sense I think finance will be more driven by stuff that is developed outside of our industry than ever before. So meaning that of course these nice theories that are still around and are applied and some of them successful, others not that much, they Usually came from finance professors or finance practitioners, but now people start using stuff maybe developed within, let's say Google. And major point was to have a good algorithm to play Go or to build a self driving car. So completely different background but in that sense the background can easily be changed to a financial background with some adjustments. And in this regard I'm happy that we have since March this year the first proper book about financial machine learning. Because many algorithms can easily be transferred but we have some specifics to consider when we apply the algorithms with regard to the data, how we crunch the data, how we manage the data, what are kind of special things when we look at the financial time series in contrast to let's say a physical time series of whatever kind. And in that sense it will be commonplace, not something special, it will become a core thing. As you say, well, what do you think of technology in the banking industry? Many people say banks are essentially technology companies. And I think the data driven AI first finance future is not too far away from my point of view.

So my final question is, do you have a final call to action for all our listeners out there?

Yeah, sure. I mean focusing again on the Python for finance side and it started some, let's say three years ago when people reach out and say well I'm interested in machine learning, I want to apply it in finance, I, I having this and that background, want to make a move. And today when people try still today, I must say when people try to get into the field and to apply data science and machine learning and try to profit, be it within corporate context or as a retail algo trader, let's say trading their own cash positions there, it's still hard to get up and running. What I can only recommend you to experience and since we are doing it, is to allow look for kind of an appropriate integrated training program. You have that many things for free out there, you have that many university based things that might be remotely related to what you're looking for. But many people told me and confirmed that they have wasted months and months trying to look for that stuff on their own. So we have done a living out of putting this all together into integrated training programs documentation. So for example, with our program we get a 1200 page documentation only about Python for finance and algo trading. So it's quite a bit and once you have found something like this, it doesn't need to have to do with what we do. Not everybody's interested in algo trading, but something like this where I said well this is a good starting Point this provides me with the product overview but also with the details that I need to get started. Then you should focus on the basics. And this is where many people are in our program. I wouldn't say really complain but when I say well, I'm having a really hard time and it's with mastering the basics. So for example, for the first time people may be just having a Windows background, having used the regular tools there and so forth, setting up a droplet, moving around in a Linux environment and so forth might see kind of a really difficult thing. But I mentioned it before, some people after a few weeks said well, now it feels like second nature. So this means you can get to the point where you're, where you know how to use VIM via SSH access to a cloud instance, you deploy your automated code and work with sockets and whatever. So master the basics, have a look at the nice tools that are there in the Linux world and master the processes there. And I think then once you have get around the training, have the basics ready, be it more on the finance side, more on the programming side, you should get started with implementing many little projects. So that interests you. They say well, well this is what we typically do as training examples that you might say, well, I want to have a little app where I simply put in, let's say a stock symbol and my flask app that I host on DigitalOcean Troublet then shows me a graph with some simple moving averages. So for somebody who knows what this is, this might be. This takes less than an hour if you know what you do. But if you get started with these things, this might take like a week or two. But to have these projects where you have indeed at the end a result that you might even showcase to your colleagues, friends that you might put on GitHub or whatever this is I think where the learning curve gets really steep and towards the end I'm pretty sure everybody who wants to enter this field and wants to learn about all these fantastic technologies and the challenges in the field, they might have something in their head like building their own algorithmic trading operation or coming up with their own derivatives pricer or coming up with a machine learning application for portfolio management, for example, then they should, after that they should come up with such a project, scope it, specify it and get started and then work over weeks or even months and build something huge. I think this is then where you, where you collect on the way all the pieces that might have been missing before because you have been really focusing on a few things. But once you get to the point where you master such a huge project, I think you can say, well now I, I'm proficient and work in the industry, I'll get my major project then up and running.

Fantastic. So to recap, to select an appropriate integrated training program, then master the basics, then get your hands dirty with as many little Python projects as possible and then do a larger, more challenging project.

Perfect.

Fantastic. Eve, it has been such a pleasure having you on the show.

Thanks for having me. It's a pleasure as well.

Thanks for joining. Our conversation with Eve, we saw that one of the main drivers for the adoption of Python and open source software and finance has been the amount of tools available and in constant development from Numpy to Pandas and beyond. EVE made clear that all the big banks and funds are now using such tools. On top of this, we saw that banks are essentially technology companies and that the data driven AI first finance future is approaching quickly. With the masses of data, computational power and tools we now have, along with the pattern recognition capacities of machine learning, there is less need for mathematical modeling and more for data science pipelines. And for all you learners out there, if you're interested in this type of work, select an appropriate integrated training program, then master the basics, then get your hands dirty with as many little Python projects as possible, and then do a larger, more challenging project. Also make sure to check out our next episode, a conversation with Christy Bally, Assistant professor at Kent State University, about data science, ecology and the adoption of techniques such as machine learning in academic research. What are the biggest challenges in ecology that data science can help to solve? What does the intersection of open science and data science look like in scientific research? What is happening at the interface between data science and machine learning methods, which are pattern based, and traditional research methods, which are classically hypothetical deductive? Is there a paradigm shift occurring here? Join us next week to find out. Now don't forget that for those interested, we've also got a special offer this week for data framed listeners. The opportunity to try data Camp yourself. All you need to do is email salesdatacamp.com that's salesdatacamp.com with the subject line podcast and redeem your free two week trial. Well, I'm your host Hugo Bown Anderson. You can follow me on Twitter hugobown and datacamp@datacamp. You can find all our episodes and show notes@datacamp.com Community podcast.

Sam.