The Problem with Trading Probabilities | Trading Pychology

Hello and welcome to lesson seven of the Advanced Psychology course for High Performance trading. This video is looking at probability and probabilistic thinking, and in my opinion, this is by far the most important topic related to trading success. Despite everything I've talked about in the last six lessons and all the many maladaptive mechanisms that sabotage our trading, a lack of understanding of probabilities is often the the root cause that triggers all of those faults. To start with, if there was only one thing you could work on to improve as a trader, probabilities would be it. I originally intended to put this as the first video in the course. I'm so convinced that nothing else makes sense in trading without probabilistic understanding, but I decided it would actually be more beneficial if we look back at this in the context to the other trailing problems that we've already covered so far. Now split the topic into two videos, with this first one being more on the theory side and the next video showing you practical examples, especially as it relates to trading directly. Now, in terms of System two oversight objectives, what we're focusing on is System one's false assessment of probabilities and its need for certainty. Any system 1 assessment of probabilities in trading will usually be presented as a strong feeling that something about to happen. It feels like good intuition. And of course we're used to going along with what feels right, particularly with our intuitive assessments of probability, because actually our intuitive feel of probability is usually right. For instance, if you see a black cloud building on the horizon and the temperature suddenly drops and the wind picks up, your intuition will tell you that it's probably going to rain soon. And most of the time you it's correct because it's been exposed to a lot of weather and it's learned to evaluate the chance of rain in the next hour or so reasonably accurately. But in trading, it's much harder to build accurate intuitive probability assessments because our brains did not develop to function in financial markets. So let's start with looking at how system one struggles with probability. Lets say I've got a deck of cards and I pick two cards randomly from the deck. What do you think the odds are that the first card is red and the second card is black? Pause the video if you want to try and work this out. Well, the first part of the question is quite easy. The odds of picking a red card first is simply a half or 50% or 0.5, depending on how you want to write that. That's easy because half of the cards in the deck are red. But what about the second part of the question? Well, if the first card I picked was red, then the beck is now one red card down. So the odds of picking a black card second are now 26 over 51 or 50.9%. And to answer the original question on the probability of picking a red card followed by a black one, we need to multiply those two probabilities together, which gives us 25.5%. Now, of course, this takes time to work out, and your intuitive or your initial assessment of the probability may have been completely blank. But if you did have any idea at all, it was probably false until you've done the hard work using System 2. But historically, we haven't used maths to calculate probabilities accurately. We've used intuition or experience to make estimates like that ring cloud example. Even if we did have maths available to us, it's just too slow. In most real life situations, System one wants to act right now, not get out a pen and paper and work this kind of stuff out. Throughout nearly all of the long human history, our understanding the probability was very simple, and it thought more in terms of certainty. We would intuitively conclude, look, this lion is almost definitely going to kill me unless I run away. That is far more useful to us than sitting down and trying to calculate the exact specific odds. That could be 8 to 7.56%. As far as we know, the earliest record of humans playing gambling games was only about 3,000 years ago, which in real terms is extremely recent. That would have been the first time that we even attempted to think about probabilities more mathematically. Financial markets for speculation have only really been in our culture for a few hundred years. So historically, System one has barely had any exposure to accurate probabilistic thinking, which is why we don't really have a particularly advanced mechanism for calculating or assessing probabilities. Instead, System one has three seconds, yes, maybe, and no. Or in other words, it definitely will. It might or it definitely won't. But as I explained in the second video of this course on decision evolution, System one does not like the maybe state. It hates uncertainty because it doesn't know how to react. And that makes the maybe state highly stressful. When faced with a tiger, it's no good thinking that it may or may not attack us and wait in there in indecision. We need certainty quickly, so we'll assume that it will attack and we will take precautionary and defensive measures. This really sums up System 1's Probabilistic Assessment Mechanism and out of the box. It's just not refined enough for trading. If we zoom in on our more recent history of the past 2,000 years, we can look at how we've tried to cope with this poor understanding of probability in an increasingly complex world. In the first 1500 years of this period, we were playing games that involved chance, but we didn't see things probabilistically as such. The outcomes were usually attributed to fate, luck, or divine intervention. And we very much saw things as a cause and effect. We were winning because our lucky charm was working, or the sacrifice we'd just made to the gods was being rewarded, or our skill at rolling the dice. And if we were losing, we had the wrong lucky charm, or because we'd offended our favorite God, or we'd rolled the dice poorly. System one always looks for cause and effect and has never really grasped the randomness of the world around us. Odds and probabilities as we understand them today just weren't a thing back when probability proper only appeared in 1654, when mathematicians Pascal and Fermat were trying to work out how to improve their chances at winning a simple getting of dice. That was when the mathematical age of probability was born, allowing humans to calculate probabilities with much greater accuracy using System 2. This is what allows us to calculate the exact probabilities. Like the card question I gave you a moment ago, we can be precise because we have all of the necessary information and data points. When financial markets came along, we might have hoped that we could simply apply a formula and calculate the exact probability of a trade working out. But in markets, we don't have all of the information needed. Financial markets are far too complex, with far too many unknowns and different variables that change from one second to another. It's impossible to be mathematically precise in trading, so we've got no option but to fall back on System one's intuitive estimate. So we're back where we started with this. A system that only understands probabilities in terms of it will, it might, or it won't. And as I said, that maybe state is uncomfortable, so we'll do anything to reduce that uncertainty. Our ancestors survived because of this mechanism. They weren't taking any chances with maybes. And we still do this every single day. When you get into a car, you put a seat belt on. But if you think about it, Matt produces an error 99.999% of the time. In the overwhelming majority of those situations, we drive around strapped into an irritating harness that provides no benefit. It isn't needed or used. But the maybe state of wondering if you might need it just this time, well that's too uncomfortable for us. Especially when the consequences are are so severe they could be fatal. So rather than trying to accurately calculate the odds of the belt being required on any given individual journey, it's just far easier and more comfortable to remove the uncertainty and just assume it is needed, no matter how improbable. Now because this is such an important and detailed topic, I'm going to summarize this as we go along to help this information sink in. We understand now that for most of our existence we we have only really required a two or three stage perception of probability. It will, it might or it won't. This is great if you're trying to work out whether you're at risk of being eaten by a predator. Because the uncomfortable maybe state pushes us to make a quick decision and take action. The bad news is trading is by definition an activity is always in that maybe state. It is more like a dialogue with the possible outcomes in a continuous range, varying degrees of maybe, which is not comfortable for us at all. A system one seeks certainty and it will do anything to get out of this maybe state. This is the major reason why retail traders, even with profitable systems, don't succeed. They assume that the uncomfortableness they feel indicates that something is wrong. So they spend most of their time trying to reduce this uncertainty by by layering more tools, indicators and filters onto their system. And we all have this tendency towards wanting certainty. It's not just in trading, but in everyday life. The best example of this is weather forecasts. Weather forecasters the world over get very frustrated at their viewers lack of understanding of probability. But more importantly, they need for certainty. They get complaints all of the time that their forecasts were wrong. But people mistake high probability for certainty. If the forecast says there's a 90% chance of ring, most people will just take that to mean that it will ring or that a 10% chance of rain means that it won't ring. That's a classic case of system one moving things from the uncomfortable maybe category into the it will or it won't category. Whatever the forecast and whatever the weather, everyone can easily believe the forecast got it wrong. It's not that we're all stupid or that we all enjoy complaining. We are simply grappling with System 1's inability to accept the true nature of probability. The US election back in 2016 was another fantastic example of the general population's difficulty with probabilities. There were long periods where Clinton was way ahead in the polls. And by the Election Day, she was forecast by with over 70% chance of winning. And most people, especially the media, took that 70% to mean certainty. When Trump won, both sides, Democrats and Republicans, turned on the pollsters, claiming that they've got it all spectacularly wrong. Now, whether the pollsters had failed to take some information into account or not is irrelevant. Claiming they were wrong is in itself wrong. The probabilistic assessment of a single event cannot be wrong by definition. If you thought there was a 95% chance that a trade would move in a certain direction and it didn't, the assessment wasn't wrong because obviously there was a 5% chance that it wouldn't go that way. And that is what happened this time. The only way to say a probabilistic prediction is wrong is if you can look at a track record of the same prediction under the same circumstances or over multiple occurrences. We all struggle with probability, not least because System one mistakes high and low probabilities for certainty. If I put 700 gold coins and only one copper coin in a bag and then asked you to pick one out randomly, you might feel absolutely certain that you were going to pick out a gold coin. But if you didn't, you would find that hard to accept. You know, what are the chances? This can't be a coincidence. And that's the point. We really don't understand or accept randomness. Here's a picture of someone who wanted the floor tiles to appear random, so they shuffled the tiles and they picked them out at random, and this is what they got, and they weren't happy. Why, if so many dark tiles clustered around the edge instead of being distributed more evenly? It turns out we don't even understand what random looks like. This is random, but this is what people expect when they ask their bathroom or kitchen tiler to create a random pattern. We expect an evenly distributed pattern, which is not random. So why don't we accept randomness or at least recognize it? Well, System one is a pattern spotting machine. I would argue one of its main functions is to look for patterns in the chaos to make meaning and order in the environment. You can test it almost whenever you like, simply by looking up at the clouds. If I ask you to look for animal faces, eventually you will see them. And the more you see, the more you'll find you don't derive meaning from the clouds because you are aware that this is just an enjoyable optical illusion. But in our past, there's certainly been many situations where we faced random information spotted patterns, and we have seen meaning from it that have shaped our lives. Look at the way our ancestors have interpreted stars. For millennia, in every culture, people have seen shapes in the random positions of stars, literally joined the dots to create constellations, characters and people, not just for a good campfire story, but to actually help us make sense of our mysterious world and remove the uncertainty so that we have more control and can predict the future. But here's one of my favorite examples of System 1's struggle with randomness. It's from when the ipod was first released. They had tens of thousands of complaints saying that the random shuffle feature wasn't working correctly. Certain songs were being favored again and again, and other songs were never played at all. And Steve Jobs tried for over a year to explain that this was a problem with our perception of probability. But ipod owners weren't listening. So in the end, they created an algorithm to evenly distribute the songs and the complaint stopped. Now, every single device that has a random mode is not random at all. And this really highlights a big problem for traders. If System One spots a pattern in random information, by default it will assume that it is not random and it will assign mean to it. Now, most price moves in all markets are random, so you can just imagine the false patterns that traders are seeing on their chart and falsely using to make decisions about the future. Now, I'll cover pattern recognition in detail in lesson 11 of this course. We'll come back to that. But the way people play lotteries is another great example of System 1's maladaptive understanding of probabilities. Let's say I have five lotto tickets and I offer you one. Pick the one you want, but order all of them from the least likely to the most likely to win. And again, you can pause the video if you want some time. Now, if you did place them in any sort of order, even if you intuitively felt you did, you've been tricked by your own system 1. All of these tickets have exactly the same odds of about 14 million to 1 in this case. But when we see any kind of pattern, we assume that it must be ordered, not random. Let's be honest, who buys a lottery ticket with 1, 2, 3, 4, 5, 6 as the numbers? But that sequence is just as likely to be picked out randomly as any other sequence. And if you won with those numbers, you'd likely keep all those winnings yourself. Whereas sequences that seem more random tend to actually be picked by more people. This next slide is part of a common university question that is given to Students studying statistics Here are three possible daily results of the sex of the babies born in a real maternity ward picked randomly from one day of the year. Only one of those sequences is real. The other two are false. Which do you think is the real one? Remembering that the odds of a baby being born, a boy or a girl, are 50 50. You're probably onto me by now, and you know not to go with your instincts. The answer was B on that random day, all of the babies were girls. But nevertheless you will agree, even if you know what I was up to, that your intuition by itself would immediately think that C looks more random and B looks like it has order to it. In truth, all of the sequences are just as likely to appear randomly. But you can see how difficult it is for system one to randomness. Let's have a look at a series of eight coin flips. Let's say the first comes up heads, but then the following six all come up tails. If I offered you a hundred dollar bet to guess the next result, do you fancy your chances? Well, you might, because your intuitive understanding the probability may have assumed that because the last six were tails, it's much more likely to be heads next. You may not, but the vast majority of people would take that bet or think that way confidently assuming that the odds are now in their favor. For then they think there's only a slight risk of being wrong. And this is one of the simplest, yet one of the hardest concepts for system one to grasp. The past does not affect the future. The last six conflict B entails does not change the odds of the next one. And we may all agree with this intellectually, but even so, many people admit that although they know that this is true, it just doesn't feel true. What's really happening There is System 2 agrees with the odds, but System 1 disagrees to some extent. This is a form of mental accounting. You've remembered the results from the last few flips and their mental accounting is distorting your perception of of actual probabilities. But the odds on every single coin clip is always 50 50. To believe otherwise is called the gambler's fallacy. Traders need to be very careful of this. You'll often see people increase their position sizes after a string of losses because they think that a win next is more likely or has a higher probability or vice versa. After a winning streak, they reduce their position size because they think a loss next is more likely. The outcome of random events like coin flips or individual trades do not affect the next outcome. You'll often hear experienced traders say that the market can remain irrational longer than you can remain solvent. What they're really referring to is this misunderstanding, the probabilities and the connection between past and future system. One is always trying to reduce uncertainty and look for cause and effect. So it's always on the lookout for patterns and, and it will find them even in random information. And it will then assume meaning and even make us think that the odds are changing. The problem is even worse when the numbers are large. We intuitively get 1 in 4 or 1 in 20 because we can relate to those small numbers from experience in everyday life. But with very large numbers, we just can't comprehend what they even look like. If you place a 1cm coin anywhere on a tennis court, what are the odds of a randomly dropped ball landing on that coin? Is it 1 in 1,956,277, 1 in 12,355,776 or 1 in 759,221? Well, it's impossible for you to calculate that. Intuitively used in system one, each of those numbers is so big as to mean nothing. As far as our understanding of probabilities, one guess is as good as another. The answer is a by the way. So when it comes to probabilities, odds with large numbers, even just over a thousand, are very hard to grasp. We just lump them all together, making us bad at differentiating between an unlikely event and and a really, really unlikely event. And when unlikely events do happen, we can't accept that it was random. We assign meaning to it. Cause and effect. If you've just had 15 losses in a row, rather than accept it as being the random distribution of results, your system one is going to infer that unlikely is the same as impossible, and it will instantly assign some meaning, such as you, your system or indicator must be broken and you need to look for something new. Now an interesting, if not somewhat tragic, consequence of this, especially for financial markets, is our reliance and trust in talking heads when it comes to any prediction of market activity or direction. Somebody somewhere will get it right, not because of any skill or insight, but simply because if enough people are making predictions, someone will be correct. The media especially struggle with this basic concept. When an event happens, the TV companies scan their records for anyone who supposedly made the right prediction and will question them endlessly about what they think is going to happen next on the false assumption that their next insight is worth following simply because of the success of their last one. One of the most famous Examples of this was when Elaine Garzarelli, a financial analyst, was suddenly catapulted into the media spotlight after apparent the Black Monday crash in 1987. Financial journalists went crazy dubbing her the wizard of Wall Street. And for the next 10 years she was a regular on various shows with everyone hanging off her predictions. Now, as you might guess, someone who actually understood probabilities, a guy called William Sheridan, did some analysis of those 10 years worth of predictions and it turns out she was wrong 62% of the time. The point is, in the short term, with single events, predictions and probabilities are almost meaningless. Somebody will get it right purely from randomness. Outcomes that appear to follow patterns and appear to have a cause and effect are illusions. One of the best programs you can watch to demonstrate and understand dysfammarcy is Derren Brang's the System. It is on YouTube and I'll put a link in the description. I won't spoil it. But it is a fantastic explanation of how people fool themselves about their own trading system or get away with convincing other people that they have over your edge and predictability, when really it is all random luck. Now you've obviously grasped that System 1 and System 2 process will evaluate probabilities very differently. System 1 relies on its experience, its ability to recall previous situations and maybe even simulate the possibilities. So it's easy to intuitively grasp the probability of rolling a six on a dice because the odds are in a simple range and you've done it many times, so you have experience to call on. But in most cases, intuitive probability is only an estimate. For instance, if you were trying to work out the probability of hitching a ride on a road, you wouldn't come up with a specific number like 1 in 36 or 2.77%. Instead, System 1 would estimate that your best chance might be in the morning. If you were dressed smartly and you had a well written sign, you would intuitively calculate the better charms, not the specific value. Intuitive probabilities are just an estimate. They're first, but they are limited by our experience and have a complexity limit. Objective probability, on the other hand, is what we use to assess more complex probability problems. It's more methodical mathematical and Therefore is a System 2 process. Intuitively estimating the chances of rolling the six may be relatively easy, but what about two sixes in a row? Pause the video if you want to have a go at that. You'll notice that you didn't instantly come up with an intuitive answer. System two is needed and that is much slower and it requires much more effort. To get the answer, you need to multiply the probabilities together, which gives us 1 in 36 or 2.77% chance of getting two sixes in a row. And the more the complexity grows, the more methodical we need to be. For example, what are the odds of rolling four sixes in a row? Well, now even system two struggles and we need to involve a calculator. And the answer this time is 1 in 1296, or 0.0007%, which incidentally is the kind of number that system one can't appreciate. So it just calls it unlikely. And we might take that to mean definitely not going to happen. Now, before you begin to think that being a successful trader, you need to be an expert on objective probabilities and mathematics, you don't. To make accurate objective probability calculations, you need access to all of the available information, and that's never available to you in trading. Financial markets are far too complex with too many constantly changing variables to ever be able to do accurate calculations. All you need is a better trained, intuitive understanding. It's something we developed way before matters even taught or understood. You may have heard of the children's game called Poppet Pirates. So you start by pushing the pirate into the barrel and twisting which primes one of those slots. Then each player takes turns to slide a sword into one of those slots, and at some point, one of the swords triggers the spring and the pirate pops out at the top. Kids love the suspense, because the longer the pirate is not popped, the more the anticipation grows. In a sense, the child feels that probability increasing. They intuitively know that it's unlikely the pirate will be popped on the first turn. And if it does, the they're amazed. But as the game goes on, pop in becomes more and more certain. So intuitive probability doesn't involve max, but it only grasps probability easily in simple tasks like this, because it knows what has already happened and it knows what must happen. But when the same mechanism is applied to trading, it simply falls over. Mainly because the edge in your system is likely to be slight and it needs a very long time to play out. When you play poppet Pirates, the odds of the next move popping the pirate does change. It gets more and more likely. But in trading, the odds of the next trade doing anything do not change just because of what happened on the last few trades. A winning trade is not more likely after a series of losses. You have to be especially aware of this when because your system one will intuitively view trading the same way it views poppet pirates. And again, what's happening here is an issue with our memory recall. We can't calculate the objective probability of a trade, so we fall back on other mechanisms. The issue with System 1's probability intuition is that it stores emotional, anecdotal and unusual data and as a priority over the mundane information, so you don't get an evenly balanced view. And it will infer a generalization after anecdotal information. If your friend told you that he got mugged on a certain street, your brain infers that street is now unsafe, even though it is only a single data point for you to go on. If a friend that you trust told you that she had food poisoning after eating at a certain restaurant, it's highly likely that this would alter your probabilistic assessment of whether you would ever eat there. Emotionally impactful and vivid accounts like that are really what drive intuitive probability assessments. It's a mental shortcut to help make decisions when you don't have good statistical information. But even when we do have good statistical information, we tend to disregard it because it doesn't have the power and the conviction that comes with a first hand emotional account. And that makes sense. We've evolved getting our information from friends, family and fellow villagers, not from poor and overt statistical spreadsheets. It's why we fear terrorism and play the lottery. The odds of experiencing either are minuscule, but System one still thinks both are more likely than they are because of the availability, the vividness and the repetition, and importantly, the emotional content of the information. Much more common and likely events are not conveyed to us in the same manner. You don't get wall to wall news coverage every time there's a road traffic accident or someone falls down the stairs. And of course there's no news stream constantly announcing all of the people whose ticket didn't win the lottery again for the thousandth time. So our intuitive assessment of probability, risk and reward is completely distorted by the availability and the emotional content of the information. In fact, after the Twin Towers were hit and a whole nation was repeatedly shown traumatic pictures of planes crashing for days on end, understandably we all concluded that plane travel was completely unsafe and we chose to drive instead. But Tragically, more than 1600 more Americans died on the roads in the 12 months following 911 than any other year as they tried to avoid the perceived risk of flying. Emotions tend to overweight or underweight probability assessments. To put this into perspective, look at these four potential threats. A terrorist attack, a fatal Car crash, a fatal accident at work, or being struck by lightning. What you fear the most may not be what's actually more likely to kill you. A car crash is the most dangerous. Then it's an accident at work, then lightning, then a terrorist attack. What I'm trying to underline here is how probabilistic assessments are distorted by emotion. If you're a naked when you win money or you're sad when you lose money, you're distorting your future. Probabilistic assessment of trade outcomes. It's one of the reasons your post trade analysis is so important. Your emotions should be tied to whether or not you traded well, whether you followed the process and you stuck to your rules, not what the triggered outcome was. It should be process emotion, not outcome emotion. You need to try and train system one to accept that the outcome of individual trades are irrelevant by default. System 1 assumes that all actions have a cause and effect. We talked about this in lesson two. It's great for survival, not so great for trading. No analysis you do, nor decisions you make have any impact on the market. But system one will always assume cause and effect, something psychologists call the illusion of control. We tend to delude ourselves that the outcome is the direct result of our actions. We're back to my analogy of fishing. If you've had a good day and you catch lots of fish, system one will try to establish what it is that that you're doing that got this bumper catch. You try to remember exactly where you were, what bait you were using, what time of day it was, et cetera, et cetera. So you can try and repeat that outcome. So the next day you go to the same location, use the same bait at the same time of day, but you catch nothing. To the inexperienced fisherman, this can be frustrating and unfair. And that frustration is evidence of the illusion of control. And it can send you into a cycle of meddling and fiddling with your fishing system. In fishing you can improve your chances, but you can't control whether the fish will bite. In other words, you can control the process but not the outcome. When you trade, if you evaluate the trade according to the outcome, you're liable to be under the illusion that your system is causing the result. If you think a winning trade is is good and a losing trade is bad, you are definitely under the illusion of control. Trades are considered good if you followed the rules and process those. Good trades can win or lose, vice versa. A bad trade is one which did not follow your rules. Bad trades can also win or lose. You will only improve as a trader when you focus on the process of taking good good trades and you ignore the individual outcomes. Your system causes good process, it does not cause the outcome. By not accepting randomness and probability system 1 tends to default to assuming cause and effect. It thinks it controls outcomes. When the outcome is positive, we think it was something that we did, and when the outcome is negative, we think it was something that we didn't do. Or perhaps we blame it on somebody or something else. You need to be on the lookout for several signs. Firstly, any signs that you're finding patterns in randomness. What meaning do you see after strings of wins or losses? Do you imagine clusters of results mean something? Remember, random doesn't look random as we understand it. Do you ever assume that previous outcomes change the probability of the next outcome? After four or five losses in the row? Do you think that you're in line for a win? Ask yourself if you're falling for the illusion of control, thinking that a positive or negative outcome on any individual trade has anything directly to do with your actions, your execution or your system. And be aware of and monitor and recognize your need for certainty and more importantly, your discomfort for randomness. There is a lot to unpack from this topic, but we're not done with probabilities yet. This video really just covered the background and the theory. But the next video is more practical and indirect context to trading. It will simplify things and it will give you a method by which you can slowly reassure system one that single or small clusters of outcomes are largely irrelevant in a probabilistic environment. It's a great video, really interesting to watch, so jump through to that and I'll see you there.