TU Wien Rendering #21 - Tone Mapping Basics

Now to the second part. It's quite different. So it's tone mapping. This is concerned in our so before we had optimization for the light simulation. Now we will look at the very end of our rendering pipeline. So the issue of showing the image output on a display, because the problem is that a light simulation outputs radiance. So it's how much light travels along one direction when it comes from a small surface patch. So you collect radiance with your camera, record it, but somehow your display expects RGB values, so the radians can carry some color information. So you could, for example, trace the R, the G, the B value general, independently, you can do something with more fidelity, like spectral ray tracing. So you trace rays where not only radiance is carried along the ray, but spectral radiance. So radiance of a certain wavelength of the light. This, for example, you can see this effect in prisms where you split white light into rainbow colors. So if you do not perform spectral ray tracing there and just assume we have white radiance, then you cannot simulate this effect because then the refraction, so the ray geometry changes with the wavelength that's associated to it. So your refraction angles get different and this causes the split into the rainbow colors. But in some way you have radians as output. So to show them on displays or to print them, you need to convert your images somehow from radiance to rgb. And there is an inherent problem with that, because radiance of light simulation have a huge range because they try to simulate real world physics. And in real world, you have a huge difference between dark and bright. So, for example, if you take the ratio between a surface at, say, ground level in the earth atmosphere that is either illuminated by the sun or the moon, the difference is a factor of 800,000. And then imagine that the patch on the ground is either white or black. This causes another difference in the reflected radiance quantities of Approximately a factor 100. So if you want to do, say, a general light simulation, and you can expect that illumination by sun and moon should be in there, white and black surfaces should also be in there. You have to somehow cope with a ratio between the darkest and the brightest values of 80 million. So is it relevant to do that? So can people even see the difference between that and. Yes, they can. So since these are highly relevant features of our world to differentiate dark from bright, especially under very bright and dark conditions. So imagine a caveman in the woods at night. It would be very good to see small contrast differences that could indicate Predators. So there was an evolutionary forcing to, to develop a visual system that also can take advantage of this huge dynamic range of radiance values. So for example, how this is built up. So photoreceptors in our eyes, they have chemical bleaching when they are hit by light particles. So this can be regulated biochemically to enable adaptation in a range of two orders of magnitude. So just by regulating the biochemical properties of photoreceptors, you can adapt the eye to a difference in dark and bright of 100. The pupil size gives you another order of magnitude. So a factor of 10. And the neural adaptation is more or less the signal processing. So what to do actually with the changing signal because your receptors get bleached. So the dynamic range of human vision is approximately 100 million. So that means that we can perceive in fact the dynamic range of realistic conditions in our atmosphere. But then you have. So the output of a light simulation, if it takes this into account, can have a difference of 80 million or effect of 80 million between dark and bright. And then you want to show this on the display. So the technology for a standard display gives you a dark to bright ratio of approximately 1000. And if you use 8 bit encoding of your values, then you only have 265 values for 8 bit channels. That means that somehow our technology is immensely inadequate to show realistic scenarios in some way. This is also good because you get not blinded by your display, but we have to somehow account for that. So just taking the radiance output and converting it trivially to images will carry some problems with it. Yeah, but why an 8 bit image file? I thought usually bitmaps files are 24 bits of image, but usually you have, you split them into the different color values. So you have eight bits for R, eight bits for G, eight bits for P. And you can somehow imagine that this gives you how much red light is at this pixel. And yeah, you got 260. I mean of course for the whole color game mode you have more values. But if you just look at the dark bright ratio of a single color, this is approximately what you have. So now tone mapping is the method that was developed to combat this problem. So the output of a light simulation, as already said, high dynamic range because of real world dynamic range of brightness values. And display devices usually have a low dynamic range. So what we need to do is to compress the range of our output somehow. And this is referred to as either tone mapping or tone reproduction. So these are the names that you find in the literature. So There are two sub issues here. One is range compression. So how to convert high dynamic range luminance to low dynamic range luminance? This is the content of this lecture. And then you still have luminances, but usually there are standardized color spaces in which images are stored. So you don't store your own home rune format, which takes certain wavelengths and then gives the luminance values for those. But there are standardized ways to do that. And this is covered in a different lecture about color. You see the lecture number here. So I refer you to this lecture if you want to know more about color here. Now I only explain range compression to some extent. So now a graphic example. So here we have a single bright light source with no ambient light in this scene. That means that all contrast comes only from this one light source via global illumination. So if you take, if you would photograph this scene and take a short exposure, this is what you get. So all the dimly illuminated parts completely disappear in black. But you see details in previously too bright scenes. So that led to overexposure of your camera. So for example, here you now see the outline. So the silhouette of the bulb, for example, a very long exposure leads to overexposure of most of the image. So you do not see any detail anymore close to the lamp or at directly illuminated surfaces. But what you see is that previously dimly illuminated objects in the background now are, well, perceivable. So this is what you get if you take only small parts of your dynamic range and map those to an image. And what could you do to combine all this information into one image so that the most trivial thing you can imagine is I just divide by the maximum of the scene. This gives me a nice luminance values from 0 to 1, and this I can map to whatever range I want. The problem with this is that usually the maximum is not indicative of the whole scene. The maximum is usually extremely bright. So when you divide by the maximum, the only thing you see is a single reflection close to the light bulb if you clamp it. So let's say I have a range of luminances that I'm interested in. Everything below is clamped by a zero. Everything above my maximum that I decide to have is completely white. This is exactly the problem that that becomes obvious with high dynamic range, that parts are underexposed, other parts are overexposed. So clamping also doesn't get you something. You have to take the whole range. You cannot just ignore certain parts. So one approach that gives you nice results is exponential mapping. There you Assume exponential distribution in your luminance values and then rescale this exponential function into a linear one. That means that very bright values get scaled down, low values get scaled up, but you account for the very bright spots. So these are the things that get scaled the most. So the bright values at the bulb get some reasonable white, but they do not dominate the whole scene. A more sophisticated approach is the Reinhard tonemapper developed by Reinhard the Guy. And this is the one example that I will present in this lecture. As you can imagine, there are many tone mappers. So everyone tried some approach that works very well for a certain subset of scenes. Some methods were optimized for parallel hardware or for certain hardware architectures to give more speed there some presented rough approximation that work in real time. So there is already a quite wide bunch of tone mappers to choose from. The Reinhardt tonemapper is one of the most common. So if you look into some open source or commercial rendering software, this is usually one of the things that is implemented. So and there is also some additional information to put this into context. There is some. Approach in digital photography to take multiple exposure images so to combine exposures of the same scene that are taken with different exposure times into one image. So this is called high dynamic range photography. And usually they use similar methodologies. So the Reinhardt tonemapper can also be used to combine or for HDR photography. But if you put HDR photography into Google, you usually end up with images like that. And this is not what tone mapping is about. Tone mapping is a perceptually and physically validated approach that gives you realistic impressions of the scene and is not an artistic effect. So it's not that you take tone mapping, look at the image and say I would like to have it to have more contrast in there. And then you change your tone mapping. I mean, this is not its standard use. And the community of HDR photography to a large extent overuses this capability of the system. For example, this photograph here is a completely botched implementation or use of HDR range compression. For example, things that you see that are not correct. The halos around the balloons. So suddenly why the sky is brighter around the silhouettes of the balloons. This is an effect that is present in the visual system to some extent, but not to this extent. Some people think that this is nice HDR photography, but this is just wrong. Simply as that if you like that your images look like that, then this is artist decision, not tone mapping. This is just contrast enhancement, you can call it like that. Also the colors in this image get screwed up because they are now oversaturated. But this is just a warning. The tone mapping that you encounter for light simulations should not look like that because then you have a error in your application. So if you would take this photo with different exposure. I mean this, this is a combination of different exposures. It's not just a combination of different exposures, but also some. No, it's. I mean it is a combination of different exposures and it's taken with a tone mapping approach. I mean, you have to do the combination somehow. As said before, you can either divide by the maximum and combine them clamping. But usually the tone map will give you a more realistic result. And for example, why is it obvious that here already multiple exposures were combined? Because in the very left of the left balloon you see that. You still see some kind of detail in the wood behind, but at the same time you can more or less directly look into the sun without it having blurred half the screen. So that means that already a huge dynamic range was compressed here. But it should not look like that. It should look like that if it's done correctly. And this would be the correct application for this photograph. So it looks somewhat realistic. It doesn't over saturate the colors. It only has a very light halo around the silhouettes, which is also an effect that's present in the visual system. So this is what you should aim for. So now about tone mapping itself. There are two large different classes. So one are the global tonemappers and the other the local. So the global tone mappers use a mapping function that converts radians at a certain pixel to say RGB value. If you select these colors space. And this mapping function is uniform. What I mean here is not that it's uniform, it produces the uniform output value. So you don't get an image with a single color value, but that the function itself just takes as input the radiance at a certain pixel and outputs RGB value. And this function is then used for all pixels in the image. More complex methodologies are local tonemappers, which not only take a single pixel into account, but also its neighbors. This is, this is perceptually motivated because as you have seen before, this contrast or brightness adaptation of your eyes in the photoreceptors, for example, this is done locally. A single photoreceptor adapts to different brightnesses. That means that tone mapping in the human eye is a local behavior. But there are, there are reasons to employ both of them. So the global tone mappers, they are fast because they have a single mapping function. So you take you can execute this function parallel on each pixel, so this makes it perfectly usable for GPU approaches, for example, but you incur some loss of detail because you cannot locally look as is this already a dark patch of my scene so I can enhance the contrast more there? Or is it at a dark bright boundary where I do not do it that much? The local tonemappers allow this, so they allow a local contrast enhancement, but they are slow because you not only have to say look at the pixel, but also.