A new interactive green-screen keying method lets artists scribble dominant colors once, and the system propagates them through the video. It solves the hard problems: separating hair from background and removing color bleed. The approach outperforms both automated techniques and manual frame-by-frame work.
Dear fellow scholars, this is two minute papers with. In the film industry we can often see footage of a human walking on the moon, fighting underwater or appearing in any environment without actually going there. To do this, a piece of footage of the actor is recorded in front of a green screen and then the background of the scene is changed to something else. This process is called green screen keying. And and in theory this sounds simple enough, but make no mistake, this is a challenging problem. Here's why. Issue number one is that separating the foreground from the background is non trivial and is not a fully automatic process. Let's call this semi automatic. Because the compositing artist starts drawing these separation masks. And even though there is some help from pre existing software, it still takes quite a bit of manual labor. For instance, in this example it is extremely difficult to create a perfect separation between the background and the hair of the actor. Our eyes are extremely keen on catching such details, so even the slightest inaccuracies are going to appear as glaring mistakes. This takes a ton of time and effort from the side of the artist. And we haven't even talked about tracking the changes between frames as we are talking about video animations. I think it is now easy to see that this is a hugely relevant problem in the post production of feature films. And now onto issue number two which is subtracting indirect illumination from this footage. This is a beautiful light transport effect where the color of different diffuse objects bleed onto each other. In this case the green color of the background bleeds onto the karate uniform. That is normally a beautiful effect, but here it is highly undesirable because if we put this character in a different environment, it won't look like it being belongs there. It will look more like one of those super fake Photoshop disasters that we see everywhere on the Internet. And this technique offers a novel solution to this keying problem. First we are asked to scribble on the screen and mark the most dominant colors of the scene. This we only have to do once, even though we are processing an entire video. As a result, we get an initial map where we can easily fix some of the issues. This is very easy and intuitive, not like those long sessions spent with pixel by pixel editing. These colors are then propagated to the entirety of the animation. The final results are compared to a ton of already existing methods on the market and this one smokes them all. However, what is even more surprising is that it is also way better than what an independent artist produced which took 10 times that long. Similar comparisons are also made for removing indirect illumination, which is also referred to as color and mixing in the paper. It is also shown that the algorithm is not too sensitive to this choice of dominant colors, so there is room for amazing follow up papers to make the process a bit more automatic. Thanks for watching and for your generous support and I'll see you next time.
Sam.
