A Computational Study of the Internal Structure of Historical Papers: Leonardo da Vinci's Notebooks

Hi, my name is Bill Cefaris, and I'm with the Department of Electrical and Computer Engineering at the University of Wisconsin in Madison. And today I'm going to be talking with you about some papers of Leonardo da Vinci, a computational approach to historical papers. Leonardo was a genius. He was also not very well organized. He was kind of a perfectionist. He never finished anything. He had the Mona Lisa with him at the very end of his life. He'd been working on it for 20 years, and he was sort of like every few days go over and add a little bit to it. So throughout his notebooks, there's many very charming passages that say things like, I ought to write a book about this mechanical stuff. I've been doing all these gears. I ought to write a book about all the. All the. All the beautiful internal organs I've dissected from people and animals. I ought to write a book about how birds fly. And he didn't do any of that. But fortunately, what he did do was he kept copious notes of all sorts of things. And when he died, he left about 4,000 pages of stuff not very well organized in a big case. And his secretary, manservant, whatever the term was, agreed well, to take care of this, to sort things out, organize things a little bit, and spent years doing it. And he did as good a job as he could. He put lots of the mechanical drawings together, he put lots of the hydraulic studies together. He put. And any, you know, things that were made on the same size paper he tended to put together. And this is in 1519. And then a century goes by, and some of the notebooks go over here, and some of the notebooks go up to Germany and some go to England, and one of them makes its way to the United States, and they're spread out all over the place. And sometimes maybe there's pieces of them sold off and then rebought and shuffled. And so how confident are we that what we see in these, in the codices is what Leonardo might have intended? Are they in fact, in any kind of order at all? Are they chronological, for example? So how might we go about looking at this? Well, art historians and conservatives have been very interested in this question for a long time, and they've thought about it deeply, and they've come up with a lot of ideas, and all based on essentially the content of the pages. Looking at what. What's written on the pages, I mean, that makes a lot of sense. We're looking here at complementary information. We're going to look at the structure of the paper and see if there's anything that can be learned from this. Okay, so what does paper look like? Well, okay, in historical paper from about the 1850s back was all made the same way. You have this, okay, they're about this big. It's wooden frame. It's a sieve. You see, there's a bunch of vertical lines about an inch apart here, lots of horizontal lines. And what you do is you dip that sieve into a vat of paper slurry, okay? Rags, pulp, water. It's been heated, boiled for a long time. You shake it around a bit and you. Oh, yeah. Oh, so that's the. This is the useful part. The chain lines in the vertical direction, the laid lines in the horizontal direction. And they would often annotate it and decorate it with something that proved the authenticity of the paper. We are the eagle paper manufacturers. For example, here's a graphic from Diderot's encyclopedia of the process. You have one of these molds here. You have a second mold here. Here's where you dip it. This guy's the vat man, dips it in the paper slurry, shakes it around a bit, hands it to this guy who lays down a piece of felt, tips the mold over the piece of paper, goes in there. While he's doing that, this guy's taken a second mold, started working on that. Once this fills up, they bring it over to this guy who sort of shakes it out, gets rid of the water, they put it into this press, they squeeze it down, get drip all the water out, and then it goes up to the. Usually the attic where they hang the paper till it dries. And then they take it and they put it in piece by piece, and they sell it to Leonardo da Vinci or whoever's purchasing it at the time. So these are because of the era. These molds are handmade. So even though they're making them the same, they're never exactly the same. You know, the chain lines will be a little bit further or narrower apart. The laid lines might get bunched up in places or spread out in places. The watermark, you know, maybe the claw of the eagles. This way in one of them and that way in another one. Maybe we can look inside the pieces of paper and extract information about these three things. Here's page one of the Lester Codex. What you see here is we're looking at the page, actually two pages through transmitted light held behind it. We have zoomed in on a little portion here that's of particular interest to paper scholarship because well, okay, can you see it here? There's sort of this outline here buried inside all this stuff. This, you know, the surface features that we want to ignore. And it's a little hard to see. I mean, I think if you look at this particular page, you can, you can decide it's an eagle. But some pages are even more difficult to see. And wouldn't it be nice if we could somehow do some sort of processing on this to, like, get rid of all this junk here that's obscuring our view of the watermark? You see what paper historians cringe when I give a talk like this. So here's sort of the idea for that little portion of the Lester 1 eagle watermark. So here's the corresponding surface image, just a photograph of that page, the back of the page. Here's the photograph of the front of the page. And so what you see here is a mixture of all the. Well, you can kind of see the thing here. And there's that line from up that side. So here's, here's, that same piece of paper. And what I've done is I've lined the two. Well, the three images up perfectly. Those of you who use Mathematica might recognize the image align function. One word. It's a brilliant function. It lets you line up the surface image number one with whatever part of this is represented by. Surface image number one lets you line up surface image number two with the second part of whatever's there. And then we're just going to do a weighted average of these things. So if you look at my code here, M is the transmitted light. Image A is one side, B is the other side. And I'm going to control the two numbers by moving the slider. Ooh, ooh, isn't that one? Now, you can't always get everything out right. I mean, you look down here, you see this sort of little embossing effect or something that's some sort of leftover effect that's not quite matched up. So this, to do this, the alignment has to be really precise. You have to have sub pixel registration better than single pixel registration. And fortunately, by playing with the various options in image Align and ripping your hair out a little bit, coaxing the function a little bit, you can, you can, you can do this alignment pretty reliably throughout the various codices. Here's three examples. You see two eagles, one upside down, and that one, if you look at it, if you don't look below for a second, if you look at the one on the right, there is that an eagle might be. No, maybe not. Who knows? But once you denoise it, remove the surface. Yes, Yes, it's a fleur de lis. Yes, as they say in watermark terminology. But, but the point is, before the denoising or surface removal, it's really very hard to tell what that is. They, they, they had agreed on most of the eagles, but this was actually sort of a question mark. Could it be another eagle? And one. And remember what we're trying to do, we're trying to look for exactness of representation between two separate pages. If you can't see the thing really clearly, you're never going to be able to make such a determination. So the denoising step is a key piece of this puzzle. So what do we do with this once we've got them denoised? Well, we code them. Okay. Now maybe there's some way to automate this, but I doubt it and we haven't done it. So basically what we do is we go into the watermark and we pick a bunch of points that we think are, are really significant points for this watermark. So here's for this eagle, for instance, we do from the tip of the wing to the tip of the wing. We do from the top of the wing to the top of the wing. Maybe you do from where this vertical chain line here intersects the top of the crown. Anyway, you choose some points and then that gives you a code. We actually code it by looking at the ratios because the photographs are not always taken at the same scale, they're not always taken to the same distance. Especially when you take photographs of one codex and compare them to photographs of another codex. Everything's different in terms of the photographic technique. But these numbers are pretty invariant. It takes a few minutes to do, but you know, there's 46 or so different watermarks in the thing. So if it takes you five minutes apiece, that's a lot less time than it takes you to do anything else. So here's an example of three different sets of eagle watermarks from the Arendelle Codex. And you see the blue ones, there are two of them. They're pretty much the same thing. You see, there's the orange ones, there's four of them, pretty much the same thing. And then that black one there. 208. Those are the page numbers. Two hundred and eight. 209 is all by itself. It's not the same as any of the others. So right away we can say without a doubt the orange ones are different from the ones coded Blue and different from the other one. So this greatly reduces our search space. And you can formalize this kind of thing by comparing one. This compares this particular one to what we called. Okay. We named them group A, B, C, D for these four particular watermarks. So these are all eagles, A, B, C, D. And it shows you that the distance between the current one we're measuring and the others look like that. So that tells us, with some degree of assurance. Well, it indicates that we really ought to look carefully at those pages that come in groups. Yeah. Why are there only, like, seven curves as opposed to presumably, like hundreds? Oh, because these are just the eagles. There's no sense in comparing an eagle to a fleur de lis or. Or a bull's head or. Or a cross, because we. I mean, they can't be the same. Yeah. One codex, there might only be seven. For example, you know, we've dealt with two codices fully, the Lester Codex and the Arendelle Codex. We've just about two weeks ago received a subset of images from the Madrid Codex. And so there's a lot more codices around. And these ones we've analyzed so far are smaller ones. So we don't know the whole story. But I think you can see where we're going here. We are spending time begging museums to give us images. We promise we won't sell them and put them on T shirts. No, really, we won't. I mean, museums are very concerned about intellectual rights and property and stuff like that. You know, things that computer scientists think information should be free. Okay, clash of cultures. So here's a place if this stuff interests you. We've created this website called LEO Code because we're coding up the codices of Leonardo. And if you go to this website, you can see it's leocode.org and what you see is, well, here's all the information we have so far about the Arendelle and the Leicester Codex. We'll be putting up the Madrid information as soon as we get permission to do so. What you're seeing here in this video here is we are essentially morphing between Arendelle66667 and. Oh, it's off the bottom. Where'd it go? Okay. And one of the others, and this is what we call an overlay video. It's somehow. There we are. And what we've done is to essentially overlay the denoised images right on top of each other. And it's a little hard to see, but if you look at these, you can see there's no place where they actually differ. You know, watch the tip of the tip of the wing. Watch the claw. Watch the nose of the eagle. Watch the. There's three little spikes on the eagle's head. And so these overlay videos, overlay animations are what we show to the art historians and the conservators to make the argument that this is really the same. They're not impressed by graphs with orange and blue lines in them. Oh, and there's a bunch more stuff here if you want to source. Our software is here. So software overview. Here you can see all the programs that we have created to do this. Essentially all of them are. Here's the little plug for Wolfram. All these software modules are written in Mathematica and can be run on Macs and PCs. And so we instruct people who want to do this themselves. I mean, they don't need to send us. In fact, it would be great if some. If someone would analyze one of these codices and send us all the information so we could just put it up on the. On the website. That would save us a lot of work. People have looked a bit at. They've used this for other. There's a study on Durer, and they recently discovered a brand new Durer print. And indeed we verified that the trident watermark that's on that was the same as another trident watermark also found in Durer, which is one piece of evidence in the authenticity of. Of that. Of that piece. So, okay, yeah, so there's the various pieces of software. They are publicly available. Whoop. Sorry. Oh, come on. Where's the. There's too many Wolfram. Okay. Icons. So, yeah, so here's some of the stuff you'll see at that website. But let me go on here. Let me go on. Okay, I'm trying to advance. So, yes, you see the landing gear. You see the. You see, okay, this is actually a watermark called the Cardinal's Hat. But, yes, I called this the Flying Saucer for many years until our conservator friends said, no, you have to stop doing that. So I have now stopped and I've admonished you to do a stop as well. Okay, so the watermarks, as convincing as they are, is just one piece of evidence. If it's really true that the watermarks are from the exact same physical mold, then not only must the watermarks be the same, but the chain lines have to be the same. Not only must the chain lines be the same, but the laid lines have to be the same. Now, okay, unfortunately, in many of the, in many of the Lester pages, the laid lines are not visible enough to actually get any decent measurements on. So a third of our potential evidence is gone. Here's an outline of the program that we developed to extract the chain lines. It basically. Okay, look at these, look at these wonderful Mathematica functions. Radon transform. It gives you a little picture that you can then process a little bit and it tells you how to, how to straighten the thing up and down. Once it's straightened, you filter it with a large vertical filter. And then you do this is, this is image lines. Wonderful. Another wonderful thing which. What does it do? It picks out the lines. Okay, great. Thank you. Image lines. And then once you know what the lines are, you can do that same kind of coding thing we did. You look at the ratios between all the successive, successive lines and that gives you a set of features that you can then match numerically. We also have a little interface when the previous thing fails. And you can just go in and mark it yourself. You can say, oh, look here's. I put these little red plus signs down where the chain lines are. And then it automatically calculates the codes which look something like this. And we've essentially represented the ratios of the two of the successive successive lines in a code like in a chart like this. And you can see that, well, these, these particular six happen to group into those two. And then you make sure that the grouping you have from the watermarks is the same as the grouping from the chain lines, Blade lines. These are the really dense ones that were, that are the small ones. Okay, here's a little. You can see the magnifying glass, little blow up of it. You can see two chain lines in the vertical direction and a whole bunch of laid lines in this direction. You can't count them by eye. I mean, there's just too many of them. Makes it a perfect application for Fourier transforms. You're looking for sort of something that's mostly a repetitive thing. And indeed, if you think of any vertical slice on that, well, down that page, then it's basically an undulating thing, not really sinusoidal. If you look at one of the spectra, you'll see it's got a sort of fundamental frequency which corresponds nicely typically to what you see when you look at that thing. And then it's got a whole bunch of harmonics that represent something like the shape of the actual wires that are, that are represented here that you can see. There's some technical details of how you do this calculation to get sufficient Resolution. There's this thing called the phase vocoder that allows you to combine measurements from multiple overlapping FFTs to refine the frequency estimation. So we actually do that because otherwise it's a little bit coarse. In any case, once you make the frequency estimation, you end up with something that looks like this. So here it's simply color coded. Red means the lines are close together. Blue means the lines are relatively far apart in that piece of the, in that piece of paper. So this represents this whole piece of paper here. And so what does this look like when it comes down to. Oh, yeah. So here's the kind of thing you have. Say there's four candidates here, you know, four different eagles, say in the. And here's another one. Is there a match? Is there not a match? I think you can kind of see that the second one actually looks pretty good. Huh? You might want to come up with some sort of numerical, mathematical way of describing sort of the difference between these pictures. Image difference is no good. Don't even think about it. See, if you, if you take the features of an image, it basically looks at a, at his, at an RGB histogram. Okay, that's useless because it's really important where the, you know, it's important that the reds be here rather than here if you're going to match something that looks like that. So there's no built in match. Laid lines function into mathematics. What's, what's up with you guys? So we've been doing stuff like. So the other thing about the laid lines is there's often portions of the image where you can't get a picture. These particular ones are from nice clean pieces of paper that we did a study on rather than the actual, quite a bit more noisy ones that you see from the actual codices. And we've been. The major problem we're having with this particular piece of the puzzle is suppose we only had this much of it. Now we want to know, does this part of it exist anywhere in the above four? And that's something you really can't do by eye. And so we're playing around with various kinds of metrics to come up. There's some that seem to work, but we'd like something that we're convinced is robust pixel by pixel Euclidean distance of RGB values. Just normalize it and see which one is the minimum. Yeah, but what if the scale's a little different? Yeah. Or what if, what if what? What if you, you know, you don't know and you have to then scan it around and find it. You know, you're looking to. Correlation is always a bet. It's a. It's a great way to look for something if you know exactly what you're looking for. It's not such a great way to looking for if it's sort of the same. I mean, even a slight tilt. Right. So, yeah, so it's not as simple as it appears. And then using all this information, you can build what's called a collation diagram. And this is for the Arendelle, this is for the Lester Codex, and these are how the pages are aligned throughout the. Well, this is the whole lester Codex. It's 36 pages and these are the first 30 pages of the Arendelle. There's not much. We don't have any matches in the rest of the Arendelle except. Well, no, there's a couple. There's like 206 and 208, I think, but they're almost nothing in the rest of it. Isn't that kind of interesting? If you look back through the literature, art historians have been telling us for years that they think that the first bunch of pages in the Arendelle Codex really form a distinct thing. So the people who say that are really happy with this result, the people who disagreed are fairly, well, they don't talk to us. So that's really what I wanted to say. You know, there's a lot more to do in this kind of investigation. I mean, first of all, there's nothing specifically Leonardo ish about this project. We've done some stuff with Rembrandt. I mentioned the Durer earlier. We're currently working on the Madrid Codex. Madrid has about 30 watermarks that look like what are called bull's heads and only a couple of anything else. So we don't have. So one thing that was fascinating about the Arendelle and the Lester is that there were a couple. There were two watermarks which are identical in the different codices. Cross code, not just mold mates within the codex. Right. That's what you would expect to see. Right. Remember that A.B. a.B. Thing that the VAT men and the Kutcher are doing, but rather this is a. So a piece of paper sort of that might have. You might think should be in the Lester Codex is sitting over there in the Arendelle Codex. Does that mean anything? I don't know. I mean, it means that someone really needs to take a look at it closely, I think, because I don't think that computer scientists are the right people to be making those decisions. I think we should present evidence, like a little court of law maybe or something. We think this might be the same. So the software that we have currently available does most of what I've showed you, except for the laid line stuff, because that's, I guess I would say, in, in progress. And this, this stuff here. Okay, I, I was singing the praises of Image Align moments ago. Right. What a great function. It saved my life many times. It doesn't always work. And not only does it not work, you have, you have to kind of futz with it. You know, there's like many Mathematica commands there are. It says, you know, options goes to, or method in this case, goes to. And there's a whole list of things and you can try different ones and try to get it to. You can try to get it to give you more key points. And you can do it like that. You can try filtering the papers a little bit one way or another to try to get more key points or. The problem is, once we got, we fine tuned everything for the Lester Codex, we got it all working. We can do the whole thing with one set of parameters. And then when the Arendelle Codex came along, we had to rejigger everything and change all the parameters. So now we have some, like, default settings that work for, like, both Arendelle and Lester. And when the Madrid codecs came along, we have to rejigger everything again. So we have no confidence that this, the Image align function is going to be able to work generically and across new codices that we haven't tuned it for. So that's why we're holding off on that. We want to. We. Yes. Okay. Yeah. So honestly, we do break it apart. I mean, there's fine corresponding points and then there's. Yeah. And then there's fine geometric transformation and there's. I didn't mean to imply that we only looked at Image Align. Yeah. But thank you for warning. Don't trust the first function you find because there might be hidden functions sneakily built inside of that one that you could. Yes. Oh, okay. No. Okay, so what are these things? Let's go back to this picture here. So all of these things are wires. So there's a wooden frame around the thing. There's these wires here, wires here. And this is not wire. And it's just woven, sort of woven into the chain lines and the laid lines. So these are just wires. And then when the liquid drains out, you have in this thing, you flip it up and then there's the impression of the wires left in the paper. And that's what we're seeing is a slight indentation, if you will, in the paper. It's a local thickness of the paper that we're seeing. Yes. So paper has thickness. Have you done anything with dreams trying to analyze the. What's at what depth? So, so we have not. This is something that people do, though. What they don't like, though, is that most of such analysis techniques require taking samples. Yeah, that would be bad. Yeah. And so there's a real hesitation in. I mean, I mean, I've never touched any of these pieces of paper. Right. One of, one of my collaborators, Peggy Ellis, has a ways to. Yes, that's right. Well, we have a professional photographer who handles artistic items. Thanks a lot. And let me know if you have any further questions or anything.