88 - Mysteries of the Scrolls — with Nat Friedman

Most of those attempts in the late 1700s were quite destructive. People did things like they took a dagger and they would just sort of like scrape off the layers of papyrus or slice it.

Just amazing to think how much human brainpower had to come together to make this thing remotely possible.

If it didn't work, people would say like, of course it didn't work. This was like an insane idea. And people have been trying for hundreds of years to open these scrolls and you had horrible hubris by getting involved at all.

People look as far back into the past as they want to look into the future. Nat Friedman is an entrepreneur and investor. He's a co-founder of Zimian and Zamrin. I hope I said those right. The latter of which was acquired by Microsoft in. 2016, Nat stayed on at Microsoft, and in 2018, he led Microsoft's acquisition of GitHub, an important open source platform of which he became the CEO of in 2018 until 2021. Then he moved on to other projects, which brings us to maybe the number one reason and occasion for our conversation. One of these cool projects that you've Um, that's come across your desk, Nat, the Vesuvius Challenge, which is in some— we're going to get into the details— a mission to scan and digitally unroll and read a large trove of ancient scrolls that are preserved through the famous Pompeii-destroying eruption of Vesuvius, but have never been— it's never been possible to read them before. Until now. So, you know, at The Cost of Glory, we're all about bringing to life great ancient figures and texts and ideas from the past, from the Greek and Roman world that you can use today. So for that reason, I'm really thrilled to have you. Thanks for joining us, Nat.

Thank you for having me. I love your podcast.

Great. Well, why don't we start by just you telling us, for those who don't know the story, how did a guy like you get interested in a project like this?

It was really pretty random, honestly. One of those sort of sequences of events that seem so contingent and unlikely that you have to wonder how predetermined anything that ever happens really is. But yeah, I mean, it sort of started back in early 2020. I was out here in California, I was running this company called GitHub and COVID struck and the lockdowns kicked off. And we were in that early period of COVID when there was so much uncertainty about kind of what it was. I don't know if anyone really remembers, but there was obviously COVID was not good, but there was a lot of concern at the time that it was actually much, much worse than it ended up being. People were disinfecting their groceries and, you know, a lot— there was just a lot of worry in the world. And, you know, the company I was running was a great company. GitHub is a fantastic company, amazing people, but man, not always easy. And so I was just sort of found myself, you know, looking for an outlet, looking for other things to occupy my mind in the evenings. And I stumbled upon this book that I think Amazon recommended to me. which was called 24 Hours in Ancient Rome. And, you know, as a kid, I'd never actually been very interested in history, to be honest. I, um, I was sort of a math and science kid, and, you know, learning about math and learning about science seemed to me like you were sort of in pursuit of universal truths of the universe and these kind of, um, you know, these things that would be important and true in any world. And history seemed very kind of contingent and parochial to me. And, you know, something that just was sort of like a sequence of events that had no, you know, had, had like no sort of universality to it. That's what I thought at the time.

Aristotle would've agreed with you actually. He says, yeah, history is all about particulars rather than the philosophical truths. It's very interesting.

But yeah, I mean, I did later, I think after college, I did realize that even if that were true, this world with this history is the only one I would ever live in. And so even if all these things were contingent, I would only have to learn them once. And so I did— that sort of did wake me up to the fact that maybe it was worth paying some attention to, but I didn't really do it. I mean, I did enjoy— I took some classics in high school and I did enjoy reading Plato and Homer and It was just sort of fun food for thought, but to me, it almost seemed like reading fiction. And so anyway, for whatever reason, in this period in early 2020, this book appeared in my Amazon recommendation list, and I decided to give it a shot. And I ended up staying up all night reading it. It's a fantastic book, actually, 24 Hours in Ancient Rome. I really recommend it. Someone later told me that it was written for schoolchildren, which might be true, I don't know, but it doesn't matter. It was fantastic. It sort of really drew me into thinking about that period. And so I found myself over the sort of subsequent evenings kind of, you know, in these different Wikipedia chasing and surfing exercises. And I think there was something about reading about ancient history at a time, you you know, when the world was unsure about its own future, that was just deeply calming. You know, all your problems in either deep space, you know, in either large amounts of time or large amounts of space, your problems seem small.

I love this image of you as a CEO of this $7 billion acquisition, right? Like huge enterprise, like taking the time to reflect on the past. Maybe it's the kind of chaos of COVID you know, it's really not sure what you're supposed to do, but maybe that's the answer, that it was, it was somehow calming.

It was, it really was. Uh, you know, your, your own, your own issues or problems seem pretty small if you realize that, you know, your, your life is short and there have been many lives before you and there will be many lives after you. And, um, the other thing that I think has a similar effect is, uh, you know, astronomy, uh, just thinking about like the size of the universe. And, you know, it's— you maybe hopefully become slightly less petty if you pay attention to these things. So maybe it was that, or maybe it was just interesting too. So, you know, there's this chapter in the book that talks about the sort of way Roman bread was made, and everyone was into making bread back then. So I went out and sought out like the heirloom wheat varietals, this Red Fife flour and tried to make my own Roman bread. And anyway, so one night I'm in Wikipedia and I stumbled on this article about the Herculaneum Papyri, and I'd never heard of them before. And I thought it was just really incredibly fascinating. Obviously, Vesuvius kind of covered the whole region of the Bay of Naples in mud and ash and incredibly hot gases. And everyone knows about Pompeii, and I'm sure I'd heard about Herculaneum before, but I'd never heard of this. And so there was this enormous kind of villa, private villa, that was just on the outside of Herculaneum that had been buried by the eruption as well. And it was thought to have been built and owned by the father-in-law of Julius Caesar, this guy Calpurnius Piso. And what made it really special is that in the mud, as it was excavated in the 1700s, late 1700s, these kind of lumps of charcoal were discovered. And I think at first no one knew what they were until someone noticed on a broken-off flake of one of them a little bit of writing. And the excavators realized that they'd stumbled upon a library, a library of papyrus scrolls. made from the Egyptian papyrus plant. And this was a unique find because somebody gave me this great phrase, humidity is the enemy of history. And so to the extent that there were libraries along the Bay of Naples at that time, if they had not been buried by a volcanic eruption, they would've just eventually sort of rotted away. And so therefore we have really almost no intact, you know, books or works of writing from that period unless they were copied by, you know, a cascade of monks through the Middle Ages and preserved through copying. And anything that was sort of original is gone. There are some exceptions in really arid places like Egypt, I later learned. But, you know, those are unfortunately the exception and not the rule. And so Hundreds, actually thousands of scrolls were excavated from there, and then there were attempts to open them, and most of those attempts were in the late 1700s were quite destructive. You know, people did things like they took a dagger and they would just sort of like scrape off the layers of papyrus or slice it, and maybe they'd see a few letters and they'd try to write those down along the way. But, you know, this was, this was very, very destructive.

It's horrifying to think now with what we can do, you know.

Yeah. Well, the worst part about it is, as I later learned, the scrolls that were attempted to be opened in these like really just physically destructive ways were the best ones because, you know, obviously the excavators looked at the ones that were in the best shape and thought, you know, these will be the easiest to open and read. And so it's probably just through the selection bias the worst ones that we have now that were not destroyed along the way. There was a monk, this guy named Piaggio, in the late 1700s who ended up designing this machine to attempt to unroll them. And it used like cat gut and pig bladder and it was semi-successful. So like it would, in the case of the best scrolls, be able to unroll parts of them, but it was also pretty destructive overall too. So layers would stick together and there'd be gigantic holes in the unrollings that he did. But we were able to apparently determine from that that these were primarily Epicurean writings of this Epicurean philosopher who's a follower of Epicurus, this guy named Philodemus. And so, so there had been like a little precedent, but, um, you know, for, for looking inside these things. But, um, what was left couldn't be opened, and what was left was huge— hundreds and hundreds of scrolls. And in fact, it is the largest, um, library that we know of from antiquity and is a mystery. I mean, and I loved that. It's a mystery. What's in there? No one knows. And, uh, there were no known techniques for, for either opening them or figuring out what was in the papyrus without opening them.

It's amazing. And I had, I did not know the part about cat gut and pig bladder.

Um, yeah.

Yeah. Yeah. So before, you know, the last 5 years, there were some number of scrolls, maybe the easier ones to open, uh, many destructive losses of text in the process. Um, but I guess in, uh, in recent years, it seems like The technology has sort of like, and I remember when I was in grad school, uh, hearing whispers about this, you know, like, oh, they're, they're getting close, they're getting close, but it's always, it always seems like kind of asymptotal, like we're not actually getting closer.

Um, yeah. Well, first I think it's just like intrinsically exciting to have this opportunity. Um, and I think everyone's sort of had these dreams about like having a time machine and going back to some period that's particularly interesting to you and being able to see what it was actually like, or, I remember when I was a little kid either thinking of or reading about this theory that maybe you could extract audio from pottery wheels. And, if there had been some reed that was brushing against the pot when it was being spun, then maybe it would have like a phonograph needle recorded audio from the room and just being totally fascinated by that idea. I had this idea that maybe as paint was drying, the pressure waves from the sound in the room, you know, could end up being encoded in like the physical structure of the dried paint. I think that's— I no longer think that makes any sense at all, but like when I was 11 or something, I thought that was a neat idea. So we all have had these like theories about, or fantasies about how cool it would be to get like a real window into the past. And so I think that's what fascinated me about this was like, wow, if you could actually get all these scrolls and read them, um, it's a lot. It's a lot of text. And it would add in terms of the total, you know, megabytes of information we have about the past. it would add like a really material amount. And so I think that was one of the things that excited me. And then I think it was also just this like anti-authority impulse of, um, there are these mystery scrolls, they're unopenable, you can't open them. And someone tells you you can't open them and you think like, we've got to find a way to open them. So I just thought it was really cool. I'd never heard of it before and started talking to my friends about it. And it turns out none of them had either. And so somehow this was like, You know, my friends are from tech and software, and so, you know, maybe we just aren't educated on this. But I thought it was just such a neat mystery that people have worked on for hundreds of years. And then I think like maybe later that night, I found an article about an effort that was underway to try to read the scrolls. using 3D imaging and that there was this professor at the University of Kentucky named Brent Seales who had been working for years actually, I think at that time for about 18 years, to scan the scrolls in 3D at high resolution with something like a CT scanner, like you would, you know, if you have an injury, you might have been in a CT scanner, part of your body was. to be able to look inside them and then hopefully be able to sort of use software to not physically but digitally unroll them and then recognize the ink and read what's in there. And he'd been working on this for years and years. And so that was like, you know, when I found out about the Herculaneum Papyri, I thought that was cool. When I found out about Brent's project, I thought that was like the coolest thing I'd ever heard. And I just became a fanboy and thought I'd I'd follow along.

Interesting.

Yeah.

So, had he scanned a scroll, a full scroll at that point, or had it just been fragments?

Yeah. No, he had over several years. I think his first scans he did were in 2009 and they were on a low-resolution, I think maybe it was a medical-grade CT scanner. And, when you do a medical CT, there's you don't really need to see things at like the, you know, millimeter or submillimeter level most of the time in order to see if your wrist is broken or whatever. And so those were not— they were— that was an important step, but those were not usable to try to read the scrolls. And then the thing that really excited me, because I found this in early 2020, was that I read that in 2019 he had scanned 2 of the scrolls in a particle accelerator. And the particle accelerator was helpful because it allowed him to get much higher resolution and theoretically a resolution that was good enough to be able to read the text. And so when I discovered this, you know, just months after those first scans, I thought, okay, a breakthrough is imminent. And, you know, I'll just— I went and set up some Google alerts just to, you know, for Brent's name and for the papyri so that, you know, I'd get an email if anything was announced and I wouldn't miss it. And that was kind of, uh, that was, that was where I was in early 2020.

And when we're talking about particle accelerator, was he using the one at Oxford?

Yeah. So it's a synchrotron at Oxford, near Oxford, uh, called the Diamond Light Source. And, um, you know, the thing you'd get with a particle accelerator that's helpful is, um, It's really bright, so you get a high flux of photons from that, and that makes the scanning faster, which is helpful. And then the other thing you get from a particle accelerator that's nice is the beam is parallel. And so that, because the scrolls are pretty big relative to most things you would scan in like a benchtop CT scanner at high resolution, the parallel beam allows you to get uniform resolution across the entire scroll as opposed to a conical beam, which would give you high resolution near the source of the photons and much lower resolution kind of on the other side of the scroll. And so, um, the particle accelerator, well, first it's just friggin awesome to be able to use a particle accelerator to do something.

This thing is gigantic, right? It's like, yeah, yeah. Oh yeah.

They're like giant buildings.

Yeah.

Yeah, it's like one donut-shaped building that's— I don't actually know its diameter. I think it's, you know, it's certainly hundreds of meters. You know, maybe it's a— yeah, I'm not quite sure. We can look it up. But yeah, it's a building. It's a big building that's a big circle. And particle accelerators use You know, use magnets to accelerate electrons. And then they, at the end, they like hit something and they cause it to shoot off photons. And if you do it at the right energy, you get X-rays coming out of it. And those X-rays then pass in a parallel beam through, through the scroll. And then you have a little detector on the other side of the scroll. that's sort of like a camera, like a CCD sensor in a camera, and that can see the variations in light intensity that the photons have after they've passed through the scroll. And so then what you do is you kind of shoot a bunch of those and you spin the scroll in place. You just sort of slowly rotate it. And that allows you to get these variations in intensity at different cross-sections through the scroll. And then you do some math. There's an algorithm called the inverse Radon transform, which allows you to reconstruct based on all these different images the 3D structure of what's inside the scroll. And that whole process is called tomography. And that's where the T in CT comes from. It's a computed tomography scan.

Yeah. I love the technical explanation that you do it so simply and well. It's, um, just amazing to think how much human brainpower had to come together to make this thing remotely possible. And yet it is possible. And as you said in a tweet a while back, how would the ancient Romans feel if they knew that 2,000 years later we would be using particle accelerators and supercomputers to read their words, preserve them for eternity, And whisper them into the ear of a baby god. Yeah.

Yeah. I mean, they might be horrified by everything about us, actually. But there's probably, I think, I mean, you would know a lot better than me, but it seems to me from listening to your podcast and reading a little bit that, you know, the ancients were pretty concerned about posterity and, you know, legacy. And, um, so maybe the idea that we're working incredibly hard Um, to resurrect their words and then, you know, preserve them on the internet and maybe put them into AIs. Uh, maybe they'd like that. I don't know. I'd certainly find it interesting.

Yeah, I think it's very much in the spirit of, um, of antiquity, you know, like people look at, they say, oh, well, Plato has already had this classical culture. You know, they have, they've had Homer for 300 years. Homer himself is already classical, you know, like he's, he's, he's speaking this language that is very old and archaic. And there's a kind of like deep desire to preserve this and to pass it on and immortality. The bard is kind of like the vector of immortality for these heroes. Um, and I always think that, you know, people look as far back into the past as they want to look into the future. So yeah, there is something really kind of like classical and frontier spirit, I think, about bringing this stuff to life.

Yeah. So anyway, just picking up the story, that's how it started. And then a couple of years passed and my Google Alerts did not inform me of any like major breakthrough. We didn't know what was in the scrolls yet. And I had this sort of excuse, which was that My friends and I organize these kind of get-togethers occasionally in California, small groups of people who get together to kind of share what they're working on, interesting projects, and we use it as a way to just try to invite folks who are doing cool things and, you know, whatever kind of we think is fascinating that's out there in the world. And I thought, you know, gosh, I should really invite the Scroll guy, Brent, and he can talk about his projects. I know none of my friends know about this, and I think everyone will find it as fascinating as I do. And so I reached out to Brent and invited him to come out to California for a weekend and spend it with us and chat about what he was doing. And it was actually a little bit hard to get him on the phone, but when I finally did, he was game. And he came out and we had— I was really excited. I got him to do a talk. And I had sort of lured him out with this promise that if he came, you know, there'd be a bunch of great AI people there. Uh, maybe they would get excited about the project and want to help or have some ideas that could be useful for him. And, you know, there were some successful, you know, tech founders there. And, you know, I knew his project was, um, was funded, but, um, you know, relatively small team and maybe, you know, maybe somebody would get inspired and, and write him a check or something like that. And so that was sort of my, um, debate that I used, uh, to get to meet him. I was just starstruck and I thought he was doing this cool thing. But the weekend passed, and though it was a lot of fun, nobody else was as excited as I was. I was totally scroll-pilled by the end, but somehow it hadn't worked on anyone else. And so then I basically just felt bad because I had invited Brent out and sort of promised something would happen and nothing had happened. And so before he left, we got together. And I'm just trying to kind of salvage his trip. And so out of embarrassment, basically, I suggested to him, like, why don't we, you know, work together and we could launch a contest and try to get people on the internet who are smart to work on this? Because it seems like once you have the scans, which, you know, he had done, it's just— it's quote unquote just a software problem. And someone who's working from their laptop anywhere in the world might be able to make some progress and contribute to this and get it done faster. And, um, I was, you know, partly aware that as an academic who'd been working on this for almost, you know, 20 years, um, Brent might not be open to that, uh, because, you know, academics want to finish their work and receive credit. But I think Brent's a really unique person, and I think he was probably also getting impatient and just like wanted to figure this out. Um, and, um, and so he said yes. And, uh, I got excited and I, you know, I promised I'd chip in the prize money. I offered to put up $100,000 or $125,000. And then my friend Daniel Gross, who was with me, said he'd match that. So we had, you know, a handshake with Brent and a quarter million dollars. And, you know, I left, I drove home pretty excited about this. But then it pretty quickly dawned on me what, like, a huge project this was going to be. I mean, the scans are big. The techniques are complicated. There's a lot of things you need to know to make any progress on this at all. It's not like a simple math puzzle or something like that. There's how the scanning works, how the tomography works, the, you know, file format of the scans, the tools that need to be used to unroll the scrolls, some of which Brent's team had already built that were, you know, instrumental. They had made enough progress that I was confident enough that this could work, it might work, but there was just a lot that hadn't been done yet. No one had yet found a single letter inside of a scroll, and there were lots of technical challenges in even identifying the ink. And I should probably say something about that. So interestingly, you know, if you get an X-ray of your arm or something like that, it's pretty easy to see where the bones are because they're denser than the other tissues in your arm. And so usually on an X-ray, they show up as white, and then your other tissues are darker. And so that's what you would hope for the scrolls is that you sort of scan them and the letters show up, you know, like bones, like they show up, you know, white against a dark background. There'd be some contrast. But unfortunately, as it happens, the scrolls in Herculaneum were written using a carbon-based ink and then on papyrus, which itself is also carbon-based. And so the, you know, radiocontrast of those things was nil, basically. So when you scan one of these scrolls, you see no letters, you see no ink in X-ray, that you don't see it at all.

One scholar described it to me as like the, the stuff that collects at the bottom of your lamp, like stick your finger in that, add some gum arabic, and that's your ink. And it's like, it's actually not that different from like a burnt scroll. It's like, it's already like carbonized.

Someone told me that too, that it was like soot and, you know, soot on papyrus basically. And so Brent had some results from not whole scrolls but broken-off fragments that indicated it was possible though. He'd done some really painstaking work to show that it was possible to do using machine learning and that the machine learning algorithms could possibly pick up subtle patterns in the shape of, you know, not in the contrast but in the sort of shape, like the papyrus surface is got all these ridges in it from the fibers in the papyrus plant, and then they crisscross. And so there's like ridges and divots, and the ink, it turns out, kind of fills those in a little bit and smooths them out. And so you have in parts like a little bit of a smoother surface. It maybe stands slightly proud of the papyrus. And what we later learned was that when the ink dries and when it's thicker, we think when it's thicker, It will form this kind of, you know, pattern of what we call cracked mud or crackle, which looks sort of like the surface of a dried lake bed a little bit. People, you know, who have been part of the contest, like this one guy, Casey Hanmer, who made this sort of critical discovery of this cracked mud pattern and is a friend of mine, you know, he spent so much time looking at these things that now whenever he's sort of going around, if he sees cracked mud, he's like, it's like a game of Tetris to him. He sees letters everywhere. So. Yeah, you have that sort of afterimage. But, um, yeah, so we had all that, but no, you know, no one had sort of found letters in the scrolls before. And so anyway, I realized it was like super complicated problem and it was going to take a lot of work and there's no way I could do it myself. And Brent said, look, I've got a couple of grad students, but like they're incredibly busy. So we needed, we needed to get help. And I, I actually put out this very mysterious job description to try to hire someone to work on this project. And I ended up getting Actually about 60 applicants. Um, I didn't say what it was. I just said, this is like an Indiana Jones style adventure for a programmer.

Dangerous mission, return not likely.

Yeah, exactly.

Yeah.

I promised like, if it works, this will be a story you tell your grandchildren. That was sort of my promise. And, um, I think we've lived up to that one. So that's really cool. But I ended up hiring this great guy, JP Pozma, who helped basically me put together the contest at the beginning of last year of 2022. And, you know, I guess the rest is history. You know, we launched the contest and it was a heck of a year. We'd set a deadline for ourselves of December 31st. And I was pretty confident it could be solved, but not at all confident that it could be solved in the 9 and a half months that we gave ourselves. And so I was sort of biting my nails the whole year and constantly asking, what can we do to increase our probability of success? And, you know, how do we— is there anything we can do to move the ball forward, basically?

And so the structure you guys came up with, I think is really interesting because, you know, GitHub is to Microsoft, it's like open source versus, you know, the kings of IP and patents and suing you if you infringe. But so there's, there's like, I think there's an interesting kind of paradigm shift there. And then the way that the Vesuvius challenge works with the kind of incumbent way is institutional funding, apply for a grant from a government or a foundation, and then it has to get channeled through academia. And obviously that has come up with some good for society, but it also can be a little bit slow and inefficient and politicized. And so I think just this idea of doing a competition open source like you did is fascinating. Yeah. So how did you kind of come up with that and tell us about that?

Yeah, I mean, people think this was a really creative idea for me. Like, sort of the most obvious idea because that's what I'd done my whole life. I mean, my whole life, I basically worked in kind of online internet communities. I kind of grew up on the internet. I got on the internet in the early '90s and I was a kind of introverted science kid. So, by the time I was 15 or 16, most of the people I knew were on the internet. This is much less unusual now than it was then. Then it was really strange. But the people who were on the internet then were not normal people. They were obsessive technology people and programmers. And so we had this emerging phenomenon of what we now call open source already then, which was super smart people working on projects together, writing software, sharing their work, learning from each other. And I loved it. And, you know, as a 15-year-old in a small town in Virginia, I could kind of join like these incredible communities online and learn from them and try to contribute. And so that was amazing to me. And so I, um, I did, and then I, you know, I was running GitHub. So like, you know, GitHub basically hosts now a lot of these open source projects. And so it was sort of like, to me, it was like the most obvious thing was why don't we open it up to the world? And, uh, you know, many eyes can make the problem shallower potentially. And— but there were some serious sort of contest design problems that needed to be solved. So the main issue was that the problem was, you know, though Brent and his team had laid this extremely good foundation and had pioneered a lot of key techniques, there was just like an unknown amount of work left. And it turned out to be a you know, a pretty decently large amount of work left. And so my, you know, there's sort of this continuum of collaboration and competition and, and the problem was to pick a point along that continuum. So our original idea, what I first pitched to Brent was we should do a big XPRIZE style, you know, grant, you know, some, just like a grand prize for someone who can like read the whole scroll or something like that. And as I was starting to try to figure out how to put that together and looking deeper into the problem, a friend of mine said, Nat, there's like 4 or 5 PhDs left in this project. This is like not a weekend hack. This is not something somebody could do in a few evenings. This is like a really hard problem. And I agreed. And so the concern there with doing a grand prize and only a grand prize for people working on this part-time is that, you know, like, let's say there's 10 teams and each of them makes, you know, a little bit of progress. Maybe they get 5% or 7% or 15% of the way there. If that's all you do, you're probably going to give up. You know, you're going to feel like your chances of winning are low. And in the case where this is just a contest, you have like no incentive to share your little bits of progress with anyone else. And so we decided that— actually, first I went looking for a solution to this problem. I ended up calling the folks at XPRIZE and talking to people at other kind of online contests, distributed contests, to try to find an answer. And basically I couldn't find that anyone had solved this. I mean, it's possible that I missed some antecedent for this, which where someone had already come up with the solution, but we ended up just sort of banging our heads against it and then struck upon this idea, which was to basically have 2 systems of prizes. One would be this gigantic, charismatic grand prize for doing something amazing and cracking the problem seriously. And we knew that a grand prize would just excite people. It would Have prestige, you know, if you won it, you'd get famous. Um, and if we put a lot of money on the stage and— Yeah, exactly.

Yeah.

And if, and, you know, we could put a lot of money behind it as an incentive too. But then I had this idea of having progress prizes, which, uh, would be like a series of smaller prizes that you could win for making little bits of progress towards the goal. conditioned on you open sourcing your work. And so the idea was that if you were one of those, you know, teams that got 5% of the way there and you thought, well, I can't get any further because I got to go back to work or school, you could release that 5%. And then we would, if you kind of qualified for the Progress Prize, we would give you an award and recognize you and you'd get some money. And you get some dopamine along the way and everyone would benefit. You know, your little 5% of progress would accrue to everyone.

And, um, you open source it after you win the prize, right?

Like you definitely, yeah.

It's not like as you're going, you can peer over people's shoulders.

Yeah. I mean, you can open source it before and still be eligible, but most people would choose not to because they might want to, it's, it's this blend of competition and, and cooperation. You're in an adversarial relationship for the grand prize, and you're in a collaborative relationship for the progress prizes. So most people would choose to kind of wait till after they had won to release their progress as open source, but actually not all did. So that was the design, and we hoped it would work, and it turned out to work much better than we expected. It was incredibly powerful as a kind of mechanism design for creating this community and creating a ton of progress that I think happened much faster than it would have otherwise. The alternative might've been to have like an enormous grand prize. Um, like if there was like a $20 million grand prize or something like that, that might've worked, but we didn't have that kind of money. So that was like not an option for us. Um, the other challenge we had is that a lot of the talent The new skills that were needed to work on this are also the same skills that are needed to work on AI. And AI is having this gold rush right now. And so, you know, with our little archaeology project, we were competing with, you know, vast riches and commercial success that all these other people could, could go chase for themselves as well. And so the Progress Prize has created a community and little bits of prestige and progress. And actually what we didn't anticipate is that they were really encouraging. And so the people who won progress prizes early would become progressively more obsessed with the project, and they would have received this dopaminergic reward for having done something, and then they would want that again. And so that was— and in fact, all the people on the grand prize-winning team, all 3 of them, Youssef, Luke, and Julian had won multiple progress prizes before. And, you know, I think if maybe if they hadn't, they not only would the work not be out there for others to build on, but they might not have been so encouraged. Yeah.

And I, I think that that's one of the striking things about the project. Like, and I, I do encourage people to go to scrollprize.org to check it out, check out the website. And right now you have another round of competition going on and we, we can talk about that more if you'd like. Um, but you know, one of them is go back and reproduce the results that was, that were done before, but, but more efficiently, more or less something like that. So, so like, uh, it seems like so much of the kind of hard thinking leadership and, you know, let's say dictatorial control, and it comes into the very careful thinking about the incentives. I'm trying to think of lessons that other people might use to construct their own kind of competitions and rethink how we—

I think, yeah. I mean, maybe the Meta thing here is we treated the contestants as our customers basically. And so we didn't want to just succeed, but we wanted them to have like this really good experience and have a high chance of having a good experience. And so I do think a lot of people think with contests or bounties that it's fire and forget. You just sort of put it up there, it's a million bucks, and that's all you have to do. And it was nothing like that at all. I mean, I think that's a very bad way of thinking about it. For example, we knew that it was really hard to get into this and that people have a lot of other things they could do with their time, especially the people who can make progress on this. And so we actually, before launching the contest, worked really, really hard to produce a website that kind of made it exciting and told the story, and then also had technical tutorials that provided an on-ramp for someone to get the context they needed to start working on it at all. And that was a lot of work. You know, there was a lot for people to learn. And, you know, one of the things we realized is like you're reasoning in 3D and that's sort of hard to even describe in text. And so we needed to make like animations. I ended up hiring an animator and we spent a couple months with him making like explanatory animations, which are now all over the Scroll Prize website. And so I do think it, you know, I've built products before and I've run companies before and this actually felt remarkably similar even though it was a contest. You know, you have to think, how do I make it so when someone comes to my webpage they're interested and then they know what to do next? And so yeah, contests are a really high-leverage tool, but, you know, you can't be— they don't give you a reason that you can be lazy and not think about all of these Right.

It's not all about the money. And it's like that, uh, Patrick Collison speaking in an interview recently about how, you know, beauty is important in design because among other things, it shows that the people in charge really care. And that's really motivating for a customer to say nothing of a, of a contestant. Yeah.

We wanted to speak to them too. Like, uh, we wanted to write the text on the webpage. So it was clear that the people on the other side of the contest were like them, were technical and also excited by this sort of cross-domain project. And so anyway, yeah, I mean, the big, um, the big couple decisions we made along the way were after figuring out this basic design for progress prizes, um, were, were 2. The first is, um, we realized that there was this one step in the process that was just like really, really, really hard. And the contestants a few months in were making no We're making no progress on it. And so just to describe how it works, you know, the first thing you do is you scan a scroll and you get this basically this 3D image of it. The next thing you have to do actually is to— the scroll is like a wound spiral of papyrus. They can be really long. They can be like 15 meters long. And, you know, then they get rolled up and the writing is on the inside. And so, and then unfortunately when Vesuvius erupted, the hot mud really distorted them. They're super messed up. The heat caused the papyrus to blister in places or to delaminate or to sort of fuse together some of the layers, and then they got bent as well. And so it's not like this perfect spiral that you can just easily unroll with software. What we have to do instead is to trace the surface of the papyrus inside the scroll, and unfortunately we don't have good algorithms for doing that yet, and so that process ended up being really, really manual. There are some tools that Brent's team had built before, this guy Seth Parker, and those were a good start, but those tools also needed to be greatly improved, and so we What we ended up doing basically was, um, and this was a controversial decision inside the team. What we did is we ended up hiring a team of people full-time to just, just trace papyrus all day on these giant monitors.

Yeah. And, um, this is the segmentation, right?

Yeah. This is, we have the term we use for this is segmentation. Um, and this is, it was sort of like, it's not in the spirit of a contest, right? Like, wait, hold on a second. You're, You're supposed to be the referee. Now you're running out on the field and kicking the ball a little bit. Like, what are you doing? But I was just desperate to make sure we were successful by the deadline. And this seemed like essential. It was just not going to happen otherwise. And it turned out to be really, really important. And we ended up spending actually a lot of money on this, hundreds of thousands of dollars on paying people to do this meticulous work. We hired 15 people initially. It turned out to be so difficult to do that really most of them were not good at it, and we ended up with a full-time 3-person team of segmenters who were awesome, and, you know, David and Ben and Conrad, and they worked, you know, till their eyes hurt, honestly, on this. We were in a race because we knew in order to meet the grand prize threshold, we needed a certain amount of papyrus segmented quickly, and so that was sort of one key thing, and then the other key thing we decided was that We had this grand prize which was for finding 4 passages inside of a scroll of sufficient length, which we picked 140 characters as that length, and then we realized that was a pretty high bar, we needed some intermediate milestone, and so we decided to have a first letters prize for finding the first 10 letters inside the scroll in a certain, you know, 4-square-centimeter region. I think that was early summer when we put that kind of progress prize out there. And that was critical. I mean, that's when the whole sort of avalanche started to run, essentially, because the first thing that happened was that Casey Hanmer found this cracked mud pattern and he found a couple of letters. He didn't find 10, but he found a couple of letters in the scroll and he decided to share that with the community. And that kicked off a race and people start— he found those letters by looking just manually with his own eyes. He called it direct persistent inspection, just looking at it a lot in the X-rays.

And taking time off from his Nat Friedman-backed space-related energy startup, right? Like he's not just doing this in a basement.

No, he's a Father of three, and is running a startup. It's also very hard, and I don't know how he did it honestly. But he's amazing. So he found this; he put it out there, and then subsequently Luke Ferritor, who was at the time twenty-one-year-old University of Nebraska undergrad, saw Casey's pattern that he'd found and said, "I can train an AI model on this to find more instances of this throughout the segment, you know, the flattened segments that we." produced with our segmentation team. And absolutely miraculously, he found the first word that had been seen in 2,000 years from inside an unopened scroll. He was the first person to see that handwriting in millennia. And then even more incredibly, it was a cool word. It was this word, perforus, which I hope I'm saying it right. Yeah, close enough. Okay, thanks. It means Purple. And, uh, you know, it could have been like a super normal word, um, like although or something, but, you know, it was purple and that was so mysterious, uh, to us. And so that was the first, first word that was found.

The color of royalty. Yeah.

Yeah, exactly.

So we're going to have Brent, uh, Dr. Brent Seales on here in a little while on the show. Um, everybody stay tuned, uh, coming soon, but Um, so we want to get more into the technical details, but we could zoom out a little bit. You know, one of the things that I think is so exciting about this project is, um, it's kind of this meeting of, uh, humanities and great, like great knowledge from the past, but like activated with a whole lot of high agency determined people. Dr. Seales, very determined person. I mean, he's been working on this for 20 years.

Definitely.

Yourself and Daniel and all the people on the prize team. Like, you know, I think of what Paul Graham says about, you know, being relentlessly resourceful as like a key for a startup founder. I wonder if this is kind of like, you know, a project like this just has to filter for that kind of person and kind of train that in them, which to me is just this classical spirit of competition that you see in the Olympic Games. Um, so, so if I could ask, like, what are some of the— because when I think of, um, the great men that we study on the Cost of Glory podcast, you know, somebody like Julius Caesar, okay, he's, he's very ambitious, but he's also, uh, interested in technical matters. He rewrites the calendar. He's a great, you know, kind of field engineer in warfare. He's a poet, he's a writer, he's this kind of Renaissance man in a lot of ways. And it's like, like Andrew Roberts was saying on a show recently, like Napoleon would've been a tech founder today. I think you'd probably say the same thing about Julius Caesar. What are some of the qualities that you would like to see this project inspire in people today that you'd like more of?

That's a good question. Yeah, it would be great if more people did crazy projects. that might fail. And I think this is— this was, at least in my world, this was like a really unusual thing to work on. And I got a lot of funding from friends who just thought it was crazy and they hoped it would work. And that was key. And so, yeah, I do think people should do more things that just seem strange, but that like they have a lot of personal enthusiasm about. And Should be slightly more impulsive about that. I mean, the story for me psychologically of doing this is that I kind of got into it impulsively, and then once it was out there and I had kind of my name on it, I really wanted to do whatever it took to make sure it succeeded. And so I had this like relatively low threshold for starting this project, and then once I was in it, I was in a way I was stuck. I trapped myself because it was going to be really embarrassing if this didn't work. If it didn't work, people would say, like, of course it didn't work. This was like an insane idea. And, you know, people have been trying for hundreds of years to open these scrolls and you, you had horrible hubris by getting involved at all. And, um, I, uh, so I do think that's part of it is just like you should do something like that. And then I think the other thing is, um, there's a lot of areas of the world that are just not picked over, you know, by people, at least, uh, of like my, my tech background. There's, so I would say like, it's probably a good idea to look in the just less obvious areas to some extent. I mean, bringing software to archaeology, maybe it's obvious to a lot of other people, but at least in my world, it wasn't. And so I think that's, you know, there's $100 bills lying on the sidewalk there. And then, yeah, just do whatever it takes. You know, do whatever it takes. do it quickly. I think, I think that's really important too. So, um, but yeah, I hope like the— this will be good if it inspires other people to do more crazy projects, not because, you know, you can tell like the story of super high impact. Um, you know, the, the kind of classic thing for a tech founder to work on, you know, after they have a slight amount of free time is something in health, you know. cure diseases, etc. I think that's very important, obviously, but I would like to see more kind of surprising projects from people and things that are slightly stranger and only make sense in retrospect. I think that's what you want to do. You want to do things that make sense in retrospect, whether or not they make sense at the time. And the ones that make sense at the time don't always make sense in retrospect. And so that would be my, if this, if this is inspiring to people, I hope, you know, from my, my world, I hope it inspires them in that way to do something that seems like strange, doesn't make sense now, but if it works, will make a lot of sense in retrospect.

This, uh, project could unearth another Renaissance worth of, of ancient texts. Would you think people will be more or less receptive to such a discovery than they were in the Renaissance today?

I don't know. I think, I think it's, I don't really know exactly. I guess they were pretty receptive during the Renaissance.

Maybe we overestimated with the rosy-colored lenses looking back at the past. Yeah.

My guess is they'll be slightly less receptive now, um, just because we have so many sources of new information every day. Um, And there is a sense of freshness from a lot of that information, you know, on Twitter or whatever. And so we're not as bored, is my guess, as we might have been in a lower information era. However, I think we will be very interested in it now. I do think there's a way in which people— the world is sort of changing a lot and society has fewer kind of bedrock spiritual values that we really believe in and that we all have in common. And so there's a way in which I think some people at least are looking to the past and the ancient world for comfort and guidance and maybe other ways of thinking and living. In a search for something that's not a fad diet or ephemeral.

Yeah. And so many of these texts that we've already discovered really deal with some of the hardest questions of life. One of the best passages of Philodemus is on how do you, how do you like meditate more on death so you live a better life? And so when death comes and he's, you know, he's living this period where everybody important is dying violently, actually Julius Caesar dies violently. Pompey dies violently. Cicero dies violently. All these guys In the period that, you know, are Philodemus's peers, his clients, his, uh, his patron Piso, I don't think died violently, but, you know, maybe because he took Philodemus's advice. But yeah, they, I think, you know, some of these texts really, you know, I would love to discover dialogues of Aristotle. I would love to discover letters of Julius Caesar. I think we really could. But, um, but the villa's secrets, I mean, and there's of course there's great art and we're going to do a more of this content on the show to talk about why this is so cool. Um, yeah, even what we've already discovered to me is very inspiring because it tells us about these people that are, um, some of the most fascinating figures in Roman history. And, and like, you know, a guy like Philodemus is trying to help those great leaders connect with some of the profound questions and kind of, I don't know, the values that are really going to make their society and their own lives better. Like, that's, I really think that's his mission. And so I find it like, it's already an inspiring story to me as I'm getting into it. I'm kind of going down the rabbit hole with you guys as a non-technical person.

Yeah, there's a way in which, you know, it seems sort of random that what we've got is a bunch of Epicurean stuff, but I was wondering, like, maybe Epicureanism is sort of what we need right now. And, uh, you know, maybe there's like, like, what if actually there's some amount of, um, meaning that we can find in Epicureanism? Stoicism is super famous. Um, and a lot of people in out here in California are into Stoicism. Um, but, uh, but I like the idea of just exploring this alternative frame of Epicureanism too, and what makes life worth living from an Epicurean point of view. Um, yeah.

Yeah. And in, in, in times of political turmoil, like they lived in, in Philodemus's day, maybe especially so. Well, Nat, thank you for taking the time. Uh, where can people find you, uh, before, before you go?

Oh, sure. Um, well, you can read more about the Vesuvius Challenge at scrollprize.org, right? Um, where we've read 5% of one scroll. We have hundreds of full scrolls left to read. And we're working really hard on doing that. And I hope we do uncover a dialogue of Aristotle or a new history or something incredible, but whatever happens, it'll be great. And then, geez, where do you find me? My website is nat.org. And so you can read a little bit more about me there, or I'm on Twitter as Nat Friedman.

Great. Well, thanks again. We'll see you soon.

Thank you for having me.