88 - Mysteries of the Scrolls — with Nat Friedman

Julius Caesar's father-in-law owned a villa on the shore of the Bay of Naples in a little town called Herculaneum. This is the father of Calpurnia, who was the longest-standing and most respectable of Caesar's wives, his final wife, the one who is a character in Shakespeare's play Julius Caesar. Well, this man, Calpurnia's father, Piso was his name, Calpurnius Piso, He was a personal enemy of Cicero. He was the consul in 58 BC. Cicero wrote a speech against him, delivered it in the Senate. We still have that one. But Piso was also a dedicated student of Epicurean philosophy, and he invited a famous philosopher named Philodemus to come and reside at his villa in Herculaneum. And Philodemus brought a vast library of Greek philosophy with him. From Athens. 100 years or so later, that library Philodemus curated was still in that villa when Mount Vesuvius erupted. And along with the city of Pompeii, the Vesuvius eruption buried that villa in Herculaneum too, carbonizing those scrolls in pyroclastic mud and ash, and thereby sealing a treasure trove, a time capsule, to be uncovered centuries later Scholars have known about this library and these scrolls for more than 2 and a half centuries. Hundreds of scrolls have been unearthed, and possibly thousands more still lie buried in that villa. But for the most part, we've been unable to read these scrolls, or we've only been able to read certain bits of them, and with extreme difficulty.

Until now.

We are on the verge of some of the most incredible discoveries about the ancient world since the Renaissance. And in this interview, I am talking with one of the people who's at the center of the story of decoding the Herculaneum Scrolls. His name is Nat Friedman. He's not an archaeologist or a historian, but a technologist, a company founder, a former CEO, and we'll get to the details in the interview.

Hi, Nat.

I'm Alex Petkus. You are listening to The Cost of Glory, where it is our mission to retell the lives of the greatest Greek and Roman heroes, following the lead of the ancient philosopher and classic writer, Plutarch, in order to try to unearth the characters of these men and their insights, if we can, to resurrect some of the ancient spirit of greatness those real-life heroes exhibited, and with which They have over the centuries, periodically in times of great trouble, inspired the more active classes of society. I'm happy to announce, by the way, that The Cost of Glory is now an Infinite Media collaboration. Well, without further ado, here is the interview with Nat Friedman, where Nat tells the story of Herculaneum, the scrolls, and this fascinating contest that he co-engineered to help push the scientific research and technological innovation Past what seems to be a tipping point in the effort to decode these texts. And we'll talk about some of the other very important people involved—

modern people—

and also about Epicurean philosophy, about Philodemus, and many lessons that you can apply for bringing out the best in your own work and life. I hope you enjoy it and find it as inspiring as I did.

Nat Friedman is an entrepreneur and investor. He's a co-founder of Zimian and Xamarin. I hope I said those right. The latter of which was acquired by Microsoft in 2016, Nat stayed on at Microsoft, and in 2018, he led Microsoft's acquisition of GitHub, an important open source platform of which he became the CEO of in 2018 until 2021. Then he moved on to other projects, which brings us to maybe the number one reason and occasion for our conversation. One of these cool projects that's come across your desk, Nat. The Vesuvius Challenge, which is in some— we're going to get into the details— a mission to scan and digitally unroll and read a large trove of ancient scrolls that are preserved through the famous Pompeii-destroying eruption of Vesuvius, but have never been— it's never been possible to read them before until now. So, you know, at The Cost of Glory, we're all about bringing to life great ancient figures and texts and ideas. from the past, from the Greek and Roman world that you can use today. So for that reason, I'm really thrilled to have you. Thanks for joining us, Nat.

Thank you for having me. I love your podcast.

Great. Well, why don't we start by just you telling us, for those who don't know the story, how did a guy like you get interested in a project like this?

It was really pretty random, honestly. One of those sort of sequences of events that seem so contingent and unlikely that you have to wonder, how predetermined anything that ever happens really is. But yeah, I mean, it was sort of started back in early 2020. I was out here in California. I was running this company called GitHub and COVID struck and the lockdowns kicked off and we were in that early period of COVID when there was so much uncertainty about kind of what it was. I don't know if anyone really remembers, but obviously COVID was not good, but there was a lot of concern at the time that it that it was actually much, much worse than it ended up being. People were disinfecting their groceries and there was just a lot of worry in the world. And the company I was running was a great company. GitHub is a fantastic company, amazing people, but man, not always easy. And so I was just sort of found myself looking for an outlet, looking for other things to occupy my mind in the evenings. And I stumbled upon this book that I think Amazon recommended to me, which was called 24 Hours in Ancient Rome. And, you know, as a kid, I'd never actually been very interested in history, to be honest. I was sort of a math and science kid. And, you know, learning about math and learning about science seemed to me like you were sort of in pursuit of universal truths of the universe and these things that would be important and true in any world. And history seemed very kind of contingent and parochial to me. And something that just was sort of like a sequence of events that had like no sort of universality to it. That's what I thought at the time.

And Aristotle would've agreed with you actually. He says, yeah, history is all about particulars rather than the philosophical truths. That's very interesting.

But yeah, I mean, I did later, I think after college, I did realize that even if that were true, this world with this history is the only one I would ever live in. And so even if all these things were contingent, I would only have to learn them once. And so that sort of did wake me up to the fact that maybe it was worth paying some attention to, but I didn't really do it. I mean, I did enjoy, I took some classics in high school and I did enjoy reading, you know, Plato and Homer and it was just sort of fun food for thought. But to me, it almost seemed like reading fiction. And so anyway, for whatever reason in this period in early 2020, this book appeared in my Amazon recommendation list and I decided to give it a shot and I ended up staying up all night reading it. It was a fantastic book, actually, 24 Hours in Ancient Rome. I really recommend it. Someone later told me that it was written for schoolchildren, which might be true. I don't know, but it doesn't matter. It was fantastic. It sort of really drew me into thinking about that period. And so I found myself over the sort of subsequent evenings kind of, you know, in these different Wikipedia chasing and surfing exercises. And I think there was something about reading about ancient history At a time, you know, when the world was unsure about its own future, it was just deeply calming. You know, all your problems in either large amounts of time or large amounts of space, your problems seem small.

I love this image of you as a CEO of this $7 billion acquisition, right? Like huge enterprise taking the time to reflect on the past. Maybe it's the kind of chaos of COVID you know, it's really not sure what you're supposed to do, but maybe that's the answer, that it was somehow calming.

It was, it really was. You know, your own issues or problems seem pretty small if you realize that your life is short and there have been many lives before you and there will be many lives after you. And the other thing that I think has a similar effect is, you know, astronomy, just thinking about like the size of the universe. And you maybe hopefully become slightly less petty if you, if you pay attention to these things. So maybe it was that, or maybe it was just interesting too. So, you know, there's this chapter in the book that talks about the sort of way Roman bread was made, and everyone was into making bread back then. So I went out and sought out like the heirloom wheat varietals, this red fife flour, and tried to make my own Roman bread. Anyway, so one night I'm, I'm in Wikipedia and I stumbled on this article about the Herculaneum Papyri, and I'd never heard of them before. And I thought it was just really incredibly fascinating. Obviously, Vesuvius kind of covered the whole region of the Bay of Naples in mud and ash and incredibly hot gases. And everyone knows about Pompeii, and I'm sure I'd heard about Herculaneum before, but I'd never heard of this. And so there was this enormous kind of villa, private villa that was just on the outside of Herculaneum that had been buried by the eruption as well. And it was thought to have been built and owned by the father-in-law of Julius Caesar, this guy Calpurnius Piso. And what made it really special is that in the mud, as it was excavated in the 1700s, late 1700s, these kind of lumps of charcoal were discovered. And I think at first no one knew what they were until someone noticed on a broken-off flake of one of them, a little bit of writing. And the excavators realized that they'd stumbled upon a library, a library of papyrus scrolls made from, you know, the Egyptian papyrus plant. And this was a unique find because somebody gave me this great phrase, humidity is the enemy of history. And so, you know, to the extent that there were libraries along the Bay of Naples at that time, if they had not been buried by a volcanic eruption, they would have just eventually sort of rotted away. And so therefore we have really almost no intact books or works of writing from that period unless they were copied by a cascade of monks through the Middle Ages and preserved through copying. And anything that was sort of original is gone. There are some exceptions in really arid places like Egypt, I later learned. But, you know, those are unfortunately the exception and not the rule. And so, Thousands of scrolls were excavated from there, and then there were attempts to open them. And most of those attempts were in the late 1700s, were quite destructive. You know, people did things like they took a dagger and they would just sort of like scrape off the layers of papyrus or slice it, and maybe they'd see a few letters and they'd try to write those down along the way. But, you know, this was very, very destructive.

It's horrifying to think now with what we can do, you know.

Yeah. Well, the worst part about it is, as I later learned, the scrolls that were attempted to be opened in these like really just physically destructive ways were the best ones because, you know, obviously the excavators looked at the ones that were in the best shape and thought, you know, these will be the easiest to open and read. And so it's probably just through the selection bias, the worst ones that we have now that were not destroyed along the way. There was a monk, this guy named Piaggio in the late 1700s, who ended up designing this machine to attempt to unroll them. And it used like a cat gut and pig bladder and it was semi-successful. So like it would, in the case of the best scrolls, be able to unroll parts of them. But it was also pretty destructive overall too. So, you know, layers would stick together and, you know, there'd be gigantic holes in the unrollings that he did. But we were able to apparently determine from that that these were primarily Epicurean writings of this Epicurean philosopher who's a follower of Epicurus, this guy named Philodemus.

Yeah.

So there had been like a little precedent for looking inside these things, but what was left couldn't be opened. And what was left was huge, hundreds and hundreds of scrolls. And it in fact is the largest library that we know of from antiquity and is a mystery. I mean, and I loved that. It's a mystery. What's in there? No one knows. And there were no known techniques for either opening them or figuring out what was in the papyrus without opening them.

It's amazing. I did not know the part about cat gut and pig bladder.

Yeah.

Yeah. So before, you know, the last 5 years, there were some number of scrolls, maybe the easier ones to open. many destructive losses of text in the process. But I guess in recent years, it seems like the technology has sort of like, and I remember when I was in grad school hearing whispers about this, you know, like, oh, they're, they're getting close. They're getting close. But it's always, it always seems like kind of asymptotal. Like we're not actually getting closer.

Yeah. Well, first I think it's just like intrinsically exciting to have this opportunity. And I think everyone's sort of had these dreams about like having a time machine and going back to some period that's particularly interesting to you and being able to see what it was actually like. Or I remember when I was a little kid, either thinking of or reading about this theory that maybe you could extract audio from pottery wheels. And if there had been some reed that was brushing against the pot when it was being spun, then, you know, maybe it would have like a phonograph needle recorded, you know, audio from the room. And just being totally fascinated by that idea. I had this idea that maybe like, as paint was drying, the pressure waves from the sound in the room, you know, could end up being encoded in like the physical structure of the dried paint. I no longer think that makes any sense at all, but like when I was 11 or something, I thought that was a neat idea. So we all have had these like theories about, or fantasies about how cool it would be to get like a real window into the past.

Yeah.

And so I think that's what fascinated me about this was like, wow. if you could actually get all these scrolls and read them, it's a lot. It's a lot of text. And it would add, in terms of the total, you know, megabytes of information we have about the past, it would add like a really material amount. And so I think that was one of the things that excited me. And then I think it was also just this like anti-authority impulse of there are these mystery scrolls, they're unopenable, you can't open them, and someone tells you you can't open them and you think like, we've gotta find a way to open them. So I just thought it was really cool. I'd never heard of it before. Started talking to my friends about it, and it turns out none of them had either. You know, my friends are from tech and software, and so, you know, maybe we just aren't educated on this, but I thought it was just such a neat mystery that people have worked on for hundreds of years. And then maybe later that night, I found an article about an effort that was underway to try to read the scrolls using 3D imaging, and that there was this professor at the University of Kentucky named Brent Seales who had been working for years, actually, I think at that time for about 18 years, to scan the scrolls in 3D at high resolution with something like a CT scanner, like you would, you know, if you, if you have an injury, you might have been in a CT scanner, part of your body was, to be able to look inside them and then hopefully be able to sort of use software to not physically but digitally unroll them. And then recognize the ink and read what's in there. And he'd been working on this for years and years. And so that was like, you know, when I found about the Herculaneum Papyri, I thought that was cool. When I found out about Brent's project, I thought that was like the coolest thing I'd ever heard. And I just became a fanboy and thought I'd, I'd follow along.

Interesting.

So that's how it started.

Yeah. So had he scanned a full scroll at that point or had it just been fragments?

Yeah. No, he had over several years. I think his first scans he did were in 2009 and they were on a low resolution. I think maybe it was a medical-grade CT scanner. And when you do a medical CT, you don't really need to see things at the millimeter or submillimeter level most of the time in order to see if your wrist is broken or whatever. And so that was an important step, but those were not usable to try to read the scrolls. And then, The thing that really excited me, because I found this in early 2020, was that I read that in 2019, he had scanned 2 of the scrolls in a particle accelerator. And, the particle accelerator was helpful because it allowed him to get much higher resolution and theoretically a resolution that was good enough to be able to read the text. And so, when I discovered this just months after those first scans, I thought, okay, a breakthrough imminent. And, you know, I'll just— I went and set up some Google Alerts just to, you know, for Brent's name and for the papyri so that, you know, I'd get an email if anything was announced and I wouldn't miss it. And that was where I was in early 2020.

And when we're talking about particle accelerator, was he using the one at Oxford?

Yeah, so it's a synchrotron at Oxford, near Oxford, called the Diamond Light Source. And You know, the thing you'd get with a particle accelerator that's helpful is it's really bright. So, you know, you get a high flux of photons from that, and that makes the scanning faster, which is helpful. And then the other thing you get from a particle accelerator that's nice is the beam is parallel. And so that, because the scrolls are pretty big relative to most things you would scan in like a benchtop CT scanner at high resolution, the parallel beam allows you to get uniform resolution across the entire scroll as opposed to a conical beam, which would give you high resolution near the source of the photons and much lower resolution kind of on the other side of the scroll. And so the particle accelerator, well, first it's just friggin awesome to be able to use a particle accelerator to do something.

This thing is gigantic, right? It's like, yeah, yeah. Oh yeah.

They're like giant buildings.

Yeah.

Yeah, it's, it's like one donut-shaped building that's, I don't actually know its diameter. I think it's, you know, it's certainly hundreds of meters. Yeah, I'm not quite sure. We can look it up. But yeah, there, it's a building. It's a, it's a big building. That's a big circle. And particle accelerators use magnets to accelerate electrons. And at the end, they like hit something and they cause it to shoot off photons. And If you do it at the right energy, you get X-rays coming out of it, and those X-rays then pass in a parallel beam through the scroll. And then, you have a little detector on the other side of the scroll that's sort of like a camera, like a CCD sensor in a camera, and that can see the variations in light intensity that the photons have after they've passed through the scroll. And, So then what you do is you kind of shoot a bunch of those and you spin the scroll in place. You just sort of slowly rotate it, and that allows you to get these variations in intensity at different cross-sections through the scroll. And then you do some math. There's an algorithm called the inverse Radon transform, which allows you to reconstruct based on all these different images the 3D structure of what's inside the scroll. And that whole process is called tomography. And that's, that's where the TNCT comes from. It's a computed tomography scan.

Yeah. I love the technical explanation that you do it so simply and well, it's just amazing to think how much human brainpower had to come together to make this thing remotely possible. And yet it is possible. And as you said in a tweet a while back, how would the ancient Romans feel if they knew that 2,000 years later, we would be using particle accelerators and supercomputers to read their words, preserve them for eternity, And whisper them into the ear of a baby god. Yeah.

Yeah. I mean, they might be horrified by everything about us, actually. But there's probably, I think, I mean, you would know a lot better than me, but it seems to me from listening to your podcast and reading a little bit that, you know, the ancients were pretty concerned about posterity and, you know, legacy. And so maybe the idea that we're working incredibly hard to resurrect their words and then, you know, preserve them on the internet and Maybe put them into AIs. Maybe they'd like that. I don't know. I'd certainly find it interesting.

Yeah.

I think it's very much in the spirit of, um, of antiquity, you know, like people look at, they say, oh, well, Plato has already had this classical culture. You know, they have, they've had Homer for 300 years. Homer himself is already classical, you know, like he's, he's, he's speaking this language that is very old and archaic and there's a kind of like Deep desire to preserve this and to pass it on and immortality. The bard is kind of like the vector of immortality for these heroes. And I always think that people look as far back into the past as they want to look into the future. So yeah, there is something really kind of like classical and frontier spirit, I think, about, about bringing this stuff to life.

Yeah. So anyway, just picking up the story, that's how it started. And then a couple of years passed and my Google alerts did not inform me of any, any like major breakthrough. We didn't know what was in the scrolls yet. And, and I had this sort of excuse, which was that my friends and I organized these kind of get-togethers occasionally in California, small, you know, small groups of people who get together to kind of share what they're working on, interesting projects. And we, we use it as a way to just try to invite folks who are doing cool things and, um, and, you know, whatever, whatever kind of we think is fascinating that's out there in the world. And I thought, you know, gosh, I should really invite the Scroll guy, Brent, and he can talk about his projects. I know none of my friends know about this, and I think everyone will find it as fascinating as I do. And so I reached out to Brent and invited him to come out to California for a weekend and, you know, spend it with us and chat about what he was doing. And it was actually a little bit hard to get him on the phone, but when I finally did, he was game. And he came out, you know, I was really excited. I got him to do a talk and I had sort of lured him out with this promise that If he came, you know, there'd be a bunch of great AI people there. Uh, maybe they would get excited about the project and want to help or have some ideas that could be useful for him. And, you know, there were some successful, you know, tech founders there. And, you know, I knew his project was funded, but, you know, relatively small team and, you know, maybe somebody would get inspired and, and write him a check or something like that. And so that was the bait that I used to get to meet him. I was just starstruck and thought he was doing this cool thing. But the weekend passed and though it was a lot of fun, nobody else was as excited as I was. I was totally scroll-pilled by the end, but somehow it hadn't worked on anyone else. And so then I basically just felt bad because I had invited Brent out and sort of promised something would happen and nothing had happened. And so before he left, we got together and I'm just trying to kind of salvage his trip. And so out of embarrassment, basically, I suggested to him, like, why don't we, you know, work together and we could launch a contest and try to get people on the internet who are smart to work on this, because it seems like once you have the scans, which, you know, he had done, it's just, it's quote unquote, just a software problem. And someone who's working from their laptop anywhere in the world might be able to make some progress and contribute to this and get it done faster. And I was, you know, partly aware that as an academic who'd been working on this for almost, you know, 20 years, Brent might not be open to that because, you know, academics want to finish their work and receive credit. But I think Brent's a really unique person. And I think he was probably also getting impatient and was just like, wanted to figure this out. And so he said yes. And I got excited and I, you know, I promised I'd chip in the prize money. I offered to put up $100,000 or $125,000. And then my friend Daniel Gross, who was with me, said he'd match that. So we had a handshake with Brent and a quarter million dollars. And I left, I drove home pretty excited about this, but then it pretty quickly dawned on me what a huge project this was going to be. I mean, the scans are big, the techniques are complicated. There's a lot of things you need to know to make any progress on this at all. It's not a like a simple math puzzle or something like that. There's how the scanning works, how the tomography works, the file format of the scans, the tools that need to be used to unroll the scrolls, some of which Brent's team had already built that were instrumental. They had made enough progress that I was confident enough that this could work. It might work, but there was just a lot that hadn't been done yet. No one had yet found a single letter inside of a scroll, and there were lots of technical challenges in even identifying the ink. I should probably say something about that. So interestingly, if you get an X-ray of your arm or something like that, it's pretty easy to see where the bones are because they're denser than the other tissues in your arm. And so usually on an X-ray, they show up as white, and then your other tissues are darker. So that's what you would hope for the scrolls is that you sort of scan them and the letters show up like bones, like they show up you know, white against a dark background, there'd be some contrast. But, uh, unfortunately, as it happens, the scrolls in Herculaneum were written using a carbon-based ink and then on papyrus, which itself is also carbon-based. And so the, you know, radio contrast of those things was nil, basically. So when you scan one of these scrolls, you see no letters, you see no ink in X-ray that you don't see it at all.

One scholar described it to me as like the, the stuff that collects at the bottom of your lamp. Like stick your finger in that, add some gum arabic, and that's your ink. And it's like, it's actually not that different from like a burnt scroll. It's like, it's already kind of like carbonized stuff.

Someone told me, me that too, that it, that it was like soot and, you know, soot on papyrus basically. And so Brandt had some results from not whole scrolls, but broken off fragments that indicated it was possible though. He'd done some really painstaking work to show that it was possible to do. using machine learning and that the machine learning algorithms could possibly pick up subtle patterns in the shape of, you know, not in the, uh, contrast, but in the sort of shape, like the papyrus surface is got all these ridges in it from the fibers and the papyrus plant, and then they crisscross. And so there's like ridges and divots and the ink, it turns out, kind of fills those in a little bit and smooths them out. And so you have in parts like a little bit of a smoother surface. Maybe it stands slightly proud of the papyrus. And what we later learned was that when the ink dries and when it's thicker, it— we think when it's thicker, it will form this kind of, you know, pattern of what we call cracked mud or crackle, which looks sort of like the surface of a dried lake bed a little bit. People, you know, who have been part of the contest, like this one guy, Casey Hanmer, who made this sort of critical discovery of this cracked mud pattern, And as a friend of mine, he spent so much time looking at these things that now whenever he's sort of going around, if he sees cracked mud, he's like, it's like a game of Tetris to him. He sees letters everywhere. So yeah, you have that sort of afterimage. But yeah, so we had all that, but no, you know, no one had sort of found letters in the scrolls before. Anyway, I realized it was like super complicated problem and it was going to take a lot of work and there's no way I could do it myself. And Brent said, look, I've got a couple grad students but like they're incredibly busy. So we needed, we needed to get help. And I, I actually put out this very mysterious job description to try to hire someone to work on this project. And I ended up getting actually about 60 applicants. I didn't say what it was. I just said, this is like an Indiana Jones style adventure for a programmer.

Dangerous mission, return not likely.

Yeah, exactly.

Shackleton thing.

Yeah, I promised like, if it works, this will be a story you tell your grandchildren. That was sort of my—

Oh, nice.

My promise. And I think we've lived up to that one. So that's really cool.

Spades. Yeah.

I ended up hiring this great guy, JP, helped basically me put together the contest at the beginning of last year of 2022. And, you know, I guess the rest is history. You know, we launched the contest and It was a heck of a year. We'd set a deadline for ourselves of December 31st, and I was pretty confident it could be solved, but not at all confident that it could be solved in the 9 and a half months that we gave ourselves. And so I was sort of biting my nails the whole year and constantly asking, what can we do to increase our probability of success? And, you know, how do we, is there anything we can do to move the ball forward? Basically.

And so the structure you guys came up with, I think is really interesting because GitHub is to Microsoft. It's like open source versus, you know, the kings of IP and patents and suing you if you infringe. So I think there's an interesting kind of paradigm shift there. And then the way that Vesuvius Challenge works with the kind of incumbent way is institutional funding, apply for a grant from a government or a foundation, and then it has to get channeled through academia. And obviously that has come up with some good for society, but it also can be, you know, a little bit slow and inefficient and politicized. And so I think just this idea of doing a competition open source like you did is fascinating. Yeah. So how did you kind of come up with that and tell us about that?

Yeah, I mean, people think this was a really creative idea for me, like, sort of the most obvious idea because that's what I'd done my whole life. I mean, my whole life I basically worked in kind of online internet communities. I kind of grew up on the internet. I got on the internet in the early '90s and I was a kind of introverted science kid. So by the time I was 15 or 16, most of the people I knew were on the internet. This is much less unusual now than it was then. Then it was really strange. But the people who were on the internet then were not normal people. They were, you know, like obsessive technology people and programmers. And so we had this emerging phenomenon of, of what we now call open source already then, which was, you know, super smart people working on projects together, writing software, sharing their work, learning from each other. And I loved it. And, you know, as a 15-year-old in a small town in Virginia, I could kind of join like these incredible communities online and learn from them and try to contribute. And so that was amazing to me. And then I, you know, I was running GitHub. So like, you know, GitHub basically hosts now a lot of these open source projects. And so it was sort of like, to me, it was like the most obvious thing was why don't we open it up to the world? And, uh, you know, many eyes can make the problem shallower potentially. But there were some serious sort of contest design problems that needed to be solved. So the main issue was that the problem was though Brent and his team had laid this extremely good foundation and had pioneered a lot of key techniques, there was just like an unknown amount of work left. And it turned out to be a pretty decently large amount of work left. And so there's sort of this continuum of collaboration and competition, and the problem was to pick a point along that continuum. So our original idea, what I first pitched to Brent, was we should do a big XPRIZE-style you know, grand, you know, some, just like a grand prize for someone who can like read the whole scroll or something like that. And as I was starting to try to figure out how to put that together and looking deeper into the problem, a friend of mine said, Nat, there's like 4 or 5 PhDs left in this project. This is like not a weekend hack. This is not something a couple of somebody could do in a few evenings. This is like a really hard problem. And I agreed. And so the concern there with doing a grand prize and only a grand prize for people working on this part-time is that, you know, like let's say there's 10 teams and each of them makes, you know, a little bit of progress. Maybe they get 5% or 7% or 15% of the way there. If that's all you do, you're probably going to give up. You know, you're going to feel like your chances of winning are low. And In the case where this is just a contest, you have like no incentive to share your little bits of progress with anyone else. And so we decided that— actually, first I went looking for a solution to this problem. I ended up calling the folks at XPRIZE and talking to people at other kind of online contests, distributed contests, to try to find an answer. And basically, I couldn't find that anyone had solved this. I mean, it's possible that I missed some antecedent for this, which— where someone had already come up with this solution. But we ended up just sort of banging our heads against it and then struck upon this idea, which was to basically have 2 systems of prizes. One would be this gigantic charismatic grand prize for doing something amazing and cracking the problem seriously. And we knew that a grand prize would just excite people. It would have prestige. If you won it, you'd get famous. And if we put a lot of money—

Big check on the stage.

Yeah, big check. Exactly. Yeah. And we could put a lot of money behind it as an incentive too. But then I had this idea of having progress prizes, which would be like a series of smaller prizes that you could win for making little bits of progress towards the goal. conditioned on you open sourcing your work.

Mm-hmm.

And so the idea was that if you were one of those, you know, teams that got 5% of the way there and you thought, well, I can't get any further because I got to go back to work or school, you could release that 5%. And then we would, if you kind of qualified for the progress prize, we would give you an award and recognize you and you'd get some money and you get some dopamine along the way. And everyone would benefit, you know, your little 5% of progress would accrue to everyone.

And, uh, and you open source it after you win the prize, right? Like you definitely, yeah, it's not like as you're going, you can peer over people's shoulders.

Yeah, I mean, you can open source it before and still be eligible, but most people would choose not to because they might want to, it's, it's this blend of competition and, and cooperation. You're in an adversarial relationship for the grand prize. And you're in a collaborative relationship for the progress prize, you know, and so, so most people would choose to kind of wait till after they had won to release their, their progress as open source, but actually not all did. So that was the design and we hoped it would work and it turned out to work like much better than we expected. It was incredibly powerful as a kind of mechanism design for creating this community and creating a ton of progress. that I think happened much faster than it would have otherwise. The alternative might've been to have like an enormous grand prize. Um, like if there was like a $20 million grand prize or something like that, that might've worked, but we didn't have that kind of money. So that was like not an option for us. The other challenge we had is that a lot of the talent, the new skills that were needed to work on this are also the same skills that are needed to work on AI.

Right.

And AI is having this gold rush right now. And so you know, with our little archaeology project, we were competing with, you know, vast riches and commercial success that all these other people could, could go chase for themselves as well. And so the progress prizes created a community and little bits of prestige and progress. And actually what we didn't anticipate is that they were really encouraging. And so the people who won progress prizes early would become like progressively more obsessed with the project. And they would have received this dopaminergic reward for having done something, and then they would want that again. And so that was— in fact, all the people on the grand prize-winning team, all 3 of them, Youssef, Luke, and Julian, had won multiple progress prizes before. And I think maybe if they hadn't, not only would the work not be out there for others to build on, but They might not have been so encouraged.

Yeah. I think that that's one of the striking things about the project. Like, and I do encourage people to go to scrollprize.org to check it out, check out the website. And, um, like right now you have another round of competition going on and we can talk about that more if you'd like. But, you know, one of them is go back and reproduce the results that was, that were done before, but, but more efficiently, more or less something like that.

That's right. Yeah, exactly.

Seems like so much of the kind of hard thinking leadership and, you know, let's say dictatorial control, and it comes into the very careful thinking about the incentives. I'm trying to think of lessons that other people might use to construct their own kind of competitions and rethink how we—

Well, I think, yeah, I mean, maybe the meta thing here is we treated the contestants as our customers basically, and So we didn't want to just succeed, but we wanted them to have like this really good experience and have a high chance of having a good experience. And so I do think a lot of people think with contests or bounties that it's fire and forget. You just sort of put it up there, it's a million bucks, and that's all you have to do. And it was nothing like that at all. I mean, I think that's a very bad way of thinking about it. For example, we knew that it was really hard to get into this. And that people have a lot of other things they could do with their time, especially the people who can make progress on this. And so we actually, before launching the contest, worked really, really hard to produce a website that kind of made it exciting and told the story, and then also had technical tutorials that provided an on-ramp for someone to get the context they needed to start working on it at all. And that was a lot of work. You know, there, there was a lot. for people to learn. And one of the things we realized is like you're reasoning in 3D and that's sort of hard to even describe in text. And so we needed to make like animations. I ended up hiring an animator and we spent a couple months with him making like explanatory animations, which are now all over the Scroll Prize website. And so I do think it, I've built products before and I've run companies before, And this actually felt remarkably similar, even though it was a contest. You know, you have to think, how do I make it so when someone comes to my webpage, they're interested and then they know what to do next. And so, yeah, contests are a really high leverage tool, but you know, you can't be, they don't give you a reason that you can be lazy and not think about all of these things.

Right. It's not all about the money. And it's like that Patrick Collison speaking in an interview recently about how You know, beauty's important in design because it, among other things, it shows that the people in charge really care. And that's really motivating for a customer to say nothing of a, of a contestant. Yeah.

Yeah. We wanted to speak to them too. Like, we wanted to write the text on the webpage. So it was clear that the people on the other side of the contest were like them, were technical and also excited by this sort of cross-domain project. So anyway, yeah, I mean, the big couple decisions we made along the way were after figuring out this basic design for progress prizes were 2. The first is we realized that there was this one step in the process that was just like really, really, really hard. And the contestants a few months in were making no, we're making no progress on it. And just to describe how it works, you know, the first thing you do is you scan a scroll and you get this basically this 3D image of it. The next thing you have to do actually is to— the scroll is like a wound spiral of papyrus. They can be really long. They can be like 15 meters long. And, you know, then they get rolled up and the writing is on the inside. And so, and then unfortunately when Vesuvius erupted, the hot mud really distorted them. They're super messed up. The heat caused the papyrus to blister in places or to delaminate. were to sort of fuse together some of the layers, and then they got bent as well. And so it's not like this perfect spiral that you can just easily unroll with software. What we have to do instead is to like trace the surface of the papyrus inside the scroll. And unfortunately, we don't have good algorithms for doing that yet. And so that process ended up being really, really manual. There's some tools that Brent's team had built before, this guy Seth Parker, And those were a good start, but those tools also needed to be greatly improved. And so we, what we ended up doing basically was, and this was a controversial decision inside the team when we did it, is we ended up hiring a team of people full-time to just trace papyrus all day on these giant monitors.

Yeah. This is the segmentation, right?

Yeah. This is, we have the term we use for this is segmentation. And this is, it was sort of like, it's not in the spirit of a contest, right? Like, wait, hold on a second. You're, You're supposed to be the referee. Now you're running out on the field and kicking the ball a little bit. Like, what are you doing? But I was just desperate to make sure we were successful by the deadline. And this seemed like essential. It was just not going to happen otherwise. And it turned out to be really, really important. And we ended up spending actually a lot of money on this, hundreds of thousands of dollars on paying people to do this meticulous work. We hired 15 people initially. It turned out to be so difficult to do that. Really, most of them were not good at it. And we ended up with a full-time 3-person team of segmenters who were awesome. And, you know, David and Ben and Conrad, and they worked, you know, till their eyes hurt, honestly, on this. We were in a race because we knew in order to meet the grand prize threshold, we needed a certain amount of papyrus segmented quickly. And so that was sort of one key thing. And then the other key thing we decided was that We had this grand prize, which was for finding 4 passages inside of a scroll of sufficient length, which we picked 140 characters as that length. And then we realized that was a pretty high bar. We needed some intermediate milestone. And so we decided to have the first letters prize for finding the first 10 letters inside the scroll in a certain, you know, 4-square-centimeter region. And I think that was early summer when we put that kind of progress prize out there. And that was critical. I mean, that's when the whole sort of avalanche started to run essentially, because the first thing that happened was that Casey Hanmer found this cracked mud pattern and he found a couple of letters. He didn't find 10, but he found a couple of letters in the scroll and he decided to share that with the community. And that kicked off a race and people start— He found those letters by looking just manually with his own eyes. He called it direct persistent inspection, just looking at it a lot in the X-rays.

And taking time off from his Nat Friedman-backed space-related energy startup, right? Like he's not just doing this in a basement.

No, he's the father of 3 and is running a startup. It's also very hard. And I don't know how he did it, honestly, but He's amazing. So he found this, he put it out there, and then subsequently Luke Ferritor, who was at the time a 21-year-old University of Nebraska undergrad, saw Casey's pattern that he'd found and said, I can train an AI model on this to find more instances of this throughout the segment, you know, the flattened segments that we produced with our segmentation team. And absolutely miraculously, he found the first word that had been seen in 2,000 years from Inside an unopened scroll, he was the first person to see that handwriting in millennia. And then even more, you know, incredibly, it was like a cool word. It was this word "porphorus," which I hope I'm saying it right. But yeah, close enough. Okay, thanks. It means purple. And you know, it could have been like a super normal word, like although or something. But you know, it was purple, and that was so mysterious to us. So that was the first.

The color of royalty. Yeah.

Yeah, exactly.

So we're going to have Dr. Brent Seales on here in a little while on the show. Everybody stay tuned, coming soon. But so we want to get more into the technical details, but we could zoom out a little bit. You know, one of the things that I think is so exciting about this project is kind of this meeting of humanities and great, like great knowledge from the past, but like activated with a whole lot of high agency determined people. Dr. Seals, very determined person. I mean, he's been working on this for 20 years.

Definitely.

Yourself and Daniel and all the people on the, on the Prize team. Like I think of what Paul Graham says about, you know, being relentlessly resourceful as like a key for a startup founder. I wonder if this is kind of like, you know, a project like this just has to filter for that kind of person and kind of train that in them.

Yeah.

Which to me is just this classical spirit of competition that you see in the Olympic Games. And so if I could ask, like, what are some of the— because when I think of the great men that we study on the Cost of Glory podcast, somebody like Julius Caesar, okay, he's, he's very ambitious, but he's also interested in technical matters. He rewrites the calendar. He's a great, you know, kind of field engineer in warfare. He's a poet, he's a writer, he's this kind of Renaissance man in a lot of ways. And it's like Andrew Roberts was saying on a show recently, like Napoleon would've been a tech founder today. I think you'd probably say the same thing about Julius Caesar. Like, what are, what are some of the qualities that you would like to see this project inspire in people today that you'd like more of?

That's a good question. Well, I think, yeah, it would be great if more people did crazy projects. that might fail. And I think this is, this was, at least in my world, this was like a really unusual thing to work on. And I got a lot of funding from friends who just thought it was crazy and they hoped it would work. And, and that was key. And so, yeah, I do think people should do more things that just seem, seem strange, but that like they have a lot of personal enthusiasm about and should be slightly more impulsive about that. I mean, the story for me psychologically of doing this is is that I kind of got into it impulsively. And then once it was out there and I had kind of my name on it, I really wanted to do whatever it took to make sure it succeeded. And so I had this relatively low threshold for starting this project. And then once I was in it, I was, in a way, I was stuck. I had trapped myself because it was going to be really embarrassing if this didn't work. If it didn't work, people would say, of course it didn't work. This was like an insane idea. And people have been trying for hundreds of years to open these scrolls and you had horrible hubris. getting involved at all. So I do think that's part of it is just like you should do something like that. And then I think the other thing is there's a lot of areas of the world that are just not picked over, you know, by people at least of like my tech background. There's, so I would say like it's probably a good idea to look in the just less obvious areas to some extent. I mean, bringing software to archaeology, maybe it's, obvious to a lot of other people, but at least in my world, it wasn't. And so I think that's, you know, there's $100 bills lying on the sidewalk there. And then, yeah, just do whatever it takes, you know, do whatever it takes and do it quickly. I think that's really important too. So, but yeah, I hope like this will be good if it inspires other people to do more crazy projects. Not because, you know, you can tell like the story of super high impact, you know, the, the kind of classic thing for a tech founder to work on, you know, after they have a slight amount of free time is something in health, you know, cure diseases, et cetera. I think that's very important, obviously, but I would like to see more kind of surprising projects from people and things that are slightly stranger. and only makes sense in retrospect. I think that's what you want to do. You want to do things that make sense in retrospect whether or not they make sense at the time. And the ones that make sense at the time don't always make sense in retrospect. And so that would be my— if this is inspiring to people, from my world, I hope it inspires them in that way to do something that seems like Strange, doesn't make sense now, but if it works, will make a lot of sense in retrospect.

This project could unearth another Renaissance worth of, of ancient texts. Would, do you think people will be more or less receptive to such a discovery than they were in the Renaissance today?

I don't know. I think, I think it's, I don't really know exactly. I guess they were pretty receptive during the Renaissance.

Maybe we overestimate it with the rosy-colored lenses looking back at the past.

Yeah, my guess is they'll be slightly less receptive now just because we have so many sources of new information every day and there is a sense of freshness from a lot of that information on Twitter or whatever. And so we're not as bored, is my guess, as we might have been in a lower information era. However, I think we will be very interested in it now. I do think there's a way in which people, the world is sort of changing a lot and society has fewer kind of bedrock spiritual values that, you know, that we really believe in and that we all have in common. And so there's a way in which I think people, some people at least are looking to the past and the ancient world. for comfort and guidance and maybe other ways of thinking and living in a search for something that's not a fad diet or ephemeral. Yeah.

And so many of these texts that we've already discovered really deal with some of the hardest questions of life. One of the best passages of Philodemus is on how do you like meditate more on death so you live a better life. And so when death comes and he's, you know, he's living this period where Everybody important is dying violently. Actually, you know, Julius Caesar dies violently. Pompey dies violently. Cicero dies violently. All these guys in the, in the period that, you know, are Philodemus's peers, his clients, his patron Piso, I don't think died violently, but, you know, maybe because he took Philodemus's advice. But yeah, they, I think, you know, some of these texts really, you know, I would love to discover dialogues of Aristotle. I would love to discover letters of Julius Caesar. I think we really could, but the villa's secrets, I mean, and there's, of course, there's great art and we're going to do more of this content on the show to talk about why this is so cool. Even what we've already discovered to me is very inspiring because it tells us about these people that are some of the most fascinating figures in Roman history. And like, you know, a guy like Philodemus is trying to help those great leaders connect with some of the profound questions and kind of, I don't know, the values that are really going to make their society and their own lives better. Like, that's, I really think that's his mission. And so I find it like, it's already an inspiring story to me as I'm getting into it. I'm kind of going down the rabbit hole with you guys as a non-technical person.

Yeah. There's a way in which, you know, it seems sort of random that what we've got is a bunch of Epicurean stuff, but I was wondering, like, maybe Epicureanism is sort of what we need right now. And Maybe there's like, what if actually there's some amount of meaning that we can find in Epicureanism? Stoicism is super famous and a lot of people out here in California are into Stoicism, but I like the idea of just exploring this alternative frame of Epicureanism too and what makes life worth living from an Epicurean point of view.

Yeah. And in times of political turmoil, like they lived in, in Philodemus's day, maybe especially so. Well, Nat, thank you for taking the time. Where can people find you before you go?

Oh, sure. Well, you can read more about the Vesuvius Challenge at scrollprize.org, where we've read 5% of one scroll. We have hundreds of full scrolls left to read, and we're working really hard on doing that. And I hope we do uncover a dialogue of Aristotle or a new history or something, you know, incredible. But it'll, whatever happens, it'll That'd be great. And then, geez, where do you find me? I'm— my website is nat.org, so you can read a little bit more about me there, or I'm on Twitter as Nat Friedman.

Great. Well, thanks again. We'll see you soon.

Thank you for having me.