Luís and João Batalha: Fermat's Library and the Art of Studying Papers | Lex Fridman Podcast #209

The following is a conversation with Luis and Joao Batala, brothers and co founders of Fermat's Library, which is an incredible platform for annotating papers as they write. On the Fermat's Library website, Justice Pierre de Fermat scribbled his famous Last Theorem in the margins. Professional scientists, academics and citizen scientists can annotate equations, figures, ideas and write in the margins for Mars. Library is also a really good Twitter account to follow. I highly recommend it. They post little visual factoids and explorations that reveal the beauty of mathematics. I love it. Quick mention of our sponsors Skiff, SimpliSafe, Indeed, Netsuite and Four Sigmatic. Check them out in the description to support this podcast. As a side note, let me say a few words about the dissemination of scientific ideas. I believe that all scientific articles should be freely accessible to the public. They currently are not. In one analysis I saw more than 70% of published research articles are behind a paywall. In case you don't know, the funders of the research, whether that's government or industry, aren't the ones putting up the paywall. The journals are the ones putting up the paywall while using unpaid labor from researchers for the peer review process. Where is all that money from the paywall going in this digital age? The costs here should be minimal. This cost can easily be covered through donation, advertisement, or public funding of science. The benefit versus the cost of all papers being free to read is obvious. And the fact that they're not free goes against everything science should stand for, which is the free dissemination of ideas that educate and and inspire. Science cannot be a gated institution. The more people can freely learn and collaborate on ideas, the more problems we can solve in the world together, and the faster we can drive old ideas out and bring new, better ideas in. Science is beautiful and powerful, and its dissemination in this digital age should be free. This is the Lex Friedman podcast, and here's my conversation with Luis and and Joao Batalla. Luis, you suggested an interesting idea. Imagine if most papers had a backstory section the same way that they have an abstract. So knowing more about how the authors ended up working on a paper can be extremely insightful. And then you went on to give a backstory for the Feynman QED paper. This is all in a tweet, by the way. We're doing tweet analysis today. How much of the human backstory do you think is important in understanding the idea itself that's presented in the paper or in general?

I think this gives way more context to the Work of scientists. I think a lot of people have this almost kind of romantic misconception that the way a lot of scientists work is almost as the sum of eureka moments, where all of a sudden they sit down and start writing two papers in a row. And the papers are usually isolated. And when you actually look at it, the papers are chapters of a way more complex story. And the Feynman QED paper is a good example. So Feynman was actually going through a pretty dark phase before writing that paper. He lost enthusiasm with physics and doing physics problems. And there was one time when he was in the cafeteria of Cornell, and he saw a guy that was throwing plates in the air, and he noticed that when the plate was in the air, there were two movements there. The plate was wobbling. But he also noticed that the Cornell symbol was rotating, and he was able to figure out the equations of motions, the equations of motions of those plates. And that led him to kind of think a little bit about electron orbits in relativity, which led to the paper about quantum electrodynamics. So that kind of reignited his interest in physics and ended up publishing the paper that led to his Nobel Prize, basically. And I think there are a lot of really interesting backstories about papers that readers never get to know. For instance, we did a couple of months ago an AMA around a paper, a pretty famous paper, the Gans paper, with Ian Goodfellow. And so we did a AMA where everyone could ask questions about the paper. And Ian was responding to those questions, and he was also telling the story of how he got the idea for that paper in a bar. So there was also an interesting backstory. I also read a book by Cedric Villany. Cedric Villany is this mathematician, the Fields medalist. And in his book, he tries to explain how he got from, like, a PhD student to the Fields Medal. And he tries to be as descriptive as possible about every single step, how he got to the Fields Medal. And it's interesting also to see just the amount of random interactions and discussions with other researchers, sometimes over coffee, and how it led to fundamental breakthroughs and some of his most important papers. So I think it's super interesting to have that context of the backstory.

Well, the Ian Goodfellow story is kind of interesting, and perhaps that's true for Feynman as well. I don't know if it's romanticizing the thing, but it seems like just a few little insights and a little bit of work does most of the leap required. Do you have a sense that for a lot of the stuff You've looked at just looking back through history, it wasn't necessarily the grind of like Andrew Wiles or the Fromaz Last theorem, for example. It was more like a brilliant moment of insight. In fact, Ian Goodfellow has a kind of sadness to him almost in that at that time in machine learning, like at that time, especially in for Gans, you could code something up really quickly on a single machine and almost do the invention, go from idea to experimental validation in like a single night. A single person could do it. And now there's kind of a sadness that a lot of the breakthroughs you might have in machine learning kind of require large scale experiments. So it was almost like the early days. So I wonder how many low hanging fruit there are in science and mathematics and even engineering where it's like you could do that little experiment quickly. Like you have an insight in a bar. Why is it always a bar? But you have an insight at a bar and then just implement and the world changes.

It's a good point. I think it also depends a lot on the maturity of the field. When you look at a field like mathematics, it's a pretty mature field, a field like machine learning, it's growing pretty fast and it's actually pretty interesting. I looked up the number of new papers on Arxiv with the keyword machine learning and like 50% of those papers have been published in the last 12 months. So you can see just the sense 5,0 or 5,0,50%. So you can see the magnitude of growth in that field. And so I think as fields mature, those types of moments I think naturally are less frequent. It's just a consequence of that.

The other point that is interesting about the backstory is that it can really make it more memorable in a way. And by making it more memorable it kind of sediments the knowledge more in your mind. I remember also reading the sort of the backstory to, to Dijkstra's shortest path algorithm, where he came up with it essentially while he was sitting down at a coffee shop in Amsterdam. And he came up with that algorithm over 20 minutes. And one interesting aspect is he didn't have any pen or paper at the time. And so he had to do it all in his mind. And so there's only so much complexity that he can handle if you're just thinking about it in your mind. And that like when you think about the simplicity of Dijkstra's shortest path finding algorithm, it's, you know, knowing that backstory helps sediment that algorithm in your mind so that you don't forget about it as easily.

It might be from you that I saw a meme about Dijkstra. It's like he's trying to solve it and he comes up with some kind of random path. And then it's like, my parents aren't home. And then he does. He figures out the algorithm for the shortest path strike through words to convey memes. But it's hilarious. I don't know if it's in post that we construct stories that romanticize it. Apparently with Newton, there was no apple. Especially when you're working on problems that have a physical manifestation or a visual manifestation, it feels like the world could be an inspiration to you. So it doesn't have to be completely on paper. Like you could be sitting at a bar and all of a sudden see something and a pattern will spark another pattern and you can visualize it and rethink a problem in a particular way. Of course, you can also load the math that you have on paper and always carry that with you. So when you show up to the bar, some little inspiration could be the thing that changes it. Is there any other people almost on the human side, whether it's physics with Feynman, Dirac, Einstein, or computer science touring anybody else? Any backstories that you remember that jump out? Because I'm also referring to not necessarily these stories where something magical happens, but these are personalities. They have big egos. Some of them are super friendly, some of them are like self obsessed, Some of them have anger issues, some of them. How do I describe Feynman? But he appears to have appreciation of the beautiful in all its forms. He has a wit and a cleverness and a humor about him. So does that come into play in terms of the construction of the science?

Well, I think you brought up Newton. Newton is a good example also to think about his backstory, because there's a certain backstory of Newton that people always talk about, but then there's a whole another aspect of him that is also a big part of the person that he was. But he was really into alchemy and that he spent a lot of time thinking about that and writing about it. And he took it very seriously. He was really into Bible interpretation, trying to predict things based on the Bible. And so there's also a whole backstory then. And of course you need to look at it in the context and the time when Newton lived. But it adds to his personality. And it's important to also understand those aspects that maybe, you know, people are not as proud to teach to little kids. But it's important. It was part of who he was. And maybe without those, who knows what he would have done otherwise. So.

Well, the cool thing about alchemy, I don't know how it was viewed at the time, but it almost to me symbolizes dreaming of the impossible. Like, most of the breakthrough ideas kind of seem impossible until they're actually done. It's like achieving human flight. It's not completely obvious to me that alchemy is impossible, or like putting myself in the mindset of the time and perhaps even still everything that, you know, some of the most incredible breakthroughs would seem impossible. And I wonder the value of believing, almost like focusing and dreaming of the impossible, such that it is actually is possible in your mind. And that in itself manifests whether the accomplishing that goal or making progress in some unexpected direction. So alchemy almost symbolizes that for me.

I distinctly remember having the same thought of thinking, you know, when I learned about atoms and that they have protons and electrons, I was like, okay, to make gold, you just take whatever has an atomic weight below it and then shove another proton in there, and then you have a bunch of gold. So, like, why don't people do that? It seemed like conceptually it's like, you know, this sounds feasible. You might be able to do it.

And you can actually. It's just very, very expensive.

Yeah, yeah, exactly, exactly. So in a sense, we do have alchemy, and maybe even back then it wasn't as crazy that he was so into it, but people just don't like to talk about that as much. Yeah, but Newton in general was a very interesting fellow.

Anybody else come to mind in terms of people that inspire you, in terms of people that you just are happy that they have once or still exist on this Earth?

I think, I mean, Freeman Dyson, for me.

Yeah, Freeman Dyson was. I've had a chance to actually exchange a couple of emails with him. He was probably one of the most humble scientists that I've ever met, and that had a big impact on me. We were actually trying to convince him to annotate a paper on Fermat's library. And I sent him an email asking him if you could annotate a paper. And his response was something like, I have very limited knowledge. I just know a couple of things about certain fields. I'm not sure if I'm qualified to do that. That was his first response. And this was someone that should have won a Nobel Prize and worked on a bunch of different fields, did some really, really great, great work and then just the interactions that I had with him. Every time I asked him a couple of questions about his papers and he always responded saying, I'm not here to answer your questions, I just want to open more questions. And so that had a big impact on me. It was like just an example of an extremely humble yet accomplished scientists. And Feynman was also a big, big inspiration in the sense that he was able to be, you know, again, extremely talented and, and scientist, but at the same time, socially he was able to. To. He was also really smart from a social perspective and he was able to interact with people. He was also a really good teacher and was also to. Did an awesome work in terms of explaining physics to, to the masses and motivating and getting people interested in physics. And that for me was, was also a big inspiration.

Yeah, I like the childlike curiosity of some of those folks. Like you mentioned, Freeman, I've. Daniel Kahneman. I got a chance to meet and interact with some, some of these truly special scientists. What makes them special is that even in older age, there are still like, there's still that fire of childlike curiosity that burns. And some of that is like not taking yourself so seriously that you think you've figured it all out, but almost like thinking that you don't know much of it. And that's like step one in having a great conversation or collaboration or exploring a scientific question. It's cool how the very thing that probably earned people the Nobel Prize or the or work that's seminal in some way is the very thing that still burns even after they've won the prize. It's cool to see. And they're rare humans, it seems.

And to that point, I remember the last email that I sent to Freeman Dyson was in his last birthday. He was really into number theory and primes. So what I did is I took a photo of him picture and then I turned that into a giant prime number. So I converted the picture into a bunch of one and eights, and then I moved some numbers around until it was a prime. And then I sent him that.

Also the, the visual, like it still looked like the picture and it was made up of a prime. That's tricky to do. It's hard to do.

It looks harder than it actually is. So the way you do it is like you convert the darker regions into eights and the lighter regions in ones, and then there's.

Then just keep flipping, flipping.

Yeah, but there's like some primality tests that are cheaper from a computational standpoint. But what it tells you is it excludes numbers that are not prime. Then you end up with a set of numbers that you don't know if they are prime or not. And then you run the full primality test on that. So you just have to keep iterating on that. And it's funny because when you got the picture, he was like, how did you do that? It was super curious to. And then we got into the details and again, it was already 90, I think 92 or something. And that curiosity was still there. So you could really see that in some of these scientists.

So could we talk about Fermat's library?

Yeah, absolutely.

What is it? What's the main goal? What's the dream?

It is a platform for annotating papers in its essence. And so academic papers can be one of the densest forms of content out there and generally pretty hard to understand at times. And the idea is that you can make them more accessible and easier to understand by adding these rich annotations to the side. Right. And so we can just imagine a PDF view on your browser and then you have annotations on each side, and then when you click on them, a sidebar expands and then you have annotations that support latex and markdown. And so the idea is that you can, say, explain a tougher part of a paper where there's a step that is not completely obvious, or you can add more context to it. And then over time, papers can become easier and easier to understand and can evolve in a way. But it really came from myself, Luige and two other friends. We've had this long running habit of kind of running a journal club amongst us. We come from different backgrounds. I studied cs, we studied physics, and so we'd read papers and present them to each other, and then we tried to bring some of that online, and that's when we decided to build Fermat's library. Then over time, it kind of grew into something with a broader goal. And really what we're trying to do is trying to help move science in the right direction. That's really the ultimate goal and where we want to take it now.

So there's a lot to be said. So, first of all, for people who haven't seen it, the interface is exceptionally well done. Execution is really important here.

Absolutely.

The other thing, just to mention, for a large number of people, apparently, which is new to me, don't know what latex is, so it's spelled like latex, so be careful Googling it if you haven't before, it's Sorry, I don't even know the correct terminology. It's a typesetting language where it's. You're basically program writing a program that then generates something that looks, from a typography perspective, beautiful.

Absolutely.

And so a lot of academics use it to write papers. I think there's like a bunch of communities that use it to write papers. I would say it's mathematics, physics, computer science.

Yeah, that's. Yeah.

Because I'm collaborating currently on a paper with two neuroscientists from Stanford, and they don't know that. So I'm using Microsoft Word and Mendeley and like, all of those kinds of things. And I'm being very Zen, like, about. About the whole process. But it's fascinating. It's a little heartbreaking, actually, because it actually, it's. It's funny to say, but. And we'll talk about open Science. Actually, the bigger mission behind from ours Library is like, really opening up the world of science to everybody is these silly two facts of, like, one community uses Latex and another uses Word. Is actually a barrier between them that's like. It's like boring and practical in a sense, but it makes it very difficult.

To collaborate just on that. I think there are some people that should have received a Nobel Prize but will never get it, and I think one of those is Donald Knuth. Because of tech and latex, then? Because it had a huge impact in terms of just making it easier for researchers to put their content out there, making it uniform as much as possible.

Oh, you mean like a Nobel Peace Prize?

Maybe a Nobel Peace Prize. Maybe a Nobel Prize. Freestyles. Yeah, I think so.

I mean, he, at a very young age, got the Turing Award for his work in algorithms and so on. So like an incredibly. Like, when I think it's in. It might be even the 60s, but I think it's the 70s that. So when he was really young and then he went on to do, like, incredible work with his book and. Yeah. With tech that people don't know.

And going back just one, on the reason why we ended up. Because I think this is interesting, the reason why we ended up using the name Fermat's Library. This was because of Fermat's Last Theorem. And Fermat's Last Theorem is actually a funny story. So Pierre de Fermat, he was a lawyer and he wrote on a book that he had a solution to Fermat's Last Theorem, but that didn't fit the margin of that book. And so Fermat's last theorem basically states that there's no solution. If you have integers A, B and C, there's no solution to A to the power of N +B to the power power of n equals to C to the power of N if N is bigger than two. So there's no solutions. And he said that. And that problem remained open for almost 300 years, I believe. And a lot of the most famous mathematicians tried to tackle that problem. No one was able to figure that out until Andrew Wiles, I think, was in the 90s, was able to publish the solution, which was, I believe, almost 300 pages long. And so it's kind of an anecdote that there's a lot of knowledge and insights that can be trapped in the margins, and there's a lot of potential energy that you can release if you actually spend some time trying to digest that. And that was the origin story for the name.

Yet you can share the contents of the margins with the world. That could inspire a solution or a communication that then leads to a solution.

And if you think about papers, papers are, as Joan was saying, probably one of the densest pieces of text that any human can read. And you have these researchers, like some of the brightest minds in these fields, working on new discoveries and publishing this work on journals that are imposing them restrictions in terms of the number of pages that they can have to explain a new scientific breakthrough. So at the end of the day, papers are not optimized for clarity and for a proper explanation of that content because there are so many restrictions. So there's, as I mentioned, there's a lot of potential energy that can be freed if you actually try to digest a lot of the contents of papers.

Can you explain some of the other things? So margins, Librarian, Journal club.

So Journal Club is what a lot of people know us for, where every week we release an annotated paper in all sorts of different fields. Physics, cs, math. Margins is kind of the same software that we use to run the Journal Club and to host the annotations, but we've made that available for free to anybody that wants to use it. And so folks use it at universities and for running journal clubs. And so we've just made that freely available. And then Librarian is a browser extension that we developed that is sort of an overlay on top of Archive. So it's about bringing some of the same functionality around, comments, plus adding some extra niceties to archive, like being able to very easily extract the references of a paper that you're looking at, or being able to extract The Bibtex in order to cite that paper yourself. So it's an overlay on top of Archive.

The idea is that you can have that commenting interface without having to leave Archive.

It's kind of incredible. I didn't know about it and once I learned of it it's like, holy shit, why isn't it mo more popular? Given how popular Archive is, like, everybody should be using it. Archive sucks in terms of its interface, or let me rephrase that, it's limited in terms of its interface.

Archive is a pretty incredible project. Right? And it is, in a way, it's. The growth has been completely linear over time. If you look at number of papers published on Arxiv, it's pretty much a straight line for the past 20 years. Especially if you're coming from a startup background and then you are trying to do archive, you'd probably try all sorts of growth acts and try to then maybe have paid features and things like that. And that would maybe ruin it. And so there's a subtle balance there and I don't know what aspects you can change about it.

Yeah, for some tools in science, it just takes time for them to grow. Arxiv just turned 30, I believe. And for people that don't know, Arxiv is this kind of online repository where people put preprints, which are versions of the papers before they actually make it to journals.

Arkansas xiv.

Exactly.

For people who don't know. And it's actually a really vibrant place to publish your papers. And in the aforementioned communities of mathematics and computer science.

It started with mathematics and physics and then over the last 30 years it evolved and now actually computer science, now it's a more popular category than physics and math on Arxiv.

And there's also, which I don't know very much about, like biology. Medical version of that.

Bioarchive. Yeah, bioarchive, more recent. It's interesting because if you look at these platforms for preprints, they actually play a super important role because if you look at a category like math for some papers in math, it might take close to three years after you click upload paper on the journal website and the paper gets published on the website of the journal. So this is literally the longest upload period on the Internet. And during those three years, like it's, you know, that content is just, you know, locked. And so that's why it's so important for people to have websites like Archive so that you can share that before it goes to the journal with the rest of the world that was actually on archive that Perelman published the three papers that led to the proof of the Poincare conjecture. And then you have other fields like machine learning for instance, where the field is evolving at such a high rate that people don't even wait before the papers go to journals, before they start working on top of those papers. So they publish them on Arxiv, then other people see them, they start working on that. And Arxiv did a really good job at building that core platform to host papers. But I think there's a really, really big opportunity in building more features on top of that platform apart from just hosting papers. So collaboration annotations and like having other things apart from papers like code and other things. Because for instance, in the field like machine learning, there's a really big, you know, as I mentioned, people start working on top of preprints and they are assuming that that preprint is correct. But you really need a way, for instance to maybe it's not peer review, but distinguish what is good work from bad work on Arxiv. How do you do that? So like a commenting interface, like librarian, it's useful for that so that you can distinguish that in a field that is growing so fast as machine learning. And then you have platforms that focus for instance on just biology. Bioarchive is a good example. Bioarchive is also super interesting because there's actually an interesting experiment that was run in the 60s. So in the 60s the NIH supported this experiment called Information Exchange Group, which at the time was a way for researchers to share biology preprints via mail or using libraries. And that project in the 1960s got canceled six years after it started. And it was due to intense pressure from the journals to kill that project because they were fearing competition from the preprints for the journal industry. Creek was also one of the famous scientists that opposed to the Information Exchange group. And it's interesting because right now if you analyze the number of biology papers that appear first as preprints, it's only 2% of the papers. And this was almost 50 years after that first experiment. So you can see like that pressure from the journals to cancel that initial version of a pre print repo had a tremendous impact on the number of papers that are showing up in biology as preprints. So it delayed a lot that revolution. And but now platforms like bioRxiv are doing that work. But there's still a lot of room for growth there. And I think it's super important because those are the papers that are open that everyone can read.

Okay, but if we just look at the entire process of science as a big system, can we just talk about how it can be revolutionized? So you have an idea, depending on the field, you want to make that idea concrete. You want to run a few experiments in computer science, there might be some code, there'd be a data set for, you know, some of the more sort of biology, psychology, you might be collecting the data set. That's called, you know, a study. Right. So that's part of that, that's part of the methodology. And so you are putting all that into a paper form and then you have some results and then you, you submit that to a place for review through the peer review process. And there's a process where. How would you summarize the peer review process? But it's really just like a handful of people look over your paper and comment and based on that decide whether your paper is good or not. So there's a whole broken nature to it. At the same time, I love the peer review process when I buy stuff on Amazon, but like, for like the commenting system, whatever that is. So, okay, so there's a bunch of possibilities for revolutions there. And then there's the other side, which is the collaborative aspect of the science, which is people annotating, people commenting, sort of the low effort collaboration, which is a comment. Sometimes, as you've talked about, a comment can change everything. But you know, or a higher effort collaboration, like more like maybe annotations or even like contributing to the paper. You can think of like collaborative updating of the paper over time. So there's all these possibilities for doing things better than they've been done. Can we talk about some ideas in this space? Some ideas that you're working on, some ideas that you're not yet working on but should be revolutionized because it does seem that Arxiv and like Open Review, for example, are like the Craigslist of science. Yeah, okay. I'm very grateful that we have it, but it just feels like it's like 10 to 20 years. Like it doesn't feel like that's a feature. The simplicity of it is a feature. It feels like it's a bug. But then again, the pushback there is. Wikipedia has the same kind of simplicity to it and it seems to work exceptionally well in the crowdsourcing aspect of it. So sorry, there's a bunch of stuff going on the table. Let's just pick random things that we can talk about.

Wikipedia, you know, for me it's the cosmological constant of the Internet. It's. I think we are lucky to live in the parallel universe where Wikipedia exists.

Yes.

Because if, if someone had pitched me Wikipedia like a publicly edited encyclopedia like a couple of years ago, like it would be. I don't know how many people would have said that that would have survived.

I mean it makes almost no sense. It's like having a Google Doc that everybody on the Internet can edit and like that will be like the most reliable source for, for knowledge.

And I don't know how many, but hundreds of thousands of topics.

Yeah, it's insane.

It's insane. And like you have. And then you have users, like there's one, a single user that edited one third of the articles on Wikipedia. So you have these really, really big power users that are a substantial part of what makes Wikipedia successful. And so no one would have ever imagined that could happen. And so that's one thing. I completely agree with what you just said. I also.

Sorry to interrupt briefly. Maybe let's inject that into the discussion of everything else. I also believe, I've seen that with Stack Overflow that one individual or a small collection of individuals contribute or revolutionize most of the community. Like if you create a really powerful system for archive or open review and made it really easy and compelling and exciting for one person who is a 10X contributor to do their thing, that's going to change everything. It seems like that was the mechanism that changed everything for Wikipedia and that's the mechanism that changed everything for Stack Overflow is gamifying or making it exciting or just making it fun or pleasant or fulfilling in some way. For those people who are insane enough to like answer thousands of questions or write thousands of factoids and like research them and check them, all those kinds of things or read thousands of papers.

Yeah, no, Stack Overflow is another great example of that. And those are both two incredibly productive communities that generate a ton of value and capture almost none of it. In a way it's almost like counter. It's very counterintuitive that people. That these communities would exist and thrive. And it's really hard to. There aren't that many communities like that.

So how do we do that for science? Do you have ideas there? What are the biggest problems that you see? You're working on some of them just.

On that there are a couple of really interesting experiments that people are running. An example would be like the polymath projects. So this is kind of a social experiment that was created by Tim Gowers, fields medalist and his idea was to try to prove that is it possible to do mathematics in a massively collaborative way on the Internet. So he decided to pick a couple of problems and test that. And they found out that it actually, it is possible for specific types of problems, namely problems that you're able to break down in little pieces and go step by step. You might need, as with open source, you might need people that are just kind of reorganizing the house every once in a while. And then people throw a bunch of ideas and then you make some progress, then you reorganize, you reframe the problem, you go step by step. But they were actually able to prove that it is possible to collaborate online and do progress in terms of mathematics. And so I'm confident that there are other avenues that could be explored here.

Can we talk about peer review, for example?

Absolutely. I think in terms of the peer review, I think it's important to look at the bigger picture here of what the scientific publishing ecosystem looks like, because for me, there are a lot of things that are wrong about that entire process. So if you look at what publishing means in a traditional journal, you have journals that pay authors for their articles and then they might pay reviewers to review those articles and finally they pay people to or distributors to distribute the content. In the scientific publishing world, you have scientists that are usually backed by government grants that are giving away their work for free in the form of papers. And then you have other scientists that are reviewing their work. This process is known as the peer review process, again for free. And then finally we have government backed universities and libraries that are buying back all that work so that other scientists can read. So this is, for me, it's bizarre. You have the government that is funding the research, is paying the salaries of the scientists, it's paying the salaries of the reviewers, and it's buying back all that product of their work again. And I think the problem with this system, and it's why it's so difficult to break this suboptimal equilibrium, is because of the way academia works right now and the way you can progress in your academic life. And so in a lot of fields, the competition in academia is really insane. So you have hundreds of PhD students, they are trying to get to a professor position and it's hyper competitive. And the only way for you to get there is if you publish papers, ideally in journals with a high impact.

Factor in computer science. Often conferences are also very prestigious or actually more prestigious than journals now.

So interesting.

So that's the one discipline where, I mean that has to do with the thing we've discussed in terms of how quickly the field turns around. But like NeurIPS CVPR, those conferences are more prestigious or at the very least as prestigious as the journals. But doesn't matter. The process is what it is.

And so for people that don't know, the impact factor of a journal is basically the average number of citations that a paper would get if it gets published on that journal. But so you can really think that the problem with the impact factor is that it's a way to turn papers into accounting units. And let me unpack this because the impact factor is almost like a nobility title because papers are born with impact even before anyone reads them. So the researchers, they don't have the incentive to care about if this paper is going to have a long term impact on the world. What they care, their end goal is the paper to get published so that they get that value up front. So for me that is one of the problems of that and that really creates a tyranny of metrics because at the end of the day, if you are a dean, what do you want to hire is like people, researchers that publish papers on journals with high impact factors because that will increase the ranking of your university and will allow you to charge more for tuition, so on and so forth, especially when you are in super competitive areas, people will try to gamify that system and misconduct starts showing up. There's a really interesting book on this topic called Gaming the Metrics. It's a book by a researcher called Mario Biagioli. It goes a lot into how the impact factor and metrics affect science negatively. And it's interesting to think especially in terms of citations. If you look at the early work of looking at citations, there was a lot of work that was done by a guy called Eugene Garfield and this guy, the early work in terms of citation, they wanted to use citations as from the descriptive point of view. So what they wanted to create was a map. And that map would create a visual representation of influence. So citations would be links between papers and ideally what they would show they would represent is that you read someone else's paper and it had an impact on your research. They weren't supposed to be counted.

I think this inspired Larry and Sergey's.

Exactly right for Google. Exactly. I think they even mentioned that. But what happens is like as you start counting citations, you create a market. And the same way like, and this was the work of Eugene Garfield, was a big inspiration for Larry and Sergey and for the PageRank algorithm that led to the creation of Google and they even recognized that. And if you think about it, it's like the same way there's a gigantic market for search engine optimization, SEO where people try to optimize, you know, the page rank and how high a web page will rank on Google. The same will happen for papers. People will try to optimize like the impact factors and the citations that they get. And that creates a really big problem. And it's super interesting to actually analyze the. If you look at the distribution of the impact factors of journals you have like Nature, with Nature, I believe is like in the low 40s. And then you have, I believe science is high 30s. And then you have a really good set of good journals that will fall between 10 and 30. And then you have a gigantic tale of journals that have impact factor below 2. And you can really see two economies here. You see the universities that are maybe less prestigious, less known that where the faculty are pressured to just publish papers regardless of the journal. What I want to do is increase the ranking of my university. And so they end up publishing as many papers as they can in journals with low impact factor. And unfortunately this represents a lot of the global south. And then you have the luxury good economy. And there are also problems here in the luxury good economy. So if you look at the journal like Nature, so with impact factor of like in the low 40s, there's no way that you're going to be able to sustain that level of impact factor by just grabbing the attention of scientists. What I mean by that is like for the journals, the articles that get published in Nature, they need to be New York Times great. So they need to make it to the big media. They need to be captured by the big media because that's the only way for you to capture enough attention to sustain that level of citations. And that of course creates problems because people then will try to again gamify the system and have titles or abstracts that are bigger, make claims that are bigger than what actually can be sustained by the data or the content of the paper and you'll have clickbait titles or clickbait abstracts. And again, this is all a consequence of metrics and science on metrics. And this is a very dangerous cycle that I think it's very hard to break. But it's happening in academia in a lot of fields right now.

Is it fundamentally the existence of metrics or the metrics just need to be significantly improved? Because like I said, the metrics used for Amazon for purchasing, I don't know, computer Parts is pretty damn good in terms of selecting which are the good ones, which are not. In that same way, if we had Amazon type of review system in the space of ideas, in the space of science, it feels like those metrics would be a little bit better, sort of when it's significantly more open to the crowdsourced nature of the Internet, of the. Of the scientific Internet. Meaning as opposed to like my biggest problem with peer review has always been that it's like five, six, seven people, usually even less. And it's often nobody's incentivized to do a good job in the whole process. Meaning it's anonymous in a way that doesn't incentivize, like doesn't gamify or incentivize great work. And also it doesn't necessarily have to be anonymous. Like there has to be. The entire system doesn't encourage actual sort of rigorous review. For example, like open review does kind of incentivize that kind of process of collaborative review, but it's also imperfect. But it just feels like the thing that Amazon has, which is like thousands of people contributing their reviews to a.

Product.

It feels like that could be applied to science where the same kind of thing you're doing with Fermat's library, but doing at a scale that's much larger. It feels like that should be possible given the number of grad students, given the number of general public that's getting. For example, I personally, as a person who got an education in mathematics and computer science, like I can be a quote unquote like reviewer on a lot bigger set of things than is my exact expertise. If I'm one of thousands of reviewers, if I'm the only reviewer 1 of 5, then I better be like an expert in the thing. But if, if I, and I've learned this with COVID which is like you can just use your basic skills as a data analyst as a. And to contribute to the review process and a particular little aspect of a paper and be able to comment, be able to sort of draw in some references that challenge the ideas presented or to enrich the ideas that are presented. Or, you know, it just feels like crowdsourcing the review process would be able to allow you to have metrics in terms of how good a paper is that are much better representative of its actual impact in the world, of its actual value to the world, as opposed to some kind of arbitrary gamified version of its impact.

I agree with that. I think we. There's definitely the possibility, at least for a more resilient system than what we have today. And I think that's kind of what you're describing, Alex, to an extent. We kind of have a little bit of a Heisenberg uncertainty principle. When you pick a metric, as soon as you do it, then maybe it works as a good heuristic for a short amount of time, but soon enough people would start gamifying. But then you can definitely have metrics that are more resilient to gamification and they'll work as a better heuristic to try to push you the best direction.

But I guess the underlying problem you're saying is there's a shortage of positions in academia.

That's a big problem for me.

Yeah. And so they're going to be constantly gamifying the metrics.

It's a bit of a zero sum.

It's very competitive. It's a very competitive field. And that's what usually happens in very competitive fields.

Yeah.

But I think some of the peer review problems, scale helps, I think, and it's interesting to look at what you're mentioning, breaking it down, maybe in my small, smaller parts and having more people jumping in. But this is definitely a problem. And the peer review problem, as I mentioned, is correlated with the problem of academic career progression. And it's all intertwined. And that's why I think it's so hard to break it. There are a couple of really interesting things that are being done right now. There are a couple of, for instance, journals that are overlay journals on top of platforms like Arxiv and Biorxiv that want to remove the more traditional journals from the equation. So essentially a journal is just a collection of links to papers. And what they are trying to do is removing that middleman and trying to make the review process a little bit more transparent and not charging universities. There are a couple of more famous ones. There's one discrete analysis in mathematics. There's one called the Quantum Journal, which we are actually working with them. We have a partnership with them for the purpose that Kat published in Quantum Journal. They also get the annotations on formats and they're doing pretty well. They've been able to grow substantially. The problem there is getting to critical mass. So it's again convincing the researchers, and especially the young researchers that need that impact factor, need those publications to have citations to not publish on the traditional journal and go on an open journal and publish their work there. I think there are a couple of really high profile scientists, people like Tim Gowers, that are trying to incentivize famous scientists that already have tenure and that don't need that to publish that, to increase the reputation of those journals so that other maybe younger scientists can start publishing on those as well. And so they can try to break that vicious cycle of, of the more traditional journals.

I mean, another possible way to break this cycle is to like raise public awareness and just by force, like ban paid journals. Like, what exactly are they contributing to the world? Like basically making it illegal to forget the fact that it's mostly federally funded. So that's, that's a super ugly picture too. But like, why should knowledge be so expensive? Like where everyone is working for the public good and then there's these gatekeepers that, you know, most people can't read most papers without having to pay money. And that's, that doesn't make any sense. Like that, that should be illegal.

I mean, that's what you're saying is exactly right. I mean, for instance, right. I, I went to school here in the us we studied in Europe. And you would sit like, you'd ask me all the time to download papers and send it to him because he just couldn't get it. And like papers that he needed for his research. And so.

But he's a student. Like he's a grad student. He was a grad student. But that, you know, I'm even referring to just regular people.

Oh yeah, okay, that too.

And I, I think during 2020, because of COVID a lot of journals put down the walls for certain kind of coronavirus related papers. But like that just gave me an indication that like this should be done for everything. It's, it's absurd. Like people should be outraged that there's these gates because so the moment you dissolve the journals, then there will be an opportunity for startups to build stuff on top of archive. It'd be an opportunity for like, for mass library to step up, to scale up to something much even larger. I mean, that was the original dream of Google, which I always admired, which is make the world's information accessible. Actually, it's interesting that Google hasn't. Maybe you guys can correct me, but they have put together Google Scholar, which is incredible, and they did the scanning of books, but they haven't really tried to make science accessible in the following way. Like besides doing Google Scholar, they haven't delved into the papers. Right.

Which is especially curious given what Luis was saying.

Right.

That it's kind of in their genesis. There's this research that was very connected with how papers reference each other and building a network out of that.

Interesting enough, like Google, I think there was a. There was not intended. Google was like the Google social network that got canceled was used by a lot of researchers. Yes, it was, I think was just a, you know, side kind of the side effect. But. And a lot of people ended up migrating to Twitter, but it was not on purpose. But yeah, I agree with you. Like, they haven't gone past Google Scholar. And.

Well, that said, Google Scholar is incredible for people who are not familiar. It's one of the best aggregation of all the scientific work that's out there, and especially the network that connects all of them, what sites, what. And also trying to aggregate all of the versions of the papers that are available there and trying to merge them in a way that one particular work, even though it's available in a bunch of places, counts as, you know, like a central hub of what that work is across the multiple versions. But that almost seems like a fun pet project of a couple of engineers within. Within Google, as opposed to a serious effort to make the world's science accessible.

But going back to just the journals, when you're talking about that, Lex, I believe that in that front, I think we might be past the event horizon. So I think the model, the business model for the journals doesn't make sense. They are a middle layer that is not adding a lot of value and you see a lot of motions. Whereas in Europe, a lot of the papers that are funded by the European Union, they will have to be open to the public. And I think there's a lot of Bill Gates too.

Like what the Gates foundation funds. They demand that it's accessible to everybody.

Oh, interesting.

So I think it's a question of time before that wall kind of falls. And that is going to open a lot of possibilities, because imagine if, if you had that gigantic layer of papers all available online, that unlocks a lot of potential as a platform for people to build things on top of that.

But to what you're saying, it is weird. You can literally.