Improving Software Engineering in Biostatistics with Daniel Sabanés Bové

Welcome to the Axial Podcast. Axial is an early-stage investment firm based in San Francisco. We partner with great founders and inventors investing in early-stage life science companies, often when they are no more than an idea. Axial is fanatical about helping the rare inventor who is compelled to build their own enduring business. Okay, we're recording. Okay, Daniel, I think we had a really pleasant conversation. Mainly me talking, so hopefully you're talking now. Um, great to have you just do a conversation here and talk more about, you know, software and biotech. I really enjoyed the preprint you put out recently, uh, it was just— it resonated with me. I think it resonates with a lot of folks, and so I think your message should get amplified more. So hopefully this serves as a starting point for you becoming more influential in biotech. But maybe you can just jump right into it and give people context on your general perspective on software and biotech and statistics and reproducibility.

Yeah, thanks, Josh. Thanks for the invitation for this podcast interview. So I'm really, honestly, I'm totally surprised somebody even found this preprint and even more surprised that somebody wants to talk about it.

So it's pretty important. You don't know how important this is.

Like, yeah, that's pretty cool.

Important science.

Yeah, no, no, thanks. That's really cool. And so hope we're going to have an interesting conversation. And I mean, the topic, just to get to the point of the topic, I feel today, you know, the way we do coding and programming and so on in statistics, and in particular biostatistics, but I would say in any statistics kind of field, is still kind of underrated and kind of underappreciated and kind of neglected in some sense. I think people are still very much thinking like, oh, everybody can, you know, program, you know, everybody writes some R programs or Python programs and they run and they produce some results and then they're happy and they just stop there. And some people, of course, go a little bit further and they produce R packages or Python modules and, and recently, more recently, Julia packages, for example. Um, and but, but even those people, I think most of them don't really think about long-term sustainability of these, of these software pieces and, um, also the, you know, correctness and kind of maintainability of those things. And, and that's why both in academia as well as in industry Many of these pieces then just die after some time because person just moves on. You know, maybe somebody wrote something during the PhD, then they finished the PhD, then it just dies there. Or somebody writes a piece of software in the job, but then they change the role or something and it just dies there, right? And that's kind of a pity because a lot of, let's say, knowledge is lost. A lot of, let's say, work is kind of not living any longer. Maybe somebody is actually using it, but that's not maintained anymore, so it would just, you know, not be usable anymore after some time. And that's, I think, that's kind of the, the problem that I'm still seeing now, and, and what we're trying to get a little bit more awareness on this topic. So, and hopefully kind of try towards changing this a little bit.

I have my own experiences in terms of why this is an issue. I think it's just incentives. But what do you think the maybe causes are of why maybe science isn't reproducible, especially biology, and why the software is just poorly maintained? And then what are the solutions? Could it be better training? Or is it like, one person I really admire is a guy named Brad Chapman. He maintained a lot of open source packages in Biopython and it has an immense impact. So do we just need more Brad in the world? Where I think you pointed out in past presentations, like you have this list of R packages that are really well maintained. Do we just need a better open source software?

Yeah, so I think, I think, yeah, I agree that incentives are the, let's say, one of the root causes because especially in academia, of course, there's incentive is to publish a paper and whether the paper contains then methodology that's then actually easily to apply with this piece of software, that's another kind of question. And even though many journals nowadays ask for code to come as a supplemental material, or to be referenced in the paper, usually that's— it's enough to just have a little script, you know, just an R script, a Python script, or whatever. It's not really reusable for, let's say, different applications very easily usually. So that's kind of one reason. I think the other reason is, is also education, because I have heard also from others and also from my own experience, I mean, while, let's say, quantitative science degrees like statistics or physics and so on and similar ones, there is some element of computer science courses that you need to take and that people take, and of course people learn to program, but then it usually just stops there, I think. And in particular, when you then come back to your, let's say, core degree topic, then you don't really think about this anymore. And you don't think about unit tests, you don't think about code style or readability or code review or anything like that. That's kind of not really, I think, a topic so far in education. So I think education is definitely a factor there. And that's also, I think, how can we— what could be potential kind of ways to improve this? Yeah, I think it's a lot about education and I think we cannot— it will take time to change, let's say, curricula in universities, grad schools, and so on. But we try— what we're trying, for example, is to bring this a little bit closer to the people with, yeah, articles like this preprint. Or we also have, of course, this working group which is called OpenStatsWare, openstatsware.org. You can check it out. And we try to work together across companies and across institutions to kind of bring this message out there. Make people aware of it and then of course also offer some kind of workshops and YouTube kind of webinar pieces to actually educate people how to do this. And of course there is a lot of material out there, right? I mean, it's not like there's no material to learn about this stuff or that there's a lot of material. But I think it's more like about the fact that people get aware of the issue and then actually start looking at this material, right?

I think people are just— a lot of people are too busy. And then if you're not— if you're not rewarded for the work, like, why the hell if I publish a paper? It's hard enough to publish a paper. Why the hell am I gonna also like make the data really nice and format the spreadsheet, you know, like CSV file? Like, I'm done. I don't care about the future. And so, you know, so on that point, you have way more— you have way more experience than I do. Is it better in academia or is it better in industry in terms of like maintaining software packages, or are they about the same?

Yeah, that's a hard question. I think it's a bit different, different setting.

You know, because you think in industry there's like a reward system. It's like, let's make sure this stuff is reproducible. It's like, let's make sure the software is well maintained because 10 years, 20 years from now, this has a huge outcome on our drug.

Trial. Exactly, exactly. So I think there's that aspect for sure. And of course, let's say we have— especially let's say when it's more about, you know, clinical trial phase statistics, then of course we have often the objective then to submit these things also to FDA or EMA or other regulatory agencies. And of course we have to make sure that this stuff is actually, you know, reproducible and the software is good enough, right? And this is especially more coming now with the move towards open source languages in comparison to past where everything was more in proprietary systems. So we are shifting more from proprietary to open source and therefore the focus is increasing in industry. I think probably at the moment, probably academia is lagging a little bit behind maybe and they don't have, let's say, the stakeholder objectives so much, right? So I think what I heard from academic co-authors also when we discussed about the paper and the conferences and so on, then it might even be more an issue. And it might even be more an issue there because you have this, again, PhDs finishing, postdocs finishing their work, and then it's kind of not really clear who inherits that. Whereas more and more in industry, I think you have these little groups and kind of crystallization points of people coming together. And then even if one person would leave to another part of the organization, then it would still kind of carry on.

Maybe like some of the best practices will go, you know, often from academia to industry, IP and whatever talent. Maybe like reproducibility best practices will go from industry to academia. So somehow like creating that channel. And academies usually resist industry, they're skeptical. But you have a really hardcore math background and like a really hardcore math background. How did you get into like statistics and medicine? It seems like you— I have some friends who are like hardcore math people like at Princeton and whatnot. And there's like social risk. You know, you get judged for like doing something that like is seemingly lower, you know what I mean? Like, how did you, you know, do hardcore math and then decide, hey, I want to do this, I want to work on these problems, I want to get out of the math matrix. There's social risk.

Maybe. Yeah. Okay. So, so thanks. Good question. Maybe, maybe just to clarify a little bit. So, I mean, I actually didn't study hardcore math, I wouldn't say. I studied statistics with, let's say, of course, math, let's say, classes and stuff. And it's actually, you know, a cool thing that this is actually possible in certain universities that you can directly study statistics. So you don't have to have a math undergraduate and then only can specialize in statistics. And because of this, I had basically from the first year mandatory classes in other subjects where to apply this to. And I always was interested, I was always been interested, I have always been interested, sorry for the grammar. I've always been interested in medicine. And that's why I enrolled basically for the medicine, can let's say application subject, right? So I had the most of this time was math and statistics class. And then of course there was some medicine classes from the beginning. And this kind of carried on bachelor, master. And then when I did my PhD, It was also in the biostats group, and that's— I was also consulting with medical doctors and stuff. So it always kind of carried through basically throughout the education part. And then, sorry, then it was kind of natural to, to think about joining, you know, pharma, joining Roche after the PhD. But that's kind of— so it kind of was a natural path already. Almost from the beginning.

What was the origins of like the medical interest? Because also one thing, there's temptation when you're a stats, applied math, math major. Uh, the temptation is quant finance. How did you resist the hedge fund world? Or their overtures to you? Or maybe you just told me, uh, I don't—

honestly, that's a very hard question. Um, I think, I don't know, maybe I just had in, maybe I just, and you know, and you're in high school or something when this kind of interests are forming, maybe I just had very bad economics kind of teachers or something. But I was, I was honestly never really into economics or anything like that. I mean, even though my mom was doing some stock trading, you know, when, when this whole kind of thing started, um, end of the '90s and so on. But I was also not that much into that. So that was, I mean, the whole quant stuff was never really so interesting for me. On the other hand, I think I always enjoyed, you know, I always enjoyed biology a little bit. Chemistry, I was not too bad. Physics, I was pretty bad. So I was not so interested in that. So I think, I don't know, but that's kind of, on the other hand, I wasn't really considering becoming a medical doctor myself. But then I thought, ah, cool, you know, stats on the one hand you can apply to many different subjects. That's really cool. And on the other hand, you can also apply to medicine. And I thought, oh, that's kind of a cool combination. So it's like, you know, something very practical. You can help people a little bit, maybe in indirect ways. And on the other hand, you can— I can use my math and my kind of programming interests. And so that's, yeah, that's the reason basically.

Oh, so you move to Switzerland, you join Roche. It was fast. What really fascinated me was like, you can't even work at Roche for 4 years, join Google for 2 or something like that, and then you're back at Roche. And so it's like, it takes a lot of courage. It also takes like a lot of confidence to do that, you know. It's like leave a job, join something else, you know what I mean? Like, and so what was the kind of the initial experience at Roche? And then like, what was the motivating factor to join Google for a while?

Right. Yeah. Yeah. So that was really, really exciting. Um, let's say dangerous, I would say. Um, you know, Roche is a great company and, and I'm still working for it, so I'm biased, but, um, Google is also a great company, but so they're both great companies, but, um, It's a completely different business environment, right? In Roche, you are kind of embedded in this multidisciplinary kind of, I don't know, mosaic of like, you know, different expertise, right? So you're bumping always, you know, into different kind of, I don't know, chemists, biologists, medical doctors, bioinformaticians, or whatever, or regulatory people, or safety science people. So it's, it's very diverse. And of course within that you still have this data science or statistics or more nerdy kind of people that are working with data and working with code. Whereas on the other hand, in Google is much more, I felt it's much more homogeneous. I mean, you all just have only those nerdy people with physics, math, or computer science background and some stats as well. So it's much more homogeneous. You have much, um, I feel like Google, it's much more pre-selected group of people, um, so it's much more filtered down, and because they do so many interviews and they only let through a few, very few, and so on, whereas Roche, I feel it's a bit more, I don't know, a bit more normal people, you know, um, not everybody is super crazy and, uh, super whatever intelligent, uh, I mean, they're all intelligent people, of course, right, but it's not like so pre-selected, I feel. It feels like more normal. But what was the reason to go to Google? Honestly, I was headhunted on LinkedIn. So very, very normal thing. And then first I thought, well, I would just say no because I didn't even think about leaving Roche. But then, you know, I also discussed with my wife and then we said, hey, why don't we give it a try, right? This is Google. I'm still not too old. I'm still quite young. I want to try it out, right? And then I thought, okay, you know, I just try it out. So I tried it out and I really enjoyed it. And I really enjoyed this technical expertise of everybody and the kind of the whole kind of, you know, software engineering, totally different world, of course, in Google, right? Everybody doing this all the time. You have all the greatest and latest kind of, you know, continuous integration, continuous delivery systems available to you. Things are just running very quickly. You have this big mono repository of code and everything just works very flawlessly, right? On the other hand, after some time I noticed I'm not so much interested in the business as such. So I was working in advertisement there and, and there was no real way to work more on healthcare stuff. So that's why I thought, okay, if I can combine a little bit both worlds, if I can do more software engineering stuff, um, of some sorts, um, in pharma. If I can maybe bring some of the skills that I learned and some of the tricks I learned in, in Google, if I can bring this back to pharma, that could be really cool. Um, and that's why I came back. Cool.

And so like, that's pretty— that's a pretty formative experience. You get to like see how the best software company, one of the best software companies in the world, like actually works, and then you go back to something maybe a little more passionate about in terms of like problems you're working on. What kind of— how much of what Google does can be transferred to biotech, and maybe what can't be transferred? You know, maybe there's some stuff that big tech companies do in terms of like producing code and software products that just won't work because maybe biotech companies are too multidisciplinary. But what do you think could be transferred and what do you think can't?

Yeah, yeah, great question. Um, I think there's a lot that cannot be transferred. Um, for example, I mean, basically Google is like a big, you know, money-making machine in some sense. Of course, it's— I mean positively sad. It's this big machine and connects advertisers and users, or you can also say it connects users and products via this magic machine in the middle. And of course it's a great tool, right? And of course I'm still using it kind of every day. But that said, so because it's like the central machine and it's basically software there. And so the software is the product, right? In Google, the software is the product, right? And that's of course a completely different world than in Roche, because in Roche, the product is, at least so far to 95% or 99%, is not the software, is not the product, but the software is just a tool like any other of the tools that they have, that we have, to bring medicines and diagnostics to people, to patients, right? And to improve the life of the patients. So that's a kind of very different framing. So that's why many things cannot be transferred. On the other hand, again, I was working as a statistician within Google. So I wasn't like computer engineer or software engineer, but I was working closely with software engineers. And I still remember in the first weeks, I learned the very hard way how to write unit tests almost— I would not say the whole day, but I wrote a lot of unit tests. And I was very surprised about the level of rigor that my manager, who was doing the code review, was actually insisting on. For example, if there was a comment and there was not a full stop at the end of the comment, then it was like a review comment, like, oh, you need to put a full stop at the end of the comment, right? Or things like naming. Meaning of functions or things like edge cases and unit tests and all of that stuff, all of the nitty-gritty details was kind of something really useful to learn and see how they're doing it and to understand why it's so important. And I think a lot of that stuff can be transferred to other, let's say, other software domains, including statistics and pharma.

Makes sense. I think most scientists have never written a unit test, and industry, not academia. So I see from your experience, maybe some of your peers, you know, when pharma companies or biotech companies use software as a tool, how much consideration is there when you do some sort of statistical analysis or build a script or even build some sort of like bespoke package, how much is it— how much do maybe the company cares about getting the result versus making something that's reusable? How much stuff has— have you done? How much work have you done that's been reused over time? Like, either it's like a, a workflow or some sort of software package.

Yeah.

And, you know, I'm assuming not much, but maybe it's— I'm totally off. And then either way, like, how do you think— do you think that's a problem, reusability? Or is there some sort of other lens to look through that, like, how do you make biotech companies value software more as a tool? Is it usability or just something else?

Yeah, good question. I think, I think it's also changing at the moment a little bit that, I mean, I think there is more, it's coming more the, the valuation for the software piece and the whole kind of engineering piece. And I think it's coming mainly through the whole AI kind of sometimes I would say hype, but also applications. I mean, there are also some, let's say, on-the-ground applications that actually make a lot of sense, right? And I think that's kind of a driving factor that actually, let's say, also high-level management or senior management in biotech is more realizing that actually computation is actually super important. It's not just some random part, you know, that we just have to do. And we see that, for example, also in Roche and Genentech and so on. So, but that said, That said, I think— I think— sorry, what was the question? What was your question?

Like, maybe I had a two-part question, but the first part was like, what, you know, what's the lens to think about how to increase the value of software? And maybe AI is the lens. Maybe enough hype will get the management and investors we should put more money into software, we should pay more attention. At least that's happening in Silicon Valley where most biotech companies have to have a computational angle, and that probably flows up to pharma sooner rather than later. The other part of the question was more like actually building statistical models, building scripts, packages. For drug developers, like how much is that just bespoke? Like just you have to build something problem by problem, like for this clinical trial we have to solve it versus building a framework where, you know, how do I say this? Yeah, yeah, framework for every trial.

Yeah, yeah, okay, great, great, great. So yeah, sorry. Uh, okay, cool. So, so I think It is changing, and I think in the past, again, many things were using proprietary systems, and the sad thing was I think every company kind of was reinventing the wheel. Every pharma company was reinventing the wheel to write kind of macros to run in those kind of proprietary systems. And then those were then also submitted to regulators and so on afterwards. Nowadays we are much more shifting towards this open source languages as mentioned before. And there the great thing is also that companies start to work together on frameworks, right? So one keyword, for example, that to mention is PharmaVerse. So Pharma and then verse, like Tidyverse but PharmaVerse. That's like, you know, open source collaboration between different companies. Towards particularly kind of the whole kind of reporting. How do we actually report clinical trials kind of thing, right? How do we produce all these tables and plots that we need for our submissions? And I think that's kind of great because we can share the, let's say, the cost between companies. We can still make it very transparent to the public and to any stakeholders. Including regulators and payers and patients, how we're actually analyzing the data. So that's actually much more transparent than in the past. And in addition to that, but let's say in addition to that, I think we still need some— need to do some more improvements towards, let's say, stuff that's happening within the companies, more like on the exploratory analysis level, right? So when, you know, when there's any kind of study reading out or any kind of preclinical experiments, how do we actually get from the raw kind of experiment data to the actual conclusions? I think there, there's probably still some way to improve, you know, some leverage that we can improve, you know, things like reproducibility, but also, yeah, simple things like code review, right? I mean, many cases what we have seen, and not just, you know, not just me, but also others in other places that many people just write a script with one, one person's writing the script and not even really double-check that this is actually working correctly. Again, when we're in exploratory context, right? Um, this is not really checked and nobody's actually reviewing that, and maybe later nobody actually still remembers how was this actually done and so on. So I think there's a lot of stuff we can improve there, very simple things. Not nothing too— nothing too like rocket science.

Yeah, where as software gets increasingly more valuable for pharma companies, they'll start investing in those best practices. It also seems like, back to your original point, like, while maintaining open source packages, like, you know, somebody's gonna have to fund that. I think, I think that's what big tech companies do pretty well. They, they fund, you know, open source development pretty substantially, maybe our companies will have to step up and fill the gap because otherwise you're just relying on the goodwill of people and people are busy. So one point on this open source thing though, from your experience, you build these macros and little scripts for a particular prompt and at least last 5 years in particular in SF, there's been this proliferation of like SaaS and biotech. That's one way to put it. Analytics, ELNs, LIMS, and a few other things. What's been your experience, at least on the pharma side, of like interacting with open source tools versus maybe proprietary closed source software services? And like, what would be your preference as somebody in a big pharma company helping drug developers.

No.

And how do you make the trade-offs between something that's open source versus something that, like, you'll have to pay a subscription for, or maybe it's a little closed off, but maybe it's better convenience. One example is Veeva. You know, Veeva has— is a dominant player for, you know, CRMs and vaults. So Definitely closed source.

Yeah, yeah, okay, great question. Yeah, so I think there are definitely, let's say, you know, products, and I will not mention any names of course, but there are definitely products that, let's say, big pharma companies are licensing and using because it just wouldn't make any sense to develop that in-house or, you know, trust open source solutions, or maybe there's not enough open source solutions for that, for those kind of things. So that's definitely happening and But on the other hand, I think as we are, let's say, if we think more specifically about statistics and data science in general, I would say then what I have seen is that while there are interesting products out there that are closed source proprietary, I think, I think at least from the Roche side, I think we And when I say roadside, I mean, of course, that's in my department and the usual disclaimers apply. But what I've seen, what I have been seeing is that we are more careful whether we are locking ourselves in another proprietary system. Again, we are coming from decades of using certain proprietary systems for that, right? And we actually want to get out of that. We want to get to an open source state. So that's why I think it's definitely, it will require even much more advantages of any kind of proprietary systems to really go there and use that. We even prefer, let's say, you know, building stuff from the ground in the open source world if it's not yet existing, just because we don't want to be locked in anywhere, right? And we, because we want to have this transparency, right? And that's the other thing, right? I think we realize, and definitely pharma companies are starting to realize, that open source doesn't mean it's free, right? Often you have to do a little bit different work and you have to do things like checking that things are actually working as you want. There's nobody you can really call, of course, if something goes wrong, right? So you have to take certain responsibility yourself, evaluate the tools you're using, build certain trust, maybe build some testing suites and so on. But we are taking, rather those steps than, than let's say going too much into proprietary, uh, statistics or data analysis solutions.

That makes sense. I was amazed you didn't know about Benchling. Like, Benchling is a— where, like, in my network, Benchling is a massive company, like, on value, paper valuation. And I was like, no, but Benchling, that's incredible. So that kind of maybe reinforces that maybe at least for scientific software, open source is definitely the default preference. And it is always going to be something proprietary. But on this note, maybe to get out of the biotech, just science in general for software, you think about some of the most successful scientific software packages— SAS, MATLAB, a few other things. Like, but there are open source equivalents. I use MATLAB. MATLAB is horrible, but Python, Python is so much easier, but you still get stuck with MATLAB and that gets like, why do you think proprietary scientific software exists in the first place? Convenience, you know, some sort of way to certificate, you know, certificate, you know, like a lot of people are certified as people, SAS people. Versus R, SAS, R is better. SAS is huge. So maybe just for scientific software in general, why do you think proprietary solutions exist? Is it— do you think they help solve that reproducibility problem? Is there an economic incentive to build software that—

Yeah, that's okay. That's a big question. I mean, I definitely— I mean, I would say— I think it's a good question.

My instinct would be like SAS exists because they make money, and so they're going to maintain their software really well. And that's the best— that maybe that's the counterargument, that there should be more proprietary software in biotech to solve this reproducibility problem. The open source, you know, would require large amounts— not large, but some amount of funding for pharma. So either or, I think.

Yeah. And as I mentioned before, I think, I think we're seeing the increasing funding in the open source. It's, it's currently not so much through, let's say, funding certain, let's say, I don't know, open source developers that are freely running around or something. It's more like by actually having contracts with developers that are kind of in-house in the pharma companies, and then they help build this stuff. But to come back to the original question, I think it's a cool question. I think, so as always, the world is not black and white, and I think it's great that there are also, let's say, you know, proprietary solutions, and there are companies that are doing this for money. I think that's good, but I think we need to have both, and I think the question is also what are these let's say, products kind of serving, right? I think why it has developed over time, and I think we have both seen develop over time, right? We have these companies seen develop over time. We have seen open source languages developing more over time. I think it's maybe sometimes really the risk, let's say, profile that customers want to have, right? If you, if you want to have the least risk solution, if you don't want to invest, let's say, variable amounts of money into developer developing stuff yourself, then probably buying something is much easier, right? You buy SAS or you buy MATLAB or buy anything else, that's much easier, and you can call somebody if there's a problem, right? On the other hand, if you are more into innovation. If you want to maybe use the latest and greatest methodology, if you are anyway having developers in your maybe large organization, if you want to be more transparent, but on the other hand maybe take some risks, some calculated risks, then maybe you're better off with open source. And I think we will always see, let's say biotech and pharma companies, there will always be both. Right.

So I agree. I think there's— it's a complicated subject in general. And just like software in general, open source is closed source, and there's room for both. But particularly for science, you would think a lot of it would be open source just because of the ethos of science. Yes. You know, a lot of it's public. You would think most of the software in science would be public, but it's not. And it's a longer topic. MATLAB is a case study where every person, at least in the US, every academic is trained with MATLAB, and then they carry that forward in their career versus they probably should be trained in Python. I think it's changing, and so maybe MATLAB is in trouble. But I agree, I think there's, you know, it's going to be— I think it just depends on pharma, how pharma values software. Are they willing to at least put some money towards it? And if they don't, I agree, all this open source stuff deprecates. And I think, I think it's changing. I think you're, you're on the front line, you definitely see it changing, and I'm in back of the pack and I'm definitely seeing it as well. Um, one, one topic maybe on this point is, you know, for software, particularly statistics, where can— like, where's all the— what kind of innovation is there on the statistical package side, maybe software in general, that gives pharma companies advantages? Are there, like, from your experience, like, new statistical packages or techniques that have made it— made a trial more efficient, that have, you know, increased the precision of a trial versus the past? And looking forward, how do you think statistics software actually being not only just something that like helps with reproducibility but actually gives you an advantage like as a drug developer, like you can help you, you know, be more efficient. I think, know the answer more quickly so you spend less money on a failure and spend more money on a success. Yeah, what kind of maybe anecdotes you have or maybe technology methods you see or software you see coming down the pipeline that can help the broader industry?

Yeah, very, very, very big questions.

But I can start. I think one is these kinds of digital twins. I think we discussed this as well, and it's kind of a black box. I don't fully— maybe you can explain it to me. I don't fully understand it. And it's, you know, the market leaders are Learn out in San Francisco. They're definitely— they have approval in Europe. And that's kind of one example of like a statistical software product that could reduce the placebo arms of trials, at least.

Yeah, I mean, you know, I think we're definitely seeing a lot of enthusiasm and activity in the space of dynamic borrowing of control arms. For reducing, as you mentioned, control arms in pivotal trials. And I think especially for that also, I think it's important that we have methodology developments and corresponding software kind of going hand in hand and being transparent, I think, for researchers. And then we're also working in that space. So for example, PSBORO-2, that's like an R package that we are developing actively in that space. We see definitely a lot of energy in the covariate adjustment space, which is basically very much close to the digital twin stuff. A lot of enthusiasm about that. And we also actually with OpenSTUDS, we're also working in that space towards having like, let's say, reliable solutions also in the open source domain. I think I think there's a lot, there's a lot that is kind of not maybe also so much related to the methodology as such, but there's also a lot that is related to how we actually deliver the solutions, right? Of course, with the whole web-based developments in the last two decades, you know, we're shifting from very like static kind of PDF documents to something more like web-based and you can click around and you can, you can explore the data. I think that's also super important for pharma companies, especially for the internal decision-making and for exploratory analysis. And I think that's again where I think open source is very, very strong in that. And I think, yeah, I think otherwise, I think I'm I think I'm a little bit not so optimistic at the moment about the whole LLM hype that is being performed. There's a lot, I think, that's maybe a bit too optimistic. On the other hand, again, for specific solutions and actually also for software engineering as such, I think, of course, there's a huge potential for LLMs to help and make things actually better.

On this point, initially I was skeptical of these LLMs. And, uh, but then once ChatGPT-3 came out and I started using it, and then I use Claude as well, you know, it's like 2.5 or something, or is it on the 4? I forgot what version of it, but nevertheless, it's transformed the way I read patents. At least I read a lot of patents for diligence, and it's made my life so much easier. And if I'm having this small benefit, I can imagine like for people who deal with a lot of paperwork, actually like structured paperwork, patents, regulatory stuff, these LLMs probably add a lot of value. Yes. Like beyond this point of overhype, I agree a lot of it, like I get pitched a lot of transformer models for biology. Oh, we're gonna train on this thing and make a prediction about drug targets. And it's like, probably not. Good luck with that. Yeah, I don't, I don't like buy this whole transformer models for a particular vertical until I see some results. I think that's a— I think, I think in combination for other things maybe, but not as a standalone. Exactly. Platform. But there also are benefits. I mean, in— I can read like 100 patents a day and like know what they do. What's been your experience with Roche, maybe just personally, where you think these LLMs, maybe just a transformer model in general, can add a lot of value?

Yeah, I'm probably not the best person to ask about that, but I mean, I definitely see that again, I see possible applications to make, again, coding faster and better., right? And that maybe can also be one, you know, one small solution to maybe infuse those best practices because maybe your LLM, when you're coding, maybe it can tell you, hey Daniel, maybe you should add a test here, or maybe, you know, you should refactor this function a little bit like that, right? So it also can come in through that angle. And of course, we see, I think we see a lot of internal activity improving workflows, as you mentioned, right? Reducing the manual labor work, making more time for, you know, creative thinking. I think that's also great that LLMs can help with that, reducing the manual kind of boring work. And of course, there are, let's say, let's say, also activities with the early research biology kind of domain and so on. But, but again, I think for that one, I'm personally, I'm personally thinking that there's just so much complexity in biology. I think we probably need to be very focused in what we are trying to achieve with what. And, and personally, I would be maybe more betting on like things like quantum computing and, you know, in a few decades to really kind of be able to simulate, you know, biological processes maybe more reliably, or very complex things that are happening on any chemistry space or biology space, rather than maybe LLMs. I think LLMs, again, are good for like text. That's what they're made for. And I think anything kind of text-based probably can help us with that. But there's many other things, right?

I think one thing I I agree with you in terms of the complexity of biology and chemistry. And so you need just new forms of computing to get there. The counterargument I've heard is text represents the state of the world. Everything is in text. And so if you understand text, you understand the world. And that's the counterargument. A lot of those people are from that AI foundational. We'll see. I mean, this whole— in San Francisco, you should come to San Francisco. There's a whole AGI religion out here. It's crazy. Like, certain parts of SF, it's a religion in some ways. It's very interesting to see. But one little point, I think on the— we've had portfolio companies use LLMs to ask questions about their preclinical data, and none of it's been right, but it's been provocative. And so I know, I think this transformer models for biology, maybe they don't give you right answers, but they give you non-consensus starting points where it's like, oh, I didn't consider this, and then you have to go and do the work yourself versus it's not going to give you an answer that's relevant for complexity of biology, but it might give you an interesting take. And that That might be good enough, actually, because sometimes science, my experience, you're so stuck in what you learned. You're so stuck. It's like you're calcified in terms of like, oh, this gene only does this. Maybe it doesn't. Maybe it does something else. And so maybe LLMs help you kind of—

that could be. Yes.

One of my old mentors, Gary, brilliant guy, and he had that kind of zaniness. He was always willing to like consider a nonsensical idea. And so maybe LLMs act as that, like, nonsensical scientist. Yeah, that's been my experience using it. That makes sense. It's never been right, but it's been interesting. So, okay, since I never— but one last point would be, um, you know, in terms of GitHub Copilot, I've used it I use Replit and it gives a copilot and it's like, man, coding it. I don't code for a living, but I was like, I just wrote some scripts with it. I'm like, this is so easy. Do you think all these LLMs for coding, I definitely think that's like the first application that's worked. Do you think that can help software for science?

Maybe that can help.

Yes, I think, I think that can help. Have you seen it in practice? Have you any examples or maybe?

I have. Well, I had some tries, not sure if it was GitHub Copilot or GPT-3 or something, where it was not so successful, but I guess I was like too greedy with the kind of tasks I gave it so that it didn't succeed on those ones. But, but I think as we, again, as, as we, as these things are getting better over time and as we see more native integration into the IDEs and so on, I think, I think we will see Yeah, as we see today when we type an email and things are being auto-completed and most of the time actually it makes sense and you just accept it, I think we will see that. And actually coming back maybe to the basic things, but I think as we are seeing that happening, I think it's even more important that statisticians and others learn about how do we actually specify certain requirements for our software and how do we actually encode that maybe in some basic tests, or at least can we tell the LLM to give us this and that kind of test because that's kind of what we are, what we think is important to have, right? So I think maybe the LLM in the end can even write a program if we give it the corresponding requirements and if we give, if we give it the kind of framework in terms of the test cases and so on, right? So maybe writing the actual code itself will, be actually less important or less difficult in the future, will be much easier. But I think in addition, yeah, on the flip side, that means things like best practices on more like the design and more like the high-level things about the software will become even more important.

I think especially for science, especially for a clinical trial, specifying the requirements, because like, they could be life or death for some people. So you kind of like really create the specifications up front and know they're correct. Because if not, bad things can happen. But on that point of best practices, so the group you're part of or leading, OpenStatsWare, right? OpenStatsWare.org. I will put in the link and then we'll promote it. Hopefully more people need to join it. I'm also very— I always admire people who maintain R packages. I had a friend in college. He's an economics major. His claim to fame was he took a lot of pride in it. He didn't get paid for it, though. He maintained the most used economic R package in the world. And I would hang out with him. What are you doing this weekend? I got to freaking code up my R package. I got to do some work on it. And it's like, do you make money? So shout out to all the R people out there. I think they do it for street cred. I think— I don't know your experience. Why do you think people maintain open source code, especially R? Is it just for the glory of it or like the respect or— that's what it seems like.

Well, I think it depends. I think there is definitely this enthusiasts and that's great. I guess as long as you maybe don't have a family yet, you can do those kind of things. Fortunately, I'm doing it as part of the job. So I think, again, that's where it comes to PharmaCom. Companies investing in this kind of activities. And of course, R in particular has a very important academic, let's say, core team, right, that is really maintaining the inside of R and all of the C libraries that are sitting underneath and how that is linked to our interface and so on. So in that sense, yeah. I think it's relatively solid and I think it will stay around for a while, I think. But other things are also getting interesting, like Julia, for example, is also really, really cool stuff.

Julia, I remember Julia came out like 10 years ago, 2011, 2010 or something. I remember that Julia was really cool because I like Python elements of Python, but then the power of R graphics or like, no, no, the power of C. Julia was like Python. Yeah, hybrid. Okay, I'm happy that Julia's taken off. I never— I hadn't heard about it since for like a long— I had to look. I don't know. I think it's not as widely used as R or Python.

No, no, no, it's very niche at the moment still.

Yeah, I always thought Julia was cool. I used to play around with it when it was— we were playing around with lists. I was trying to learn lists. I don't know if I have a hobby of learning new programming languages, and Lisp was so hard. A lot of coding, from my experience, is about bragging rights. A lot of it is just bragging rights, like, you know, I know how to code this up, I know this language, you don't, right? A lot of it is— that's what it feels like. That was my experience of, like, existing on the fringes of coding. You know, I was on Hacker News. I don't know if you explore Hacker News too often. You don't? You should. Hacker News, I don't know, I'll DM it to you. But, uh, good. Yeah, Hacker News. Yeah, yeah, Hacker News is— it used to be really cool 10 years ago. It's like a Reddit for engineers. Okay, cool. It used to be really cool 10 years ago, and it's kind of relocated because they got too big. But maybe it's not too big. We don't know about it, you know. So, okay, Hacker News is pretty good. They have some pretty— I'm gonna— let me, let me stop this recording and then I'll just talk offline. But Daniel, it's great talking. I think a lot of people are gonna find this useful.

I really appreciate it. Thanks, Josh. It was fun.