Sovereign AI in Poland: Language Adaptation, Local Control & Cost Advantages with Marek Kozlowski

Hello, and welcome back to The Cognitive Revolution.

While we often discuss sovereign AI in the Silicon Valley AI bubble, we rarely hear directly from the technical leaders who are actually leading national AI projects. And so today, I'm very glad to share my conversation with Marek Kozlowski, who's leading Project PLUM, which stands for Polish Large Language Models, in his role as head of the AI lab at the National Information Processing Institute, Of Poland. Poland, with a population of 38 million and GDP of roughly $1 trillion— roughly 10% and 3% of the United States respectively— is an interesting and in some ways a representative case study. It clearly doesn't have the resources required to compete with the US and China at the AI frontier, but it does have strong technical talent, a real sense of pride in its language and culture, and a deep desire to control its own technological destiny And avoid domination by global superpowers. So what does that mean in practice? As you'll hear, Marek's strategy relies on the core belief that by training small models for a particular local language and cultural context, countries like Poland and projects like PLUM can compete with the latest frontier models, all while retaining control, preserving data privacy, and achieving a major cost advantage. In this conversation, we dig into the strategic realities that motivate projects like PLUM And the technical challenges that they have to overcome to succeed, including how today's frontier models, which are trained on overwhelmingly English and Chinese data, fall short in other languages. Why this problem is actually getting worse from one generation to the next as frontier model developers prioritize things like coding performance above support for niche languages. How EU regulation prevents European AI builders from conducting massive web scrapes and instead forces them to rely on more focused data curation projects. How the Polish government is thinking about investing its finite resources across data, compute, and talent. The language adaptation techniques that Marek's team layers on top of LLaMA and Mistral base models so as to inject local knowledge without needing to start from scratch. Why they haven't yet had to worry about developing a constitution or other explicit articulation of values for Polish AI systems. And why government agencies and national champion companies are often better served by by smaller models fine-tuned for specific tasks and served locally than by massive generalist models served from the cloud. Overall, Marek's mix of realism about the challenges of competing with global leaders and his positive vision for transparently created, locally controlled AI is a great window into what AI leaders around the world are thinking and doing to maintain AI sovereignty. So with that, I hope you enjoy this deep dive into the meaning, and training of Polish AI with Marek Kozlowski.

Marek Kozlowski, head of the AI lab at the National Information Processing Institute of Poland. Welcome to The Cognitive Revolution.

Welcome, everyone.

I'm excited for this conversation, too. We met not too long ago at an AI event in Las Vegas, the Enterprise Technology Leadership Summit, and I thought it was really interesting to double-click on everything that you're doing, because in the United States and in the sort of Silicon Valley AI circles that I spend most of my time in, there is this ongoing conversation about sovereign AI. And I think it's funny that a lot of this conversation happens in the Silicon Valley bubble and sort of makes a bunch of assumptions about what other countries feel the need to have or aspire to create. what's driving those decisions. And I don't hear too much from primary sources of people that are actually doing the sovereign AI projects around the world. So I was excited to meet you and learn more about what it is that you're doing in Poland. Poland, obviously, I think obvious to me, is a country with a lot of technical skill and very distinct culture, obviously its own language, proud tradition. And so I'm really interested to get into it and figure out what sovereign AI means in the context of Poland.

Once again, thank you for the introduction and for introducing my person and showing the idea. The idea is that I called it slightly broader, not only the sovereignty, but also creating the localized LLMs. Because the localized, it can be the national LLMs, but also the domain-oriented LLMs. Yeah, and I create the— maybe not I create the idea, but I am promoting the idea of the localized LLMs. It means the LLMs adapted to the language or domain, because they can be also a domain, and they are in this domain or the language, they have higher quality understanding text in this language or domain and have the higher quality and they are able to create the higher quality text in the generation step. It means that building the localized LLMs, they can be, of course, adapted to the language or domain, has 2 goals. First of all, to improve the understanding in this domain of the language, but also give the possibility to generate the higher quality texts, of course, in the aspects like the linguistic and cultural aspects. There are the idea, and today— and our goal is to create the models that are an order of magnitude smaller than the closed popular now LLMs, but in the aspects of the language and the culture or the domain, they have the same quality as the 10 times bigger models, and they are open source, transparent, secure, and as much organic as we can.

Yeah, okay, great. That's a great start. Can we maybe take one step back? If we can and just talk about why this is needed, first of all, from a capabilities perspective. Famously, I think it was— gosh, it's been a minute, but I think it was the GPT Instruct series. I think the model originally was Text-DaVinci-002, if I recall correctly, one of the first models that OpenAI trained to follow instructions. They reported basically, we just trained this thing to follow instructions in English, and lo and behold, it seemed to be able to follow instructions in other languages too, which was obviously a strong example of emergent capabilities and transfer learning, positive generalisation, all these sorts of phenomena that had been kind of elusive, but I think in many ways characterised the phase change that we've gone through from earlier AI systems to these more general, AI systems, positive transfer being obviously a huge one. But that's where they started in terms of just English and, oh my God, it works in other languages. Since then, of course, they've gone and done a lot of work to try to collect data in other languages, to try to even things out. And my sense from just kind of benchmark data is that they have made pretty good progress, but still, performance is best in English. And then you can kind of think of performance getting worse the farther a language is from English in the language tree, and also just correspondingly how many resources it has, right? Few low-resource languages are obviously going to be a bigger challenge than higher-resource languages. That's my sense of—

Yeah, at least because I have to correspond to your insights. First of all, 90% of the training data is English and Chinese. Yeah, when you— even if you look at the biggest open source or the biggest closed LLMs, 90% or 90% plus data are English and Chinese ones. Only 10% or less are the other languages. For example, it varied, but for example, in some models, the Polish language— I use this example— there is about 1% of the corpora or even smaller. And this decides that the that the vast majority of the skills and the competencies are gained by the English and Chinese instructions. And of course, if you have the large model, it has a huge competency transfer learning. You can say they extrapolate. You can very easily extrapolate between the tasks. But for example, even if I have the— in the instructions, I have lots of mathematical calculations prompted or commanded by the English language, And even if I ask to do it in Spanish, the very large model can do it in the intermediate steps. It can translate on the fly the commands from Spanish to English and somehow map the knowledge from the English to resolve the solutions, even if it was not learned on the English— on the Spanish examples of how to calculate some mathematical formulas. But what is the most important, that this works very good. But this works the same way as we or our kids learn the language. First of all, we learn how to understand and hear, listen. Next, how to write and how to speak. And of course, if you learn a new language and we get some command, in our minds we try to map these commands to what we know from our primary language, our native language. And the same is going inside the LLMs. For example, I can give you the example. That the models that were not trained by the huge volume of Polish texts, they are still able to be communicative and create the text that is understandable. But there are some statements and some phrases that are very easily identified that they are not the natives, right? Like, for example, I give you the example about writing the emails in Polish. And for example, they have such a formula typical for English, I hope you stay in good health and condition. This is typical for English but not typical for Polish. And even if you translate this word to word, it's communicative, understandable, but not typical for our language and culture.

So is that— is there more to say about how the leading commercial models are underserving the Polish market than that? I mean, I have the sense that there is a little bit more to it than just the cultural idiosyncrasy, because even when I look at an MMLU benchmark, it does seem like performance degrades across the language spectrum, right? Like, the highest MMLU score is in English. It does seem to get worse in other—

I know.

But for example, when you look at the benchmarks, we live in the world that we are biased by the benchmarks. It means, for example, the NLU benchmark is mostly you choose the solutions A, B, C, D, yeah, the multiple choice questions. They are not testing the ability to communicate fluently in the language. Most of the benchmarks don't test how good is the model in producing the longer forms, longer writings, or longer sentences. They usually test the understanding, extractive competencies, summarizing competencies and, and many knowledge about the facts in the world. But there are, I know, little or almost little, very little, very few benchmarks that test how good the model is in generating the longer forms of text in the other languages than English and Chinese, for example, the niche languages. Yeah, because it's much harder. For example, we in Poland create the benchmark PLCC, Polish Linguistic and Cultural Competency Benchmark, and this benchmark enables us to evaluate how good is the model in different subcategories. And for example, they're not only categories of the grammar vocabulary, but also about our culture, tradition, and history, and many, many others. But we would like to not only evaluate how good is the wordings of the model, how good the model is in, in some typical for our tradition wordings and the phrases, about the history also, about the— but we also trying to check how good the models are in the general spectrum of the using some ambiguous words, for example, in Poland. But there is still— but there is still not— this benchmark still doesn't validate how good the longer sentences are in Polish, how good the model is in producing the longer structures in Polish language.

Yeah, interesting. So is it fair to say that the primary focus of your work in creating Polish native models is on this, on these sort of softer skills? Like, it doesn't sound like you're focused on closing the benchmark gap or the sort of reasoning gap that exists between English and Polish. It's more about as you said, like culture, values, tradition, history, cultural competence.

But it's the— because I think that the language is not only the wordings. The models can have the very broad vocabulary, but they should be able to use it properly in the context. And sometimes the language is not only the words. There are the culture, tradition, history. Everything is mixed in it. And in order to create the model that behaves as natives, You have to inject not only the knowledge about how to create grammatically correct sentences, but also how to use special idioms or phrases in special contexts, or what places are typical for Polish history, or maybe what places are viral now. Yeah, generally it's like that. You have to mix the history, grammar, vocabulary, art, entertainment, culture, and tradition, everything into the mix. To create the language ability that is somehow similar to the natives. But as you ask me why we are doing that, first of all, because as I mentioned sometimes, that we believe in the idea of the localization LLMs, the localized LLMs, it means the LLMs adapted to the language, that they are as much similar to natives as possible. The second issue that the competency gap. But for example, we believe that, for example, we have to develop our people or our engineers to have the skills to build our own models, because maybe in a few years the market will change. Maybe the more, more models will be closed, or maybe some models will be forbidden. There are plenty of models currently in the European Union, but they are not— we are not able to use it because of the AI Act. Yeah, even in the licenses, the Llama 3.4 The KIMI and many, many other models, they have the sentence, the statement in their license that they are prohibited to use in the European Union. Maybe we will be forced to use this knowledge to build our own models. Maybe they will be a little bit worse than the Chinese or USA, but they will be our own. Yeah, and sometimes it's better to have the competencies to build even something a little bit worse, but have the ability to do it. then don't have this. Yeah, we can. Sometimes it means more than you think. But also we think that there is in this approach, in the PLUM family, because we create the family of models, we also believe in the transparency because we show how we built it from scratch. We released a few weeks ago, or 2 weeks ago, sorry, we released the publication, almost 100 pages, how we build these models. And also, we not only release the publication, the recipe book, the cookbook, but also we publish on Hugging Face the samples of our datasets, the instructions, preferences, because we would like to show not only the open weights, but more than— because the open source are not only the open weights, there are also some samples of open data and the cookbook, how we do it step by step. In a very detailed manner. And what is for us important, because I think even now they are the most popular open source models, are the Chinese now, but they are only open weight. There are no samples of the instructions or preferences they use to train the models. And here we would like to go a step further to be as transparent as possible. And also we invest lots of in their organic or dynamic data. Because we believe that we also proved that, that when you, uh, you have 3 stages on the, you know, when you learn the models. First, the pre-training. It's somehow similar to learning the kids a new language that you identify the words, how to create structures from these words, and some pieces of information. But children after this type of learning is not able to resolve the mathematical calculations or write the essay. Yeah, it's like the— you learn the language, but you don't learn the competencies. Next stage is the SFT, supervised fine-tuning. If you learn how to resolve some tasks, downstream tasks, write the poem, write the essay, summarize this article, perform some calculations, you learn the competencies. Like the children in the school, you have the math, the geography, chemistry, and many others. And after all, you have the alignment, the preference learning. That you mark what the children have done during the test, for example, and this information, these marks, makes them know what should be corrected or not. Yeah, and with the same what we have done with the children, with the kids, that we learn them the language, the piece of information, the wordings, and the structure. Next, we learn the competencies, and then you evaluated them, and during the feedback loop, we try to improve their abilities We have the same— the same things are done with the LLMs. And for example, when you're doing the pre-training, you show the model the hundreds of billions of tokens to learn the language. And after all, in the SFT, the supervised fine-tuning stage, you— we show the model the syntactic instruction. Syntactically, it means that they are produced by the other LLMs. They usually, if they are linguistically poor, they also degrade the model. Yeah, because if you— in any stage of the learning, and you will see that when the model sees the poorer data, it will degrade. It means it's going down in the quality of the linguistic creation of the sentences. And so we focus mainly on creating the organic datasets, organic instructions and preferences, or even if you use the LLMs to produce such a instructions, uh, we check it by the humans to improve the structure and quality. And I think there are the novelty in— and the second is that, first of all, there is the open source, open data, and open cookbook. The second is transparent because it's written in the cookbook what we have done step by step, and we show the samples. We also focus on the organic data, organic instructions, organic, uh, references. And I think this is the reason why the GPTs and other big models are so good, because they are also— they have plenty of manual instructions and they don't show them because there is intellectual property of these companies. And we also, we secured our models on our own because we discovered that these models, when they are secured for the English speakers, they can be much easier hacked than the secured for the Polish speakers. I think there are the novelties, maybe briefly speaking. Yeah.

Hey, we'll continue our interview in a moment after a word from our sponsors. I have like 7 follow-up questions I want to ask about various parts of that. And maybe we can kind of break it down by inputs to AI for one thing, right? Obviously the big inputs are data, compute, and talent. And you touched on certainly data and talent there. I also do want to come back to the safety training because that's always a keen interest of mine. But maybe let's start with the goal. You've spoken about it somewhat, but I think one big challenge that we have, certainly in the United States, and we have all this talk, right, of especially in the context of the geopolitical competition in AI, there's a lot of talk about, well, we want to have AI with democratic values win. We don't want to have Chinese values, or maybe we even are bold enough to say we want American values to be the values that the AIs embody and kind of propagate through the world. That obviously brings a big question, which is, well, what are those American values? And I can certainly say that There's no single agreed-upon answer for that, right? What American values are is hotly contested on an ongoing basis. And that leaves basically the AI companies to try to come up with their own best guess of what that should be. And that too is often sharply criticised because it's too woke, or it's not woke enough, or it's right-wing extreme, or it's describing itself as Hitler.

So the LLMs, they are somehow the compressed representation of what we have in our web in the internet. Yeah, there's somehow what topics are the most important, what information is the most popular ones, it somehow is reflected by the LLMs. Yeah, if you have the problems, the political problems, the religious problems, the everything what is there is also reflected somehow in the compressed LLMs because LLMs, they are somehow, they are compressed memory repositories. Yeah, they are the compressed stores of the memory of the internet.

Certainly, that's, you know, all that stuff is baked in. Sometimes I don't know how far American, the leading companies have come today in terms of filtering the training data. I know that there are some techniques that are like, we're going to get rid of all the bad pre-training data, you know, and just try to show this.

Yeah, there is a typical step. Yeah, even we in our project, there are these called data curation. That even because, as I mentioned, in the pre-training stage, the 90% of the data, the web data, and the web data, plenty of them is creepy. Yeah, you are not able to use them because the model, your model will be not stable. And in this data curation step, there are 2 sub-stages. The duplications, you remove the same information that is repeated very often in the internet. It's sometimes, it's scaled 2 times your Because there is plenty of duplicates in the internet. And also there is a filtering out. We filter out the data that are very crappy. It means they are low quality. It means, for example, there are plenty of special characters, plenty of interpuncions, plenty of not words from not being recorded in our vocabulary. Yeah, there are plenty of such disturbed data. It should be truncated because it will have impact on the stability of the quality of models. And I think you mentioned that the big companies, they have such tools that are not even able to eliminate some poor quality data, but they even eliminate some, for example, the theories, some points of views, and much broader section Not only the linguistic aspects of the data. So it's the same like censorship. The Chinese models, if you ask about the, you know, what has happened in Tiananmen Square, they are not able to give you any information. Yeah, they generalize. The, the people who built models, they can able to isolate, or how is called, the black, the band some important information that for people who are not aware about that will be the reflection of the world without some part of it.

Yeah, but there's at least like 2 layers to this, right? I mean, there is all this data pre-training or pre-filtering, and I genuinely, I could certainly believe that the Chinese models are trained on data that's so thoroughly filtered as to never have seen any document about Tiananmen Square.

I feel that even not about the pre-training step, I think in the last stage, because as I mentioned, there are 3 stages during the learning models: the pre-training, the SFT, supervised fine-tuning, and the preference learning, sometimes called the reinforcement learning with the human feedback. But there are other methods like DPO or ORPO. And I think in this stage, they secure the model how to not behave.

Right. So that's what I want to get at for what you're doing in the Polish context, because I don't know what the Chinese companies are doing. I do know that the American companies are developing their model specs or their constitution. It's basically this super long document that says, this is how we want our AI to behave. And to their credit, they're starting to be reasonably transparent about what those are, so that at least the public has a sense of what they're going for. But again, in the US context, it's like pretty contentious because, you know, everything is contested here. In the Polish context, is it like that? Or is it, you know, is it an easier time? Like, do you have a constitution for what you want Polish AI to be?

There are some strategies, like there were the strategy how the AI should behave, or no, maybe not, how should not behave. Maybe in this case, for example, they should be ethical, that should not blame anyone, or for example, to avoid some Topics that are, for example, very risky topics. Yeah, how to say the topics about the hate speech. Yeah, there are some places that they're typical. The same for the models from the Chinese or the USA. There are the place when there is a risk that the model behaves in an unethical way, or in a way that we can be blamed. That is not— it's not the— it's not— it's rude, at least rude. But of course, there we go, some political tensions. Yeah, but generally, I think we don't have such a huge— now, currently, very huge constraints, as you mentioned. Yeah, we don't have a constitution where they have plenty of points that you have to obey. I feel we are mostly that the models should be ethical as much as we can, but we don't give too many constraints to the model because I think we are on the other level of development of LLMs than China, Chinese, or the USA government or the companies, because we are, I think, a few years behind them, or 2 years or 3 years. It's hard to say, but generally, I think we are not— don't have such a— we have our own regulations, but not the regulations containing how the model should behave, but rather what kind of data we are able to use for training. We have many constraints, time, something like that. focus on the data than on how the model should behave.

Yeah, interesting. So, I mean, I've never even been to Poland, so obviously, you know, should be very humble in terms of my ability to describe it. But one high-level fact that I know is that the large majority of Polish people identify as Catholic.

Yes, it seems to be. For many years, it used to be a good statement. Currently, I think it depends on how the city is big, that the inhabitants of the cities versus inhabitants in other villages, I think they should be varied.

Yeah. So how do you think about that dimension? Should the— I just happened to have done an episode not long ago about Catholic AI with a company that is literally building AI that embodies Catholic values specifically for religious Catholics. But in your context, you've got this sense that, like, okay, well, maybe a majority of people are Catholic, but maybe that's on the decline, and maybe it depends on an urban-rural divide. Is there some sort of decision-making process where you think, okay, how Catholic should our Polish AI be? And does it vary in different situations? Are you guys getting explicit about Articulating goals there?

I think we have much more liberty now. I think now we don't have such ideas to create the reflection of our world. But as you mentioned, I think the models can be very easily constrained by the preference learning, and you can learn it to behave in such a special way. But now, currently, when we produce the family of the models, we produce not only the chat models but also the instruct models, the base models. We give the possibility to companies to use any kind of the models because we know that some constraints may have a disruption effect on some business cases. But generally, I don't think that we, as the producer of the— as the releaser or the builder of the models, we should get all of them, the people, and people will decide how to skew them.

Gotcha. Okay. Yeah, very interesting. Do you envision that this will become something, as you presumably go on to train more in future models and they become even more powerful and potentially, I don't know to what degree you sort of aspire to serve a consumer use case versus empowering businesses in the country. But do you think that this becomes a challenge at some point? Do you envision a future where there is a sort of Polish constitution for AI that actually seeks to answer that question? And if not, how do you think you ultimately get around that? Because it seems to be a very central thing that the American companies feel they need to grapple with. So if you think you can avoid that problem indefinitely, I'm kind of wondering.

I think we have much harder problems, because if, for example, you have the AI constitution, solutions for the companies in the USA market. But for example, you can very easily use all data you have without any constraints. Of course, there is a problem with the— with some suits and many other cases, but there is like maybe it's a long process, not very— I think for most of the companies in USA, they can't take this risk because they are still profitable enough to even to pay some Some, to say, some, there are some bad decisions by the judges and so on. I want to say the, the arbitral decisions. But generally, I think in Poland and European Union, we have the AI Act and our local regulations, like for example the acts concerning the, the, how it's called, the rights, the authorship rights. And there I think these problems, legally speaking, these documents may have much harder impact on our quality of models, then, then, then we can— I think it's enough, it's enough huge constraint not to going further. Yeah, because as I mentioned, in the European Union you have the AI Act concerning the general purpose LLMs. We have also our local regulators, like for example the AI Act about the— the Act about the authorship rights. And both of them combined create much harder constraints than, for example, any kind of constitution that is much, I think, flexible, more flexible than our regulations. And I think we don't think currently about the AI constitution, but I know that maybe in 1, 2 years something like that appears. But currently in European Union and Poland, we fight with the currently existing and obeying the regulations.

Yeah.

And I think they are much harder, much harder. And they are much more impactful than those you mentioned in the USA. Because, for example, you have the AI Act or the authorship rights regulations. For example, they can eliminate 80% of the data from your training datasets. And so it's a huge— this has a huge impact on the quality of models.

Yeah, okay, that's interesting. Hey, we'll continue our interview in a moment after a word from our sponsors. So turning to data, then we can check back in on the state of the Polish AI constitution in a year. On the data front, you had mentioned that in the biggest open source models, maybe 1% of the data is Polish. Quick back of the envelope math. I think the LLaMA models are like maybe up to 15 trillion tokens that they've been trained on. I don't know if they disclose their data mix, but that would cash out to something roughly on the order of 100 billion tokens in Polish that the biggest projects might be using. I understand you have quite a bit more data than that, but also your last comment made me think like—

We don't have the trillion of tokens. Yeah, because even as I mentioned, even in the LLaMA, there was about, as I mentioned, 1% was the Polish language or maybe less. We have now several hundred billion tokens. We don't have even a trillion tokens because, as I mentioned, the duplication stage and the filtering out stage, they eliminate lots of data. And we don't have 1 trillion tokens after these data curation steps.

So where are you getting your data? And your comment about the The difference, the sort of regulatory arbitrage that the American companies are potentially taking advantage of. Are they able to use some Polish data that is on the internet, but not even the US?

Because remember, I remember that it was some time ago, there was the people who analyze the crawlers that are going on the webs, on the websites in Poland. They identified there are plenty of Anthropic crawlers and there was plenty of robot takes the pages and the websites that they disallow these Anthropic crawlers to get data. I mean, it means there are plenty of the crawlers from the US companies, US-origin companies that are crawling the Polish data, even if they are not, if they are, even if they are not allowed to do it. Because as I mentioned, it's much harder, for example, to be the, I know they have the court in the USA and to have some accuse them. for using the data and able to fight with them in the USA court. Even if they have some proofs that they use data, that they have the desire clauses.

And so in that way, you're— if there's 100 billion tokens that they're getting off the internet, it sounds like you can only use a fraction of that. And then you had to go elsewhere to find the 100, a few hundred billion tokens that you've—

Yeah, for example, they're using the central libraries with some, some sources of data that are not web. Yeah, because as I mentioned, the vast majority of data used by the, the big vendors, also by us, my web data. But for example, you have also some data that are not published in the web, and you can use to somehow, of course, to some extent use them. But as I mentioned, there is a, there is a minority of data. Yeah, still, even for us, even if you have some access to the the local organizations and so on. The vast majority of data we use, there is web data, and the problem is the same for all other players. Yeah, maybe we can much easier identify some websites that are not easily crawled by the external crawlers, but generally I think that most of the companies, the OpenAI or the Anthropic, they have, I don't know, maybe 80 or 90% of our data still. Hmm.

Yeah. So where else are you going to get data? Like, what is your data process?

We have, for example, the first of the Mass Image Web Data, but also we have, for example, the— there is a called— how is it called— the Library of Science. There is plenty of publications and so on. We also have done some private bilateral agreements with the publishers that do not publish on the web. But as I mentioned, there's only a fraction of data we have in our corpora.

And are you also— you mentioned kind of doing a lot of human review.

Yeah, is there— the huge— what we have, our advantage is not in the data used for the pre-training stage, because as I mentioned, I think that 80% of them is still in the Anthropic or OpenAI repositories, but that we have dozens or even hundreds of annotators that create manually the instruction preferences. Yeah, because they give us the ability to create the new data that is not published on the internet still.

Is there like a Polish equivalent of like Scale AI or Labelbox that you're working with to do this, or is this a project?

No, we have built our internal tools. Not so crowdsourcing ones.

So you guys have built your own platform for human preference data?

Yeah, human preferences and the human instructions are maybe locally, internally. Of course, some of them, some samples of them we publish to show what is the structure of our instruction preferences and some examples, but most of them is still the closed asset.

So how do you think about that? I mean, one question that I've been thinking about in the context of This whole sovereign AI discourse is obviously, as a national government, you can have different strategies or different goals and different strategies for what you're trying to do. One goal is, as you alluded to, we want to make sure we have our own base, our own data, our own talent base, compute we'll get to in a minute, so that if we get cut off or who knows what might happen, we have some sovereignty over, you know, what's going on, right? I think that's—

Yeah, some possibilities to develop in a different way. Yeah. But also there is the second, because this is the one, as I mentioned, we can something, we can mean something more than you think. Yeah. If you can do something even a little bit worse, there is still the competency and possibility to do a new way, new movements. But I think there's also the second issue. That I believe that the AI agentic revolution will be based on the smart localized models. Why? Because first of all, there are some branches or sectors in the economy or even in our public sector that we are not allowed to use the cloud-based solutions. Yeah, there are some regulations or the risk is too high. And there are some demands to have the on-premise models. When you have the on-premise models, you always have some challenges, like for example, the GPUs you have to buy, the energy consumptions. And usually when you realize that, for example, you need to buy the 16 GPUs and pay for energy, you always go to the downscaling, yeah, to use the as small model as possible to achieve the expected goal. And from our experience that, for example, the people, especially the businesses, but also the public sector, they don't demand exactly the ChatGPT, the general-purpose LLM that is able to resolve 1,000 tasks. Usually in the business and the public sector, you have the demands for the 10 or 20 use cases, and you are able to create the smaller models that are able to resolve these tasks on the same level as the few-shot use big LLMs and hosting them on the previous solutions. And I think when we go to the AI agentic solutions, there are plenty of agents. It means there are plenty of models that are used to resolve some complex scenarios. You really have to downscale the models to using only the as small as possible models to be energy effective. And also the economic aspects are crucial now. And I think this is the place where the small localized models can play well.

Yeah, that makes a lot of sense in the business context. And yeah, I, you know, in my experience, I would say the same has been true. Like when I'm really trying to dial in performance for a particular use case and that's all I care about, and I kind of know that this model is gonna be deployed in a controlled environment where you know, because of the way the system is set up, I know what the inputs are going to be, I know what the outputs are going to be, I know that, you know, I have other layers of control, then yeah, I can just kind of dial into one task or a few tasks. And often a cheaper model, you know, with the right training can do just as well.

Yeah, especially when you have the— because mostly the people are now using the cloud LLMs in a few-shot manner. Yeah, because they are very powerful, they are able to resolve the very broad number of tasks, thousands of tasks, and they use them as the— in a zero or few-shot scenario. It means they are integrating the API of their own systems with the cloud-based LLMs, and they use them as they can out of the box easily, because you could create the prompt and use the output. That's all. But when you have the— when you need to create the much more controllable solution, the closed solutions, the on-premise solutions, You are not able to use cloud solutions. You have to go with the different, as I mentioned, aspects of the decisions. Do I need the multimodal or textual-only model? Do I have training datasets? Maybe if I have training datasets, I can supervise, fine-tune the smaller model and achieve the same quality as the few-shot applied cloud-based solutions. And mostly, when I— we can create many deployments currently. We identified that when you have the few or 10 different use cases and you create for them at least 1,000 instructions or a few thousand instructions and you SFT, supervised fine-tune, the smaller models, you achieve almost the same quality or maybe sometimes higher quality than using in a few-shot approach the big, very big, large cloud-based LLMs. But of course, you have to prepare training datasets at least 1,000 instructions. The best— the higher number of instructions, better, but 1,000 is enough. Mostly organic ones, but— or maybe the semi-automatically created with the human factor. And then you have 1,000 or more of the instructions, you can SFT smaller models, and in this one task or a few tasks, you will have the same quality as using the zero or few-shot cloud-based LLMs. And I think this is the future. Because I think if they, if they will go to the business and the business will calculate the risks, the money, the possibilities, the, the how they control the solutions, how, what is the impact of their decisions, they will finally, this AI agentic environments, choose the small localized models and fine-tune them to their demands. And there is also one risk I will show you there because we discussed it last time, lastly with the, uh, and of my colleagues, that for example, in the Atropix models, the Claude, or for example, the LLMs from Atropix, they, for example, their quality in the ability, in knowledge about the Polish language and cultures going down because, for example, they decide to be focused mostly more on the software developer assistance. And if they focus on this, they might have a small fraction of the market, they go down with the qualities in the, for example, generating texts in the niche languages. And for example, imagine that, for example, in Poland you apply such a model from Anthropic, you integrate it with your environment, with your ecosystem, and after the next months or next years, the next releases of this model going down on your competencies that you demand, and you have to choose the other model Or you roll back if you can, because sometimes you aren't able to roll back, revert the previous models, or you have to choose another vendor. I think creating the huge integrations based on the cloud LLMs is a huge risk because, as I mentioned, during the next years they can change their target objectives and don't need to focus on the Polish or Czech Republic languages. Because there is not the market for them.

Yeah, that's really interesting. I've never heard that before. So just to make sure I understood correctly, you are seeing worsening performance over time in Polish on like top issues of sort of Polish culture, general world knowledge, as the Claude models have progressed through generations.

Yeah, I, because we have the, I don't, I can say that there's the Anthropic-based models, I think the Clout and on the haiku, or maybe I think the haiku. But also we identified this problem with GPT models. For example, the GPT models that don't going up with the equality of the Polish language and cultural competencies. There are some of them even going down with the next releases. I think the problem is much more broader. I think that there are the creators of the models, they analyze the market. And for example, if they need to focus on some competencies and they improve these competencies, sometimes there is a trade-off that other competencies are going down. For example, in this case, the Polish cultural and linguistic competencies. I can check it. It was the Claude model from Anthropic, and this version is going down on this, our PSCC benchmark, when you compare it to the previous releases.

Wow. Okay, that's a really interesting data point. Yeah. And I guess it maybe sort of answers the next question that I had for you, which is, because my general working model has been that the AI frontier model developers want as much data as they can get, and if you had any data for them, they would be happy to take it and maybe even pay you for it. What you're saying sort of suggests, well, Maybe not always, because they're trying to do the smallest models that they can as well, while they're doing all this sort of distillation. They're trying to go for efficiency. They're trying to obviously serve the core use cases that they're getting paid for, which is a lot of coding. And so maybe if you showed up at their doorstep with a few hundred billion tokens worth of Polish data and said, hey, would you like to use this? Maybe they at this point might say not really, because we aren't that focused on that use case versus we'd rather, you know, go do another however many billion, you know, generations of token— of coding tasks and use those tokens instead. I guess, would you guys ever consider— I know you have some, it sounds like you have some open data, but not all of it is open. If I was the— if I'm thinking as the government, I guess another goal that I might have is, I want my users, retail users, just my general public, to be as well served by AI as possible. And I don't know if you have stats on what the Polish retail consumer is using right now. Are they going to ChatGPT? Are they going to Gemini or something else? Mistral?

Who knows?

You may know. I don't know. But if I was the Polish government and I was saying, okay, here's what my people are doing, they're using these other Companies, we've gone and collected all this data. Is there some sort of deal to be made with the AI companies where you might say, hey, we'll either give you this data or perhaps license you this data, you pay us for it, and that way you can incorporate it into your process, and that way you can serve the Polish market better? It's— I've wondered if there's some trade that—

It's a good point of view that is, I think, the next natural step. Yeah, that we are not able to get more data without some assist or the cooperation with other players. But as I mentioned about the previous statement about the to serve as good as we can for our citizens, and I think that this is still there. There is one of the goals of our project because about the PLUM, the family of models that they call PLUM, Polish Language Models. It's a family of models, but not only the family of models, but also the assistant and chatbots for citizens and city inhabitants. Yeah, they will not only focus on the models as itself, because the models are very good asset, but how to build based on those models the chatbots, the RAG approach solutions that can work for the citizens nationwide, but also in the— for city inhabitants, for the, for the local chatbots, for example, in the cities halls. And as I mentioned, it was the It's very good to mention that, that sometimes we are trying to create better and better models, but the problem is, for example, on the digital level, on another place. For example, there is not a problem that the model is a little bit worse or better, but for example, there is no chatbots for the cities, municipal halls, and so on. And there are 2 issues. First of all, the deployment issue. This model should be a little bit customized, supervised, fine-tuned to be able to work as part of the citizens and provide some assistance for the city inhabitants. And you may— I bet this is the one issue. And the second issue, as I mentioned, is that sometimes maybe there's time for some cooperation because only the cooperation gives you the ability to improve your datasets and improve your models. I think it's a very good step, and I think, as I mentioned, it's it's we are we are developing in our pace, but we know that there is a place where you are not able to go further, and you have to be supported by someone else. Yeah, it's normal in each the same is in the business. Yeah, to some level of your development, you are you you reach some points, some but you have to be supported by some better players or diverse players. to be better finally.

Yeah.

So do you know what that kind of market share breakdown is today? Um, and is there a sort of established goal that you have to win market share with the models, or is it—

It's very hard about the winning. And I think because we are the— the, the models are not the corporate initiative. It's not the private money and the other funds. This is the project supported by and funded by the Ministry of Digital Affairs. It's a consortium of the 6 institutes and now the 8 because we emerged in the second year, institutes and universities. And we are the public initiative. If you are the public initiative, you don't think too much about the return on investment. Or the number of customers. We are much more focused on the how to be the as open as possible, how legal, because you have to be legal compliance with 2 regulatory holders, transparent as possible, organic because it improves the linguistic possibilities and secured, and how to can be used by the public sector as much as we can, because mostly in the public sector the models can be, should be closed on-premise deployments.

Yeah, gotcha.

How—

you've shared a lot about how you train these models, but what is the base? You're not doing all the pre-training from scratch, right? My understanding is—

Starting with the face, I can say about it more. I can say it and explain it much more in detail. We're trying to create the models from scratch, it means from the random weights. But the problem is in the pre-training stage, the number of tokens you have. As I mentioned, even there, when you look at the April reports from the DSLM models, they proved that even if you have the 8 billion parameters models, you have at least 1 trillion tokens to have the stable training. It means stable that gives you the moderate or high quality base model. And in our case, as I mentioned, the duplication stages and filtering out, give us around 200 billion tokens. It was too little to create the model from scratch, and we used, of course, the LLaMA. Of course, now the LLaMA are much more closed, but one year ago they were still open, and the LLaMA license was not so prohibited in European Union as it is now. But usually we use the LLaMA-based models and the Mistral-based models, and we continue pre-training. We perform the language adaptation, Language adaptation means we continue pre-training them on our corpora of Polish texts, and after all, we have the new base model, and this new model, new base models can be SFT and difference optimization learning in the next 2nd and 3rd stage. But as you mentioned, we are not able to create the, I would say, the moderate quality or the good enough quality model without 1 trillion tokens, and we don't have 1 trillion tokens in Polish language. Now we have made some experiments with the mixture of languages. Yeah, we made not only the Polish language, but we also use the other languages, the mixture of language to get this 1 trillion tokens and to make some random pre-training from scratch. But the efforts will be in a few weeks.

Hmm, okay. On this language adaptation step, I have a couple questions there. One is, do you continue to mix in English? Do you try to preserve the model's ability to speak English? Or after this language adaptation, does it just only speak Polish?

Of course, there is— when you have the cascade learning, because if you use the base model that was already created and you perform the language adaptation, of course, there is always in the cascade learning the problem of the forgetting. Some knowledge is forgotten in the previous— from the previous learning stages. But there is still some persistence still now. Even if we pre-train a few epochs on our Polish data, the models still have the competencies, for example, to write something in English. There's nothing that we don't prune. We don't prune any other competencies on the other languages. They have— they somehow, they are forgotten because there is a problem of forgetting in the cascade learning, but generally, speaking, we don't prune it manually. We only take the base model and perform the language adaptation, continue pre-train a few epochs, and in this how we make better the abilities in the Polish language. Of course, there's a trade-off. Some other abilities in other languages are going down, but they are not pruned at all.

And that's also where the world knowledge comes from, right? And by world knowledge, obviously there's know, all these sort of local details of life, right? And I— you can maybe give better examples. I'm sure you can give better examples than what I would give, but I'm thinking just like, you know, what are the names of the Polish candies that kids like? And, you know, how does one file a document if you want to sell a car to somebody else? There's, you know, surely some filing process. All these sort of little details, they're absorbed in that stage as well, right? Yeah.

And then you have to know that that usually when you— the knowledge, the general knowledge is pertained, but then you ask me about the factuality, how model is good in some facts, for example, regulatory issues, some law issues that they are changing in the time, it's always the problems in any kind of LLMs. Yeah, because you pre-train the LLM, for example, on the data that are till March 2025, and you don't have in this memory store information about the changes in the law or the some regulations or even some situations, accidents, and the names of the new politics after this time point. But generally, you— we use it in— when using RAG approaches or data, it means you have this retrieval stage and you have the database, the knowledge base up to date. It's much easier to be updated. And after all, when you get the retrieval stage, you use the models to synthesize or generate the answer based on that. And we, in this case, we use this factuality issue because I think none of the providers of the LLMs, even the big ones, are not able to pre-train it and a new interval of data is coming up.

Yeah. Do you think this would work for companies? This has been sort of a— this is a bit of a digression, but it's been a question I've had in my mind for a long time. Like almost 2 years ago now, I did an episode of the podcast with a company called MosaicLM. And what they were doing, among other things, was this sort of continued pretraining for businesses. They would go into a business and say, let's get all your tokens. And this could be all the Google Docs that you've got and the Slack history and all these various things. Let's compile that. And now we can pretrain on that. And hopefully the model will start to speak your internal native dialect of Whatever language you're speaking.

I call it, as I mentioned, the domain adaptation. Yeah, for example, you have some closed data. Yeah, there's a closed— for example, I have in my internal closed data for the documents about my insurance, my customers, about some reports that are not open. And I would like to pre-train the model on this data to make it more adapted to the domain. And we have done such a project now for the biggest bank in Central Eastern Europe, the PKO BPE. It's one of the biggest banks in Europe, the biggest one in Central Eastern Europe. And we have performed what you mentioned, the domain adaptation. We adapt the models to their domain. They have some— they have their own data, domain data that includes cross-data, and we pre-train, continue pre-training the models on their data. And I think it's a very good, very good approach. We proved that this, in different tasks, it varied, but there are some tasks that this domain adaptation gives you a huge gain. in the quality and in some, in some financial measures and, and so on. But, but, but I think there is one remark I have to mention, and I think that only the huge companies has enough data to be worth to perform a domain adaptation. Because we know that, for example, you have to at least have, after the duplication and filtering out, you have to reach at least around 10 billion tokens. If you don't have 10 billion tokens, it's not worth to performing adaptation. And I think if you would like to have the 10 billion tokens in domain corpora, you have at least 30 billion tokens before the duplication and filtering out stage. And the 30 or 40 billion tokens, I think there are only a few companies, maybe not few, but less than 100 in Europe that have 100 billion tokens.

They closed Internal data.

Yeah, I guess that my intuition is that, I guess it depends what you count, right? I mean, excuse me, I've, I started a company that's, you know, 40 people and I don't know how many tokens we have, but over all the Slack messages, all the Google Docs, all the Jira tickets, all the, you know, contract proposals that we've sent and the revision history on all of those, I do feel like it adds up pretty quick. So I guess maybe one of the barriers is just like, exactly how deep are these companies willing to mine into their own data? Like, if they're actually willing to go get email data from their employees.

I think if you count this data, there may be plenty of billions of conversations, the hundreds of thousands of agreements or the proposal of agreements. But if you sum up them, and then count them, and after all, deduplicate them and filter out, Maybe it's very hard to get 10 billion tokens. Hmm.

Yeah.

Interesting. Yeah. You can, you can count it. 10 billion tokens is about 10 billion words. And it's, you can imagine that after the duplication, but you have to at least 30 or 40 billion tokens to have finally 10 billion tokens in a domain corpora. I think it's not so easy. It seems to be easy for many, many, but when we start counting them, it's not so easy to get 10 billion tokens.

And I assume that, that sort of what you're counting probably doesn't include like individual employees' email histories, all that sort of stuff. That stuff is like kind of out of scope.

I think the emails, when they are emails between the, for example, the, the sales forces, for example, between the, the help center, the call center, or the sales emails, you can use them. But for example, there are also some kind of emails that are not keen to be used because of some undefined problems with the intellectual property or the risky cybersecurity risks and so on. I think it's not so easy to use any kind of email because always is the, the risk that some emails are too, yeah, I would say too risky or too, not too delicious.

Sensitive.

Yeah, yeah, yeah, sensitive.

Yeah, that's really interesting. Yeah, that also does sort of make me wonder if new organisation structures are going to be advantaged in some of these dimensions, because I totally understand the difficulty that would arise if you said, okay, hey, everybody, we know you've been working here for all these years and sending all these emails. By the way, we're going to take all that and put it into our training process. You might have a revolt.

bit different, that there is a— most of them, even the bigger organizations, they are not aware about what kind of data they have. And it's before you adapt the AI or you be— before you train the AI, you have to, first of all, you have to clean your data stores, identify what data you have, what is there, creepy or not creepy, they're high quality or low quality. And this, the data curation process or data organization process Everything what is around the topic how to organize data to find the high-quality fraction of the data, I think is the problem itself. Very often the company tries to integrate or deploy AI or even train the AI without this data curation, data organization process, and usually it collapses.

Yeah.

I think this is the most important, to be aware about what kind of data inventory you have, what kind of data level of quality is, what kind of data you can use without any regulations, internal regulations, external regulations. And this is the most important step. After this step, when you have the properly identified datasets, good, described, and well organized, you can start— go up to invest in the AI training and even AI deploying based on that.

That's really all very interesting. Thank you. Definitely great food for thought for me. What about just going back one more question on models? I know you had mentioned that some of the Chinese models have licenses that don't allow you to use them in the EU. Is that—

Mostly they were— they missed the Llama 3.4, the first model. In the license, there is a prohibition to use them in the— they are forbidden to use them in the European Union. But now there are KIMI models, the Chinese models, they have the same. They are not able to use them in the European Union. I think it's the problem with the AI Act, because the AI Act that was then released, the second chapter was released August 2025. They demand from you to create for the general purpose models, the, the model, the card of model, what data were used on training, how it was secured, what are the datasets used, and what are the resources used for creating them. And so many, many other points in the model card. And I think those kind of models, they don't want to evidiate what data was used by them and what kind of even the step tagging stages look like.

Yeah.

If that weren't a problem, would you be open to using Chinese models, or are Chinese models, um, not appealing for other reasons?

I think it depends on the task you have. For example, I will be very, as you say, sensitive. I will have a huge aversion to the risk when I would like to use the Chinese model to create the long structures of histories or essays and emails and so on. Because I think that when you create the longer forms of text, the ability that the censorship evidence will be, how to say, easily noticeable, it is going up. But when you, for example, use some model for the task like the understanding, analytical extractive, for example, to extract key information from the documents, or for example, just maybe something that you get the information, specific information information from the documents, I am open to use the chains model. Yeah, because the risk of the data censorship in the tasks that are typical analytical extractive ones is very low.

Yeah, that makes sense. Turning to compute and talent, the other 2 big legs of the AI stool, I guess how, How AGI-pilled would you say the Polish government is? I mean, it's pretty remarkable that all this is going on at the governmental level already. I would say that speaks to a pretty situationally aware and generally agile government. But how committed is the government, or how big of a deal does the government understand this to be? And downstream of that are going to be questions around, like, How much funding is there? And is there—

There are different, as you mentioned during the first minutes of our speech, that for example, there are 3 pillars of the AI revolution. The data, as mentioned before, the data organization, data curation, and generally looking at the data as the crucial point for the AI training. Next, the compute powers. I mean, the GPUs and the AI factories and any others, the data centers and so on. And the talents, it means the people. Yeah, the 3 pillars, they together combined create the fuel for the AI revolution. And as I mentioned, I think in Poland we are focused currently mostly on the AI factories. It means buying as much as we can the GPUs and creating the data centers for GPU tasks. Of course, we have some projects like the PLUM. That is a very good example. One of the— I think that maybe there are 3 or 2 projects similar to the PLUM in the European Union, that the Ministry of Digital Affairs, they are funds— they have funds, and they funded the consortium of the universities and institutes that are able to develop the models and the competencies and so on. And somehow they support the talents in this guide we have because you have the money for the people.

somehow.

But I think we don't be able to be competitive for the— against the US market. Yeah, because in the US market, the AI engineers, they are paid like the NFL players. Yeah, I heard something like that, that the best AI engineers or AI researchers, they have the contracts like the quarterbacks in NFL. Yeah, they are treated as stars. And I think we don't have such a maturity inside the decisions to pay people, to overpay people. For their niche competencies, because I think it's much harder in European Union, especially in Poland, to say how do we be— what is the final objective function we would like to get. For example, 10 million customers, or I don't know, the $10 million each week in our subscriptions. It's much harder for public sector to define such objectives that are very easily monetized And very easily evaluated. And I think in USA they pay some so huge contracts because they are able to evaluate in somehow even making some margins buffers for the future how this talent can give you what kind of innovation and how this innovation will pay pay you back. And I think this is the problem that in USA everything is there as I mentioned when I mentioned when I was in Las Vegas in ETS. When I mentioned that we are working on the LLMs in the public ecosystem, that the Ministry of Digital Affairs, they funded us, they could create the consortium, and this is funded by the public sector. Most of the people I met in Las Vegas, they were surprised that the public sector invests in LLMs. But in the USA, it's something— it's hard to imagine that the public sector is the they have enough, I don't know, the intuition, enough knowledge, enough money to invest in such a sexy and very revolutionary topic like AI.

So how do you think this will evolve over the next couple of years? Obviously, the amount of resources that the frontier companies are putting into their current and future models just continues to grow, right? That's expected to be— It's exponential.

Even now we see that when we look at, for example, on the GPT models, and when you compare them, of course, the reasoning abilities are going up in different kinds of models. But generally, the GPT-5, it was not such a huge— there's not a huge improvement over GPT-4. Of course, there are some reasoning capabilities, but generally there's a plateau. Yeah, the models are going up, But since some level, they are going— the improvements are very steady. There is like the horizontal improvement, not the vertical one. It isn't going up, but much more plateau, plateaux. And their development is not so sexy as it used to be. Remember ChatGPT 2022, ChatGPT-4, multimodality 2023, 2024. There are some places in our history of the AI revolution that they were so shocking and they also, they are so making our imagination work so that each year we have something huge, something changing the rules of the game. Now I think the models are going much more steadily. There are not huge bumps. But generally, I think now we are starting to count the costs. Yeah. Because of the energy and because of what their models are used for. There is the place of their— I don't know how to say— the same is needed in people. That when you meet someone, for example, you have some 5 minutes that you will be adored, or maybe not adored. And the same, after all, you have to evaluate, is it worth to meet with these people or not? Maybe some 5 minutes, maybe several hours. But generally, I think now is coming to the time where we're trying to evaluate what is the real cost of this, of these tools, and how we, how we can use them. And if they— if you use them properly, what they give us. Yeah, like the, the verification stage. Yeah. And I think the verification stage will give us a lot of information that we don't need to have a very huge LLMs, we have to invest in the smart localized LLMs, especially when you are working on on-premise solutions.

Yeah, I sort of have mixed feelings about that. I mean, on the one hand, I do think already the models that exist are amazing artifacts, for one thing. And very often, especially if you do take the time to do the supervised fine-tuning and really dial in their performance, they can totally work perfectly well for all sorts of use cases. At the same time, of course, we've got the leading companies saying we're nowhere near done. This is definitely going to keep going. You should expect more progress. We're going to have AI scientists, we're going to have AI researchers. How much do you think your strategy depends on or will change If it turns out that there's not so much a plateau, but that you do still see significant capabilities jumps, albeit with even exponentially more resources required to achieve them? How do you think you navigate that world if it really is the case that a $10 billion training run is actually that much better than a $1 billion training run?

I think it's much— the most important issue is the first one, the demands and what is expected by our customers. Because I also mentioned that we are very often biased by the benchmarks, general-purpose benchmarks, but they don't match the benchmarks and expectations that the business has. And for example, if you have then your business, the company, and you know that, for example, you would like to use the AI, And in these places, this AI should have at least this kind of metrics. It gives you very good information what kind of benchmarks you should create to evaluate whether— what kind of model is able to reach these benchmarks, these expected metrics. And I think this is the most important, that very often we analyze the general-purpose benchmarks, which are, for example, the factual, the reasoning, I know the extractive inferential competencies they are evaluating there. But for example, for the business, the problem is slightly different. For example, they need something that writes that beautiful email to the customer or something that makes the email that enables you to do cross-selling. That's very often we don't know what should be done properly because there is no business, they don't define their requirements very explicitly. I will start from there. First of all, what kind of— what would we like to improve in your business? What kind of tasks you would like to send to the AI? Next, create the benchmarks for these tasks. And next, you choose the LLMs, because I think that most of the business cases I have seen, they don't demand reasoning. It's— you can do it with your— with the normal LLMs without the reasoning stages. I think we should We should try to— we should slow down a little bit and analyze what is needed to be done and what especially has the huge business factor. Not only because it is sexy and the public relations likes that, but what gives the money to the business or what makes some savings.

Yeah, there's a massive disconnect, I think, often between General business culture and the culture of painting?

There are 2 trains, but they are not in the sequence, one after one, but they are in the— they are next to each other. One is much faster, the second is going on their pace, but they are not, as you mentioned, very often there is no crossing between them. Yeah, there are 2 roads, but the crossing is very far before us.

Yeah, that's really interesting. Who are your allies in this? You mentioned using multiple languages, and I assume that that's sort of in some sort of partnership with maybe other neighbouring country national institutes. I guess I'm curious as to how you think the international dynamics will play out. Historically, in the Cold War, we had the US and the USSR and these 2 great powers were sort of engaged in proxy conflict and whatever all over the world. And a lot of other countries understandably said, this is bullshit from our perspective. And there was a movement of countries that were like, we don't really want to be in either of your camps. We would rather be just independent. And the beef that you guys have between yourselves, we don't really want to be a pawn in that game. Now it's the US and China, obviously, that are kind of the two big poles of AI power. How do you think countries that are— I sometimes say countries 3 through 193 on the AI power rankings— how do you think they will react? Do you see alliances forming or countries working together to share resources, share datasets, to try to create some sort of a third way in the AI space?

There are some movements in the European Union. For example, there are projects that are international. They gather the different kind of people from different countries and try to do something together. But I think it's, it's a huge problem that generally when you would like to get very fast products or very fast outcomes, you have to centralize. The problem is always the same. Generally, the best way is to have the federation. Everything should be spread out. You have different people in different countries, they collaborate each other, they build the The, how do you say, the, the, the wealthiness is going up everywhere in somehow distributed way, but about a normalized way. But generally, when you would like to get very fast incomes, very fast outcomes, and, and have the products in months, not in years, usually you have to centralize the assets in one place. And, and there is a problem, yeah, because there are 2 different ways, yeah. If you would like to do it in ideal way You should create the unions, yeah, the unions of countries, the unions of states, the unions of the partners, the networking, the consortium with the consortia with the plenty of hundreds of stakeholders to get this knowledge everywhere and to distribute this knowledge and this power everywhere. But usually you have to get some outputs very, very fast, you have to centralize. And the problems are there are 2 opposite ways. You're not able to do it the same way, both of them. And I think this is the problem, yeah? Because generally when the— for example, when you have some very huge pressure on the outcomes of new models, you always prefer the centralizations, yeah? Like with the Silicon Valley, yeah? You have the huge USA, but most, 90% of the startups are in Silicon Valley, yeah? The centralization in one place where there are the money, assets, and everything. But from the economic point of view, the best place would be to distribute these companies across the whole USA.

Yeah.

And I think that that is the problem, yeah, that if you are— if you would like to monetize something and you would like to get the very fast outputs, you have to centralize. But generally, the best for the economy and social aspects is to distribute in a normalized way across the country. And also the the continent. What about geopolitical? I think there are still two players: the Chinese and the USA. They have the two biggest economies. They have money. The Chinese companies, I heard that they pay the same money to the researchers as the USA ones. Yeah, the the contracts are now currently they are somehow similar. It means they they pay very well. They don't have to compete. For example, there is not the risk that the people will be Taken over by the USA companies because they are well paid in China. In Europe, I don't think there is still the— there is a Mistral, the European-funded startup. Now it's not the startup, but it was the startup 2 years ago or 3 years ago. But now I heard that the shares, 30 or 40% of the shares are in Microsoft. Yeah, there is not so open and as it used to be. because there are some stakeholders from the USA. I think the— generally the problem is slightly different. There is— the question is whether the Chinese and the USA, they are going to be the rivals still, or there also there's a chance for the cooperation. This is the question. Maybe it's the chance for cooperation still.

Yeah, from your lips to God's ears. Maybe just one little follow-up, and I think this has been excellent. I really appreciate all your time and all these thoughtful answers. Is there anything that you have seen that on sort of a technical or sociotechnical level can help with the cooperation of decentralised AI? Here I'm thinking about things like the Near Protocol. I recently did an episode with the creator of the Near Protocol, Illia Polosuhin, Or there's also the Intelligent Internet, which is Imad Mostaq's project. There's others as well. These things sort of have this idea that if we create the right scheme, it might be somewhat cryptographically enabled.

There is a topic called federation learning, federated learning, that you can use the different datasets somehow anonymized and secured and use them in a way that the networks that you are unable to identify the sensitive data, but you can use this data to train your, your models. Yeah, there are different kind of ideas about this federated learning, but generally I had not known that there is some huge deployments of such approaches. Yeah, even that there is a very good approach, yeah, to have such networks or federations to cooperate each other and to share data in some, as I mentioned, the secured way. But I think still there is— we are not on— we— if you talk about the business and economy, we are not on the level to use it. Yeah, because I think we are still— the problem is that the companies mostly, they don't know what data they have, what is the quality of their data, what is the value of their data. If you aren't able to measure your own in-house data repositories, how you can go further and create some data mixture or the network of data repositories? Yeah, I think this is that maybe in the future. Yeah, in the future I think there is a chance that maybe the, the, the, like, the, the distributed data repositories with the secure levels and anonymizations used by the huge consortia to use as much data as we can. But I think this is not the— this is not for the next year.

Yeah, so much depends on whether there really is a Plateau or whether the frontier companies are just going to continue to keep scaling successfully.

And now I heard that the plateau is caused by— because there is a lack of the organic data. Yeah, the biggest companies, they collected already almost all organic data that was able to be collected from the internet. Even some of the companies, they scan the books that they're not— that they haven't appeared and In the internet to make this organic data gains relevant. But there is still the problem that I think that there is the, this repository, the resource of the organic data is almost full.

Yeah, that's why we're now seeing all these simulated worlds and, uh, yeah, the strategies to overcome that are, are gonna be definitely fascinating to watch. I for one will bet on them working, but it does to a certain degree certainly remain to be seen. Again, really fascinating conversation. It's been awesome to get your perspective. Anything else you want to share or anything we didn't touch on you want to comment on before we break for today?

I can share that I can recommend to our arXiv paper, the Plume Family Models. It is now the arXiv paper released in November, first days of November. And I recommend readers and the viewers of this podcast podcast to look inside this paper. The Plume family is the archive, the title of this paper.

Yeah, that's P-L-L-U-M.

P-L-L-U-M, yeah. And this, the archive paper, almost 100 pages.

And the P, of course, is for Polish. It's the Baelish Large Language Model. Marek Kozlowski, this has been amazing. Thank you for being part of the Cognitive Revolution.

Yeah, thank you very much.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.