波兰主权 AI：语言适配、本地控制与成本优势——Marek Kozlowski

Hello and welcome back to the Cognitive Revolution. While we often discuss sovereign AI in the Silicon Valley AI bubble, we rarely hear directly from the technical leaders who are actually leading national AI projects. And so today I'm very glad to share my conversation with Marek Kozlowski, who's leading Project PLUM, which stands for Polish large language models, in his role as head of the AI lab at the National Information Processing Institute of Poland. Poland, with a population of 38 million and a GDP of roughly $1 trillion—roughly 10% and 3% of the United States, respectively—is an interesting and, in some ways, representative case study. It clearly doesn’t have the resources required to compete with the United States and China at the AI frontier, but it does have strong technical talent, a real sense of pride in its language and culture, and a deep desire to control its own technological destiny and avoid domination by global superpowers.

What does that mean in practice? Marek’s strategy relies on the core belief that by training small models for a particular local language and cultural context, countries like Poland and projects like PLLuM can compete with the latest frontier models, all while retaining control, preserving data privacy, and achieving a major cost advantage.

In this conversation, we dig into the strategic realities that motivate projects like PLLuM and the technical challenges they have to overcome to succeed, including how today’s frontier models, which are trained on overwhelmingly English and Chinese data, fall short in other languages. We discuss why this problem is actually getting worse from one generation to the next as frontier model developers prioritize things like coding performance over support for niche languages; how EU regulation prevents European AI builders from conducting massive web scrapes and instead forces them to rely on more focused data-curation projects; and how the Polish government is thinking about investing its finite resources across data, compute, and talent.

We also discuss the language-adaptation techniques that Marek’s team layers on top of Llama and Mistral base models to inject local knowledge without needing to start from scratch; why they haven’t yet had to worry about developing a constitution or another explicit articulation of values for Polish AI systems; and why government agencies and national champion companies are often better served by smaller models fine-tuned for specific tasks and served locally than by massive generalist models served from the cloud.

Overall, Marek’s mix of realism about the challenges of competing with global leaders and his positive vision for transparently created, locally controlled AI offers a great window into what AI leaders around the world are thinking and doing to maintain AI sovereignty.

Marek Kozlowski, head of the AI lab at the National Information Processing Institute of Poland, welcome to The Cognitive Revolution.

Welcome, everyone. I’m excited for this conversation too.

We met not too long ago at an AI event in Las Vegas, the Enterprise Technology Leadership Summit. I thought it would be interesting to double-click on everything that you’re doing because in the United States, and in the Silicon Valley AI circles that I spend most of my time in, there’s this ongoing conversation about sovereign AI.

I think it’s funny that a lot of this conversation happens in the Silicon Valley bubble and makes a bunch of assumptions about what other countries feel the need to have or aspire to create, and what’s driving those decisions. I don’t hear too much from primary sources—people who are actually doing sovereign AI projects around the world—so I was excited to meet you and learn more about what you’re doing in Poland.

Poland, obviously, is a country with a lot of technical skill and a very distinct culture: its own language and proud traditions. I’m really interested in getting into this and figuring out what sovereign AI means in the context of Poland.

Once again, thank you for the introduction, for introducing me, and for showing the idea. I call the idea slightly broader: not only sovereignty, but also the creation of localized LLMs.

Localized LLMs can be national or local national LLMs, but they can also be domain-oriented LLMs. I didn’t create the idea, but I am promoting the idea of localized LLMs. That means local LLMs adapted to a language or domain. In that language or domain, they have a higher-quality understanding of text, and they’re able to create higher-quality text during the generation step.

Building localized LLMs, which can of course be adapted to a language or domain, has 2 goals. First of all, it’s to improve understanding in this domain or language, but it also gives us the possibility of generating higher-quality text, particularly in linguistic and cultural aspects.

The idea and our goal is to create models that are orders of magnitude smaller than the closed, popular LLMs we have now, but that have the same quality as models 10 times larger when it comes to language, culture, or a particular domain. They’re open source, transparent, secure, and as much organic as we can make them.

That’s a good start. Can we take one step back and talk about why this is needed, first of all, from a capabilities perspective?

Famously, I think it was the GPT Instruct series. I think the model originally was text-davinci-002, if I recall correctly—one of the first models that OpenAI trained to follow instructions. They reported that they had basically trained this thing to follow instructions in English and, lo and behold, it seemed to be able to follow instructions in other languages too.

That was obviously a strong example of emergent capabilities and transfer learning—positive generalization, all these sorts of phenomena that had been elusive but, in many ways, characterized the phase change we’ve gone through from earlier AI systems to these more general AI systems. Positive transfer was obviously a huge part of that.

That’s where they started in terms of just English: “Oh my God, it works in other languages.” Since then, of course, they’ve done a lot of work to collect data in other languages to try to even things out. My sense, just from benchmark data, is that they’ve made pretty good progress, but performance is still best in English. Then you can think of performance getting worse the farther a language is from English in the language tree, and correspondingly, languages with fewer resources are also going to be a bigger challenge than higher-resource languages.

That’s my sense of it.

I have to agree with your insights. First of all, 90% of the training data is English and Chinese. Even if you look at the biggest open-source or closed LLMs, 90% or more of the data is in English and Chinese. Only 10% or less is in other languages.

For example, it varies, but in some models the Polish language represents about 1% of the corpora, or even less. This means that the vast majority of the skills and competencies are gained from English and Chinese instructions.

Of course, if you have a large model, it has huge competencies in transfer learning. It can extrapolate very easily between tasks. For example, even if I have lots of mathematical calculations prompted or commanded in English, and I ask it to do them in Spanish, the very large model can translate the commands from Spanish to English on the fly and somehow map the knowledge from English to resolve the solutions.

That works even if it wasn’t trained on Spanish examples of how to calculate some mathematical formulas. What’s most important is that this works very well, but it works the same way as we or our kids learn a language.

First of all, we learn how to understand and listen. Next, we learn how to write and how to speak. If you learn a new language and receive a command in that language, you try to map the command in your mind to what you know from your primary language, your native language.

The same thing is happening inside the LLM. For example, models that weren’t trained on a huge volume of Polish text are still able to be communicative and create text that is understandable. But there are some statements or phrases that are very easily identified as not being native.

For example, I can give you the example of writing emails in Polish.

For example, they have a formula that is typical for English: “I hope you stay in good health and condition.” This is typical for English, but not typical for Polish. Even if you translate it word for word, it’s communicative and understandable, but not typical for our language and culture.

So, is there more to say about how the leading commercial models are under-serving the Polish market than that? I have the sense that there’s a little bit more to it than just the cultural idiosyncrasy, because even when I look at an MMLU benchmark, it does seem like performance degrades across the language spectrum, right? The highest MMLU score is in English. It does seem to get worse in other languages.

I know, but when you look at the benchmarks, we live in a world where we’re biased by the benchmarks. For example, the MMLU benchmark is mostly about choosing solutions A, B, C, or D. They’re multiple-choice questions. They’re not testing the ability to communicate fluently in the language.

Most of the benchmarks don’t test how good the model is at producing longer-form writing or longer sentences. We usually test understanding, extractive competencies, summarizing competencies, and so on. There are very few benchmarks that test how good the model is at generating longer forms of text in languages other than English and Chinese, for example, because it’s much harder.

In Poland, we created the PLCC, or Polish Linguistic and Cultural Competence benchmark. This benchmark enables us to evaluate how good the model is in different subcategories. The categories are not only grammar and vocabulary, but also our culture, tradition, history, and many others.

We would like to evaluate not only how good the wording of the model is, or how good the model is at using wordings and phrases typical of our tradition and history, but also how good the models are across the general spectrum—for example, in using ambiguous words. But this benchmark still doesn’t validate how good longer sentences are in Polish, or how good the model is at producing longer structures in the Polish language.

Interesting. Is it fair to say that the primary focus of your work in creating Polish-native models is on these softer skills? It doesn’t sound like you’re focused on closing the benchmark gap or the reasoning gap that exists between English and Polish. It’s more about culture, values, tradition, history, and cultural competence.

I think language is not only the wording. The models can have a very broad vocabulary, but they should be able to use it properly in context. Sometimes language is not only the words. Culture, tradition, and history are all mixed into it.

In order to create a model that behaves like a native, you have to inject not only knowledge about how to create grammatically correct sentences, but also how to use idioms or phrases in a particular context. You need to know what places are typical for Polish history, or maybe which places are real now. Generally, you have to mix history, grammar, vocabulary, art, entertainment, culture, and tradition into one thing to create language ability that is somehow similar to that of a native speaker.

As you asked, why are we doing that? First of all, as I mentioned, we believe in the idea of localized LLMs—LLMs adapted to a language and made as similar to native speakers as possible. The second issue is the competency gap.

For example, we believe that we have to develop our people and our engineers so they have the skills to build our own models. Maybe in a few years the market will change. Maybe the models will be closed, or maybe some models will be forbidden. There are plenty of models currently in the European Union, but we’re not able to use them because of the AI Act.

Even in the licenses for Llama 3.4, Kimia, and many other models, there’s a statement that they’re prohibited from being used in the European Union. Maybe we’ll be forced to use this knowledge to build our own models. Maybe they’ll actually be a little bit worse than the Chinese or U.S. models, but they’ll be our own.

Sometimes it’s better to have the competence to build even something a little bit worse, and have the ability to do it, than not have that ability. Sometimes it means more than you think.

In the PLLuM family, because we’re creating a family of models, we also believe in transparency. We show how we built them from scratch. We released a publication 2 weeks ago—almost 100 pages—about how we built these models.

We not only released the publication, the recipe book, the cookbook, but also published samples of our datasets, instructions, and preferences on Hugging Face. We would like to show not only the open weights, because open source is not only about open weights. It also includes samples of open data and a cookbook showing how we do it step by step, in a very detailed manner.

What’s important for us is that, even now, the most popular open-source models are Chinese, but they’re only open-weight models. There are no samples of the instructions or preferences used to train them. We would like to go a step further and be as transparent as possible.

We also invest a lot in organic data because we believe—and we’ve also proved—that there are 3 stages when you train models. The first is pretraining. It’s somehow similar to teaching kids a new language: you identify the words, how to create structures from those words, and some pieces of information.

But children, after this type of learning, are not able to solve mathematical calculations or write an essay. It’s like learning the language, but not learning the competencies. The next stage is SFT, or supervised fine-tuning. You learn how to resolve downstream tasks: write an essay, summarize an article, or perform calculations. That’s like children learning math, geography, chemistry, and many other subjects in school.

After that, you have alignment, or preference learning, where you evaluate what the children have done during a test. This information, these marks, show what should be corrected and what shouldn’t. We do the same thing with children: first we teach them the language, pieces of information, wording, and structure; next we teach them competencies; and then we evaluate them. During the feedback loop, we try to improve their abilities.

The same things are done with LLMs. When you’re doing pretraining, you show the model hundreds of billions of tokens to learn the language. Then, in the SFT stage, we show the model synthetic instructions. “Synthetic” means they were produced by other LLMs.

If those instructions are linguistically poor, they also degrade the model. If, at any stage of learning, the model sees poor data, it degrades. It means the quality of its linguistic generation goes down. So we focus mainly on creating organic datasets, organic instructions, and organic preferences. Even if we use LLMs to produce such instructions, we have humans check them to improve their structure and quality.

I think these are the novelties. First of all, there are open-source data and an open cookbook. The second is transparency, because the cookbook explains what we’ve done step by step, and we show the samples. We also focus on organic data, organic instructions, and organic preferences.

I think this is one of the reasons why GPTs and other models are so good: they also have plenty of manual instructions, and they don’t show them because they’re the intellectual property of those companies. We also aligned our models ourselves because we discovered that models aligned for English speakers can be much more easily hacked than models aligned for Polish speakers. I think those are the novelties, briefly speaking.

I have 7 follow-up questions I want to ask about various parts of that. Maybe we can break it down by inputs to AI. Obviously, the big inputs are data, compute, and talent, and you touched on data and talent there. I also want to come back to the safety training, because that's always a keen interest of mine.

Maybe let's start with the goal. You've spoken about it somewhat, but I think one big challenge that we have in the United States—and we have all this talk, especially in the context of the geopolitical competition in AI—is that we want AI with democratic values to win. We don't want Chinese values, or we might even be bold enough to say that we want American values to be the values that the AIs embody and propagate through the world.

That obviously brings up a big question: What are those American values? I can certainly say that there's no single agreed-upon answer for that. What American values are is hotly contested on an ongoing basis, and that leaves the AI companies to come up with their own best guess of what that should be. That, too, is often sharply criticized because it's too woke, not woke enough, right-wing extreme, or describing itself as Hitler in some cases.

Somehow, they are the compressed representation of what we have on the web, on the internet. They somehow reflect which topics are the most important in the world and which information is the most popular. That is somehow reflected by the LLMs.

If you have political, religious, or other problems, everything that is there is also reflected somehow in the compressed LLMs, because LLMs are somehow compressed memory repositories. They are the compressed stores of the memory of the internet.

Certainly, all that stuff is baked in. Sometimes I don't know how far the leading American companies have come today in terms of filtering the training data. I know that there are some techniques where they're going to get rid of all the bad pre-training data.

Yes, there is typical stuff. Even in our project, there is what is called data curation. As I mentioned, in the pre-training stage, 90% of the data is web data, and plenty of it is creepy. We are not able to use it because the model will not be stable.

In this data-curation step, there are 2 substages. The first is deduplication, where you remove the same information that is repeated very often on the internet. It is sometimes duplicated 2 times because there are plenty of duplicates on the internet. The second is filtering out data that is very poor quality.

That means, for example, that there are plenty of special characters, plenty of interjections, and plenty of words that are not recorded in our vocabulary. There are plenty of such distorted data. It should be truncated because it will have an impact on the stability and quality of the models.

I think you mentioned that the big companies have tools that are not only able to eliminate some poor-quality data, but that also eliminate, for example, theories and some points of view—a much broader section, not only the linguistic aspects of the data. It's the same as censorship.

With Chinese models, if you ask what happened in Tiananmen Square, they are not able to give you any information. The people who build the models can isolate, or, how is it called, ban some political information. For people who are not aware of that, it will be a reflection of the world without some part of it.

But there are at least 2 layers to this. There is all this pre-training data and pre-filtering. I could genuinely believe that the Chinese models are trained on data that's so thoroughly filtered that they never saw any document about Tiananmen Square.

I think that this is not even about the pre-training stage. I think it happens in the last stage, because, as I mentioned, there are 3 stages during the learning of models: pre-training, SFT, or supervised fine-tuning, and preference learning, sometimes called reinforcement learning from human feedback. There are other methods, such as DPO and ORPO.

In this stage, they teach the model how not to behave.

Right. So that's what I want to get at in the Polish context. I don't know what the Chinese companies are doing, but I do know that the American companies are developing their model specs, or their constitutions. It's basically this super-long document that says, “This is how we want our AI to behave.”

To their credit, they're starting to be reasonably transparent about what those are, so at least the public has a sense of what they're going for. But in the U.S. context, it's pretty contentious because everything is contested here. In the Polish context, is it like that, or is it an easier time? Do you have a constitution for what you want Polish AI to be?

There are some strategies for how our AI should behave—or maybe how it should not behave. For example, it should be ethical and should not blame anyone. It should avoid topics that are very risky, such as hate speech.

There are some cases where there is a risk that the model behaves in an unethical way, or in a way that can be criticized as rude, at least. Of course, we could have some political tensions, but generally, I don't think we currently have such huge constraints as you mentioned.

We don't have a constitution with plenty of points that the model has to obey. I feel that we mostly want the models to be as ethical as we can make them, but we don't give too many constraints to the model because I think we're at another level of development compared with the Chinese or American governments and companies. We're, I think, a few years behind them—2 or 3 years, hard to say.

Generally, we have our own regulations, but not regulations containing how the model should behave. Rather, they specify what kind of data we are able to use for training. We have many constraints focused on the data rather than on how the model should behave.

Interesting. I've never even been to Poland, so I should obviously be very humble in terms of my ability to describe it. But one high-level fact that I know is that the large majority of Polish people identify as Catholic.

Yes, it seems to be. For many years, that was a good description. Currently, I think it depends on how big the city is—the inhabitants of cities versus the inhabitants of villages. I think those groups vary.

So how do you think about that dimension? Should the AI—I just happened to do an episode not long ago about Catholic AI with a company that is literally building AI that embodies Catholic values, specifically for religious Catholics.

In your context, you've got this sense that maybe a majority of people are Catholic, but maybe that's on the decline, and maybe it depends on an urban-rural divide. Is there some sort of decision-making process where you think, “How Catholic should our Polish AI be?” Does it vary in different situations? Are you getting explicit about particular goals there?

I think we have much more liberty now.

I think that now we don't have such ideas as creating a reflection of our world. But as I mentioned, the models can be very easily constrained by preference learning, and you can teach them to behave in such a special way. Currently, when we produce the family of models, we produce not only the chat models, but also the instruct models and the base models.

We give companies the possibility to use any kind of models because we know that some constraints may have a disruptive effect on some business cases. But generally, I don't think that we, as the producer or builder of the models, should forget all of the people. People decide how to use them.

Gotcha. Okay. Yeah, very interesting. Do you envision that this will become something as you presumably go on to train more future models and they become even more powerful? I don't know to what degree you should aspire to serve consumer use cases versus empowering businesses in the country, but do you think that this becomes a challenge at some point?

Do you envision a future where there is a sort of Polish constitution for AI that actually seeks to answer that question? And if not, how do you think you ultimately get around that? Because it seems to be a very central thing that the American companies feel they need to grapple with. So if you think you can avoid that problem indefinitely, I'm kind of wondering—

I think we have much harder problems, because if you have AI constitutions for companies in the U.S. market, for example, you can very easily use all the data you have without any constraints. Of course, there is a problem with some lawsuits and many other cases, but it's maybe a long process. I think most of the companies in the U.S. can take this risk because they are still profitable enough to pay some fines, even if there are some bad decisions by judges or arbitral decisions.

But generally, I think that in Poland and the European Union, we have the AI Act and our local regulations, like, for example, the acts concerning authors' rights. Legally speaking, these documents have a much harder impact on the quality of our models than any kind of constitution. I think this is already a huge constraint that prevents us from going further.

As I mentioned, in the European Union we have the AI Act concerning general-purpose AI. We also have our local regulations, like, for example, the act concerning authors' rights. Both of them combined create much harder constraints than any kind of constitution, which is much more flexible than our regulations.

We don't currently think about an AI constitution, but I know that maybe in 1 or 2 years something like that will appear. Currently, in the European Union and in Poland, we are dealing with the existing regulations. I think they are much harder and have much more impact than those you mentioned in the U.S.

For example, the AI Act or authors' rights regulations can eliminate 80% of the data from your training datasets, and that has a huge impact on the quality of models.

Yeah, okay, that's interesting.

So, turning to data, we can check back in on the state of the Polish AI constitution in a year. On the data front, you had mentioned that in the biggest open-source models, maybe 1% of the data is Polish. Quick back-of-the-envelope math: I think the Llama models have been trained on maybe up to 15 trillion tokens. I don't know if they disclose their data mix, but that would cash out to roughly 100 billion tokens in Polish that the biggest projects might be using.

I understand you have quite a bit more data than that.

We don't have 1 trillion tokens, because, as I mentioned, even Llama had not 1% of its data in the Polish language, or maybe less. We now have several hundred billion tokens. We don't even have 1 trillion tokens, because the deduplication stage and the filtering-out stage eliminate a lot of data. We don't have 1 trillion tokens after these data-creation steps.

So where are you getting your data? You're talking about the difference—the sort of regulatory arbitrage that the American companies are potentially taking advantage of. Are they able to use some Polish data that's on the internet?

I remember that some time ago there were people who analyzed the crawlers going onto websites in Poland. They identified plenty of Anthropic crawlers, and there were plenty of robots.txt pages and websites that disallowed these Anthropic crawlers from getting data. That means there are plenty of crawlers from U.S.-origin companies that are crawling Polish data even if they are not allowed to do it.

Because, as I mentioned, it's much harder, for example, to take them to court in the U.S. and accuse them of using the data and then fight with them in a U.S. court, even if you have proof that they used data that had disallow clauses.

So if there are 100 billion tokens that they're getting off the internet, it sounds like you can only use a fraction of that, and then you have to go elsewhere to find the few hundred billion tokens.

For example, they're using the central libraries with some sources of data that are not on the web. As I mentioned, the vast majority of data used by the big vendors, and also by us, is web data. But you also have some data that is not published on the web, and we can use it to some extent. That's still a minority of the data.

Even for us, even if you have some access to local organizations and so on, the vast majority of the data we use is web data. The problem is the same for all other players. Maybe we can more easily identify some websites that are not easily crawled by external crawlers.

But generally, I think most of the companies—OpenAI and Anthropic—still have maybe 80% or 90% of our data.

So where else are you going to get data? What is your data process?

We have, for example, the data I mentioned, but we also have what is called the scientific literature. There are plenty of publications and so on. We also have some private bilateral agreements with publishers that are not published on the web. But as I mentioned, that is only a fraction of the data we have in our corpora.

You also mentioned doing a lot of human review. Is there a huge amount of human review?

Yes. Our advantage is not in the data used for the pretraining stage because, as I mentioned, I think 80% of it is still in the Anthropic or OpenAI repositories. Our advantage is that we have dozens or even hundreds of labelers who create manual editing instructions and preferences. That gives us the ability to create new data that is not published on the internet.

Is there a Polish equivalent of Scale AI or Labelbox that you’re working with to do this, or is this a project you’re doing internally?

No, we mostly have our own internal tools, not crowdsourcing ones.

So you’ve built your own platform for human preference data, and the human preferences and instructions are built locally and internally?

Of course, we publish some samples to show the structure of our instruction preferences and provide some examples, but most of them are still closed access.

How do you think about that? One question I’ve been thinking about in the context of this whole sovereign AI discourse is that, obviously, as a national government, you can have different strategies and different goals for what you’re trying to do. One goal is, as you alluded to, making sure you have your own base—your own data, your own talent base, and your own compute, which we’ll get to in a minute—so that if you get cut off, or who knows what might happen, you have some sovereignty over what’s going on. I think that’s important.

There are some possibilities to develop in a different way. As I mentioned, we can do something that may mean more than you think. If you can do something even a little bit worse, there is still the competence and possibility to create new ways and a new movement.

But there is also a second issue. I believe that the agentic AI revolution will be based on small, localized models. First of all, there are branches or sectors in the economy, and even in the public sector, where we are not allowed to use cloud-based solutions. There are regulations, or the risk is too high, and there is a demand for on-premise models.

When you have on-premise models, you always have challenges. For example, you have to buy the GPUs and pay for the energy consumption. Usually, when you realize that you need to buy 16 GPUs and pay for the energy, you start downscaling. You try to use the smallest model possible to achieve the expected goal.

From our experience, people—especially businesses, but also the public sector—don’t demand exactly ChatGPT, a general-purpose LLM that can resolve 1,000 tasks. Usually, businesses and the public sector have demands for 10 or 20 use cases. You can create smaller models that are able to resolve those tasks at the same level as large LLMs used in a few-shot setting, while hosting them on on-premise solutions.

When you move to agentic solutions, there are plenty of agents, which means there are plenty of models used to resolve complex scenarios. You will have to downscale the models and use the smallest models possible to be energy-efficient. The economic aspects are also crucial now. I think this is where small, localized models can play a meaningful role.

That makes a lot of sense in the business context. In my experience, I would say the same has been true: when I’m trying to dial in performance for a particular use case, and that’s all I care about, I know the model is going to be deployed in a controlled environment. Because of the way the system is set up, I know what the inputs and outputs are going to be, and I have other layers of control. I can dial in to one task or a few tasks, and often a cheaper model, with the right training, can do just as well.

Especially when you consider that most people are now using cloud LLMs in a few-shot manner. They are very powerful and able to resolve a broad number of tasks—thousands of tasks—and people use them in a zero-shot or few-shot scenario. That means they integrate the API of their own systems with the cloud-based LLMs and use them out of the box. You create the prompt and use the output, and that’s all.

But when you need to create a much more controllable solution—a closed, on-premise solution—you are not able to use the cloud solutions. You have to make different decisions. Do you need a multimodal model or a text-only model? Do you have a training dataset? If you have any dataset, you can supervised-fine-tune a smaller model and achieve the same quality as with a few-shot, cloud-based solution.

When we create many deployments, we identify that when you have a few or 10 different use cases and create at least 1,000 or 3,000 instructions for them, you can supervised-fine-tune smaller models and achieve almost the same quality, or sometimes higher quality, than by using very large cloud-based LLMs in a few-shot approach.

Of course, you have to prepare training datasets—at least 1,000 instructions. The higher the number of instructions, the better, but 1,000 is enough. They are mostly organic ones, or perhaps semiautomatically created with a human factor. When you have 1,000 or more instructions, you can SFT smaller models, and for one task or a few tasks, you will have the same quality as when using zero-shot or few-shot cloud-based LLMs.

I think this is the future. If AI goes into businesses, businesses will calculate the risks, the money, the possibilities, how they can control the solutions, and the impact of their decisions. Ultimately, these agentic environments will choose small, local models and fine-tune them to their demands.

There is also one risk, which I discussed last time with one of my colleagues. For example, in Anthropic’s models—Claude, or perhaps Haiku—we have seen their quality in knowledge of the Polish language and culture going down. We identified this problem with GPT models as well. Some GPT models are not improving in their quality on Polish language and cultural competencies; some of them are even getting worse with subsequent releases.

I think the problem is much broader. The creators of the models analyze the market, and if they need to focus on certain competencies, they improve those competencies. There is a trade-off, though, and other competencies go down. In this case, those are Polish cultural and linguistic competencies.

For example, imagine that Poland applies such models from Anthropic and integrates them into its environment and ecosystem. After the next few months or years, the next releases of the model may have declining competence relative to your demands. You would have to choose another model or roll back, if you can. Unless you are able to roll back and revert to previous models, you have to choose another vendor.

I think creating huge integrations based on Claude LLMs is a huge risk because, as I mentioned, during the next few years they can change their target objectives. They may no longer need to focus on Polish or Czech languages because those are not markets for them.

That’s really interesting. I’ve never heard that before. Just to make sure I understood correctly, you are seeing worsening performance over time in Polish, on topics like Polish culture and general world knowledge, as the Claude models have progressed through generations?

Yes. We have identified this problem in Anthropic-based models, including Claude and Haiku. We also identified it with GPT models. Some of the GPT models are not improving in quality on Polish language and cultural competencies, and some are even getting worse with subsequent releases.

I think the problem is much broader. The creators of the models analyze the market, and if they need to focus on some competencies, they improve those competencies. There is a trade-off, however, and other competencies go down—for example, Polish cultural and linguistic competencies.

I can check the exact version, but it was a Claude model from Anthropic, and this version was declining on our PSCC benchmark compared with previous releases.

Wow. Okay, that’s a really interesting data point. I guess it maybe answers the next question I had for you. My general working model has been that frontier AI model developers want as much data as they can get. If you had any data for them, they would be happy to take it and maybe even pay you for it.

What you’re saying suggests that maybe it’s not always the case, because they’re trying to do the smallest models they can as well. While they’re doing all this distillation, they’re trying to go for efficiency and obviously serve the core use cases they’re getting paid for, which is a lot of coding. So maybe if you showed up at their doorstep with a few hundred billion tokens’ worth of Polish data and said, “Hey, would you like to use this?” they might say, “Not really, because we aren’t that focused on that use case. We’d rather go do another however many billion generations of coding tasks and use those tokens instead.”

I guess, would you guys ever consider—I know you have some open data, but not all of it is open—if I was thinking as the government, another goal I might have is that I want my users, retail users, just my general public, to be as well served by AI as possible. I don’t know if you have statistics on what the Polish retail consumer is using right now. Are they going to ChatGPT? Are they going to Gemini or something else—Mistral? Who knows? You may know; I don’t know.

But if I was the Polish government and I was saying, “Okay, here’s what my people are doing. They’re using these other companies. We’ve gone and collected all this data. Is there some sort of deal to be made with the AI companies where you might say, ‘Hey, we’ll either give you this data or perhaps license you this data. You pay us for it, and that way you can incorporate it into your process, and that way you can serve the Polish market better’?” I’ve wondered if there’s some trade to be made.

That’s a good point of view. I think it’s the next natural step: you’re not able to get more data without some assistance or cooperation with other players. But as I mentioned, serving our citizens as well as we can is still one of the goals of our project.

With PLLuM, the family of models called Polish Large Language Models, it’s not only a family of models, but also assistants and chatbots for citizens and city inhabitants. We don’t focus only on the models themselves, because the models are a very good asset, but also on how to build chatbots and practical solutions based on those models that can work for citizens nationwide, as well as for city inhabitants—for example, local chatbots in city halls.

As I mentioned, sometimes we’re trying to create better and better models, but the problem may be somewhere else, for example, at the digital level. The problem isn’t that the model is a little bit worse or better; it’s that there are no chatbots for cities, municipal halls, and so on. There are 2 issues.

First of all, there’s the deployment issue. The model should be somewhat customized, supervised, and fine-tuned to be able to work as part of the bots and assistants for citizens and city inhabitants. The second issue, as I mentioned, is that sometimes it may be time for cooperation, because cooperation gives you the ability to improve your datasets and improve your models.

I think it’s a very good step. As I mentioned, we’re developing at our own pace, but we know that there is a place where you’re not able to go further and you have to be supported somehow by someone else. It’s normal. It’s the same in business: at some level of your development, you reach certain points, but you have to be supported by better or diverse players to finally be better.

So do you know what that kind of market-share breakdown is today? Is there a sort of established goal that you have to win market share with the models, or is it very hard to talk about winning?

I think it’s very hard to talk about winning, because the PLLuM models are not a corporate initiative. It’s not private money and all the funds. This is a project supported and funded by the Ministry of Digital Affairs. It’s a consortium of 6 institutes and universities—8, because we enlarged it in the second year.

We’re a public initiative. If you’re a public initiative, you don’t think too much about return-on-investment analysis or the number of customers. We’re much more focused on how to be as open as possible, legally compliant with EU regulatory frameworks, as transparent as possible, organic because it improves linguistic possibilities, and secure, and on how it can be used by the public sector as much as possible.

Mostly in the public sector, the models can—and should—be closed, with on-premises deployments.

Got you. You’ve shared a lot about how you train these models, but what is the base model? You’re not doing all the pretraining from scratch, right?

I can explain it in much more detail. We’re trying to create the models from scratch, at least from random weights. But the problem is the number of tokens you have in the pretraining stage. As I mentioned, if you look at the April reports from the DSLM and DSLM models, they show that even if you have an 8-billion-parameter model, you need at least 1 trillion tokens to have stable training. By stable, I mean training that gives you a high-quality base model.

In our case, as I mentioned, the deduplication stages and filtering give us around 200 billion tokens. That was too little to create the model from scratch, so we use, of course, Llama. Llama is much more closed now, but 1 year ago it was still open, and the Llama license was not as restrictive in the EU as it is now.

We usually use the Llama base models and the Mistral base models, and we continue pretraining them. We perform language adaptation, which means we continue pretraining them on our corpora of Polish text. After that, we have a new base model, and these new base models can be fine-tuned with SFT and used for direct preference optimization in the second and third stages.

As you mentioned, we’re not able to create a moderate-quality or good-enough-quality model without 1 billion tokens, and we don’t have 1 billion tokens in Polish. Now we’re doing some experiments with a mixture of languages. We may not only use Polish, but also other languages—a mixture of languages—to get this 1 billion tokens and do some pretraining from scratch. But the results will be in a few weeks.

Okay. On this language-adaptation step, I have a couple of questions. One is: do you continue to mix in English? Do you try to preserve the model’s ability to speak English, or after this language adaptation, does it only speak Polish?

Of course, when you have continual learning—because if you use the base model that was already created and perform language adaptation, there’s always the problem of forgetting in continual learning—some knowledge from previous learning stages is forgotten. But there is still some knowledge that persists. Even if we pretrain for a few epochs on our Polish data, the model still has the competencies, for example, to write something in English.

It’s not that we prune any other competencies in the other languages. They have somehow been forgotten because there is a problem of forgetting in continual learning, but generally speaking, we don’t prune them manually. We take the base model and perform language adaptation—continue pretraining for a few epochs—and in this way make its abilities in Polish better. Of course, there’s a trade-off: some other language abilities go down, but they aren’t pruned at all.

And that’s also where the world knowledge comes from, right? By world knowledge, obviously there are all these local details of life, right? I can maybe give worse examples; I’m sure you can give better examples than I would, but I’m thinking: What are the names of the Polish candies that kids like, and how does one file a document if you want to sell a car to somebody else? Surely there’s some filing process. All these little details are absorbed at that stage as well, right?

Yes, and you have to know that usually the general knowledge is retained. But then you will ask me about factuality—how good the model is with some facts, for example, regulatory issues and laws that change over time. It’s always a problem in any kind of language model.

You pretrain the language model, for example, on data that was current in March 2025, and you don’t have information in this memory store about changes in the law or regulations, or even situations, accidents, and the names of new politicians after that time point. But generally, when using a retrieval-augmented approach with current data, you have a retrieval stage and an up-to-date database or knowledge base.

It's much easier to stay updated. After all, when you get to the retrieval stage, you use the models to synthesize or generate the answer based on that. We use this in this case to address the factuality issue, because I think none of the providers of the LLMs, even the big ones, are able to pre-train them whenever a new interval of data is coming up.

Do you think this would work for companies? This has been a bit of a digression, but it's been a question I've had in my mind for a long time. Almost 2 years ago now, I did an episode of the podcast with a company called Mosaic LM. What they were doing, among other things, was this sort of continued pre-training for businesses.

They would go into a business and say, “Let's get all your tokens.” This could be all the Google Docs that you've got, the Slack history, and all these various things. Let's compile that, and now we can pre-train on that. Hopefully, the model will start to speak your internal native dialect of whatever language you're speaking.

I call it, as you mentioned, domain adaptation. We have some closed data. For example, I have internal closed data about my insurance, my customers, and some reports that are not open, and they would like to pre-train the model on this data to make it more adapted to the domain.

We have done such a project now for the biggest bank in Central and Eastern Europe. PKO is one of the biggest banks in Europe, and the biggest one in Central and Eastern Europe. We performed what you mentioned: domain adaptation. We adapted the models to their domain. They have their own domain data that is closed, and we continued pre-training the models on their data.

I think it's a very good approach. We proved this in different tasks, and the results varied. There are some tasks where this domain adaptation gives you a huge gain in quality, as well as in some financial measures and so on.

I think there is one remark I have to mention. Only huge companies have enough data for it to be worth performing domain adaptation. We know that, for example, after deduplication and filtering, you have to reach at least around 10 billion tokens. If you don't have 10 billion tokens, it's not worth performing domain adaptation.

I think if you would like to have 10 billion tokens in a domain corpus, you need at least 30 billion tokens before the deduplication and filtering stage. With 30 or 40 billion tokens, I think there are only a few companies—maybe not a few, but fewer than 100 in Europe—that have 100 billion tokens of closed internal data.

I guess my intuition is that it depends on what you count, right? I started a company that's 40 people, and I don't know how many tokens we have. But with all the Slack messages, all the Google Docs, all the Jira tickets, all the contract proposals that we've sent, and the revision history on all of those, I do feel like it adds up pretty quickly.

I guess maybe one of the barriers is exactly how deeply these companies are willing to mine into their own data. If they're actually willing to go get email data from their employees, for example, that seems like it could add a lot.

If you count this data, there may be billions of tokens in conversations, hundreds or thousands of agreements or agreement proposals. But if you sum them up, deduplicate them, and filter them out, maybe it's very hard to get 10 billion tokens.

Yeah, interesting. You can count it. Ten billion tokens is about 10 billion words. You can imagine that after deduplication, I believe you have to have at least 30 or 40 billion tokens to finally have 10 billion tokens in the domain corpus. I think it's not so easy to get 10 billion tokens. It seems easy for many, but when we start counting them, it's not so easy.

I assume that what you're counting probably doesn't include individual employees' email histories and all that sort of stuff. That stuff is kind of out of scope.

I think emails between, for example, the sales forces, the help center, the call center, or the sales teams can be used. But there are also some kinds of emails that are not suitable for use because of undefined problems with intellectual property, cybersecurity risks, and so on.

I don't think it's so easy to use any kind of email, because there is always the risk that some emails are too risky or, I would say, too sensitive.

Yeah. Yeah. Sensitive. Yeah, that's really interesting. That also makes me wonder if new organizational structures are going to be advantaged in some of these dimensions, because I totally understand the difficulty that would arise if you said, “Okay, hey, everybody, we know you've been working here for all these years and sending all these emails. By the way, we're going to take all that and put it into our training process.” You might have a revolt.

It would be different, but there is a problem: most organizations, even the big ones, are not aware of what kind of data they have. Before you adapt the AI or train the AI, first of all, you have to clean your data stores and identify what data you have. Is it creepy or not creepy? Is it high quality or low quality?

The data curation process or data organization process—everything around how to organize the data to find the high-quality fraction of the data—is a problem in itself. Companies very often try to integrate or deploy AI, or even train the AI, without this data curation and data organization process, and it usually collapses.

I think the most important thing is to be aware of what kind of data inventory you have, what level of quality the data has, and what data you can use without violating internal or external regulations. This is the most important step.

After this step, when you have properly identified data sets that are well described and well organized, you can start moving ahead with AI and invest in AI training and AI deployment based on that.

That's all very interesting. Thank you. Definitely great food for thought for me. What about just going back one more question on models? I know you had mentioned that some of the Chinese models have licenses that don't allow you to use them in the EU. Is that mostly the Miss Dilemma 3.4? It was the first model with this license, with a prohibition on using it in the European Union. But now there are key models, the Chinese models, that have the same restriction?

They are not able to use them in the European Union. I think it's a problem with the AI Act, because the second chapter of the AI Act was released in August 2025. It demands that, for general-purpose models, you create a model card: what data was used for training, how it was secured, what data sets were used, what resources were used to create them, and many other points in the model card.

I think those kinds of models don't want to provide evidence of what data was used by them and what the pre-training stages looked like.

If that weren't a problem, would you be open to using Chinese models, or are Chinese models not appealing for other reasons?

I think it depends on the task. For example, I would have a huge aversion to risk when I would like to use a Chinese model to create long-form histories, essays, emails, and so on. When you create longer forms of text, the possibility that censorship will be evident—noticeable—is going up.

But when you use some kind of model for tasks like understanding or analytical and extractive tasks—for example, to extract information from documents or get specific information from documents—I am open to using the Chinese model. The risk of data censorship in tasks that are typically analytical and extractive is very low.

That makes sense. Turning to compute and talent, the other 2 big legs of the AI stool, how AGI-pilled would you say the Polish government is? It's pretty remarkable that all this is going on at the governmental level already. I would say that speaks to a pretty situationally aware and generally agile government.

How committed is the government, or how big of a deal does the government understand this to be? Downstream of that, there are questions around how much funding there is and what the government is doing.

As you mentioned during the first minutes of our conversation, there are 3 pillars of the AI revolution. The first is data, as you mentioned before: data organization, data curation, and generally looking at data as the crucial point for AI training.

Next are compute powers, such as GPUs, AI factories, and other resources. This means data centers and so on. And then there is talent, meaning people. These 3 pillars, combined together, create the fuel for the AI revolution.

As I mentioned, I think we are currently focused mostly on the AI factories.

It means buying as many GPUs as we can and creating data centers for GPU tasks. Of course, we have some projects like BLOOM, which is a very good example. I think there are 2 or 3 projects similar to BLOOM in the European Union.

The Ministry of Digital Affairs has funds, and it funded a consortium of universities and institutes that are able to develop the models, the competencies, and so on. They somehow support the talent we have in this way because they have money for people. But I think we won't be able to compete with the U.S. market, because in the U.S. market, AI engineers are paid like NFL players.

I heard something like that: the best AI engineers or AI researchers have contracts like quarterbacks in the NFL. They are treated as stars. I don't think we have such maturity in making decisions to pay people—to overpay people—for their niche competencies.

I think it's much harder in the European Union, especially in Poland, to say, "What is the final objective function we would like to achieve?" For example, 10 million customers, or I don't know, $10 million each week in subscriptions. It's much harder for the public sector to define objectives that are easily monetized and easily evaluated.

I think in the U.S. they pay such huge contracts because they are able to evaluate, somehow, and even build in some margins or buffers for the future: how this talent can give you what kind of innovation, and how this innovation will pay you back.

I think this is the problem: in the U.S., everything is there. As I mentioned, when I was in Las Vegas at TLS, I mentioned that we were working on LLMs in the public ecosystem, that the Ministry of Digital Affairs had funded us, and that we had created the consortium. This was funded by the public sector.

Almost all of the people I met in Las Vegas were surprised that the public sector invests in LLMs. But in the U.S., it's hard to imagine that the public sector has enough intuition, enough knowledge, and enough money to invest in such a sexy and revolutionary topic as AI.

So, how do you think this will evolve over the next couple of years? Obviously, the amount of resources that the frontier companies are putting into their current and future models just continues to grow, right?

No. Even now, when we look, for example, at the GPT models and compare them, some reasoning abilities are improving in different kinds of models, but generally, GPT-5 was not such a huge improvement over GPT-4. Of course, there are some reasoning abilities, but generally, they say it's a plateau.

The models are improving, but after a certain level, the improvements are very steady. There is a horizontal improvement, not a vertical one. They are not improving dramatically; it's much more of a plateau, and the development is not as exciting as it used to be.

Remember ChatGPT in 2022, GPT-4 in 2023, and multimodality in 2024. There were moments in the history of the AI revolution that were so shocking and that made our imagination work in a way that led us to expect something huge and game-changing every year.

Now I think the models are improving much more steadily. There are no huge bumps. Generally, I think we are now starting to count the costs: the cost of energy and the cost of what the models are used for.

There is the same thing with people. When you meet someone, for example, you might spend 5 minutes with them and be bored—or maybe not bored. After all, you have to evaluate whether it's worth meeting these people or not. Maybe it's 5 minutes; maybe it's an hour.

Generally, I think a time is coming when we will try to evaluate the real cost of these tools, how we can use them, and what they give us if we use them properly. It's like the verification stage. I think the verification stage will give us a lot of information showing that we don't need very huge LLMs. We have to invest in small, localized LLMs, especially when we are working on private solutions.

I have mixed feelings about that. On the one hand, I do think the models that already exist are amazing artifacts. Very often, especially if you take the time to do the supervised fine-tuning and really dial in their performance, they can work perfectly well for all sorts of use cases.

At the same time, of course, we've got the leading companies saying, "We're nowhere near done. This is definitely going to keep going. You should expect more progress. We're going to have AI scientists. We're going to have AI researchers."

How much do you think your strategy depends on—or will change if it turns out that there's not so much a plateau, but that you do still see significant capability jumps, albeit with exponentially more resources required to achieve them?

How do you think you navigate that world if it really is the case that a $10 billion training run is actually that much better than a $1 billion training run?

I think the most important issue is the first one: the demands and what is expected by our customers. As I mentioned, we are very often biased by general-purpose benchmarks, but they don't match the benchmarks and expectations that businesses have.

For example, if you have a business and you know that you would like to use AI in certain places, where AI should behave according to at least a certain set of metrics, that gives you very good information about what kind of benchmarks you should create to evaluate what kind of model is able to reach those expected metrics.

I think this is the most important point. Very often, we analyze general-purpose benchmarks, which evaluate, for example, factuality, reasoning, and extractive and inferential competencies. But for a business, the problem is slightly different.

For example, a business might need something that writes a beautiful email to a customer, or something that creates an email that enables cross-selling. Very often, we don't know what should be done properly because businesses don't define their requirements very explicitly.

I would start from there: What would we like to improve in your business? What kinds of tasks would you like to send to the AI? Next, create benchmarks for these tasks. Then you choose the AI models, because I think most of the business cases I have seen don't demand reasoning.

You can do it with normal AI models. You don't need the reasoning stage. I think we should slow down a little bit and analyze what needs to be done and what especially has a huge business impact—not only because it's sexy and public relations likes that, but because it gives money to the business or creates savings.

There are 2 trains, but they are not in sequence, one after the other; they are next to each other. One is much faster, and the second is going at its own pace, but they are not—as I mentioned—very often, there is no crossing between them. There are 2 roads, but the crossing is very far from us.

Yeah, that's really interesting. Who are your allies in this? You mentioned using multiple languages, and I assume that's in some sort of partnership with perhaps other national institutes in neighboring countries.

I'm curious as to how you think the international dynamics will play out. Historically, in the Cold War, we had the U.S. and the USSR, and the 2 great powers were engaged in proxy conflicts and various other conflicts all over the world. A lot of other countries understandably said, "From our perspective, we don't really want to be in either of your camps."

There was a movement of countries that said, "We don't really want to be in either of your camps. We would rather just be independent. The beef that you guys have between yourselves—we don't really want to be a pawn in that game."

Now it's the U.S. and China, obviously, that are the 2 big poles of AI power. How do you think countries—say, countries 3 through 193 on the AI power rankings—will react?

Do you see alliances forming, or countries working together to share resources and data sets to try to create some sort of third way in the AI space?

There are some movements in the European Union. For example, they are bringing together people from different countries and trying to do something together. But I think it's a huge problem that, generally, when you would like to get very fast products or very fast outcomes, you have to centralize.

The problem is always the same. Generally, the best way is to have a federation. Everything should be spread out. You should have different people in different countries collaborating with each other and building things in a way that improves well-being everywhere, in some kind of distributed but normalized way.

But generally, when you want very fast outcomes and want to have products in months, not years, you usually have to centralize the assets in one place. And there is a problem because there are 2 different ways of doing things. If you want to do it in an ideal way, you should create unions: unions of countries, unions of states, unions of partners, networks, and consortia with hundreds of stakeholders to gather this knowledge everywhere and distribute this knowledge and power everywhere.

But usually, when you have to get outputs very fast, you have to centralize. The problem is that these are 2 opposite ways, and everybody does it the same way with both of them. I think this is the problem, because when you have huge pressure to produce new models, you always prefer centralization.

Look at Silicon Valley. You have the huge United States, but 90% of the startups are in Silicon Valley. That is centralization in one place where there is money, assets, and everything else. But from an economic point of view, the best place would be to distribute these companies across the whole United States.

If you want to monetize something and get very fast outputs, you have to centralize. But generally, the best approach for the economy and social aspects is to distribute things in a normalized way across the country and across the continent.

What about geopolitics? I think there are still 2 players: China and the United States. They have the 2 biggest economies and they have the money. I heard that Chinese companies pay AI researchers the same money as companies in the United States. Their contracts are currently somehow similar, which means they pay very well.

They don't have to compete for people in the same way. People will not necessarily be taken by U.S. companies because they are well paid in China. In Europe, I don't think there is still a strong player. There is Mistral, the European-funded startup. It is not a startup anymore, but it was a startup 2 or 3 years ago. Now I heard that 30% or 40% of its shares are held by Microsoft.

It is not as open as it used to be because there are some stakeholders from the United States. I think the problem is slightly different. The question is whether China and the United States are going to remain rivals, or whether there is still a chance for cooperation. Maybe there is still a chance for cooperation.

From your lips to God's ears. Maybe just one little follow-up. I think this has been excellent, and I really appreciate all your time and all these thoughtful answers.

Is there anything that you have seen that, on a technical or sociotechnical level, can help with cooperation around decentralized AI? I'm thinking about things like the NEAR Protocol. I recently did an episode with the creator of the NEAR Protocol, Illia Polosukhin. There's also the Intelligent Internet, which is in Mod Methuselah's project [?]. There are others as well.

These things have the idea that, if we create the right scheme, it might be somewhat cryptographically enabled.

There is a topic called federated learning. You can use different datasets that are somehow anonymized and secured, in a way that enables networks to identify sensitive data while still using the data to train your models. There are different kinds of ideas around federated learning.

Generally, I don't know of any huge deployments of these approaches, even though it is a very good approach. You could have networks or federations cooperating with each other and sharing data in a secured way. But I think that, in business and economics, we are still not at the level where we can use it.

I think we are still facing the problem that companies mostly don't know what data they have, what the quality of their data is, or what the value of their data is. If you aren't able to measure your own in-house data repositories, how can you go further and create some mixture of data or a network of data repositories?

I think this may happen in the future. There is a chance that there will be distributed data repositories with secure access levels, used by huge consortia. But I think this is not for the next year.

So much depends on whether there really is a plateau, or whether the frontier companies are just going to continue scaling successfully.

I heard that the plateau is caused by a lack of organic data. The biggest companies have already collected almost all the organic data that could be collected from the internet. Some of the companies even scanned books that hadn't appeared on the internet, to make these organic data gains relevant.

But there is still a problem. I think the repository of organic data is almost full.

That's why we're now seeing all these simulated worlds, and the strategies to overcome that are definitely going to be fascinating to watch. I, for one, will bet on them working, but it does, to a certain degree, certainly remain to be seen.

Again, this has been a really fascinating conversation. It's been awesome to get your perspective. Is there anything else you want to share? Anything we didn't touch on that you want to comment on before we break for today?

I can recommend our arXiv paper on the PLLuM family of models. It was released in the first days of November, and I recommend that readers and viewers of this podcast look inside this paper. The PLLuM family is the arXiv title of the paper. It is almost 100 pages, and the P, of course, stands for Polish: Polish large language models.

Marek Kozlowski, this has been amazing. Thank you for being part of The Cognitive Revolution.

Yeah, thank you very much.