Agent Engineering with Pydantic + Graphs — with Samuel Colvin, CEO of Pydantic Logfire

This is Alessio Fanelli, partner and CTO at Decibel Partners, and I'm joined by my co-host, swyx, founder of Smol AI.

Good morning.

Today, we're very excited to have Sam Colvin join us from Pydantic AI. Welcome.

Thank you so much for having me. It's great to be here.

Sam, I heard that Pydantic is all we need. Is that true?

I would say you might need Pydantic AI and Logfire as well, but it gets you a long way, that's for sure.

Pydantic almost needs no introduction. It's almost 300 million downloads in December, and obviously, in the previous podcasts and discussions we've had with Jason Liu, he's been a big fan and promoter of Pydantic.

It's weird because I didn't create Pydantic originally for uses in AI. It obviously predates LLMs, but we've been lucky that it's been picked up by that community and used so widely. Maybe we'll hear it right from you: what is Pydantic, and maybe a little bit of the origin story?

The best name for it, which is not quite right, is a validation library. We get some tension around that name because it doesn't just do validation; it will do coercion by default. We now have strict mode, so you can disable that coercion.

By default, if you say you want an integer field and you get a string of 123, it will convert it to 123, along with a bunch of other sensible conversions. As you can imagine, the semantics around exactly when you convert and when you don't are complicated. But because of that, it's more than just validation.

Back in 2017, when I first started it, the different thing it was doing was using type hints to define your schema. That was controversial at the time. It was genuinely disapproved of by some people. I think the success of Pydantic and libraries like FastAPI that build on top of it means that today, that's no longer controversial in Python. Indeed, lots of other people have copied that approach.

It's a data-validation library that uses type hints for the most part and does all the other things you want, like serialization, on top of that. But that's the core.

Do you have any fun stories about how JSON Schema ended up being the structured-output standard for LLMs? Were you involved in any of those discussions? I know OpenAI was one of the early adopters. Did they reach out to you? Was there some kind of structured-output council in open source that people were talking about, or was it just random?

Very much not so. I originally didn't implement JSON Schema inside Pydantic, and then Sebastian Ramirez's FastAPI came along. The first I ever heard of him was over a weekend, when I got 50 emails from him—or 50 emails as he was committing to Pydantic—adding JSON Schema, long before version 1.

The reason it was added was for OpenAPI, which is obviously closely linked to JSON Schema. I don't know why JSON Schema got picked up and used by OpenAI. It was obviously convenient for us because, as I mentioned, not only can you do the validation, but because Pydantic will generate the JSON Schema, it can be one source of truth for structured outputs and tools.

Before we dive further into the AI side of things, something I'm mildly curious about: obviously, there's Zod in JavaScript land. Every now and then there's a new validation library that's in vogue and takes over for a few years, and then something else comes along. Is Pydantic done?

I've just come off a call where we were redesigning some of the internal bits. There will be a version 3 at some point, which will not break people's code half as much as version 2. Version 2 was the massive rewrite in Rust, but it also fixed all the things that had been broken since version 0 that we didn't fix in version 1 because it was a side project.

We have plans to move some of the data storage into Rust types after validation, rather than converting to Python types. If you were doing validation and then serialization, you would never have to go via a Python type. We reckon that could give us another 3–5x speedup. That's probably the biggest thing, along with changing how easy it is to extend Pydantic and define how particular types—for example, NumPy arrays—are validated and serialized.

There's also work going on in Jiter, the JSON library that does JSON parsing in Rust. It has a SIMD implementation at the moment only for amd64, so we need to add SIMD for other instruction sets. There's a bunch more we can do on performance.

I don't think we're going to revolutionize Pydantic, but it's going to continue to get faster and, hopefully, allow people to do more advanced things. We might add a binary format like CBOR for serialization, for when you just want to put the data into a database and probably load it again from Pydantic. There are some things that will come along, but for the most part it should just get faster and cleaner.

From a focus perspective, I guess, as a founder too, how did you think about the AI interest rising? How did you prioritize what was worth going into more deeply? What was your early experience with LLMs, and when did you figure out that this was something you should take seriously and focus more resources on?

I think the parallel question is that Pydantic is weird because it existed before I was starting a company. I was working on it in my spare time, and then at the beginning of 2022 I started working on the rewrite in Rust. I worked on that full-time for a year and a half, and then, once we started the company, people came and joined.

It was a weird project because that would never get signed off inside a startup: “We're going to have 3 engineers work full-time for a year in Python and Rust, writing 30,000 lines of Rust just to release a free, open-source Python library.” The result of that has been excellent for us as a company. It's made us remain entirely relevant. Pydantic is not just used in the SDKs of all the AI libraries.

I can't say which one, but one of the big foundational-model companies upgraded from Pydantic v1 to v2, and their number-one internal performance metric was time to first token. That went down by 20%. You think about all the actual AI going on inside, and yet at least 20% of the CPU usage—or at least the latency—inside requests was actually Pydantic. That shows how widely it's used.

In answer to your question about how we prioritize AI, the honest truth is that we've spent a lot of the last year and a half building good general-purpose observability inside Logfire and making Pydantic good for general-purpose use cases. AI has kind of come to us.

The appetite, both in Pydantic and in Logfire, to build with AI is enormous because it makes sense. If you're starting a new greenfield project in Python today, what's the chance that you're using GenAI? Globally, let's say 80%. Obviously, it's 100% in California, but even worldwide it's probably 80%.

Everyone needs this stuff, and there's so much yet to be figured out—so much space to do things better in the ecosystem. Going and implementing a database that's better than Postgres is a Sisyphean task, whereas building tools that are better for GenAI than some of what's available now is not very difficult, putting the actual models themselves to one side.

You released Pydantic AI recently, which is an agent framework. Early on, I would say everybody—LangChain, Instructor, and a lot of these frameworks—were trying to use Pydantic to be better. What was the decision behind doing your own framework? Were there design decisions you disagreed with, or workloads that you thought people weren't supporting well?

It wasn't so much about design or workflow, although I think there are some things we've done differently. Looking in general at the ecosystem of agent frameworks, the engineering quality is far below that of the rest of the Python ecosystem.

There's a bunch of stuff that we've learned how to do over the last 20 years of building Python libraries and writing Python code that seems to be abandoned when people build agent frameworks now. I can respect that, particularly in the very first agent frameworks like LangChain, where people were literally figuring out how to do this stuff. It's completely understandable that you would skip some standard best practices to get something built.

I'm shocked by the quality of some of the agent frameworks that have come out recently from well-respected names. It seems to be opportunism, and I have little time for that. The early ones, though, were just figuring out how to do things, and just as lots of people have learned from Pydantic, we were able to learn a bit from them.

The gap we saw, and the thing we were frustrated by, was production readiness. That means things like type checking. Even when type checking makes things hard, Pydantic AI has a lot of generics. You probably find it easier to use if you've written a bit of Rust and really understand generics. We don't claim that makes it the easiest thing to use in all cases, but we think it makes it good for production applications in big systems where type checking is a no-brainer in Python.

There are also a bunch of things we've learned from maintaining Pydantic over the years. Every single example in the Pydantic AI documentation is run as part of the tests, and every single print output within an example is checked during testing. It will always be up to date.

Then there are things that are standard best practices within the rest of the Python ecosystem but aren't followed surprisingly often by some AI libraries: coverage, linting, type checking, and so on. I think these are no-brainers, but weirdly, they're not followed by some of the other libraries.

Can you give us an overview of the framework itself? There are the LLM-calling frameworks, the multi-agent frameworks, and the workflow frameworks. What does Pydantic AI do?

I glaze over a bit when I hear all of the different sorts of frameworks. When I built Pydantic, Logfire, and Pydantic AI, my methodology was not to research and review all of the other things. I work out what I want and go build it. Then feedback comes, and we adjust.

The fundamental building block of Pydantic AI is agents. The exact definition of agents and how you want to define them is obviously ambiguous. They are probably sort of agentlets, although we wouldn't want to rename them that. The point is that you probably build them together to build something that most people would call an agent.

In our case, an agent has things like a system prompt, some tools, and a structured return type if you want that. That covers the vast majority of cases.

There are situations where you want to go further, with the most complex workflows where you want graphs. I resisted graphs for quite a long time. I was of the opinion that you didn't need them and could use standard Python flow control to do all of that stuff. I had a few arguments with people, but I basically came around to seeing that graphs are useful.

The problem is that, by default, they're not type-safe. If you have an `add_edge` method where you give the names of different edges, there's no type checking. Even if you do some of that, not all graph libraries are AI-specific. There's a graph library called Burr that does runtime type checking—ironically using Pydantic—to try to make up for the fact that, fundamentally, its graphs aren't type-safe.

I like Pydantic, but it's not a real solution to have to run the code to see if it's safe. There's a reason static type checking is so powerful. Eventually, we came up with a system of using, normally, data classes to define nodes, where you return the next node you want to call. We're able to introspect the return type of a node to build the graph, so the graph is inherently type-safe.

Once we got that right, I was—and am—incredibly excited about graphs. I think there are masses of use cases for them, both in GenAI and other development. Software is all going to have to interact with GenAI. There won't be a web department in a company anymore; all the developers will be building for the web, building with databases, and doing the same thing with GenAI.

I see on your docs that you call an agent a container that contains a system prompt function, tool, structured result, dependency type, model, and model settings. Are the graphs, in your mind, different agents? Are they different prompts for the same agent? What are the structures in your mind?

We were compelled enough by graphs, once we got them right, that we merged a PR this morning. Our implementation, without changing its API at all, is now actually a graph under the hood. It's built using our graph library.

Graphs are basically a lower-level tool that allows you to build these complex workflows. Our agents are technically one of the many graphs you could build. We just happen to build that one for you because it's a very common one. There are cases where you need more complex workflows where the current agent assumptions don't work, and that's where you can use graphs to build more complex things.

You said you were cynical about graphs. What changed your mind specifically?

People kept giving me examples of things they wanted to use graphs for, and my argument that you could do that with standard Python flow control became less and less compelling. I've maintained systems that end up with spaghetti code, and I could see the appeal of this structured way of defining the workflow of my code.

It's also really neat that, just from your code and your type hints, you can get a Mermaid diagram that defines exactly what can happen.

You do have a very neat implementation of inferring the graph from type hints. Is that what you would call it?

Exactly. If you look at the internal logic of actually running a graph, it's incredibly simple. It's basically: call a node, get a node back, call that node, get a node back, call that node. If you get an end, you're done.

We'll add support soon for storage, so you can store the state between each node that's run. Then you could distribute a graph and run it across compute.

The other bit that's really valuable is across time. It's all very well when Claude gives you an example with a lovely, enormous Mermaid chart for a workflow, say, managing returns for an e-commerce company. But some of those lines are literally one function calling another, while others mean, “Wait 6 days for the customer to print their piece of paper and put it in the post.”

If you're writing a demo project or proof of concept, that's fine because you can just say, “And now we call this function.” But in real life, that doesn't work. How do we manage the concept of being able to start somewhere else in our code?

This graph implementation makes it incredibly easy. You just pass the node that's the starting point for carrying on the graph, and it continues to run. It's things like that where I can imagine how systems I've built in the past would have been fundamentally easier to understand if we'd done them with graphs.

When you say “imagine,” does Pydantic AI actually resume 6 days later, like you said, or is that a theoretical thing we can do someday?

I think it's basically a Q&A pattern. There's an AI asking the user a question, and you then call the graph again to continue the conversation. It instantiates the node and calls the graph with that node again.

We don't yet have the logic for storing state in the database between individual nodes. We're going to add that soon, but the rest of it is basically there.

It does make me think that you're not only competing with LangChain and, obviously, Instructor; now you're going into the more orchestration-oriented tools like Airflow, Prefect, and Dagster.

We're good friends with the Prefect guys, and Temporal has the same investors as us. I'm sure my investor Bill would not be too happy if I said, “By the way, as well as trying to take on Datadog, we're also going to take on Temporal and everyone else doing that.”

Obviously, we're not doing all of the infrastructure for deploying that yet, at least. We're just building a Python library. What's crazy about our graph implementation is that, sure, there's a bit of magic in introspecting the return type and extracting things from unions, but the actual calls are literally just calling a function, getting something back, and calling the next function. It's incredibly simple and therefore easy to maintain.

The question is how useful it is. I don't know yet. We have to find out. We've had a slew of people joining our Slack over the last few days asking how good Pydantic AI is versus LangChain. I refuse to answer. That's your job—to go find that out. We've built a thing that I'm compelled by, but I'm obviously biased. The ecosystem will work out what the useful tools are.

Bill was my board member when I was at Temporal. I think that, generally, having been a workflow-engine investor and participant in this space, it's a big space. Everyone needs different flavors of orchestration.

The one thing I would say is that, as a library, you don't have that much control over the infrastructure. I do like the idea that each new agent—or whatever unit of work you call it—should spin up in isolated boundaries. Ideally, you want to spin out its own little container. With Pydantic AI, I think everything runs in the same process.

I agree with you 100%. It would work now, in theory. As long as you can serialize the calls to the next node, all of the different containers just have to have the same code.

I'm super excited about Cloudflare Workers running Python and being able to install dependencies. If Cloudflare would give me my invitation to the private beta, we'd be exploring that right now. I'm really excited about that as a compute layer for some of this stuff.

You can run everything as an individual worker function and distribute it, and it's resilient to failure. It spins up like a thousand businesses simultaneously. You want it to be truly serverless.

I know we have some Cloudflare friends listening, so hopefully they'll get you to the front of the line.

I was in Cloudflare's office last week, shouting at them about other things that frustrate me. I have a love-hate relationship with Cloudflare. Their technology is awesome, but because I use it all the time, I get frustrated. I'm sure I'll get there soon.

Is the Python support full? I wasn't fully aware of the status of that.

Pyodide, which is Python running inside the browser through WebAssembly, is supported by Cloudflare now. They're having some struggles working out how to manage dependencies that have binaries, particularly Pydantic.

Because these Workers can have thousands of instances on a given metal machine, you don't want a different copy of Pydantic in each one. You want shared memory for all the different Pydantic installations. That's the thing they're working out.

Hood Chatham, who's a friend of mine and one of the primary maintainers of Pyodide, works for Cloudflare, and that's basically what he's doing: working out how to get Python running on Cloudflare's network.

The nice thing is that your binary is written in Rust, which also compiles to WebAssembly. Maybe there's a way to build a different version of Pydantic that ships with whatever driver Cloudflare Workers need.

So you already have a different build of Pydantic that compiles to WebAssembly?

Exactly. Pyodide has builds for Pydantic Core and for things like NumPy—basically all of the popular binary libraries. You're compiling Rust to WebAssembly and calling that shared library from Python. It's unbelievably complicated, but it works.

Staying on graphs a little bit longer, I wanted to get to some of the other features in Pydantic AI. I see in your docs that there are 4 levels of agents: single agents, agent delegation, programmatic agent handoff—which seems to be what OpenAI Swarm would be—and graph-based control flow. Would you say those are the mental hierarchy of how these things work?

Roughly.

You had some expression around OpenAI Swarm.

OpenAI has gotten in touch with me and basically said—maybe I'm not supposed to say this—that Pydantic AI looks like what Swarm would become if it were production-ready.

That makes sense. The desire to give people the same feeling they were getting from Swarm led us to implement graphs. My initial answer—just call the next agent with Python code—wasn't satisfactory to people, so we needed a better answer. That led us to graphs.

It's a minimal viable graph in some sense. What are the shapes of graphs that people should know?

I think Anthropic did a very good public service, and was surprisingly influential, with its blog post “Building effective agents.” We actually have the authors coming to speak at my conference in New York, which I think you're giving a workshop at.

I'm trying to work it out, but yes.

Tell me if you're not, but I think so.

I'm curious whether you have community names or a top 5 set of patterns or graphs.

I don't have a top 5 set of graph patterns. I would love to see what people are building with them, but it's only been a couple of weeks. Because they're relatively unopinionated about what you can do with them, they don't have the structure to support specific names as much as some other systems do.

Our agents—which have a name, although I can't remember what it is—are basically a system of deciding which tool to call, going back to the center, deciding which tool to call, going back to the center, and then exiting. That's one form of graph. Our agents are effectively one implementation of a graph, which is why they're now using graphs under the hood.

It will be interesting to see over the next few years whether we end up with predefined graph names or graph structures, whether it's just, “I built a graph,” or whether graphs don't match people's mental image of what they want and die away. We'll see.

There's always an appeal to this. Every developer eventually gets graph religion and says, “Everything is a graph.” Then they probably over-rotate and go too far into graphs, learn a whole bunch of DSLs, and say, “Actually, I didn't need this,” before scaling back.

I'm at the beginning of that process. I'm currently a graph maximalist, although I can't say I've put any of them into production yet.

This also has philosophical connections with work coming out of UC Berkeley on compound AI systems. I don't know if you know or care about that. This is the Gartner world of things, where they need some kind of industry terminology to sell it to enterprises.

I probably should know about it. I should probably get better at selling to enterprises, but no, I don't know about that stuff yet.

The argument is that instead of putting everything in one model, you have more control and potentially more observability if you break everything out into small models and compose them together. Obviously, then you need an orchestration framework to do that.

It makes complete sense. One of the things we've seen with agents is that they work well when they work well, but when they go wrong, even if you have the observability through Logfire to see what was going on, you don't have a nice hook point to say, “Hang on, this has all gone wrong.”

You have a relatively blunt instrument: erroring when you exceed some kind of limit. What you need to be able to do is iterate through these runs so that you can have your own control flow. You can say, “Okay, we've gone too far.”

That's one of the neat things about our graph implementation. You can call the next node in a loop rather than just running the full graph, so you have the opportunity to break out of it.

I'll drop one thing here. We're not going to resolve this, but then we can move on to the next thing. The common way that we developers talk about this is that machine-learning researchers look at us and laugh. They just train a bigger model and wipe us out in the next training run.

There's a certain amount of us fighting the bitter lesson. We're fighting AGI, and when AGI arrives, all of this will obviously go away. On Latent Space, we don't really discuss that because AGI is a hand-wavy concept that isn't super relevant. But we have to respect that you could do chain-of-thought with graphs and manually orchestrate a graph that reflects, thinks about whether you need more inference-time compute, thinks again, and scales that up—or you could train Strawberry into DeepSeek R1.

I saw someone saying recently that they were really optimistic about agents because models are getting faster exponentially. I took a certain amount of self-control not to explain that it wasn't exponential.

My main point was that if models are getting faster as quickly as you say they are, then we don't need agents or any of these abstraction layers. We can just give the model access to the internet, cross our fingers, and hope for the best.

Agents, agent frameworks, and graphs are basically making up for the fact that, right now, models aren't that clever. It's the same as if you're running a customer-service business and have lots of people answering telephones: the less well-trained they are, the less you trust them, and the more you need to give them a script.

If you're running a bank and have lots of customer-service people you don't trust that much, you tell them exactly what to say. If you're doing high-net-worth banking, you employ people you think are going to be charming to other rich people and send them off to have coffee with people.

The same is true of models. The more intelligent they are, the less we need to tell them how to structure what they do and constrain the routes they take.

I'm happy to move on. There are other parts of Pydantic AI worth commenting on, and this is my last rabbit hole, I promise. Obviously, every framework needs to have its model-adapter layer so that you can easily swap from OpenAI to Claude to Grok.

You also have Google Generative Language API, which I didn't really know about until I saw it in your docs. That's the Generative Language API. I assume that's AI Studio?

Yes. Google doesn't have good names for these things. Vertex is very clear, and that seems to be the API that some of the others use. Vertex is fine, but the Generative Language API returns a 503 about 20% of the time.

We have another example of going the extra mile in engineering: on every commit to main, we run tests against the live models. Not many tests, but a handful of them. Last week, GLA 1 was failing every single run—one of the tests would fail. We may even have commented that one out for now.

All of the models fail more often than you might expect, but that one seems to be particularly likely to fail. Vertex uses the same API but is much more reliable.

My argument is that versions of this appear in LangChain, and every framework has to have its own version of this. I would much rather you adopt a layer like LiteLLM, or in JavaScript, Portkey. That's their job. They focus on that one thing and normalize the APIs for you. All new models are automatically added, and you don't have to duplicate this inside your framework.

For example, if I wanted to use DeepSeek, I'd be out of luck because Pydantic AI doesn't have DeepSeek yet.

It does.

Okay, but you know what I mean. Should this live in your code, or should it live in a layer that's more like an API gateway—a defined piece of infrastructure that people have?

If a company respected by everyone had come along and done this at the right time—if it had said, “We're going to be the universal AI layer”—that would have been a credible thing to do. I've heard varying reports about whether LiteLLM is the truth, and it didn't seem to have exactly the type safety we needed.

Also, as I understand it—and I haven't looked into it in great detail—part of its business model is proxying requests through its own system to generalize them. That would be an enormous put-off for an awful lot of people.

Honestly, I don't think it's that much work to unify the models. I understand your point, but I think everyone is centralizing around the OpenAI API as the standard. DeepSeek supports that, Grok supports that, and Ollama supports that.

If there is a universal library right now, it's more or less the OpenAI SDK. It's very high quality, well typed, and uses Pydantic, so I'm biased, but I think it's pretty well respected.

There are different ways to do this because it's not just about normalizing the APIs. You also have to do secret management and all that stuff. There's also Vertex and Bedrock, which, to one extent or another, host multiple models. They don't unify the API, but they do unify—or at least, as I understand it, they unify—the model hosting. We're halfway through doing Bedrock, so I don't know it that well.

I'm surprised they don't unify the API. That seems like something I would do.

I guess the other side of model routing and picking models is evals. How do you figure out which one you should be using? I know you have very good support for mocking in unit tests, which a lot of other frameworks don't do.

My favorite Ruby library is VCR because it lets me store HTTP requests and replay them. I'll skip that part. You also have this test-model approach, where you try to figure out what the model might respond through Python without actually calling the model, and the function-model approach, where people can customize outputs. Any other fun stories from there, or is it just what you see is what you get?

On those two, I think what you see is what you get. On evals, watch this space. It's something we are going to be working on soon.

I was cynical about it for some time, and I still have some cynicism about some of the things called evals. It's unfortunate that so many different things are called evals. It would be nice if we could agree on what they are and what they aren't.

But it's an important space, and it's something we're going to be working on in both Pydantic AI and Logfire to support better. It's an unsolved problem.

You say in your docs that anyone who claims to know for sure exactly how your eval should be defined can safely be ignored.

I was thinking about vibe evals, which are what you do when you're building. You can't really test that many times to get statistical significance. Then there are production evals. You also have Logfire, which is your observability product. I tried it before, and it's very nice.

What are some of the learnings from building an observability tool for LLMs? As people think about evals, what are the right things to measure, and how many samples do you need before you can start making decisions?

I'm not the best person to answer that, so I'm not going to tell you that I know the answer to the exact number of samples. We can do back-of-the-envelope statistical calculations and work out that having 30 probably gives you most of the statistical value of having 200, for 15% of the work. But the exact question of how many examples you need is much harder to answer because it's deep within how models operate.

One of the reasons we built Logfire the way we did, allowing you to write SQL directly against your data, is precisely because we know we don't know the answers. Allowing people to innovate on how they're going to consume and process that data is valuable.

Even if we offer an eval framework on top of Logfire, it won't be right in all regards. We want people to be able to innovate. Being able to write your own SQL, connect to the API, and query the data like a database allows people to innovate—and it allows us to do the same thing.

We do a bunch of testing by writing SQL directly against Logfire, as any user could.

The other really interesting thing in observability is that OpenTelemetry is centralizing around semantic conventions for GenAI. It's a relatively new project, and a lot of it is still being added, but the idea is to unify how SDKs and agent frameworks send observability data to any OpenTelemetry endpoint.

That unification allows us to compare different libraries and models much better. The work is at a very early stage. One of the things we're going to be working on soon is implementing those semantic conventions properly. I suspect Pydantic AI will be the first agent framework to do that because we control Pydantic AI and can say that it's important for observability.

Most of the other agent frameworks aren't maintained by people trying to do observability, with the exception of LangChain, which has its own observability platform. They chose not to go down the OpenTelemetry route, so they're plowing their own furrow and are even further away from standardization.

Can you give us a quick overview of how OpenTelemetry fits into the AI workflow? There's the question of whether a trace and a span represent an LLM call, an agent, or the broader thing you're tracking.

There was a pull request from someone at IBM talking about remote agents and trying to support that concept within GenAI. I'm not particularly compelled by that because I don't think it's the common use case, but I suppose it's fine for it to be there.

The majority of the work is defining how you instrument a given call to an LLM: the actual LLM call, what data you would send to your telemetry provider, and how you would structure that data. Apart from the slightly odd remote-agent work, most of the agent-level considerations haven't yet been implemented or decided.

There's some ambiguity, but what's good about OpenTelemetry is that you can ultimately send whatever attributes you like.

If you think about observability traditionally, everyone would say that observability data is important and must be kept safe. But companies work very hard to ensure that nothing too sensitive ends up in their observability data.

If you're a doctor in a hospital and you search for a drug for an STI, the SQL might be sent to the observability provider, but none of the parameters would be. It wouldn't have the patient number, the patient's name, or the drug. With GenAI, that distinction doesn't exist because everything is mixed together in the text.

If that same patient asks an LLM which drug they should take or how to stop smoking, you can't extract the personally identifiable information and not send it to the observability platform. The sensitivity of the data that's going to end up in observability platforms will be a different order of magnitude from what you would normally send to Datadog.

Of course, you can make a mistake and send someone's password or card number to Datadog, but that would be seen as a mistake. In GenAI, a lot of data is going to be sent. I think that's why companies like LangSmith and others are trying to offer observability on-premises.

There are companies that are happy for Datadog to be cloud-hosted but want self-hosting for GenAI observability.

Is that what you're doing today? In each span, you have the number of tokens and the context. Are you storing everything and then going to offer self-hosting for the platform?

We offer scrubbing roughly equivalent to what the other observability platforms have. If we see “password” as the key, we won't send the value. But as I said, that doesn't really work in GenAI. We're accepting that we'll have to store a lot of data, and then we'll offer self-hosting for the people who can afford it and need it.

This is the first time that most workloads' performance depends on a third party. If you're looking at Datadog data, it's usually your application driving latency, memory usage, and all of that. Here, you can have spans that take a long time because the Generative Language API isn't working or because OpenAI is overwhelmed.

Do you do anything there? Since the provider is effectively the same across customers, are you trying to surface these things for people and say, “This was a very slow span, but all customers using OpenAI right now are seeing the same thing, so maybe don't worry about it”?

We do a few things that people don't generally do in OpenTelemetry. We send information at the beginning of a span—as well as when it finishes. By default, OpenTelemetry only sends you data when the span finishes.

If you think about a request that might take 20 seconds, even if some of the intermediate spans finished earlier, you can't place them on the page until you get the top-level span. If you're using standard OpenTelemetry, you can't show anything until those requests are finished.

When requests take a few hundred milliseconds, it doesn't really matter. But when you're making GenAI calls or running a batch job that might take 30 minutes, that latency in seeing the span is crippling for understanding your application. We've done some slightly complex work to send data about a span as it starts.

What are your thoughts on everyone else trying to build on top of OpenTelemetry in different languages? There's the OpenTelemetry project, which doesn't exactly roll off the tongue. How do you see the future of these tools? Is everybody going to have to build their own OpenTelemetry observability layer to sell?

We're not going to instrument the OpenAI SDK with the new semantic conventions because, at some point, that's going to happen and live inside OpenTelemetry. We might help with it, but we're a tiny team and don't have time to do all of that work.

OpenTelemetry is an interesting project, but I suspect that eventually most of the semantic instrumentation of the major SDKs will live inside the main OpenTelemetry repositories.

What happens with the agent frameworks, and what data you need at the framework level to get the context, is unclear. I don't think we know the answer yet.

I was on an OpenTelemetry call last week talking about GenAI. Someone from Arize was talking about the challenges they have trying to get OpenTelemetry data out of LangChain, where it's not natively implemented. They're having a tough time.

I hadn't realized this before, but we're lucky to primarily be talking about our own agent framework, where we have control, rather than trying to instrument other people's.

I actually didn't know about this semantic-conventions project. It looks like it's merged into the main OpenTelemetry repository. What should people know about it? I had never heard of it before.

I think it looks like a great start. There are some unknowns around how you send the messages that go back and forth, which is the most important thing of all. That has moved out of attributes and into OpenTelemetry events.

OpenTelemetry events, in turn, are moving from being on a span to being their own top-level API where you send data. There's still a lot of churn going on.

I'm impressed by how fast the OpenTelemetry community is moving on this project. Everyone understands that it's important, and people are crying out for instrumentation, so I'm pleasantly surprised at how quickly it's moving.

I'm just browsing through the specifics. I can already see that this basically bakes in whatever the previous paradigm was, so now they have, like, GenAI...

GenAI usage: prompt tokens and completion tokens, and obviously now we have reasoning tokens as well. Then there’s only 1 form of sampling, which is top_p. You’re basically baking in, or sort of reifying, things that you think are important today, but it’s not a super foolproof way of doing this for the future.

Yeah, I mean, that’s what’s neat about OTel: you can always go and send another attribute, and that’s fine. It’s just that there are a bunch that are agreed on. To come back to your previous point about whether or not we should be relying on 1 centralized abstraction layer, this stuff is moving so fast that if you start relying on someone else’s standard, you risk basically falling behind because you’re relying on someone else to keep things up to date, or you fall behind because you’ve got other things going on.

Yeah, yeah, that’s fair. That’s fair. Any other observations just about building Logfire? Actually, let’s just talk about this: you announced Logfire. I was kind of only familiar with Logfire because of your Series A announcement. I actually thought you were making a separate company. I remember some amount of confusion with you when that came out.

So, to be clear, it’s Pydantic Logfire, and the company is 1 company that has kind of 2 products: an open-source thing and an observability thing, correct? I was just kind of curious: any learnings building Logfire? The classic question is, do you use ClickHouse? Is this the standard persistence layer? Any learnings doing that?

We don’t use ClickHouse. We started building a database with ClickHouse, moved off ClickHouse onto Timescale, which is a Postgres extension for analytical databases, and then moved off Timescale onto DataFusion. We’re basically now building—it’s DataFusion, but it’s kind of our own database. Bill is not entirely happy that we went through 3 databases before we chose 1. I’ll say that, but we’ve got to the right one in the end, I think.

We could have realized that Timescale wasn’t right. I think ClickHouse and Timescale both taught us a lot, and we’re in a great place now, but it’s been a real journey on the database in particular.

Okay, so as a database nerd, I have to double-click on this. ClickHouse is supposed to be the ideal backend for anything like this, and then moving from ClickHouse to Timescale is another counterintuitive move that I didn’t expect, because Timescale was an extension on top of Postgres, not super meant for high-volume logging. Tell us those decisions.

At the time, ClickHouse did not have good support for JSON. I was speaking to someone yesterday who said ClickHouse doesn’t have good support for JSON and got roundly stepped on because apparently it does now, so they’ve obviously gone and built their proper JSON support. But back when we were trying to use it, I guess a year ago or a bit more than a year ago, everything happened to be a map, and maps are a pain to try and do when looking up JSON-type data.

Obviously, all these attributes, everything you’re talking about there in terms of the GenAI stuff, you can choose to make them top-level columns if you want, but the simplest thing is just to put them all into a big JSON pile, and that was a problem with ClickHouse.

ClickHouse also had some really ugly edge cases. By default, or at least until I complained about it a lot, ClickHouse thought that 2 ns was longer than 1 second because they compared intervals just by the number, not the unit. I complained about that a lot, and then they caused it to raise an error and just say you have to have the same unit. Then I complained a bit more, and eventually, I think, as I understand it now, they convert between units.

But stuff like that, when a lot of what you’re doing is comparing the duration of spans, was really painful. Also, things like you can’t subtract 2 datetimes to get an interval; you have to use the date sub function.

The fundamental thing is that because we want our end users to write SQL, the quality of SQL—how easy it is to write—matters way more to us than if you’re building a platform on top where your developers are going to write the SQL, and once it’s written and it’s working, you don’t mind too much. I think that’s 1 of the fundamental differences.

The other problem that I have with ClickHouse and, in fact, Timescale is that the ultimate architecture—the Snowflake architecture of big data in object storage queried with some kind of cache from nearby—they both have it, but it’s closed source and you only get it if you go and use their hosted versions.

So even if we had got through all the problems with Timescale or ClickHouse, we would end up with them wanting to take their 80% margin, and then we would be wanting to take our own margin. That would basically leave us less space or margin.

Whereas DataFusion is properly open source. All of that same tooling is open source, and for us as a team of people with a lot of Rust expertise, DataFusion, which is implemented in Rust, means we can literally dive into it and go and change it.

For example, I found that there were some slowdowns in DataFusion’s string-comparison kernel for doing things like string contains. It’s just Rust code, and I could go and rewrite the string-comparison kernel to be faster. Or, for example, DataFusion, when we started using it, didn’t have JSON support. Obviously, as I’ve said, that’s something we needed. I was able to go and implement that in a weekend using the JSON parser that we built for pydantic-core.

So it’s the fact that DataFusion is, for us, the perfect mixture of a toolbox to build a database with, not a database, and we can go and implement stuff on top of it in a way that, if you were trying to do that in Postgres or in ClickHouse—I mean, ClickHouse would be easier because it’s C++, relatively modern C++, but as a team of people who are not C++ experts, that’s much scarier than DataFusion for us.

Yeah, that’s a beautiful rant. It’s funny: most people don’t think they have agency on these projects. They’re kind of like, “Oh, I should use this or I should use that.” They’re not really like, “What should I pick so that I contribute the most back to it?”

But I think you obviously have an open-source-first mindset, so that makes a lot of sense.

I think if we were a better startup and faster-moving, and just headlong determined to get in front of customers as fast as possible, we should have just started with ClickHouse.

I hope that long term we’re in a better place for having worked with DataFusion. We’re quite engaged now with the DataFusion community. Andrew Lamb, who maintains DataFusion, is an advisor to us. We’re in a really good place now, but yeah, it’s definitely slowed us down relative to just building on ClickHouse and moving as fast as we can.

Okay, we’re about to zoom out and do Pydantic Run and all the other stuff, but my last question on Logfire is really: at some point, you run out of community goodwill just because, like, “Oh, I use Pydantic, I love Pydantic, I’m going to use Logfire.” Then you start entering the territory of Datadog, Sentry, and Honeycomb.

Where are you going to really spike here? What’s the differentiator?

I wasn’t writing code in 2001, but I’m assuming that there were people talking about web observability, and then web observability stopped being a thing—not because the web stopped being a thing, but because all observability had to do with the web.

If you were talking to people in 2010 or 2012, they would have talked about cloud observability. Now that’s not a term because all observability is cloud-first. The same is going to happen to GenAI.

So whether or not you’re trying to compete with Datadog or with Arize and LangSmith, you’ve got to do first-class—you’ve got to do general-purpose observability with first-class support for AI. As far as I know, we’re the only people really trying to do that.

I mean, I think Datadog are starting in that direction, and to be honest, I think Datadog is a much scarier company to compete with than the AI-specific observability platforms. In my opinion—and I’ve also heard this from lots of customers—AI-specific observability where you don’t see everything else going on in your app is not actually that useful.

Our hope is that we can build the first general-purpose observability platform with first-class support for AI, and that we have this open-source heritage of putting developer experience first that other companies haven’t done.

For all that I’m a fan of Datadog and what they’ve done, if you search “Datadog logging Python” and you just try, as a non-observability expert, to get something up and running with Datadog and Python, it’s not trivial. That’s something Sentry has done amazingly well, but there’s enormous space in most of observability to do DX better.

Since you mentioned Sentry, I’m curious how you thought about licensing and all of that. Obviously, Pydantic is MIT-licensed. You don’t have any rolling license like Sentry has, where you can only use an open-source, 1-year-old version of it. Was that a hard decision?

To be clear, Logfire is closed source, so Pydantic and PydanticAI are MIT-licensed and properly open source, and then Logfire, for now, is completely closed source.

In fact, the struggles that Sentry have had with licensing, and the weird pushback the community gives when they take something that’s closed source and make it source-available, just meant that we avoided that whole subject matter.

I think the other way to look at it is, in terms of either headcount or revenue or dollars in the bank, the amount of open source we do as a company—we’ve got to be up there with the most prolific open-source companies, per head. So we didn’t feel like we were morally obligated to make Logfire open source.

We have Pydantic. Pydantic is the foundational library in Python, and PydanticAI is our contribution to open source. Then Logfire is openly for-profit, and we’re not claiming otherwise. We’re not trying to walk a line where it’s open source, but we’re really trying to make it hard to deploy so you probably want to pay us. We’re trying to be straight about that. We could change that at some point in the future, but it’s not an immediate plan.

All right. So the first one I saw—this new, I don’t know if it’s a product you’re building—pydantic.run, which is a Python browser sandbox. What was the inspiration behind that?

We talk a lot about code interpreters for LLMs. I’m an investor in a company called E2B, which is code sandboxes as a service for remote execution. What’s the pydantic.run story?

Pydantic Run is again completely open source. I have no interest in making it into a product. We just needed a sandbox to be able to demo Logfire in particular, but also PydanticAI. It doesn't have it yet, but I'm going to add basically a proxy to OpenAI and the other models, so that you can run PydanticAI in the browser, see how it works, tweak the prompt, et cetera. We'll have some kind of per-day limit on what you can spend on it, or on what the spend is.

The other thing we wanted to be able to do was, when you log in to Logfire, we have quite a lot of drop-off. A lot of people sign up, find it interesting, and then don't go and create a project. My intuition is that they're like, "Oh, okay, cool, but now I have to go and open up my development environment, create a new project, do something with the right token. I can't be bothered." Then they drop off and forget to come back.

I think that happens to all of us. I started seeing if I could do it a week and a half ago, got something to run, and then ended up improving it. Suddenly, I spent a week on it, but I think it's useful.

Yeah, I remember maybe 2 or 3 years ago there were a couple of companies trying to build browser terminals exactly for this. You go on GitHub, see a project that's interesting, but then you have to clone it and run it on your machine, and sometimes it can be sketchy. This is cool, especially since you already make all the docs runnable, like you said—you kind of test them. It sounds like you might just have—

Yeah, so the plan is that on every example in PydanticAI, there's a button that basically says "Run," which takes you into pydantic.run. It has that code there, and depending on how hard we want to push, we could also have it hooked up to Logfire automatically. There's a, "Hey, just come and join the project," and you can see what that looks like in Logfire.

That's super cool. I think one of the biggest personal drop-offs from open-source projects is that it's kind of like, "Do this," and as soon as something doesn't work, I just drop off. It takes some discipline.

There have been many versions of this that I've been through in my career where you had to extract this code and run it, and it always falls out of date. Often, we would have this concept of transclusion, where we have a separate code-examples repo that gets pulled into our docs, and it never really works. It takes a lot of discipline, so kudos to you on this.

It was years of maintaining Pydantic and people complaining, "Hey, that example's out of date," before we eventually went and built pytest-examples, which is one of the hardest-to-search-for open-source projects we've ever built. Obviously, if you search for pytest-examples, you get examples of how to use pytest.

Pytest-examples will basically go through both your code inside your docstrings to look for Python code and the Markdown in your docs, extract that code, then run it for you, run linting over it, and soon run type checking over it. That's how we keep our examples up to date.

Now we have hundreds of examples, all of which are runnable and self-contained. If they refer to the previous example, they're already structured so that they have to be able to import the code from the previous example. Why don't we give someone a nice place to actually run that using OpenAI and see what the output is?

Lovely. All right, so that's kind of my take in the notes here. I just like going through people's X accounts, not Twitter. For 4 years, you've been saying we need a plain-text successor to Jupyter notebooks.

Yeah, I think people may have gone the other way, which is to make it even more opinionated, like with AI and all these kinds of notebook companies.

Well, yes. In reply to that, someone replied and said, "Marimo is that." Sure enough, Marimo is really impressive. I've subsequently spoken to the Marimo guys and got to angel-invest in them. I can't remember—I think it's CRV.

Marimo is very cool. Its notebooks also run in the browser, again using Pyodide.

In fact, I nearly didn't build pydantic.run because we were just going to use Marimo. My concern was that people would think Logfire was only to be used in notebooks. I wanted something that, ironically, felt more basic—more like a terminal—so no one thought it was just for notebooks.

Yeah, there's a lot of notebook hate out there. I have very strong opinions about proper notebooks—not like Jupyter notebooks. The idea that you have to run the cells in the right order, and a whole bunch of other things, is basically worse than Excel, or similarly bad to Excel.

Oh, so you are a notebook hater, then, and invested in a notebook?

I have this rant called "Notbook," which was my attempt to build an alternative, but mostly just a rant about the 10 reasons why notebooks are just as bad as Excel.

But Marimo and all the new ones that are text-based at least solve a whole bunch of those problems.

I agree with that.

Yeah, I was kind of wishing for something like a better notebook, and then I saw Marimo. I was like, "Oh, yeah, these guys are ahead of me on this."

I don't know if I would do the sort of annotation-based thing. A lot of people love the "annotate this function and it just adds magic" approach. Similarly to what Jeremy Howard does with his stuff, it seems a little bit too magical still, but hey, it's a big improvement from notebooks.

Yeah, yeah, great. Just as with LLM usage, the .ipynb file is just not good to put in LLMs.

True, it's really not. They freak out. They're not good to put in Git either.

I mean, okay, well, we'll kill .ipynb at some point.

Any other takes? I was going to ask you just about the London scene. What's it like building out there?

I'm an evening person. The good thing is that I can get up late and then work late because I'm speaking to people in the US a lot of the time. I got invited earlier today to a drinks reception about AI at 10 Downing Street with the prime minister, so I'm feeling positive about the UK right now and AI.

But I think, like everywhere, everyone in the US and China knows that we're way behind on AI. I think it's good that the UK is beginning to say this is an opportunity, not just a risk.

I keep being told I should be at more events, hanging out with AI people more. My instinct is that I'd rather sit at my computer and write code. I think that's probably a more effective way of getting people's attention. Part of me thinks I should be sitting on Twitter, not in San Francisco chatting to people. I think it's probably a bit of a mixture, and I could probably do with being in the States a bit more. I think I'm going to be over there a bit more this year.

There's definitely a risk if you're somewhere where everyone wants to chat to you about code, but you don't write any code. That's a fail.

I would say definitely. There's a scene, and one way to really fail at this is to just be involved in that scene and have it eat up your time. But be at the right events, and the ones that I'm running are good events, hopefully.

Okay, come to AI Engineer. What I say is, use those things to produce high-quality content that travels in a different medium than you normally would be able to, because there's some selectivity and a focused community around that thing. People will discover your work more, and it will be highly produced. That's the pitch for why I do conferences, at least.

In terms of talking to people, I always think about this as a 3-strikes rule. After a while it gets repetitive, but maybe the first 10 or 20 conversations you have with people—if the same stuff keeps coming up, that's an indication that people want a thing. It helps you prioritize in a more long-form way than you can get in shallow interactions online.

That in-person, eye-to-eye, "This is my pain at work," and you see the pain and you're like, "Oh, okay, if I do this for you, you will love our tool"—you can't really replace that.

It's customer interviews, really.

Yeah, I agree entirely with that. I think you're right on a lot of that. It's very easy to get distracted by what people are saying on Twitter and LinkedIn. That's another thing that's very hard to correct for: which of those people are actually building this stuff in production at serious companies, and which of them are on day 4 of learning to code?

They have equally strident opinions, and in a few characters they seem equally valid. But which one is real and which one isn't—or which one is from someone who really knows their stuff—is hard to know.

Anything else, Sam? What do you want to get off your chest?

Nothing in particular. I've really enjoyed our conversation. I would say that if anyone has looked at PydanticAI, we know it's not complete yet. We know there are a bunch of things missing—embeddings, storage, MCP, toolsets, and stuff like that. We're trying to be deliberate and do stuff well, and that involves not being feature-complete yet. But keep coming back and looking in a few months, because we're pretty determined to get there.

Whether or not you think AI is going to be the next Excel, the next internet, or the next Industrial Revolution, it's going to affect all of us enormously. As a company, we get that making PydanticAI the best agent framework is existential for us.

You're also the first serious AI company I've seen that has no open roles for now. Every founder who comes on our podcast—the go-to action is, "Please come work with us."

We're not hiring right now. I would love, bluntly, for Logfire to have a bit more commercial traction and a bit more revenue before I hire more people. It's quite nice having a few years of runway, not a few months of runway, so I'm not in any great appetite to destroy that runway overnight by hiring another 10 people, even if the whole team is rushed off its feet doing, as you said, 3–4 startups at the same time.