与 Guardrails AI 的 Shreya Rajpal 一起让 AI 革命不脱轨

Shreya Rajpal

It’s kind of insane to see the amount of activity and excitement around the space. There are people training and fine-tuning deep learning models who weren’t even in the space a few months ago, and that’s really awesome.

I’ve had people say, “I really like Guardrails. I really like OpenAI, but it’s just too expensive for what I’m trying to build. Can you make this work with an open-source model?” I do think we’re going to see a lot of that proliferation: great-performing models at different price points, with different latencies, from different providers.

Today my guest is Shreya Rajpal, a former machine learning engineer at Apple and founding engineer at Predibase, who is now best known as the creator of Guardrails AI, a new Python library that allows developers to add a layer of output validation and correction to their code.

Anyone who has spent time building AI-powered products over the last 2 years will attest that validation, and even more so reliability, are key challenges. LLMs simply don’t always follow instructions, and sometimes go entirely off the rails. Better models have helped tremendously. It’s true that GPT-4 can follow instructions far more reliably than earlier models, and Claude v1.3 is also very impressive. But with this elevated capability also comes expanded developer ambition, and so it seems that, for the foreseeable future, the problem of LLM reliability will remain both critical and ubiquitous.

Shreya’s work tackles this problem in many ways and at multiple levels. Super practically, Guardrails can ensure a reliable interface between language models and more traditional deterministic software systems. Does the language model return data with the right type and format? Did the model choose a value from the list of allowable values that we provided? These are very familiar questions for developers, and they’re still powered in Guardrails by traditional code.

At the same time—and for me, this is clearly the more novel, exciting, mind-bending, and potentially risky use case—frameworks like Guardrails allow developers to ask and answer entirely new kinds of questions: assessments of things like the quality of a summary or translation, or whether a given piece of text contains redundancies, inconsistencies, or gaps in logic. These are the sorts of things that, until recently, developers simply had no way to validate.

Thus, for product and engineering teams around the world, Guardrails is both a solution to a very practical problem at hand and an introduction, or bridge, to an emerging paradigm of AI-first software development. That paradigm goes well beyond the Copilot-style autocomplete or even chat interfaces that we’ve recently seen, and begins to use AI functions not just as development tools, but as components of the production technology stack itself.

Talking to Shreya really reinforced for me just how early we are in LLMs’ impact on the software industry. The core AI capabilities needed to transform software, as far as I can tell, mostly already exist. What remains is the work of reimagining not only how software is built, but how it functions now that intelligence can be baked in at any point.

Personally, I believe that this paradigm shift could ultimately unlock bigger productivity gains and more user value than the current generation of tools that increase developer speed but don’t yet attempt to change the kind of software that they’re building. At the same time, this is something to be approached with real care and caution. The delegation of AI output validation to other AI models is not a step to be taken lightly or taken for granted.

Shreya Rajpal, welcome to The Cognitive Revolution.

Shreya Rajpal

I’m really excited to be here. Thanks for inviting me.

My pleasure. This is the first time that I’ve invited a guest basically as soon as I got off the call recording a previous episode. Matt Welsh, CEO of Fixie.ai, mentioned your new project, Guardrails AI, and I was immediately like, “Okay, I have to learn everything I can about this.” I’m really excited to dive into it with you.

I guess let’s start with what made you say, a couple of months ago, “I need to build a system to help people keep their language models on the rails”?

Shreya Rajpal

I was really solving my own pain points and my own problems. At the end of last year, I had been doing some tinkering on my own, building some applications. A lot of what you were seeing on Twitter was about chatting over proprietary documents, and I was building that. I realized that this was pretty cool and that I could see the potential, but even as a developer testing it out, I could tell that it didn’t reliably get me the desired experience—the experience that I wanted to achieve.

These language models are really potent and powerful, but they’re also inherently very stochastic and hard to control. The other thing that makes this very interesting is that, unlike traditional machine learning, you haven’t really trained the model as a developer. You can’t just throw more data at it that could work really well for your use case.

Separately, the only knob you really have as a developer is, “Here’s this prompt.” If you want it to do something or not do something, the way developers typically deal with that is by adding a lot of verbiage to the prompt, and maybe a lot of exclamation marks, to make it listen to you. That just seems woefully inadequate.

Guardrails is this idea of a specification framework where, as a developer, you know what the right output for an LLM looks like. You’re able to decompose and deconstruct that output, and individually validate and verify each component. If any of those components or quality criteria fail, Guardrails gives you a set of tools to address that in a very extensible manner.

I was building this and thinking, “I know what responses I want a user to be able to get from this thing that I’m building. How do I ensure that I’m always able to do that for a wide variety of scenarios?” That was some of the inspiration.

I also spent some years working on autonomous systems and self-driving. It’s a similar problem there, where you have a really powerful deep-learning-based perception model that often feeds into a more rule-based decision-making system. How do you ensure that the interface between that stochastic system and that deterministic system is robust and not brittle whenever the perception system doesn’t do as well?

The idea was inspired by some techniques you would see there, but built for language models and built in a very extensible way so that it’s not domain-specific.

As much as possible, I love to get super concrete on these things. This is clearly a pain point that a lot of people have. The project has gone on its own little rocket-ship ride of GitHub stars, with 1,200 as of last check at the time of this recording, but a lot of people listening could use a more concrete example.

What kind of thing are you looking for, how is it failing, and can you tell us a couple of those examples? How does Guardrails come in and save the day in those instances?

Absolutely. My favorite prototype example is that I have this Chase credit-card agreement, which was my own credit-card agreement, and I want to extract the key terms from it and get a nice JSON object out of it.

As a user, I know that if I’m extracting something like an interest rate, it must always be a number. It might have a percentage sign in it, and there’s a reasonable range for what that number can be. If I’m extracting the name of a specific fee and I want that name to be presented somewhere, I know that the name should be very concise. It should be like a variable.

One common failure point is that if I want to extract this information and then add it to some downstream data sync, it’s hard to do that reliably and consistently from an LLM, because an LLM doesn’t behave reproducibly.

In this structured-data-extraction setting, I could enforce constraints on what I want each extracted entity to look like. The interest rate must be a number and must be within a reasonable range. There must be a description with each interest rate, and the description might need to be a certain length or relevant to whatever entity it comes along with.

All those constraints are what I want to impose on this JSON structure. That was a use case I was prototyping. Since then, I’ve added a lot of examples to the documentation, but one of my favorite ones is text-to-SQL.

Text-to-SQL is a wildly different domain, but many of the same ideas apply. You want correctness from any generated SQL query. The idea is that, as a user, you want to be able to ask natural-language queries over your data and get a SQL query that you can execute.

You need to be able to define the constraints you want to impose on that SQL query. It must actually work for the database or environment where you want to execute it. You may not want to return results from specific tables, or some tables might be private. If you’re a customer, you may never be able to query those tables, so you can filter them out.

You can add constraints such as only supporting specific SQL CRUD operations. If there are DROP operations, or perhaps UPDATE or INSERT operations, you may want to filter those out. As you’re setting up this text-to-SQL task, you can add all of those constraints.

Guardrails takes your database schema and sets up a SQL sandbox. Any SQL query that is generated is executed in that sandbox to make sure it’s executable. If it isn’t, you take all of the errors explaining why the SQL query failed to execute, wrap those errors into something, send them back to the large language model, and have it correct itself to get something that actually works for your specific database.

You can add other constraints and restrictions on top of that, such as filtering out specific tables or operations. The idea is that there’s a question or task you’re asking this large language model to solve, but as a developer, you have domain expertise about what correctness means for that task and how much you care about correctness.

If it’s incorrect, is the query totally useless to you? Or, if it’s incorrect, do you just want to know about it and handle it post hoc? Guardrails takes that idea and allows you to implement it as a developer.

It’s fascinating. One of the things I want to explore most in this conversation is the seeming spectrum, or dimension, that we’re opening up. On one end, there are explicit errors—the errors we’re used to as developers in code. That could start with a syntax error and go up to more meaningful errors that can still be found through traditional software messages, such as “This variable doesn’t exist.”

Then you have a whole domain of correctness that code has never really accessed before. What are you trying to do? Does this appear to be a reasonable approach or output? That’s a much more semantic, or, dare I say, intelligent point of view.

I find myself a little confused or lost in that space, and I see that you’re covering it in an interesting mix of ways in the library and in the validators. There’s a mix of values there. Maybe walk us through how you think about that, along with some examples of different validators that sit in different parts of that space.

Shreya Rajpal

I think that was one of the very exciting things about the library as I was building it out. The general framework works even outside of things that you can verify with an assert statement. It’s extensible as a framework, where you can have a mix of large language models, rule-based heuristics, programmatic checking, and traditional machine-learning models, such as high-precision classifiers.

You can ensemble all of these techniques together to get something greater than the sum of its parts—something much more robust and reliable compared with just using a pure large language model.

As an example, I recently added a bunch of Guardrails for summarization. If you’re summarizing multiple documents and generating an aggregated summary from them, there are a bunch of requirements you may have to ensure that the summary is accurate, concise, and not redundant.

Guardrails thinks about this general problem of correctness, when assert statements aren’t sufficient, by breaking it down into smaller ML tasks or smaller verifiable heuristics, and trying to get an aggregate assessment.

In the case of summarization, if you want to figure out whether the summary is faithful to the original source text, you can look at each sentence in the summary and do similarity matching based on which parts or passages of the source text it’s most similar to. That allows you to do finer-grained attribution and figure out where the sentence is generally coming from.

You can also assign thresholds. As a developer, you can do some experimentation to figure out how varied you want these sentences to be from the original source text, set a threshold, and then filter out any sentence that falls below that threshold in terms of similarity score.

This is a way of thinking about text outputs that may seem like a single unit. It breaks them down into smaller chunks and independently tries to verify them.

Another technique for summarization is to make sure the summary is concise and contains no redundant information. Within the generated summary, you can make sure that each sentence is diverse enough from the other sentences. If that’s not the case, you can filter out sentences that are too similar to each other.

The cool thing about LLMs is that we’re looking at many of the first-order benefits, where you can use them to do a bunch of insane tasks. But I think the second-order benefit is that you can really use these models as verification systems and critique themselves.

You can use a large language model within a validator to verify and validate whether whatever you’re getting is correct. Or, depending on your access to data and latency requirements, you can use a smaller machine-learning model that does this for you.

The philosophy behind Guardrails is to bring together a bunch of different verification strategies and add guarantees around problems that have traditionally been harder to verify.

That’s really interesting. I’m mapping this onto my own use cases in real time. I’m wearing my Waymark swag today. Waymark is in the video-creation space, and we have a multimodal problem. We ultimately take in some minimal information about a business, ask the user to tell us what they want to create, and then provide a ready-to-watch video.

We do that with an ensemble of different models working together, but the core one is the language model that writes the script and gives direction for what the visual assets should look like.

We’ve come so far in a year. It was just a year and a half ago that we got the very first fine-tuned GPT-3 model to do the task at all. It wasn’t good, but it would at least respect the nature of the inputs and outputs. Today, GPT-4 can basically do it zero-shot and respect the outputs. GPT-3.5 Turbo can do it, though not reliably, and it’s about 20 times cheaper.

I’m interested in what you see the value drivers being for this sort of thing. To some degree, maybe there’s no other way to do it, but GPT-4 is pretty reliable at this point. People might want to do it for cost savings, latency, or a lot of other reasons.

Cost savings and latency are benefits I could see. If I could move to a slightly less reliable model but still know that my stuff is going to render correctly for our users, that would be valuable. What are you seeing in terms of the value drivers from your community?

Shreya Rajpal

I think there are 2 aspects. Cost and latency—being able to use open-source models or cheaper models with the same level of reliability—are one aspect.

The more interesting aspect for me is the task that you’re truly using the model for. With Waymark, there are a lot of use cases where you’re using these models as writing assistants or to help you. One product I enjoy using is Notion AI Assistant. You give it a little bit of a draft, and it helps you shape up an outline.

For those use cases, the creativity of a large language model is actually very good, and it’s a desirable trade-off. But there’s another whole space of use cases that rely on the world model ingrained in these large language models, and use them as software abstractions to do more general-purpose reasoning.

One example is using an LLM as an AI receptionist. If you’re a small-business owner, instead of needing to hire a receptionist, you can use a large language model. Whenever you get calls, it figures out scheduling, who’s available, and so on.

That’s a powerful capability of the model, but creativity is perhaps not as desirable there. You want it to stick to a desired workflow. You want to make sure that it doesn’t ask for private information if a customer is calling, for example.

Guardrails is most useful in those situations, where you’re using the LLM not just as a text generator, but as a software instruction system. When you’re using it in that capacity, reliability becomes most useful.

Over the weekend, I shared something that a community contributor had built using Guardrails. It was a GitHub Action called AutoPR. AutoPR takes a GitHub issue and automatically creates a pull request for your codebase.

That’s one of those use cases with a bunch of really strict constraints. The files must exist, and the generated code diffs must be valid for those files. Those constraints are hard to enforce without having a validation framework such as Guardrails on top of it.

For a lot of those use cases, creativity isn’t as valuable. You want more reliability. It’s still such a powerful use case for these models that we would be underutilizing their capabilities if we didn’t build software like that.

Even with many creative use cases, I’ve found that there are still a bunch of constraints people implicitly have. Right now, the way to handle those constraints is through prompt tuning or prompt iteration.

Being able to encode these constraints and have a validation step run automatically is useful. You might not want any profanity in generated scripts, or, if you’re creating video content for a brand, you might not want to mention peer products or competitors. Those constraints are useful even with free-form text.

It sounds like you’re most excited about something that couldn’t be achieved in traditional code. I’m still trying to find that line. Maybe there’s just a lot of overlap, and there’s probably some truth to that.

I’m thinking about AutoPR. It’s cool and super technical, but those responses could presumably be validated by existing libraries. The Git package itself probably has some way to say, “You haven’t got this together, so this isn’t going to work.”

I think that’s what many people are naively doing: implementing this stuff on a case-by-case basis. At Waymark, there is no standard. Nobody else has our video standard. We define and own it, so it was up to us to figure out how to represent that in text and what validation comes back from it.

If you were advising us when we were a little earlier, how would you think about what points of validation we ought to handle through Guardrails? Should we think about structure? Should we think about whether the copy satisfies the user’s prompt? There’s probably a lot of room in between. Those stake out the most rigid versus the most semantic desires or requirements from the model.

It seems like you’re more interested in the semantic side, but there’s also this fundamental interface with computing, where it gets very syntactic as opposed to semantic.

Shreya Rajpal

I want to preface this by saying that generating something, getting feedback from the end user about whether it’s good or meets the original criteria, and doing a more qualitative assessment is a very valid way of doing things.

For a lot of domains, getting human feedback and input is essential before you can really think that an output is correct and valid for whatever use case you have.

In the Guardrails world, there’s an idea that some of that human feedback can be done by a combination of traditional machine learning, heuristics, and perhaps more large language models in the loop.

If you can take some of that qualitative criteria and codify it into something more specific, you’re then able to generate specific failure messages that help the model correct its output.

I don’t know as much about Waymark, but it might help reduce the amount of back-and-forth you have with your end customer. You can take some of their constraints, run them programmatically, and, if the output is wrong, automatically create new prompts that tell the large language model why the previous outputs were incorrect and get it to correct itself.

This is one of the new capabilities we have with large language models: the ability to get them to self-heal or self-correct if you give them enough context.

That’s where Guardrails is most useful. Under the hood, it’s good at figuring out the relevant context, packaging it up, automatically creating a new prompt, getting a new response from the large language model, and merging that new response with the old response.

It’s very efficient. It only asks about the things that are wrong, rather than sending your whole previous output again. That merged result is your corrected, validated response.

That’s the world where Guardrails is most useful, but it’s also very domain-specific. There are domains where any text that’s generated must be funny. That’s next to impossible to validate. I can’t imagine writing a validator that functions as a scoring system for humor. That’s very hard to do, so you need a human expert.

There are other domains with very high stakes and a high cost of getting something wrong. Even if you can do some prescoring, actual human confirmation that the output is correct is essential before you do another iteration with the large language model.

There are domains where this re-asking strategy doesn’t work. But I do think the idea of taking what correctness means, or what an aligned output means, in your use case and trying to codify some of that is useful.

It helps with prompting the large language model, because it allows Guardrails to construct prompts that are more effective at getting what you want. It also allows for post hoc validation and a loop where you can systematically and programmatically handle failures as they arise.

That might involve re-asking or filtering incorrect output. It’s a powerful framework for requiring less human supervision and taking away the pain of going into ChatGPT, writing, “This doesn’t work for this reason,” and trying to write something else.

Maybe it’s because we all have agents on the brain, but as I’m listening to you describe that, I’m going to the agent moment we’re in. When you dig into these agent systems, it’s usually a multi-agent system in many cases.

It might be the same language model playing different roles. We’re seeing examples of a simulated town where GPT-4 plays all the people, or a research agent where there’s a planner, a coder, and a retriever that all work together. They have their own prompts and are scaffolded together.

Of course, these things fail a lot, because there’s some probability of being wrong at any given point in the chain. In the naive implementation, you’re only as strong as the weakest link in the chain.

In a sense, what I’m learning is that the Guardrails paradigm is connective tissue between the different roles in a multi-agent system. I should probably accelerate how quickly I think these agents are going to start working.

Shreya Rajpal

Agents are very exciting. I’ve been thinking about them a lot and trying to understand how to make them more effective and reliable.

The interesting thing about agents versus how Guardrails works is that Guardrails solves the problem of adding constraints to an output so that it works for your use case. It gives developers a lot of agency to think about the specific problems they’re solving and what correctness means to them.

By contrast, many of the goals and tasks of agent frameworks are configured autonomously by a large language model itself. You’re in an interesting setting where the involvement of a human or developer—the ability to enter an agent framework and add guarantees—is usually not available at a fine-grained level.

With the agent frameworks that exist today, you typically don’t have access to the fine-grained level of task execution or goal setting, because you’re not the person configuring the agents themselves.

There’s a gap between how agents operate today and what humans would ideally like to have. If you want to employ a bunch of agents to do research for you, you want to be able to say, “This is the set of allowable things you can look at.” But you also want that configured dynamically based on where the agents are operating.

Constraints and correctness specifications are essential. They’re the only way to think about how to assess what these large language models are doing. You can stream them and evaluate each step to make sure they’re not going off the rails, but how do you do that dynamically when you’re not the person creating the agents or setting their goals?

That’s the key problem to solve. It’s a problem we don’t need to solve before these agents are employable or before we can truly use them outside of seeing how exciting they are.

For Waymark specifically, where there are constraints and a finite scope for what you want the large language model to do, have you looked at some of these agents and thought about what part of Waymark could be aided by agents?

It’s probably not such a great fit for the Waymark product experience, because we have a lot of structure.

I think of these things as different modes—I’m still working out this framework. I talked about it with Matt Welsh, and that’s part of what led him to mention your project.

In terms of how we interact with AI systems, there seems to be a real-time Copilot mode, where you’re doing things as a human and the system is there to guide or shepherd you. Then there are situations where you’re ultimately delegating more because you don’t want to do the task yourself.

You don’t want to be the person in the driver’s seat, or the entity in the driver’s seat. You want to put the AI in the driver’s seat, let it do the work, and then look at its work when it’s done—or have some other way to circle that back into your life and workflows.

Right now, those workflows are largely done with integrations of various kinds. They could be code, no-code, Zapier, or something else. With Waymark, it’s a guided experience where you’re delegating the task of writing a script, choosing all the assets for your video, and so on. Then you watch the output.

There’s so much structure that it doesn’t feel like we need an agent to come in and mix things up too much. I do see an agent as the bridge between these 2 modes.

In ChatGPT Plus today, I can use a plug-in to look for flights, but I might want to tell the system, “Go book me a flight for this date, subject to my preferences,” and have it figure out all the downstream details.

This is also very relevant to me because I’m working with a company called Athena, which is in the executive-assistant space. They have these kinds of tasks all the time, where a human today is responsible for executing a lot of web tasks for a client.

There are different aspects of the cognitive work. One is understanding what’s going on—translating the client’s language, understanding what a request means, and asking the right follow-up questions. The other part is actually being able to hit the right buttons to make it happen.

Ideally, we delegate that whole thing. In our testing, along with everybody else, we’ve found that AI may be able to understand the request, parse it effectively, ask the right follow-up questions, suggest good follow-up questions, and demonstrate a robust understanding of what the human wants.

But we’re still falling down on how to hit the right buttons. God forbid you have to make a payment or log into something. Two-factor authentication still works as a deterrent to AI login, at least for the time being.

Shreya Rajpal

That’s been my experience as well. There’s an interesting wedge where agents can be useful, but for that wedge to succeed, you need this notion of grounding.

You need to figure out, “Here’s how the agent understands what I’m saying,” but at each step of the execution, you need validation.

The question of, “Book me a flight,” is interesting, because it’s based on my schedule, my budget, and where I’m trying to go. That translates into constraints grounded in my calendar, where I’m trying to go, and what flights are available.

Validation at each decision the large language model makes, and each action it takes, becomes more important for these systems to succeed.

Having worked a little in the self-driving world, part of the question is whether they’re able to do it, and part of it is whether a human can trust them. Even if these agents were perfect, there would be a trust deficit around how much people are comfortable delegating.

To build that trust, you need a verification system that makes sure that, at each step, you’re able to maintain control and oversight over how the agents execute.

Those frameworks are essential before we see adoption outside of demo use cases.

It’s funny that you say that. I would take the other side of the bet when it comes to human behavior. If I understand you correctly, you’re saying that people won’t want to trust these systems unless there are good Guardrails in place.

I think of you as saving people from themselves, because people are going to be much quicker to say, “It seems like it works. Why not go ahead? What’s the worst that could happen?” We may find out, especially in a future world, that the worst could be quite bad.

I just recorded an episode with Zach Kahani, a medical-school professor at Harvard who wrote the book The AI Revolution in Medicine. This is exactly the kind of system he’s trying to figure out how to implement into clinical practice.

For now, of course, the human is in the loop. That’s the official recommendation. But it’s easy to get lazy and overly trusting, especially when the models are getting so good.

It’s fascinating that you can get better performance just by pointing the model at itself. One of his big findings was that GPT-4 is better at evaluating text than it is at generating text, and he’s starting to develop self-critique systems on the fly in the context of clinical exploration.

That’s part of what the Guardrails validation system is based on. It’s almost “trust but verify.” You get GPT to generate something, perhaps have a separate step that evaluates it, and add an additional layer of security.

Ideally, the more diverse evaluators you have, the more guarantees you get. Ensembling is a technique I’m pretty bullish on in this space, because it’s what will give us confidence.

The medical-AI assistant space is fascinating because it reminds me of self-driving. With autonomous vehicles that are out there today, such as Tesla’s Full Self-Driving, the idea is that you’re still supposed to be very alert. You’re not supposed to take your hands off the wheel.

There’s an expectation that a human should always be aware and present, but it’s easy for humans to get lazy. One of the things I’m excited about is good interface design, so that we can get human involvement when it’s most needed and not constantly notify people to “pay attention, pay attention,” because that eventually ceases to have an effect.

It’s about finding the balance between when human involvement is necessary and when we can offload some of the work programmatically or through code. Figuring out that balance, and the interface for surfacing the division of responsibilities, is very exciting.

I don’t have a Tesla either, but a neighbor of mine does and has the Full Self-Driving package. Knowing that I’m as obsessed with AI as I am, he was gracious enough to take me for a ride.

First of all, it works much better than people commonly realize. I got into the car in front of my house, he put the destination on the screen, and hit the drive button. The car drove. There wasn’t much fuss between entering the destination and riding.

When you talk about the reminders, that’s something they’ve put a lot of engineering into as well. There are three progressions. First, there’s a visual indicator that you haven’t done anything with the wheel in a while. Some versions also have a camera that watches you from the rearview mirror.

Then it goes to a sound, and after that it gives you a warning that it’s going to pull over if you don’t keep your eyes on the road. They’re pretty far along in that. It was a remarkable experience because it’s a delicate balance, and it’s so easy to tune out.

Eventually, they can pull over. They can also kick you out of the program if you pull over too many times. My neighbor said that if you’re truly sleeping at the wheel, they’ll retract your access to Full Self-Driving.

It’s a fine balance because people can start to tune these things out. How many warnings do we tune out? It’s crazy.

The next big thing that comes to mind is the developer asking, “What’s right for me?” The pattern quickly becomes, “Why can’t an AI just do that?”

I start to think about whether there’s a version of this that’s a plug-in. You’ve created a specification, and I’m trying to envision what it will be like to use a computer a year from now, as the tooling and plumbing mature.

One possibility is a chat interface that lets you access the world and delegate tasks to little agents that go do things and report back. It seems like a GPT-4 central process could use the Guardrails specification, or something similar, to insulate itself from problems as it delegates tasks.

Do you see that as realistic? Do you have a point of view about how the computing experience might evolve in light of that?

That’s a very good question. It could go in a bunch of directions, but my core belief is that humans will remain essential in the loop. It will be very hard to automate all of this away.

One reason is that correctness means very different things to different people and in different contexts. A common example is profanity filtering. Most people can agree that if you’re generating facts, you should make sure they don’t contain profanity.

But I’ve also talked to people building chatbots for specific audiences where authenticity is essential. If the chatbot is trying to imitate, or be in the likeness of, someone who uses profanity, filtering out that profanity is actually detrimental to the user experience they’re building.

It’s very hard to figure out what those constraints are on a global level. Domain experts and developers need to be involved to figure out what they’re building, how they’re grounding it for users, and what the desired experience is.

Those inputs will continue to be important. There needs to be a way to configure those inputs, criteria, or the experience that a user or developer wants. Offloading this entirely to the model or to a provider will be hard to achieve because of that constraint.

In terms of what the programming or developer experience could look like, one hypothetical direction is to start thinking configuration-first. There could be a configuration system that allows you to tune the outputs you want to see from these LLMs.

Even if the underlying machine-learning model stays the same, you could configure how that model is validated and corrected, and how its outputs are post-processed. When people work with large language models, it’s not just text; it’s also this configuration that they pass in every time.

This is a pattern that I think will be particularly important in engineering. The prompt isn’t sufficient by itself, and choosing the LLM and temperature isn’t sufficient by itself. It’s a configuration framework for how you want the output to be and how you want the input to be processed or formatted.

Security also jumps out to me as a major driver. As I think about where I would use this, GPT-4 is running pretty well and giving me the format I want. It certainly has errors, but it very rarely goes totally off the rails.

There are vectors that are still completely undeveloped and seem likely to come online and cause a lot of problems. Having an SLA or guarantee layer—a validation layer—seems very smart in light of prompt injection.

As users become more sophisticated and adversarial in their attempts, you gain a lot by having something like this implemented ahead of time. We’ve also seen interesting things in Bing, where a site owner—or perhaps a user—can change what the model sees.

We haven’t begun to see what SEO professionals are going to do in the AI-search era. The battle between search and SEO may look pale in comparison to what happens when you can try to trick a language model at runtime with whatever content you want.

Then there’s model risk. As impressive as GPT-4 is, my general strategy right now is to use it for everything, get the quality to an acceptable level, and then think about reducing cost or latency opportunistically as necessary.

Eventually people will say, “I heard Alpaca was just as good,” and try to drop that in instead. Then you’re in a world where you have no idea what’s going on. OpenAI and Anthropic have a certain SLA, and I think a huge misconception is that the SLA is an inherent property of language models.

It isn’t. Those companies have worked very hard to get to the level they’ve reached. You can’t take for granted that a bootlegged Llama fine-tuned on whatever data is going to be as safe or friendly to users.

All of this is just starting to emerge from the mist. How much did that motivate you? Do you see other things like that, and what do you make of all this emergent security that you’re helping people get in front of?

Shreya Rajpal

The risk of working with these models, especially prompt injection, is important. When Bing Chat was released, we saw how easy it was to manipulate these models in specific ways.

I think about this by decomposing the problem. I don’t think this is a problem that will be solved by machine learning alone. A lot of what OpenAI has managed to achieve is built on responsible scaling laws. As you add data and scale the models, you start to see emergent properties.

But, based on my experience working in machine learning, these are fundamentally stochastic systems. It’s very hard to add guarantees from the model itself. You can’t have data points for all the different exciting and weird ways that people are going to use these models.

As a consequence, it’s very hard to add validation at the model level. Security becomes essential because it isn’t something you can leave up to a stochastic system. You need more determinism around it.

My way of thinking about it is to break it down this way: the model is stochastic, so what can we do around the model to add watertight guarantees?

For prompt injection, I’m interested in both the input and output sides. You can sandwich the LLM API call with input validation and output validation to create multiple layers of security that make sure the model isn’t behaving in ways you don’t want it to.

On the output side, if there are behaviors you don’t want the large language model to exhibit, you can perform secondary checks through output validation. If you know the patterns of prompt injection that people tend to follow, you can look for those patterns in input validation.

You can almost gatekeep the queries and interactions users are able to have, limiting them to the set of things you support. The solution is to decompose the overall security problem into specific, domain-specific applications.

For each domain, think about the end goal you want your users to have, and then add constraints to filter out everything outside of that goal.

I would be very surprised if we had end-to-end machine-learning systems. There’s a reason we don’t. You generally have one machine-learning subcomponent, then a checkpoint with either a human or a more deterministic component verifying or validating its output, followed by another downstream component that may use machine learning.

Decomposing the system and adding different layers of security on either end of the language model is the pattern most people use when they try to productionize these systems. Defense in depth is going to become very important.

Another version of this is models talking directly to other models in vector form. That seems like it’s going to become a huge trend.

We had the authors of BLIP-2 on the show a while back. I think of that as an emblematic foreshadowing of what’s to come. They were able to take a frozen vision model and a frozen language model, and quickly train a connector model that converts the encoding of an image into something in the language-embedding space.

That allows you to have a dialogue with the language model about the image. I see so many examples of even simple linear maps from one space to another, where people are bridging different encoded, latent, or embedding spaces.

That’s going to be a huge trend that brings much better performance to many systems. Why would you go through a natural-language bottleneck if you have a much richer representation of an image, a medical scan, or a sound?

If you have the sound of a bird in an audio file, you’re not going to project it down into language over the long term and say, “The sound of a bird is heard,” before feeding that into the language model. You’re going to figure out how to represent it in the language model’s space in a way that is truly unspeakable.

I see a lot of force going in that direction, but the obvious worry is that you’ve opened yourself up to who knows what. The space of possibility is so vast that it’s untestable and incomprehensible on the input side.

What can be done about that? Again, it seems like the answer is to start validating your outputs. That’s really important, and it’s only going to become more important.

People may be underestimating the extent to which they can create a system today that behaves predictably, while the world underneath it changes in ways that expose vulnerabilities over time.

Software has always worked that way to some extent. We’ve seen Windows patches released long after launch. But this seems different.

Shreya Rajpal

More sensory input from more sensors leads to better performance if you’re able to include it. That goes back to the idea of grounding that I mentioned earlier.

If you have a machine-learning system that projects something into a particular space, how do you make sure that the projection is correct? If you have grounding, where another sensory input is also projected into that space, you can use it as a self-correction or self-verification system.

At Drive.ai, where I worked, we saw that using only lidar was different from using cameras as additional inputs. When you have cameras in addition to lidar, you have a better understanding of the state the car is in and are able to make better decisions overall.

I think it’s a positive thing because it allows us to enforce Guardrails across multiple dimensions. It helps us figure out where things might break down and add checks.

I’m excited to see a world where we can have all these vectors in the same space and a lot of interoperability between them.

You’re describing a validation step based on the coherence of the semantic interpretation of different input signals. I’m trying to think of an example I’ve seen in the wild. I guess you’re pointing to self-driving cars as one example.

I want to throw a few other safety-control paradigms at you and have you react to them—not necessarily in a better-or-worse way. Comparing and contrasting these approaches with what you’re building may help clarify which use cases your project is best suited for.

Most people listening will be familiar with OpenAI’s moderation endpoint. You can take an output, run it against the moderation endpoint, and receive a flag indicating whether it falls into one of a finite number of problematic content types. Then you can decide what to do with it.

That sounds pretty simple. Where does that approach fall short?

Shreya Rajpal

I think it’s great, but it’s very finite in scope. If you took something like that and expanded it to any use case you wanted to verify or validate—anything that was programmatically verifiable or verifiable with a machine-learning model—you could flag specific things within the output that might be problematic.

You could also configure how you want invalid outputs to be handled. That’s how I think about Guardrails.

Moderation, specifically profanity filtering, is one validator among many in the library. You take an LLM output and give it to the profanity-filtering validator. It tells you whether there is profanity in the output, and, depending on how you configure it, it can also correct it.

For example, it could generate the text without profanity, or filter out the sentences that contain profanity. You can take that pattern and apply it across many other use cases.

If generated code is incorrect or non-executable, you can do the same thing for code. If summaries generated from source text are incorrect or invalid, you can do it for summaries. If you’ve extracted structured data and specific parts of it are incorrect, you can apply the same paradigm.

It takes the moderation endpoint pattern and makes it a general, accessible tool that you can use in practice.

What do you see people doing most often to fix things when validation fails? One option is to return an error. Another is to rerun the whole call and hope for a better result the second time. You’re also getting more granular, perhaps fixing only part of the output or removing the profanity.

How would you advise people to think about which option they should take when there’s a problem, and what do you see people actually doing today in the community?

Shreya Rajpal

That’s hard to answer because I can’t tell people what their pain tolerance is for their applications. Fundamentally, there are cases where an LLM output that fails validation is of no use. You either want it corrected or filtered out; you can’t use it in an intermediate state.

Re-asking is valuable. It allows you to treat the process as a single API call from the user’s perspective, even if the backend makes multiple calls to the large language model. You either get a corrected output or you’re told that the result is incorrect and can’t be handled.

For important, high-stakes use cases where a failed validation means the output can’t be used, re-asking is probably the most effective option I see people using.

Filtering is another option. Specifically for summarization or profanity filtering, if there are sentences that aren’t information-dense or contain profanity, you can have granular control and filter out those specific elements.

Another option, which is the default setting if you’re using Guardrails, is a mode where you still run all the validations on an LLM output, but if validation fails, you don’t do anything. You return the output as if the user were using an LLM without Guardrails, while logging everything that went wrong.

You can access that log and decide whether to iterate on your model or prompt. There are also options to raise an exception or deterministically fix the output when that’s possible.

OpenAI recently launched its Evals library alongside GPT-4. My surface-level understanding is that it’s architected more as a benchmarking suite than as a runtime aid. What have you learned from it, and what did you think was smart about its implementation or approach to validating language-model outputs?

Shreya Rajpal

My interpretation is that it’s an offline framework that allows OpenAI to test internally how well its LLMs are doing on tasks people care about.

My favorite thing about it is not necessarily technical, but more related to product and go-to-market. It’s almost like a radio crowdsource of how people want to use their large language models, and all the ways those models are going wrong.

OpenAI takes that and makes it part of its training data set or evaluation framework, so the models themselves are trying to serve those use cases better.

I’ve seen people use it to look through the examples and find interesting prompts they can borrow. That’s a nice way to mine the library.

I also thought it was a nice touch that OpenAI created a separate lane for GPT-4 API access for people who contributed to the Evals library.

How about Anthropic’s Constitutional AI approach? That’s obviously a training protocol rather than runtime validation, but do you see commonalities there, or have you taken inspiration from that approach?

Shreya Rajpal

The Constitutional AI work is interesting, but it goes back to what I said earlier: it’s essential to have deterministic checks and deterministic post hoc validation. Training alone isn’t sufficient.

Runtime verification and runtime validation are the most exciting problems to me. Another important aspect is configurability: being able to configure what correctness means to you and enforce that specifically, rather than relying on a globally understood or standardized definition of correctness.

I like a world where developers have the agency to configure what correctness means for them. Being able to do that in a post hoc setting, without retraining a model every time your definition of correctness changes for a particular use case, opens access to a much broader audience.

I see an opportunity to bring Constitutional AI-style critique to runtime. I listened to one of the lead authors of the Diplomacy paper describing the Cicero model from Meta. He was discussing a general strategy for bringing more compute forward into runtime.

Older systems, such as Deep Blue playing chess, used extremely intensive runtime computation, essentially searching a large tree. With language models, it hasn’t been as obvious how to do that.

Rather than relying entirely on computation embodied during training, we could bring some of it forward into runtime and apply checks. The more compute you spend, the more likely you are to get good output.

You mentioned deterministic systems, but some of these approaches still aren’t deterministic. If I critique something repeatedly, there’s still an inherent layer of nondeterminism. It may reduce my problems, but there’s still stochastic behavior.

Shreya Rajpal

That’s a good point, and I should qualify what I mean by deterministic.

There are different ways to evaluate or validate outputs. One is to take an LLM output and create a new prompt asking GPT-4, “Is this output correct given these criteria?” You might ask for a yes-or-no response or another binary response that lets you assess it.

That’s one validation method, and it’s not deterministic at all. It is, however, an additional layer of security.

There are also rule-based or heuristic-based techniques, as well as validation rules that don’t use LLM APIs. You can use smaller, high-precision models trained on subsets of data. Even if the output is generated by a machine-learning model, you have more control over it and may be able to set random seeds or otherwise control its randomness.

When I say deterministic, I’m referring to an ensemble of all these techniques. Some are purely deterministic, while others are more deterministic in the sense that you have more control over the model and its behavior.

One obvious question from a developer standpoint is what overhead this creates. You could measure that in several ways: token overhead, cost, complexity, latency, and perhaps tensions in the product experience.

I think about the Bing experience, where the system streams output token by token and then retracts it if it determines that it has gone off the rails. It’s remarkable that Microsoft launched with that paradigm.

I understand why. People don’t want to sit and wait for the whole thing to be generated before seeing what’s happening. There’s a powerful draw to the streaming experience—strong enough that Microsoft was willing to emit toxic content and then swipe it away.

How do you think about all those dimensions of overhead? How would you guide developers toward minimizing it? Are there things that can be done in parallel? What’s the smart version of this, given that many people will say, “The CEO says we can’t wait for that”?

Shreya Rajpal

It’s a good question. Interestingly, for specific applications, people are more comfortable with latency than you might expect. If they’re in a chatbot, that may be different, but if correctness matters and they’re getting high-quality output at the end, people are often willing to accept additional latency.

When I was initially building this, latency was a concern, and some of the design decisions supported that hypothesis. Re-asking, for example, doesn’t happen piecemeal. It aggregates the issues and then makes a request, so it’s a one-shot request from that perspective.

But people are generally comfortable adding some latency when correctness matters.

Token overhead is interesting. I’ve found a balance between how efficiently you can write a prompt using a structured prompting strategy and how efficient your prompts are when everything is written in words.

For some use cases, being able to structure your prompts is actually more efficient, even if you would expect it to require more tokens. All of the structure and constraints that might otherwise be represented in words can instead be represented with symbols or a domain-specific language, which can be much cheaper.

I’ve seen something similar with complexity. If you’re trying to get LLM outputs structured in a particular way or to have a certain behavior, the only way to do it today is to prompt manually and iterate a lot.

Guardrails abstracts some of that exploration away. There’s a domain-specific language that’s tested and works across multiple LLM providers. As a developer, you don’t have to figure out how to write a prompt that produces a specific structured output. You write it in a known way.

Even if you don’t care about validation and only care about getting structured outputs, Guardrails is useful. It’s a pain to think about how to do that, so you can use Guardrails to get structured outputs across different LLMs without writing and iterating on prompts yourself.

Help people understand a little more about the complexity you’re taking out. How do you get the desired format back?

Shreya Rajpal

The entry point for developers is creating a spec. It’s a markup language where you specify an output schema.

If you want a particular output, you define all the components you want in that output. If it’s JSON, you can configure it as JSON. If it’s just a string with additional validation, you can configure that as well.

That’s separate from the prompt. You think about what output you want and how to define it from a schema perspective.

Guardrails has the Reliable AI Markup Language, or RAIL. In a RAIL spec, in addition to the output schema, you can have a separate prompt. All you need to add to the prompt is the higher-level task description.

Everything that describes how the output should be structured, what constraints it should satisfy, and how it should be formatted goes into the schema. That’s a programming language. If you can write XML or markup, you can write the schema without converting it all into English and experimenting to make sure the LLM API understands it.

Guardrails translates that specification into a prompt. The contract between the user and Guardrails is the spec, and it’s Guardrails’ job to translate that spec into the appropriate prompt.

As the behavior of a particular LLM changes, Guardrails can compile the same RAIL spec into different prompts. The user can provide the same high-level prompt and schema while Guardrails handles the model-specific translation.

From a developer’s point of view, you don’t have to wrangle model-version updates and model-quality issues to the same degree.

That sounds really useful. I was doing some task automation yesterday and used the format trick. Everyone who works with language models has encountered it: “Use this format in the response.”

As far as I know, that came from OpenAI, and Riley Goodside popularized it. He attributes it to Boris Power, though I don’t know whether Boris came up with it independently.

I run into situations where I’m thinking about how intricate I want the format to be. Often I’ll say, “Use this format,” and then give it XML-like opening and closing tags. That lets me use a regular expression to parse whatever is inside the tags and remove any prefix, suffix, or “I hope this was helpful” text.

I haven’t gone much further than that, because then I’m hand-coding XML in the Playground or somewhere else, and that isn’t great. There’s a straightforward convenience factor in being able to write something at a higher level of abstraction and have the system translate it into a detailed prompt that produces the parsable result you want.

I found that as well when I was prototyping. I prototyped with GPT-3, and anyone who has worked with GPT-3 or GPT-3.5 knows that you get things like, “Here’s the answer you’re looking for,” or “I hope that was helpful,” when all you want is the JSON.

Since the release of Guardrails, a community contributor has pushed an effort to add instruction tags and experiment with what works well with GPT-3.5. From a developer standpoint, you write one spec, and Guardrails handles the model-specific details.

With GPT-3.5, the prompt and output fields are handled separately, and people can get what they’re looking for without the additional filler that can make parsing fail.

You’re also translating this to other models. That brings up a market dynamic. As people try to figure out where this is going, will we have one AI to rule them all, or many providers, or an oligopoly of large-language-model providers?

One force pulling everyone toward OpenAI is that so many systems are being developed against OpenAI’s state-of-the-art behavior at a given time. If you’re Anthropic, Cohere, Google, Aleph Alpha, or another provider, there’s a strong incentive to be as much like OpenAI as possible when it comes to supporting those capabilities.

But that raises the question of how you compete. If you have to spend all your energy making sure that people can switch to you without breaking the things they currently assume will work, that doesn’t seem like a great position.

Do you see that dynamic similarly? Does your experience lead you to think we’ll see concentration, if not of providers, at least in how language models behave?

I personally believe there will be a lot of diversity among model providers. It’s almost insane to see the amount of activity and excitement in the space. I’ve worked in machine learning my entire adult life, but there are people training and fine-tuning deep-learning models who weren’t even in the space a few months ago.

That level of demand creates the need for more providers. I’ve had people say, “I really like Guardrails and I really like OpenAI, but it’s too expensive for what I’m trying to build. Can you make this work with an open-source model?”

I think we’ll see a lot of proliferation: great-performing models at different price points, with different latencies, from different providers. I’m excited about that world.

Another interesting aspect is that, because so much was unlocked by OpenAI, people are building frameworks around the most performant provider and model. That creates an incentive for other model providers to offer interoperability for those specific functionalities.

The standards are evolving at the same time as the models are getting better. There’s an interesting dynamic between them. For the foreseeable future, there’s a strong incentive for other model providers to offer similar functionality or at least avoid regressions.

That gives people an incentive to switch for a variety of reasons. The pace of innovation will eventually create a more level playing field, and then it will be interesting to see what specializations different providers and open-source models offer.

I have 3 quick final questions. Answer them quickly or not at all.

First, what are your favorite AI apps, experiences, or tools that you’re loving and would recommend to others?

Shreya Rajpal

I recently had to set up my development environment from scratch. I didn’t have GitHub Copilot, and I thought, “What am I missing? Why am I coding slower?”

I can totally relate to that. When Copilot stopped working because it became paid, I was in a panic until I realized I could simply pay for it and make it come back. I’m not going to be doing this without it now. That’s not something I’m willing to endure.

Second, imagine that at some point in the future, 1 million people already have Neuralink implants. If you could get one and control your computer and devices with your thoughts, would you be interested?

Shreya Rajpal

I would not. What if I’m in a presentation or something, controlling my laptop with my brain, and I get distracted? Does that mean my slide deck is suddenly showing whatever I’m distracted by?

There’s a need for layers of security between thinking and executing something on another system. I would skip it, but I’d be very excited to see that world. I’d be happy to inhabit that role even if I wasn’t actively participating in it.

Great answer.

The final question is about society at large. Zoom out as far as you can and think as far into the future as you can. Do you have any sense of what might happen? What are your biggest hopes and fears for society as this AI moment continues to unfold?

Shreya Rajpal

My biggest concern is job displacement. I don’t think I have any special insight here, but I think about it quite a bit.

What is the amount of work that would still be valuable in a future where a lot of knowledge work can be assisted, or where knowledge workers can be made much more efficient than they are today? What does that mean for the future of work? That’s something I think about.

One hope is that I can live a life where I don’t have to do the mundane things that take up so much of our time. In the future, maybe I won’t have to do my own taxes or book my own flights. I could simply say, “Book this flight for me on this date, find me the best price, and so on.”

Being able to automate away many of those parts of life and focus on the fun parts is something I’m excited and hopeful about.

Go check out the package if you’re building with large language models and facing issues where something was working and then stopped working. Check out Guardrails AI and follow Shreya on Twitter.

Shreya Rajpal, thank you very much for being part of The Cognitive Revolution.

Shreya Rajpal

Thank you again for inviting me. I really enjoyed talking with you.