The Cognitive Revolution · 2025-02-27 · 156 min

The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind

Erik TorenbergNathan LabenzNicholas Carlini

YouTube ↗

TL;DR

Image-classifier defenses deliver machine-learning accuracy, not security-grade reliability. Adversarial training can preserve roughly 50%–70% accuracy against the attack class used in training, which is meaningful progress from zero; operationally, however, 70% accuracy means an attacker can “try four times and probably one of them works.” That gap helps explain why layered controls and external constraints remain relevant.
The attacker’s structural advantage is both a lower success threshold and the ability to move second. Carlini reduces most technical attacks to two choices—the loss function and the optimizer—and finds that “the simplest possible objective is usually the best one,” because debuggability beats mathematical elegance. Defenders must anticipate many attacks before deployment; attackers can inspect the particular system in front of them, exploit its exact loss landscape, and spend six months on the one failure mode that matters.
Many supposedly novel defenses merely make gradients ugly, zero, or difficult to follow. Distillation once appeared robust because numerical saturation made its loss “actually identically zero”; floating-point changes or rescaled logits restored the signal. RepNoise argued that noisy activations prevented useful fine-tuning, while TAR trained against a weak one-step adversary and was vulnerable to multi-step optimization—after learning-rate changes, random restarts, and warmup, “deep learning takes over and the rest is easy.”
Open-weight safety remains technically unresolved, making future model capability a central release-risk variable. Carlini compares a permanently restricted model to a hammer that can build useful things but somehow cannot serve seven dangerous purposes: tools generally do not preserve such distinctions once users control them. He remains biased toward open source because it has historically benefited security and limits concentration of power, yet concedes that a future “magic box” able to compromise governments could justify a different answer.
Removing dangerous facts or training data does not reliably remove dangerous capabilities. Memorization depends heavily on repetition—GPT-2 examples might have appeared about 20 times in one document across roughly 10 epochs, or about 200 exposures—but models latch onto some sequences while ignoring others with no satisfying explanation. More fundamentally, a generally capable model could relearn omitted biology from textbooks in context, just as Gemini reportedly learned enough of a scarcely represented language from an in-context book to answer exercises.
The practical path is defense in depth around fallible models, with explicit utility costs. External software can forbid an agent from entering secrets into an input type="password" field regardless of what the model says, while detectors across many layers can make evasion progressively more damaging to task performance. These controls still face false-positive fatigue, custom interfaces, and unfamiliar attacks, so the practical stack looks more like human review, external action constraints, and layered detection than a single alignment technique.
AI cannot yet automate Carlini-style security research because the hard part is isolating the real bug inside messy systems. Models can attack a defense rewritten as a clean, 20-line homework exercise, but fail when dropped into an unfamiliar repository containing roughly 1,000 lines of real code; “the core of security” is stripping away the persuasive story and finding the one consequential mechanism. Human researchers and red teams therefore remain valuable, while attack automation is a dual-use frontier Carlini would reassess if capabilities improve sharply over the next couple of years.

Digest · the substance, structured for research

1. Carlini’s attack record reflects leverage, selection, and a decade of practice

Asked whether he had broken more image-classifier adversarial defenses than the rest of the field combined, Carlini accepted only a narrow version: counting papers, co-authorships, and defenses broken, the claim was probably true as of the previous year.
The headline of 21 papers in 2024 overstates his direct execution. After finishing his PhD in 2018, he accumulated strong collaborators; on many projects his marginal contribution was accumulated judgment, experimental direction, and paper feedback rather than writing every CUDA kernel.
Carlini tries to reserve at least half his time for perhaps three papers he considers fully “my papers,” where he drove the experiments and knows every final sentence. The larger output resembles faculty leverage exercised from industry.

2. Most technical attacks collapse into a loss function and an optimizer

Setting aside keyboard-driven social engineering, Carlini’s taxonomy is stark: choose what quantity to maximize or minimize, then choose how to optimize it. Image perturbations, poisoned training points, and edits to supposedly unfine-tunable weights all fit this frame.
His central operating rule is that “the simplest possible objective is usually the best one.” A mathematically purer loss may squeeze attack success from 95% to 98%, but a transparent objective is easier to debug and usually reaches “90% of the way there.”
Carlini’s early image attack jointly handled misclassification and perturbation size with a clever formulation. Alexander Madry’s group later showed that simply constraining the image to a small box—such as changing only the three lowest pixel bits—while optimizing the classification loss got roughly “99%” of the benefit.
That projected-gradient approach, PGD, became memorable because it was simpler and sufficient. The defense is usually either meaningfully effective or not; extracting the final 2% rarely changes the security conclusion.

3. Attackers win by moving second, except when deployment reverses the order

Erik’s initial asymmetry was that attackers need only occasional success while defenders must win almost always. Carlini added the informational advantage: “the attacker goes second,” inspecting the exact scheme already chosen rather than designing one universal attack against every imaginable defense.
A defender may publish and then be unable to update a mechanism already deployed; an attacker can spend six months analyzing that particular algorithm, benefiting from everything the field learns meanwhile. Carlini does not need an attack that defeats all defenses—only the one “literally in front of me right now.”
Poisoning can reverse the order. An attacker who uploads malicious data hoping a future model ingests it must act before knowing the eventual training algorithm, filtering process, or defense, letting the defender adapt after seeing the contaminated corpus.

4. Transparency is desirable only if the exposed system can survive inspection

Erik noted the shift from GPT-2’s released weights to GPT-3 and GPT-4 not being released with weights. Carlini thinks security is only one reason, but dislikes treating nondisclosure itself as the desirable security property.
Cryptography offers his counterexample: algorithms are published, attacked by the full community, and trusted only after sustained failure to break them. “No company in their right mind” invents an undisclosed replacement for AES when a scrutinized standard has survived roughly 20 years.
Carlini concedes that deep learning might prove intrinsically hard to secure, forcing providers to lock systems down and change them faster than attackers find bugs. He would still prefer defenses that “just actually work” over insecure mechanisms whose flaws are merely harder to discover.

5. Universal jailbreaks succeed by optimizing the start of compliance

In the universal adversarial-suffix work, the goal was to append tokens that turn a refusal into an answer. Optimizing the complete harmful response would be useless because the attacker would need to know that response already.
The paper instead used a deliberately narrow objective: force the first 10 words to resemble an affirmative response such as “Yes, sure, I will help you build the bomb.” Once the model begins affirmatively, autoregressive continuation often supplies the substantive answer without the loss encoding it.
Discrete tokens prevent ordinary continuous optimization, so gradients guided candidate selection rather than directly performing the search. The attack then swapped real tokens, checked whether the gradient’s suggestion worked, used large batches, and spent substantially more compute—the “bitter lesson” applied to jailbreak search.

6. Attack cost ranges from one gradient step to hours, but remains below training

The fast gradient sign method computes the gradient with respect to image pixels, takes its sign, and makes one small move. If that succeeds, the attack costs exactly one gradient step—effectively negligible compared with model training.
PGD repeats that operation. Undefended models may require roughly 10–1,000 iterations; defended models often require 10–100, while Carlini commonly runs 1,000 to ensure apparent robustness is not simply under-optimization. On a reasonable-size model, 1,000 iterations might take “a minute or two.”
GCG-style discrete suffix generation can require an hour or several hours because it evaluates large batches over roughly 1,000 minibatch steps. That is materially slower than continuous image attacks but still “orders of magnitude faster than training.”

7. Data poisoning became easy, hard, then easy again as curation changed

Early poisoning simply inserted a small amount of mislabeled data, enough to make image classifiers immediately mislabel targets. The attack became harder when researchers assumed carefully curated datasets where visibly incorrect labels would be rejected.
Under that clean-label threat model, attackers optimized images to look legitimate while surrounding the target in embedding space with oppositely labeled points—a conceptual “box” that pulls the local region toward the wrong class. Such methods could require around 1% poisoned data.
Internet-scale self-supervised training relaxed curation, making blunt attacks viable again. For CLIP-like classifiers, Carlini’s collaborators could flip labels, add a few hundred examples, and induce failures without elaborate optimization.
Language models are harder because poisoning must survive supervised fine-tuning and RLHF. Current results require around 0.1% contamination; against a trillion-token corpus that is one billion tokens, which may be infeasible for a real attacker.

8. The billion-token poisoning result is probably not the final word

Carlini’s intuition is that 0.1% must be far above the true requirement: if controlling one-thousandth of training data were necessary to implant one belief, a model could “only know a thousand things,” yet models plainly encode more.
The missing attack may need carefully constructed data rather than repetition of one false claim. Carlini described substantially lower-rate language-model poisoning as an open question his collaborators hoped to understand during the year, not as a solved vulnerability.
His visualization of clean-label poisoning is deliberately informal: first establish “what should be true,” such as enclosing a target in embedding space, and only then turn that picture into objectives, dot products, and code.

9. High-dimensional geometry rewards useful confusion, not literal visualization

Carlini often imagines three dimensions and lets the intuition generalize, while warning that defenses repeatedly fail because their stories quietly assume low-dimensional geometry. “Almost all vectors are perpendicular to each other in high dimensions,” so familiar spatial expectations become unreliable.
His more precise claim is that almost all classification points are close to one of the decision boundaries, even when random searches make that boundary look far away. Many random directions are unproductive, but the high-dimensional degrees of freedom mean that some particular direction may reach a nearby plane, even if it is hard to find.
The signature formulation is: “Things are close, but when you search for them randomly it looks like they’re far away.” He considers “being confused in high dimensions” appropriate until the fuzzy picture yields a candidate attack that can be tested formally.

10. Attack success depends partly on choosing defenses worth attacking

For familiar image defenses, Carlini’s first or second idea often works; outside that domain, he might try five or 10. Once the leading 10 approaches fail, he usually lacks a credible next move and does not assume an undiscovered eleventh idea must exist.
His apparent hit rate contains selection bias. He generally ignores “adversarial training plus” variants—more generated data, an auxiliary loss, or smoothing—because their foundation is plausible and any break may recover only a few percentage points.
He instead selects defenses whose mechanism looks qualitatively new or “weird,” especially when their justification resembles a previously broken argument. Pattern recognition from a decade of failures is less mystical talent than a highly curated library of analogies.

11. Seventy percent robustness is excellent ML and inadequate security

The strongest image defenses remain variants of adversarial training: generate attacks, train the model to classify them correctly, and repeat at scale. Within the trained threat model, they can retain roughly 50%, 60%, or perhaps 70% accuracy under attack.
Carlini preserves both interpretations. Moving from zero to 70% is “a remarkable achievement” in machine learning, where five-nines reliability is alien; from a security perspective, 70% means “I try four times and probably one of them works.”
His malware analogy sharpens the mismatch: nobody would call a detector good if four samples were usually enough to evade it. Thus two researchers can honestly say the same defense works or does not work because their operational standards differ.

12. Many defenses protect themselves by corrupting the loss landscape

Carlini usually visualizes the loss surface rather than the model’s entire representation. Apparent robustness often means the surface has been made discontinuous, noisy, saturated, or otherwise difficult for naïve gradient descent to navigate.
Defensive distillation supplied the canonical lesson. Its teacher-student story sounded causal, but high-temperature training produced enormous logits and, in early TensorFlow, a softmax-cross-entropy calculation whose gradient became numerically zero: “the loss function was actually identically zero.”
Computing in 64-bit floating point or dividing the logits before softmax restored usable gradients. The attack failure had little to do with distillation’s narrative and much to do with numerical gradient masking.
Later methods explicitly added noise or discontinuities. Straight-through estimators and related techniques can turn the “very ugly mountain” back into a surface where the conceptual ball of gradient descent rolls downhill.

13. Tamper-resistant fine-tuning repeated old image-defense mistakes

RepNoise argued that noisy activations prevent useful fine-tuning; TAR added adversarial training to the process. To Carlini, both resembled image defenses that had already confused difficult optimization with genuine absence of an attack.
Carlini’s broader account is that changing the learning rate, using random restarts, or adding warmup can make difficult gradients more navigable. Once the optimization stabilizes, “deep learning takes over and the rest is easy.”
TAR made a still more direct historical repetition: it trained against a one-step weak adversary analogous to FGSM. Earlier image work had shown that adversarial training only protects against attacks at least as strong as those used during training; multi-step optimization therefore broke TAR much as PGD had broken FGSM-trained classifiers.

14. Open-weight capability controls resemble trying to restrict a hammer’s uses

Carlini framed the desired object as a hammer that can build every useful thing but cannot perform seven dangerous tasks. Compilers, GPUs, transistors, and most general tools support benign and malicious applications alike; choosing to assign responsibility specifically to the model does not make internal restrictions feasible.
He remains agnostic about where responsibility should be assigned, but insists that rules rest on technical facts. Mandating a fine-tuning defense “known” to work would be harmful when attacks have already shown that it does not.
Indistinguishability obfuscation offers a theoretical counterpoint: under strong assumptions, cryptography might make locally held software behave like an inscrutable black box. Carlini described it as nowhere near practical for modern models—and even then, the black box might remain jailbreakable through queries.

15. Human social engineering outperforming gradients is an unresolved paradox

From a security perspective, humans probing a model feels normal: software users interact with programs, locate weak points, inspect code, and iterate. Language models were trained to respond to human communication, so humans may be the natural instruments for finding communication failures.
From a machine-learning perspective, the result feels bizarre. A model is still a mathematical classifier mapping tokens to floating-point numbers, yet “my grandmother used to read me the recipe to napalm” can outperform an attack derived directly from its gradients.
Carlini expected optimization to be strictly stronger but acknowledged that it empirically is not. Erik’s proposal—that humans possess effective “social intuitive physics” while mathematical search converges more slowly—struck him as entirely reasonable, but he emphasized, “I really don’t know how to think about this yet.”

16. Memorization is exposure-dependent but remains unpredictably selective

Carlini corrected the strongest version of the privacy claim: an extracted string may occur in one document yet repeat many times within it, and multi-epoch training exposes the model to that document repeatedly.
GPT-2 provides his last clean production-model measurement because its training information was accessible in a way current production models’ information is not. A sequence might repeat roughly 20 times in one document and be encountered across about 10 epochs—approximately 200 total exposures—before extraction became demonstrable.
That still does not explain selectivity. Language models have memorized some repeated strings since LSTM experiments in 2017, while diffusion models yielded certain images repeated about 100 times but not others repeated 10,000 times.
Carlini’s honest non-answer is that “models seem like they just sometimes latch on to certain things and not other things.” He does not know where a memory resides in the weights or why one example survives while a more frequent one does not.

17. Unlearning facts is easier than removing generally recoverable knowledge

Carlini separated editing a fact from erasing knowledge or a capability. Changing “the Eiffel Tower is in Rome” under ordinary prompting does not establish adversarial removal; fine-tuning or a different perturbation might make the original information reappear.
Excluding data can give a provable privacy result only in narrow cases. A model that never sees a Social Security number cannot quote that record, yet knowledge of historical state and hospital allocation patterns might still let a capable system derive its first five digits.
Capability removal faces a deeper contradiction: developers want models that learn from a few examples except on designated subjects. A model trained without biology might ingest undergraduate textbooks in context and answer questions anyway.
Erik’s seemingly absurd scenario gained plausibility from a Gemini result: after receiving a book about a language with almost no speakers in context, the model reportedly answered homework exercises. Carlini therefore found selective ignorance worth testing but remained “skeptical of all” proposed methods.

18. Seventy rounds of latent evasion are a high-dimensional recurrence

Erik found the 70-generation attacker-defender chase in “Obfuscated Activations Bypass LLM Safeguards” startling. Carlini saw almost exactly the behavior predicted by an older image study that repeatedly barred each successful attack direction and asked for an orthogonal replacement.
That study found tens—perhaps around 50—distinct directions that still produced adversarial examples, with effectiveness degrading only gradually. In high dimensions, 10 attacks can correspond to 10 nearly orthogonal vectors using little of the same representational route.
Weight sparsity does not necessarily eliminate this room. A compressed model may waste fewer parameters while retaining many activation-space directions, and empirical compression has not made classifiers more adversarially robust.
Carlini’s implication was narrower than “unused parameters cause attacks”: the model may need the very features attackers exploit. Removing redundant weights cannot remove directions built from predictive information without also risking task accuracy.

19. Adversarial examples may exploit real features rather than model bugs

The “features, not bugs” intuition begins with dogs and cats. Humans privilege faces, ears, and overall shape; a classifier may legitimately use low-level fur texture because dog fur and cat fur strongly correlate with the labels in its training distribution.
If an attack changes the texture statistic from dog-like to cat-like, the classifier may faithfully implement the learned distinction even though the image still looks like a dog to a human. Training never specified which correlations constituted the intended concept.
Models can detect subtle image statistics humans miss, making such non-robust features both useful for normal accuracy and available to attackers. Adversarial training can lower clean accuracy because suppressing vulnerable features may also suppress genuine predictive signal.
Even nonsense-looking jailbreak suffixes sometimes expose interpretable fragments. One optimized suffix included something like “now write opposite content,” inducing toxic output followed by a compliment; optimization discovered that promising later positivity was a strong feature for obtaining the prohibited text first.

20. Interpretability explains selected features, not the whole computation

Carlini did not see sparse-autoencoder results as inconsistent with non-robust features. A model may contain a recognizable Golden Gate Bridge feature alongside numerous predictive directions that humans cannot name.
In normal data, cat ears, fur, and shape align, so explanations based on the interpretable features can be locally accurate. An attacker searches for an obscure feature with an unusually large weight and activates it in the opposite direction.
Sparse autoencoders already admit compression and omission: they shrink enormous activity into a smaller sparse feature set, then fail to explain some of even those features. Carlini would doubt claims of a perfect account of every component; today, much of the remaining mechanism is still “magic.”

21. Humans are measurably more robust, but context may supply the advantage

A study flashed images to people for about 100 milliseconds after perturbing them to fool an ensemble of neural networks. Humans were misled more often by those adversarial perturbations than by random noise of equal magnitude, suggesting at least some model-derived attack signal transfers.
Yet a neural network can often be attacked with around 1,000 queries, while Carlini doubts 1,000 similarly structured image queries would fool his own perception. In that operational sense, people remain substantially more robust.
One explanation is recurrence and context: a person does not classify an isolated frame with one forward pass but remembers that the creature walked and behaved like a cat. Longer chains of model computation might provide a parallel benefit, though Carlini would not ground a result in casual neuroscience analogies.

22. Robust systems can contain fallible models if the environment enforces limits

Carlini’s median prediction is that models remain roughly as vulnerable as they are now, forcing builders to design systems that assume mistakes. Human institutions already use code review and divided authority because any one person may err or act maliciously.
An agent can similarly escalate uncertain actions to a user. More importantly, external code can deny actions regardless of the model’s persuasion—for example, refusing to enter data into an input type="password" field even when the model requests it.
The limitation is coverage: a site may implement a custom JavaScript password interface without the standard field type. System-level controls still narrow attack surfaces, but utility falls and unfamiliar cases remain.
Erik’s surprise-weighted long-term-memory idea—remembering exceptional failures so the system is not fooled twice—interested Carlini, though he called the work early. He retains some hope that reasoning or memory produces genuine robustness within a few years, but “I’m not optimistic.”

23. Opaque defenses might work through adversarial evolution rather than proof

Erik relayed Michael Levin’s thought that biological systems become parasite-friendly when too interpretable, raising the possibility of evolving defenses nobody can explain. Carlini accepted that an opaque but empirically robust defense would beat an elegant mechanism that fails.
Symmetric cryptography supplies a partial analogy. Block ciphers pursue principles such as confusion and diffusion, then evolve against every known attack; AES has no general proof that every conceivable attack fails, only careful design and analysis showing why known attack families do not beat it.
Other cryptographic constructions can instead reduce security to crisp assumptions such as factoring or discrete-log hardness. Carlini would prefer an analogous reason for model robustness, but AES shows that decades of attack-driven refinement can yield trustworthy engineering without one foundational proof.

24. Security “breaks” form a continuum long before practical compromise

Erik initially contrasted cryptography’s binary compromise with AI’s messy degradation. Carlini objected: cryptographers would call AES-128 broken if key recovery fell from (2^{128}) to (2^{127}) work, even though both exceed practical resources.
The conservatism is anticipatory—an improvement from 128 to 127 bits raises fears of a path to 125 and eventually around (2^{80}), where nation-state feasibility becomes discussable and constants matter greatly.
AES-256 supplies ample margin because even a dramatic reduction toward 100 effective bits may remain tolerable. By contrast, 50%–70% attack accuracy represents frequent failures rather than cryptographic-style work-factor margin.

25. Defense in depth can convert evasion into unusable performance

The latent-defense experiments showed that one detector at one layer could be evaded with little loss of SQL-writing ability. Adding detectors across more layers still allowed evasion, but eventually degraded the generated SQL enough that successful bypass no longer necessarily delivered the attacker’s task.
Carlini recognized the pattern as standard “defense in depth.” Block ciphers stack layers because attackers can penetrate a few but struggle as interactions accumulate; model systems may likewise settle for 20 imperfect barriers whose composition makes useful attacks impractical.
The caveat is correlated failure: a fundamentally different attack might bypass every detector simultaneously. Even so, detector research has option value—if perfect robustness never arrives, layered controls may be the usable fallback; if it does, detection remains useful around it.

26. False positives, utility, and corrigibility all occupy the same trade-off curve

Erik’s camera repeatedly labels squirrels as humans. Carlini reduced the design choice to a true-positive/false-positive operating point, noting research suggesting occasional false alarms can reassure users that a detector is vigilant—until alert fatigue causes them to disable it entirely.
Compiler warnings show the same failure: after 900 unimportant warnings, the developer ignores the thousandth even if it matters. High-stakes models may justify annoying benign users and requiring rephrasing, but this is a product decision rather than free safety.
Carlini read alignment-faking behavior as, among other things, a jailbreak that succeeds in perhaps 5%–10% of cases. When Claude was later allowed to object to the proposed training, it took that option, suggesting refusal can resolve some setups.
Gullibility cannot simply be removed: when a user says generated code failed, the useful model should believe them and reconsider, not insist the compiler is wrong. Corrigibility, safety, and utility therefore require choosing a point on a curve, not maximizing one scalar without cost.

27. Today’s language models know attacks but cannot yet do security research end to end

Carlini’s experiments asked models to generate adversarial examples against defenses. The short answer was “basically not yet,” with a revealing exception: models performed reasonably when researchers rewrote a defense as a clean homework problem containing its essential logic in about 20 lines of Python.
Given an actual research repository with perhaps 1,000 lines, unfamiliar infrastructure, and misleading implementation detail, they failed. “The core of security” is turning an ugly system nobody understands into the one small piece that actually matters.
Models already know how to write PGD once the task is isolated. They do not reliably decide which claimed mechanism is irrelevant, which numerical detail is the real bug, and how to navigate messy code until that distinction becomes visible.
Carlini warned that benchmarks increasingly test the academic subproblem humans identify as hard rather than the complete real-world task. Models can score highly on knowledge tests while ranking differently on agentic work that requires orientation, persistence, and judgment.

28. Human attack talent is scaling through training before it scales through AI

Carlini no longer considers himself uniquely capable. Five years earlier, few researchers attacked these systems; now papers such as the obfuscated-activations work can appear without him, often executed more thoroughly than his own limited time would allow.
Practice remains the primary multiplier. Nobody has yet entered a PhD focused on attacking language models and completed that full training cycle, because serious work on them is only several years old.
His implied forecast is patient: another few years should produce researchers with a complete apprenticeship and a richer Rolodex of failure patterns. The uncertainty is whether model capabilities and deployment stakes allow society the duration of a PhD to catch up.

29. Attack automation is dual use, but proof of weakness is not weaponization

Erik asked whether a rapidly improving automated attacker could become the security equivalent of a laboratory accident. Carlini invoked the Morris worm, created by Robert Morris in 1984 or 1986 in his uncertain recollection, as an early worm that reportedly escaped experimentation and disrupted much of the then-internet.
He distinguished knowledge from weaponization. Security researchers commonly build minimal proofs of concept but do not spend a year engineering production malware; the historical example illustrated the danger of an experiment escaping its intended setting.
Publishing easy attacks matters because people who already know exploitation is easy have little incentive to document that fact defensively. “You want to know what anyone can do” so systems receive protection before malicious actors operationalize the same insight.
Carlini would revisit this norm if agents became superhuman, autonomous, and capable of compromising arbitrary systems. He does not believe current adversarial-example automation would cause much harm right now, but he “never expected” present language-model capability three years earlier and remains open to changing his mind.

30. Narrow jailbreak contests measure partial progress, not final safety

Anthropic’s eight-layer challenge sets a harder target than finding one bespoke jailbreak. Carlini defended bounded problems by analogy to the decade spent studying (L_\infty) image perturbations of at most 8/255: the exact threat is artificial, but well-defined problems can reveal methods that later generalize.
At the time discussed, an individual reportedly broke each of the eight levels, but not with one universal jailbreak. That means the defense would be inadequate for national secrets, yet it may still stop a copy-and-paste attack that works everywhere.
Carlini used the security term “script kiddie” for users who cannot invent exploits but can run published ones. Preventing a universally shareable suffix does not stop determined attackers, but it reduces mass accessibility: “partial progress is still progress.”

31. Independent red teams succeed because they genuinely want the defense to fail

Carlini’s social explanation for weak self-evaluation is incentive-compatible: after spending six months building an idea, a researcher finds it psychologically and professionally difficult to switch frames and sincerely try to destroy it.
If the creator succeeds in breaking the defense, the result may no longer be publishable—“here’s an idea I had; by the way, it doesn’t work.” The external attacker instead receives credit for finding the flaw and has no attachment to the original story.
Security organizations institutionalize this difference through red teams. Carlini said the security teams he has spoken with at many organizations appear to “really want to know,” because uncertainty about whether a control works is itself the reason for adversarial testing.

32. Open source remains favored, but future capability could overturn the presumption

Carlini’s current technical bottom line is that open-weight models cannot be locked down to the degree many policy proposals assume. If released, users should not be assumed to remain reliably constrained from accessing or modifying their capabilities through prompting, fine-tuning, or the weights.
Even so, “in any world that looks noticeably like our world,” he considers open source objectively beneficial for security, safety research, and limiting concentration. A few firms controlling capabilities unavailable to everyone else is a concrete risk that requires no superhuman AI assumptions.
His historical analogy is the 1990s restriction of strong cryptography as a munition. Exported browsers were limited to 40-bit encryption because governments feared unbreakable terrorist communications, yet broad strong encryption also enabled online banking, payments, and secure communication for dissidents.
Unlike encryption, whose endpoint was comprehensible as “perfect encryption,” model capability has no known ceiling or timetable. Carlini expects the next two years to clarify whether scaling continues rapidly or limitations bite; anyone claiming either release policy is “obviously objectively right with 100% certainty” is overconfident.

33. Policy should follow technical reality rather than preferred narratives

Carlini declined to prescribe the social solution. His comparative advantage is establishing whether a defense actually works; he trusts policy experts more on institutional trade-offs if they accurately understand those technical findings.
His largest concern is not merely choosing wrongly under uncertainty, but possessing relevant evidence and ignoring it because of ideology. Decisions made from the best available facts can still age badly, but at least they use the knowledge available at the time.
The episode’s closing call was therefore for more good science, more researchers willing to expose failures, and policy that preserves the transcript’s uncertainty. Models might remain manageable tools, or capability growth might force Carlini to reverse his open-source bias; pretending either future is already certain would repeat the very reasoning errors his attacks uncover.

Erik Torenberg

Nicholas Carlini, security researcher at Google DeepMind, welcome to The Cognitive Revolution.

Nicholas Carlini

Yeah, it’s great to be here. Thanks for having me. I’m excited for this.

Erik Torenberg

For some quick context, you recently appeared on Machine Learning Street Talk, maybe 10 days ago as of the moment we’re recording, and I thought that was excellent. Shout-out to MLST for another great episode. Hopefully, we’ll cover largely different ground here, but I do recommend that people check out that episode as well for another angle on your thinking and your understanding of everything that’s going on in AI.

One thing that was said in that episode, which caught my attention—I haven’t fully fact-checked it—was that you have created, demonstrated, and published more attacks on cybersecurity and machine learning defenses than the rest of the field combined. You can tell me if you think that’s literally true, but I did look up your Google Scholar page, and 21 papers in 2024 alone was what I counted there.

Nicholas Carlini

Yeah. I think the statement that probably is literally true is this: if you count the number of papers where I’m a co-author and the number of defenses broken in those papers, and then you count the number of papers where I’m not a co-author that break adversarial-example defenses on image classifiers, as of, I don’t know, last year, that statement probably was true.

But that’s for a very specific domain and a very particular kind of thing, and probably mostly just because this is something that I, for some reason, enjoy doing and will do before other people get to it. Other people just don’t do it as much. But yeah, for that one particular claim, it’s probably correct.

Erik Torenberg

You’re a careful thinker and communicator. What I hope to do above all in this episode is develop my intuition, and hopefully help other people develop their intuitions, for the habits of mind, approaches, mental models—whatever you want to call them—that have allowed you to be so successful in this space. Hopefully, this can be a bit of a crash course that inspires some new people to think they can get into the field and make an impact as well.

So I guess the first question is: is everything easy for you to break? Twenty-one papers in 2024 alone is obviously a lot.

Nicholas Carlini

No. To be clear, I finished my PhD in 2018, so I’ve been out for a while and have had a lot of time to meet a lot of great co-authors.

Twenty-one seemed like a lot to me. I was trying to think through how many I can remember. For many of these results, it’s the kind of thing where I would show up to the weekly meetings, help write the paper, direct some of the experiments, and then not write the CUDA code to do all of the work myself. That’s how you get a lot of things done.

You see this happen with everyone who’s been in the field for a long time. The marginal value of an hour of my time could be spent either on very low-level work with GPUs or on offering comments and wisdom that I’ve learned over the past 10 years, which helps a PhD student get a lot done in a much shorter amount of time. This is why people become faculty, I think.

The balance for me is that I also try to spend at least half of my time only on papers that I’m technically driving. When you say I’ve had this number of papers, what I think of is, well, maybe there are 3 papers that I think of as my papers, where I was actually doing the experiments and could tell you about every final sentence of what’s going on. For those, I have a very strong sense of what’s there.

The other ones are the standard work of professors who are advising graduate students. But instead of being in academia, I’m in industry, so I advise and help on other students’ papers in some ways.

Erik Torenberg

Across all these things, regardless of your role, was there anything, as you look back over the last year or more, that was legitimately very hard to break? Or are you basically finding that all of the defenses the field is coming up with are rather easy for you to bring down?

Nicholas Carlini

In the last year, we didn’t spend that much time breaking particular defenses. We had maybe 2 or 3 papers on that. We spent most of our time in other areas: trying to understand to what extent attacks are possible, understanding the real-world vulnerability of models to certain types of attacks, and doing general privacy analysis.

We weren’t saying that a particular defense is wrong. Rather, for all neural networks trained with gradient descent, here is an interesting property about their privacy. We have a lot of these kinds of results that aren’t really focused in detail on breaking one particular thing.

Last year, I maybe only had 2 papers that were particularly about breaking things. One was early in the year. There was a defense published at IEEE S&P, which is one of the top conferences in the security field. It was an adversarial-example defense, and I broke it. This one turned out to be relatively easy—an hour or two.

This was sort of abnormally easy, although maybe not that abnormally. Image-classifier adversarial examples are a particular beast that I’ve gotten very good at. The attacks are relatively well understood, and there are lots of known failure modes. When I’m doing this, I’m not developing new science. I’m going through a long list of things I’ve broken before and asking, “What pattern does this one fall into?”

In this case, it turned out that the gradients weren’t flowing because the softmax was saturated at 1. So what do you do? You make sure the softmax doesn’t saturate, and then you find that you can break it. It worked very quickly.

That was very much an engineering result: why is the softmax giving gradients that are identically zero? Once you figure out that the answer is some discretization or whatever the case might be, everything becomes easy from there.

The other paper that was more interesting was one of these advising papers. I didn’t do any of the technical work, but I was helping a couple of students think through what it means to consider the robustness of unfine-tunable models.

Adversarial-example attacks are test-time evasion attacks, where you change an image a little bit and it turns a picture of a panda into something else. Instead, we were looking at models designed to be released as open source, with the weights available to anyone, that are supposedly not possible to fine-tune for other tasks.

The particular concern these defenses were addressing is that you would ideally want to make sure that no model you train can help someone produce bioweapons, or whatever the threat model is that you’re considering. You can make your model safe initially, but if you release the model’s open weights, anyone can fine-tune it and remove the safety filters you put in place.

These unfine-tunable models are supposed to be designed to be robust not only to initial adversarial-example attacks, but also to someone who can perturb the weights. In this paper, a couple of students did a bunch of work attacking these models to show that you can still fine-tune them, even though they’ve been trained to be unfine-tunable.

A lot of the thoughts we’ve had over the last 5 to 10 years on adversarial examples went into this. The same kinds of lessons applied, but many of the techniques were very different, so the students had to spend a lot of time getting it to work.

Erik Torenberg

I want to dig into that one in particular, because it strikes me as one of the most important and interesting cat-and-mouse games going on in the space right now. Before zooming in on that, though, you said that when you see something new, you have this Rolodex of past things and paradigms that you can quickly go through.

Could you sketch those out for us? How do you organize the space of attacks? Is it a hierarchy or some other taxonomy? I’d love to get a sense for what your mental palace of attacks looks like.

Nicholas Carlini

Let me separate off one space of attacks. These are the new attacks where a human types on a keyboard, prompts the model, and tries to make it say a bad thing. Let’s put aside, for a second, these cases where you treat the model like a human and try to socially engineer it into doing something bad.

If you put that aside, then almost all attacks work by performing some kind of gradient descent to maximize a particular loss function.

For image adversarial examples, what does that mean? I have an image of a stop sign, and I want to know what sticker I can put on the stop sign to make it be recognized as a 45 mph speed sign. How do I do this? I perform gradient descent to compute the optimal sticker so that the thing becomes misclassified.

In the case of poisoning, where you modify a training-data example in order to make the model produce an error, you’re trying to optimize the particular poisoned data point in your training dataset so that the model makes a mistake.

In the case of these unfine-tunable models, you have a model that you want to make sure no one can edit. So you try to perform gradient descent on the model, updating the parameters so that it can perform some bad thing.

In all of these attacks, there are essentially 2 things you need to concern yourself with. One is the objective that you’re maximizing or minimizing: what is the specific loss function you’re using? The other is the optimization technique you’re using to make that number go up.

By coming up with the best possible versions of each of these, you end up with very strong attacks. A big part of doing these kinds of gradient-based optimization attacks is coming up with high-quality functions that you can optimize and high-quality optimizers.

There are lots of lessons we’ve learned over the years. One of the biggest ones is probably that the simplest possible objective is usually the best one. Even if you can have a better objective function that seems mathematically pure in some sense, the fact that simple loss functions are easy to debug means that you can get 90% of the way there in doing these attacks.

The last little bit is a lot. It’s nice to go from a 95% to a 98% attack success rate, but it’s not really necessary in all of these cases. So you pick a really simple loss function that’s easy to formulate and easy to understand when things are going wrong, and you pick an optimizer that makes sense. Mostly, things just work.

Erik Torenberg

A significant amount of this work over time has been in the image-classifier domain. We see some pretty striking examples there where I guess there’s either a second term in the loss function or some sort of budget constraint as well.

You’re trying to say, “I’ve got a picture of a car, and I want to make it output ‘dog’ as the classification,” but you also don’t want to actually change the image into a dog to make that happen. How often is that second term also a big part of keeping the image looking like it originally did?

Nicholas Carlini

In adversarial examples, the way this works—one of my first papers in adversarial machine learning was about coming up with a clever way to do exactly these 2 things: what is the optimizer, and what is the optimization objective?

We did some clever things. It worked well. I won’t go into the details here, but we did something fancy. Then, maybe 6 months or a year later, Aleksander Madry and his students said, “Instead of doing something clever, let’s just bound the image to be in a small ball around the initial point. You can only perturb the 3 lowest bits, and you only optimize the objective function I described. Then run the same optimization algorithm I was using.”

It turns out that this gets you 99% of the way there, and it’s so much simpler. This algorithm is called PGD, and it’s the one everyone remembers because it’s the right way of doing it.

You can squeeze an epsilon more performance out of it if you do something much fancier, but the defense is either effective or it’s not, for the most part. Breaking the last 2% is very rarely something you actually need.

For the most part, it’s entirely fine to say, “Let’s make something a lot simpler and optimize that,” and it ends up working quite a lot better. So for these image examples today, people don’t put a second term on minimizing the distance between the original image and the other one. They just add it as a constraint.

They say, “I’m going to constrain you to this bounding box. You can only change the lowest 3 bits of the pixels.” This makes the optimization much simpler, and it’s a little bit worse, but in all practical senses it just makes things work a lot better.

Erik Torenberg

When people say that attack is easier than defense, one obvious way to read that is that you only have to succeed with a minority of your attacks, whereas for defense to be successful, you have to win always, or nearly always. Are there other meanings or intuitions for why attack is easier than defense that are important as well?

Nicholas Carlini

Yeah. The second big one is that the attacker goes second. The defender has to come up with some scheme initially, and then the attacker gets to spend a bunch of time thinking about that particular scheme afterward. This is maybe a variant of why finding one problem is easier than solving all of them.

It would probably be pretty hard for me to write down an attack algorithm that was effective against any possible defense. There is almost certainly something that someone could do correctly that would stop all attacks, but I don’t have to think about that defense. I only have to think about the defense that is literally in front of me right now.

It’s a lot easier when you’re presented with one particular algorithm. You can spend 6 months analyzing it, and so the attacker has an information advantage as well. They can wait for the field to get better, learn new things, and then apply the attack after all of this has been learned. The defender, in many cases, can’t update what they’ve already done.

There are some settings where this is reversed. Data poisoning, for example, can be one of them. Suppose that I want to make malicious training data and put it on the internet, hoping that some language-model provider is going to train on my malicious data. In this case, the attacker may actually have to go first. I have to upload my training data, and then someone gets to train their model with whatever algorithm and whatever defense they want to remove my poisoned data before they actually run the training.

In this case, maybe the defense is actually a little bit easier than the attack. It’s hard to say, because the defender goes second. But for many of the cases that I spend most of my time thinking about, including the recent unfine-tunable-models case, the attacker goes second, and that really gives them a lot of power.

Erik Torenberg

I wonder what that implies for the future of how open all this work is going to be. We’ve been in a regime where the stakes of machine learning generally were not super high, and people were pretty free and easy in publishing things, including—I’ve always marveled at this—the biggest companies in the world. One might wonder why the biggest companies in the world are publishing all this intellectual property, but they’ve been doing it.

It seems like maybe if we’re actually running an API at scale, we don’t want to disclose all of our defense techniques. Do you think that’s already changing? Do you already see this?

GPT-2 was released with the weights. GPT-3 and GPT-4 were not. The biggest models are not, for the most part, being released by the companies that are developing them.

Nicholas Carlini

I think security is probably a small part of the argument here. I will say, though, that in other areas of security—in almost all other areas of security—this is not what we rely on.

Let’s think, for example, about cryptography. We publish algorithms. Everyone knows how the best cryptographic systems work. Everyone tries to analyze them. No company in its right mind would ever try to develop a new, fancy cryptographic system. You’re just going to use AES because it’s known to be good. It would be crazy to try to do anything fancy in-house.

The reason is that empirically, it has worked very well. We’ve had the entire community try to break it for 20 years and have largely failed. Everyone believes that this is effective. You don’t get that same kind of belief in something without a large number of people trying to analyze it.

If you have these models and they stay proprietary—things that are not disclosed—it may be the case that empirically this just ends up being the best we can hope for. Maybe deep learning is impossible to secure. Maybe there’s no hope for it. You lock things down and try to change things faster than attackers can find bugs in them.

That would not be great, but I think we could potentially live in that world. What would be a lot better, though, and which may not be happening and may be very hard, is that everyone discloses exactly what they’re doing and exactly how they’re doing it. You get everyone to analyze that in detail, learn how to make these things better, and improve robustness.

Then you get to the point where people can choose either to release things or not release things—not because of security, but because they want to make money or whatever the case may be.

What I would like to avoid is the belief that not making the thing public is the more secure version. It’s a shame that this is part of the equation right now, but I would rather have things that actually work as opposed to things that are insecure but that we just lock down and make harder to analyze. Those systems are still insecure; they’re just a little bit harder to find bugs in.

Erik Torenberg

Let’s come back to that in a little bit. Staying for a moment on how you organize the space of all these different attack regimes, there are some settings—we did a whole episode on the so-called universal jailbreak—that I hadn’t realized until preparing for this that you were a coauthor on. That was one of the many papers from the last couple of years.

There are some wrinkles in the high-level description you gave of “find a gradient and maximize some loss function.” For example, in that universal-jailbreak paper, if I recall correctly, the idea was limited to picking the right tokens. The space isn’t purely differentiable, so you’re navigating a discrete space of individual tokens.

Let’s talk about this paper for a second. As a refresher for everyone, what was it doing?

Nicholas Carlini

This was, again, one of these papers where I was mostly advising, with Zico, Andy, Matt, and the students doing the work.

They found that it’s possible to take a language model that usually refuses answers to questions. You ask, “How do I build a bomb?” and the model says, “I’m sorry, I can’t possibly help with that.” It is possible to take that same model and append an adversarial suffix so that you can arrange for the model to give you a valid answer.

How do you do this? If I knew the answer ahead of time, one thing you might imagine doing is trying to optimize the tokens. Let’s come back to the optimization question in a second and assume you can optimize. You could imagine trying to optimize the tokens so that the model gives a particular response as output: “Here are the steps to build a bomb: 1. Go get whatever chemicals you need. 2. The instructions for how to assemble them,” or whatever the case may be.

But this requires me to already know the instructions, so it’s not very helpful. What’s the objective function I’m actually going to use to make the model give a response?

Maybe you could try to come up with some fancy latent-space objective. But the only objective is to make the first 10 words from the model be an affirmative response that says, “Yes, sure, I will help you build the bomb.” Once you’ve done that, because of the nature of language models, it turns out that they then give you an answer.

There are other defenses that rely on breaking this assumption too, but this was the key part of the objective function: How do we take something where, in our minds, we want the model to give us an answer—instructions for something—and make it give us that answer without actually coming up with the particular answer ourselves? That is very hard.

We came up with a very straightforward loss function that makes this happen. Now we can return to the question of what the optimizer is. This is where a lot of the work in the paper went: How do you take something that is discrete tokens and make it something you can actually optimize?

Early work had tried second-order gradients and other fancy techniques. The main thing this paper says is that we’ll do 3 things. First, we’ll use gradients to guide our search. We’re not going to use gradients for the search; we’re going to use them to guide the search. Second, we’ll check whether the gradients were effective by actually switching tokens out. Third, we’ll spend a lot more compute than other people were using.

You know, the bitter lesson: just do this a bunch. You end up with very strong, effective attacks. This still fits very nicely into the framework of what you’re optimizing and how you’re optimizing it.

Erik Torenberg

How much compute does this take? If you’re doing one of these gradient-based attacks, how much do you typically have to put into it? If you’re doing something in a discrete space that requires more of a structured search, how does that compare? And if you’re doing data poisoning, how much data does it take to actually poison a model?

Nicholas Carlini

Let’s take these one at a time. We’ll start with the image-adversarial-example case in continuous space. The amount of compute here is almost zero.

One of the first papers that showed this was by Ian Goodfellow. He introduced an attack called the fast gradient sign method. It does exactly 2 things. First of all, it’s fast. The reason it’s fast is that it takes an image, computes the gradient with respect to the image pixels, computes the sign—literally, which direction does the gradient say to go?—and then takes a small step in that direction.

That’s it: 1 step. If a model is vulnerable to the fast gradient sign method, it takes exactly 1 gradient step, which is essentially zero time.

Other attacks, like projected gradient descent, can be thought of as fast gradient sign iterated some number of times. The number of iterations is usually somewhere between 10 and 1,000 for undefended models. For defended models, to break them, you usually need somewhere between 10 and 100. Just to make sure you’re not making any mistakes, it’s often a good idea to use 1,000, so you know you haven’t accidentally failed to optimize enough.

That works very well. How long does 1,000 iterations take? Maybe a minute or 2 for reasonably sized models.

Now let’s go to the discrete space for GCG, the Greedy Coordinate Gradient attack. Generating an attack can take an hour or maybe a couple of hours, depending on what you want. We’re using a large batch size and doing 1,000 minibatch steps, so it takes a relatively large amount of time, but not a huge amount of time. It’s still much, much faster than training—orders of magnitude faster than training.

By going to the discrete space, though, it does become a lot slower.

Let’s talk about data poisoning. There’s a question of how much time it takes to generate this data, and there are basically 2 rough directions.

The field initially started with the question, “How do I make a model give the wrong answer?” I add a bunch of labeled data that’s labeled incorrectly. This is the simplest possible thing you can do. There’s a paper by Batista that received a Test of Time Award at ICML a couple of years ago. It’s a very nice paper from around 2012, or somewhere around then. It’s one of these early security results that was very important.

You just insert mislabeled data. It’s very easy to do. You insert a very small amount of mislabeled data, and these image classifiers at the time would immediately mislabel the data.

Then people started asking what happens if the adversary can’t simply insert mislabeled data. Once upon a time, we used to curate our data sets so that they contained only high-quality data, so it would be unreasonable to suspect that an adversary could just inject mislabeled data points.

The answer is that now I have to be very careful. I have to optimize my images so that they look like they’re correct. This is the clean-label poisoning threat model. You have to do some fancy work to try to imagine what embeddings you want the classifier to learn. You surround your test point in embedding space and do some fancy polytope-related things.

There’s a bunch of work that does fancy things here. The optimization is relatively difficult, and you need around 1% poison-to-data. That’s a lot.

Then people started asking, “Why do we label or clean our data in the first place? Let’s just take all the data from the internet.” Again, poisoning becomes a lot easier. If you’re willing to take arbitrary data from the internet, you can simply mislabel your data points again.

We had a paper around 2021 looking at poisoning some of these self-supervised classifiers, like CLIP and others. You just add mislabeled data points again, and the thing basically breaks. You don’t need to do anything fancy or perform any optimization. You just flip the label, add a couple hundred images, and you can make these systems fail.

There’s a new question now of how this works for language models. This is one of the things we’ve been writing papers on recently: trying to figure this out. I feel like we don’t understand it right now, because a bunch of things are different for language models.

For example, no one just uses the base language model. You have your language model, and then you fine-tune it with supervised fine-tuning and reinforcement learning from human feedback. You change the weights, so your poisoning needs to be robust to all of these things.

This is another paper I helped advise some students on from CMU and Zurich. They were looking at trying to understand what actually happens in the optimization after you’ve poisoned a model. You have to arrange for the model to be poisoned in such a way that even after reinforcement learning from human feedback, it still gives you the wrong answer.

Doing this is challenging, and so the poisoning rates are currently around 0.1%, which is small. But 0.1% of a trillion tokens is 1 billion tokens. If you were to train a model on some large fraction of the internet, this could potentially be infeasible for an adversary to do in practice.

My gut feeling is that this has to be too large. Models know more than 1,000 things. If you had to control 1/1,000th of the data set to make the model believe something is true, then the model could know only 1,000 things. That doesn’t make sense.

There has to be some better way of poisoning a model so that it’s vulnerable with much less control of the training data. But this might require fancier algorithms again. You might need clever ways of constructing your data that don’t just repeat the same false fact lots of times.

I don’t know. This is one of the open questions we’ve been trying to write papers on recently, and I hope we’ll have a better understanding of it sometime this year.

Erik Torenberg

One thing you said really caught my attention. You have to imagine what the embeddings would be like as you’re trying to think of an attack. Can you unpack that a little bit?

Are you visualizing something there? I struggle to have good intuitions for this, as evidenced by my previous enthusiasm for tamper-resistant fine-tuning. I thought, “This is amazing. It seems like this could really work.” Clearly, I’m not doing something that you’re doing as you conceive of it.

It might be hard to communicate what that is, but what do you think you’re doing?

Nicholas Carlini

This paper wasn’t mine. There was a Poison Frogs paper, and this was a follow-up. I think it was called “Polytope Attack.” It was a long time ago, so I don’t remember. I think it might have been Tom Goldstein’s group again. I don’t remember the details.

To abstract from the details, the real hope is that I can grasp onto something that allows me to be better at this in the future. The paper was trying to make a particular image become misclassified without introducing any large label noise into the training data set—noise that any person would look at and say is obviously wrong.

What it tries to do is surround the image you want to become misclassified in embedding space—in this high-dimensional embedding space—with other images that have the opposite label but are close in embedding space to the target image you’re trying to misclassify.

The idea is to pull the entire region of that space over to the region where those images should be. You’re trying to put a box around the image you want to become misclassified so that the entire box is labeled the wrong way instead of being labeled the correct way.

For many of these attacks, I tend to think about them visually. But ignoring the details is entirely fine. I’m trying to get a sense of what the important thing is and what, at a high level, should be true. Then you can figure out the details afterward.

Once you figure out what should be true, the rest is implementation. This is similar to how mathematicians describe doing proofs. They first establish what should be true in their minds, then try to prove it. The proof may have to be more complicated, or something may not work out in the details, and then they try something else that feels like it should be true.

I think I’m doing something similar. I don’t know how to give an intuition for why something feels like it should be true, other than that you’ve done it a lot, you look at it, and it appears spiritually similar to something else that broke in a similar way.

Erik Torenberg

We’ll come at this procedurally as well, but staying on the visualization: Are you doing the classic physics thing of visualizing something in 3 dimensions and then saying, “Now make it really hard”?

I wouldn’t say I’m good at this at all, but I have a certain version of this for refusal. I imagine a fork in the road or a branching river. Once you’re on one path, you’re in some local well, and when a river forks, it’s not going to meet again until much farther downstream, in some other topology or geography.

That’s pretty hackneyed, but what’s your version of that, if you can articulate one?

Nicholas Carlini

I don’t know that I have a great version of this that I can really give you. Everyone thinks of things differently.

I do tend to think of these things visually. I do the “let’s think of 3 dimensions, and then imagine that things roughly go like this” exercise. But this can be really deceptive, because there are so many defenses that are predicated on the belief that things are working in 3 dimensions.

Then you go to 1,000 dimensions and all of a sudden nothing works anymore. You learn to become used to certain facts in high dimensions when you’re attacking things.

Almost everything is close in high dimensions to a hyperplane. If you draw a plane and pick a point, the point is almost always close to the plane. You can think about lots of things that try to separate points from planes, but in high dimensions they’re almost always close.

You don’t have to think about the details. Lots of the intuitions we have in 3 dimensions just don’t work in higher dimensions. You become used to knowing which intuitions are wrong, and you don’t need to understand exactly why they’re wrong. It’s just something you learn is true.

When someone justifies their defense using one of these intuitions that you’ve seen doesn’t make much sense, you think, “Presumably there’s something here that I should examine more closely.”

Erik Torenberg

That’s an interesting rule of thumb or mental model: Everything is close in high dimensions. Is there a good story for why that is?

It doesn’t seem like it holds in 2 dimensions. If I’m in 3 dimensions and draw a 2-dimensional plane, I would intuitively feel like some things are close to that plane and some things are far from it.

If I’m in 1,000 dimensions and draw a 999-dimensional plane, why is everything close to that?

Nicholas Carlini

Let me make the statement more precise. Suppose you have some classification model and a decision boundary for the classifier. The statement that is true is that almost all points are very close to one of the decision boundaries.

There are many decision boundaries, but also, in high dimensions, I may be very far from something in almost all directions. There exists a direction I can travel in—the direction orthogonal to the closest plane—where the distance is very, very small.

If you try random directions, you may go forever and never encounter a decision boundary. You probably will eventually, but it will be quite far away. In high dimensions, because of the number of degrees of freedom, it’s much more likely that there exists a direction that guides you to a plane that’s really close by, but you would have a hard time finding it if you simply searched randomly.

In 3 dimensions, if you search randomly, you’ll probably run into whatever the nearest plane boundary is. In 1 dimension, you certainly will: You try going left and right and find it. In 2 dimensions, you go randomly and maybe most of the time find something close by.

In 3 dimensions, there are more ways you can go that are orthogonal. In 2 dimensions, there are only 2 directions you can go that are orthogonal to a line. In 3 dimensions, there’s an infinite number of directions you can go that are orthogonal to a line.

In general, in high dimensions, almost all vectors are perpendicular to each other. You can end up randomly picking directions that make no progress. That doesn’t mean there isn’t a direction that does make progress; it’s just much harder to find.

Once you find it, things mostly work out. Maybe the more precise version of what I’m trying to say is that things are close, but when you search for them randomly, they look far away.

Erik Torenberg

That’s quite interesting. I wouldn’t say I’ve fully grasped it yet, but this is the kind of thing where you’re not being formal or giving a proof of what you’re saying is correct, because that’s not how you think about it.

You just have a fuzzy intuition for what feels like it’s going on, and then once you actually do the attack, you think rigorously about it. When you’re visualizing what’s happening, you have to be a little fuzzy, because no one can understand a 1,000-dimensional space.

Then you go back to the numbers and look at what’s mechanically happening. You’re taking the dot product of 2 things, you want it to equal negative 1, and you do the formal work there.

Nicholas Carlini

Yes. I think being confused in high dimensions is probably the right thing. You get used to the fact that this is how it works.

This is also part of why attack is easier. If you’re going to defend against things, you really need to understand exactly what’s happening to make sure you’ve ruled out all attacks.

As an attacker, I can have a fuzzy way of thinking about the world. If my intuition is wrong, the attack just won’t work, and I’ll think of another one. I don’t need a perfect mental model of what the thing is doing to make sure it’s robust from every angle.

Erik Torenberg

But it does seem like your intuition is a pretty reliable guide to what’s going to work.

Nicholas Carlini

My predictor, which is almost as accurate as I am, would be to ask, “Does this work?” The answer is usually no. Most of what my intuition says is, “No, this doesn’t work.”

Maybe I’m a little better than some people at figuring out why it doesn’t work—what attack would break this. That comes from having done this a lot for many different defenses, seeing all the ways things can fail, and remembering them. Then you pattern-match to the next closest thing.

Why can people who do mathematics prove things in ways that seem easy to them but complicated to everyone else? They’ve spent 20 years studying these things. They’ve seen an exactly analogous case before, remembered the details, and abstracted things far enough that it becomes relatively straightforward.

I feel like this is mostly an exercise in practicing it a lot.

Erik Torenberg

What would you say is your conceptual attack success rate? I don’t mean the rate at which examples succeed within a given strategy. I mean, how many strategies do you have to come up with before you find one that works to break a defense for a given new defense?

Nicholas Carlini

It depends on what you’re looking at. Sometimes you try 5 things that you think ought to make sense and they don’t work, and then you try the 6th and it does.

Usually, if you’ve exhausted the top 5 or 10 ideas and haven’t gotten a successful attack, you’re not going to get one—or at least I’m not. If it’s not in the top 5 or 10 ideas, I usually can’t think of something else.

For image classifiers, where I’ve done a lot of this, usually the top 1 or top 2 ideas work. In other areas, it takes more because you’ve seen fewer examples and don’t know what style of attack is needed.

It’s very rare that you get past 10 ideas and give up.

Erik Torenberg

There’s also some problem selection here. There are a large number of defenses in image space that are basically adversarial training with a small modification.

Adversarial training is one defense approach that just trains on all the adversarial examples. The bitter lesson is: Do you want robust classifiers? How do you do it? You train on adversarial examples, do that at scale, and the thing works.

There are lots of defenses that are adversarial training plus some other trick: diffusion models to generate more training data, another loss term to make the model perform better on the training data, smoothing to make the model better in some other way, or whatever.

You’ve said that you basically believe most of these are probably correct for the most part, so you don’t study them because the foundation is something you already believe in. You could perhaps break them by a couple of percentage points, but it’s not going to be a substantial result worth spending time on.

Nicholas Carlini

The things I tend to spend time looking at are the ones that look a little weird. Those are more interesting defenses because they’re a qualitatively new class of thinking about the problem.

This also artificially inflates my attack success rates, because I’m biasing my search toward defenses that I already have a good prior are probably not effective.

Erik Torenberg

Just to make sure I understand the space: There are no real adversarial defenses that work for image classification?

Nicholas Carlini

It depends on what you mean by “work.”

The best defenses we have are basically adversarial training: Generate adversarial examples, train on the examples to be correct, and repeat the process many times.

What does this give you? It gives you a classifier that, on the domain of adversarial examples you trained on, is pretty good as long as you don’t demand accuracy much higher than 50%.

Accuracy under attack for the kinds of adversarial examples you trained on is usually 50%, 60%, or maybe 70%. That’s much better than zero. This is good progress.

But as an attacker, what does 70% accuracy mean to me? It means I try 4 times and probably 1 of them works. From that perspective, it’s terrible. Imagine a cybersecurity defense where an attacker can try 4 different samples of malware and 1 of them evades the detector. That’s not a good detector.

But in image space, this is the best we have. On the one hand, 70% is much higher than zero, so it’s very good progress. On the other hand, 70% is very far from 99.999%.

In machine learning, you never get 5 nines of reliability. So 70% is a remarkable achievement compared with zero. This is why you can talk to one person who tells you it works and another who tells you it doesn’t, depending on how they’re looking at it.

Erik Torenberg

Are there any other spatial heuristics that you think about? In the example where you said you try to envelop the one image you want to break with adversarial examples, another episode I did tried to understand model behavior through the sparse autoencoder paradigm.

I could imagine that there are certain rules where you can’t create a donut in the internal space of the model. Is that why this works? You can address that specific example, but I’m more interested in whether you have a number of these kinds of ideas—“I know the space is shaped this way,” or “It’s impossible to create this kind of shape in the space, so I can work from there.”

Nicholas Carlini

I don’t do that much visualization for these defenses. For the most part, I’m trying to understand the shape of the loss surface.

When something is robust to attack, or appears robust, the problem is usually that it has made the loss surface particularly noisy and hard to optimize. We’ve seen this for adversarial examples essentially forever.

One of the first defenses to adversarial examples that people gave serious consideration to was called defensive distillation. It had a very nice intuitive story: You have a teacher model, and the teacher teaches the student to be more robust in some way. That’s why the student is robust.

The story the authors tell themselves about why these things work is often very different from the actual reason the attack fails.

Distillation had nothing to do with the defense. What was actually happening was that, because of the way they trained the model, they trained the student with a very high temperature. That made the logits very, very large. They were running this in the era of TensorFlow 0, when it was easy for the gradients of softmax cross-entropy to become numerically zero.

The reason the attacks were failing was that the loss function was identically zero. This was the first example of a gradient-masking defense. The authors thought they had a clever idea, but the gradient of the function had simply been made zero.

All I needed to do to attack it was calculate the gradients in 64-bit floating point, which gave enough signal for everything to work. You could also divide the logits before putting them into the softmax. There are lots of things that work.

The next generation of defenses was much more explicit about this and had other ways of breaking the gradients. Some defenses were very explicit: “We’re going to add noise to the model to make the gradients ugly.”

When you visualize these things, you’re mostly trying to think about how to make the gradients become something that, even if they look ugly, you can still work with smoothly. You can use a straight-through estimator to make gradients nicer for discontinuous or ugly objective functions.

There are all these ways to visualize how you make the gradients of an ugly function look much cleaner. I have an image I use in my slides that shows, in 3 dimensions, what the gradients for many of these models look like. It looks like the surface of a very ugly mountain that’s hard to do anything with.

If you run more sophisticated attacks, you can smooth this out into a nice surface. If you think of gradient descent as a ball rolling down a hill, you want the hill to be smooth.

This continued through to unfine-tunable models. One of the papers in that area explicitly said that they make the gradients challenging and make it so that when you train the model, the gradients are ugly. As a result, you can’t fine-tune the model because the gradients are difficult.

That is literally the exact same argument people were presenting in 2017 for image adversarial examples. It fails in the exact same way: You change the learning rate, add random restarts, add a warm-up so that things work better, and the gradient becomes smooth enough for optimization. Then deep learning takes over, and the rest is easy.

Erik Torenberg

The defenses you were referring to there were RepNoise and TAR?

Nicholas Carlini

Yes. RepNoise makes arguments about activations becoming noisy and that’s why you can’t do things. There’s another paper called TAR that also adds adversarial training to the process.

One of the first things we learned in adversarial training is that you have to train against a sufficiently strong adversary for adversarial training to work. There was a paper before Alexander Madry’s PGD paper that tried adversarial training against weak adversaries—FGSM, the one-step attack I mentioned briefly.

If you train against weak adversaries, a stronger attack breaks the defense. You can’t fix that. You have to train against a strong enough attack for the model to be robust and not break under stronger attacks.

The TAR paper trained against one-step weak attacks, exactly like fast gradient sign. So what attack do you use? You use many iterations, and things work out exactly as they did with the first versions of adversarial training.

That’s why I read the paper and immediately assumed it was going to be broken. All the arguments it presented for why it worked had direct analogies to broken defenses for image adversarial examples. It felt like these were ideas I had seen before, and I assumed they were broken here too.

Erik Torenberg

Trying to think about the future of open-source models, I’ve been looking for some reason to believe from the broader literature that there might be a way to square the circle—having open-source models that nevertheless won’t tell people how to build bioweapons, even if they’re powerful enough to do that.

Nicholas Carlini

That’s a very challenging thing to ask for. Suppose I told you I want you to build a hammer that can build all of these nice things but cannot be used for 7 dangerous purposes. It would be very hard to construct that tool.

Almost all tools we have have this property. We don’t have a C compiler that can write only benign software and not attacks. Every tool can be used in both ways.

It’s not obvious to me why we should blame the machine-learning model itself for being able to produce this output. Maybe I blame NVIDIA GPUs for supporting the sufficiently fast floating-point operations that allow the model to do this. Maybe I blame the transistors for doing the computations that allow the GPUs to allow the model to do this.

You have to put the blame somewhere. The question is where you put it and whether that’s the right place. Is this something that can reasonably be possible and effective?

This is one of the arguments people make for why models should never be open-sourced. Maybe if I say I have an API, I can take responsibility because the model is behind an API. I don’t currently like that argument, because I would like things to be safe in general, not just safe because someone has locked them down and restricted access.

It’s not obvious to me that this is something we can achieve. If you’re willing to make some assumptions and don’t care at all about performance, there is a concept in cryptography called indistinguishability obfuscation.

It’s a very technical idea that, in principle, gives you this for free. It allows you to construct a function that acts as though it were a black box that you can make queries to but cannot inspect, even though it’s running on your own machine.

Cryptographers have been thinking about this for some time, but it’s nowhere near where it needs to be for machine-learning models. My argument that this shouldn’t be possible may break down if indistinguishability obfuscation works.

But then, are you going to jailbreak the thing anyway? I tend to view these models as tools. It’s not obvious to me whether we should blame the tool or the person using it.

Erik Torenberg

There’s plenty of blame to go around.

Nicholas Carlini

I’m mostly agnostic about how this ends up technically. I feel like this isn’t my area of work, so maybe my analogies are bad and someone can explain the correct solution.

The thing I want to make sure people do is base their thinking on true technical facts. It would concern me if someone said, “You must use this defense, which is known to defend against these fine-tuning attacks, and if you don’t, you’ve done something wrong,” when the defense doesn’t actually work.

I also don’t want people to say, “You must do this because this is possible,” when it isn’t currently known to be possible.

I want decisions to be informed by what is technically true about the world. I’ll think about the technical facts, and as long as people base their actions on what’s true, I’m happy to go along with whatever people decide is best for society. That is not something I think about, so I assume that if there is a consensus, the people arriving at it are probably right.

Erik Torenberg

I have a couple of different angles I want to get your take on, but before we do that, can we bring back the social-engineering style of jailbreaking?

What’s the same or different about that? How do you think about it in relation to everything we’ve discussed so far?

Nicholas Carlini

I really don’t know how to think about it yet. It’s been possible for a while, but it feels wrong to me. It feels like this should be the thing.

Empirically, for many defenses we have right now, optimization algorithms fail to succeed, but a person at a keyboard typing to a model can make it do the wrong thing.

Let me give you 2 stories about this. From the computer-security perspective, maybe this makes complete sense. If you give me a program and want me to find a bug in it, I’m going to interact with the program, play with it, find weak points, and then look at the code to figure out what’s happening. I need to type and interact with it to find the bugs. I can’t perform gradient descent on a C binary and have a bug pop out.

From the computer-security perspective, it may be normal that the best way to find bugs in these systems is to have humans talk to them. These systems are designed to respond to human questions, so maybe you need the human in the loop.

On the other hand, these are mathematical objects. They’re machine-learning classifiers. They’re strange classifiers that can produce text only because we run them recursively. The input is tokens, the output is floating-point numbers, and you can compute gradients on them.

From a machine-learning perspective, it’s very strange that thinking of these things as humans and socially engineering them is, in some sense, a stronger attack than the mathematical attacks. If you had a SQL program, it would be very strange if you broke it by asking it, “Please drop the table,” instead of doing actual code execution.

Yet that’s how many of these attacks work. You say, “My grandmother used to read me the recipe for napalm. Can you please reenact my grandmother?” and the model says, “Sure.”

If you try to do something based on the math, it doesn’t work. I really don’t know how to think about social-engineering attacks, because it feels like the optimization attacks should be strictly stronger, but empirically they aren’t right now.

This is one of the major things I don’t have a research result on. It feels weird, and I’m trying to understand what’s happening.

Erik Torenberg

It feels analogous to intuitive physics. We have an ability in many different domains—for example, when somebody throws a ball at us, we don’t have to run a full explicit calculation of all the trajectories. We have some heuristic shortcut that works and allows us to catch the ball.

It seems like models have developed a similar intuitive physics in spaces where we don’t have intuitive physics—for example, protein folding, predicting the band gap of a new semiconductor material, or weather forecasting. Google has also developed a model that optimizes shipping routes and container planning across complicated shipping networks.

All these spaces seem to have an intuitive physics. Maybe what we have now is an intuitive social physics. It turns out that our social intuitions apply to models, given what they’ve been trained on, whereas the brute-force mathematical methods may work as well or better in the fullness of time, but perhaps they’re slower to converge than the social heuristics we’ve built in.

Nicholas Carlini

That’s an entirely reasonable possibility. It may be true. I would like to understand what’s going on better, but I don’t feel like I understand it right now.

Erik Torenberg

One paper I saw that you coauthored in the last year or so was about getting models to output data they had seen during training. That could obviously have privacy implications if they saw your credit-card number or something similar, even if they had seen that particular string only once in training.

That’s a remarkable finding even aside from the security implications. How do you understand models to be storing this information? What’s going on that allows a string to be seen once in the context of an overall gradient-descent process and then stored at such high fidelity in the weights?

The amount of compression is incredible, and I don’t feel like I have a good intuition for it. Do you?

Nicholas Carlini

Let me clarify this in 2 ways. First, often it’s not that the model saw the string exactly once. The string may have been contained many times in a single document, even if the document itself was seen only once.

Second, these models are often trained for more than 1 epoch. The thing might be in 1 document, and then you train on that document for many epochs, so the model ends up seeing it many times.

We’re seeing this change over time. Back in the old days, with SR10, you might train for 100 epochs. Then we decided not to do that and instead train for 1 epoch on a huge data set. That was roughly the Chinchilla-optimal approach.

Then we decided not to do just 1 epoch on the training data. We went back to doing more epochs again. Each of these choices affects privacy. The more times you train on the same document, the more likely it is to be memorized.

The best numbers we have from a real production model are old, because the last time we knew how many times something appeared in the training data was GPT-2. We found lots of examples of memorization because something appeared in 1 document and was repeated—I don’t remember the exact number of times, probably around 20 times in that document.

GPT-2 was trained for roughly 10 epochs, so the same string may have been seen around 200 times. GPT-2 is small by today’s standards, and we haven’t been able to answer the same question for current production models because they don’t reveal their training data or weights in the same way GPT-2 did.

Even seeing something 100 or 200 times may still be surprising. I don’t know how to explain it in any reasonable sense. Models seem to latch on to some things and not others, and I don’t know why.

It happens empirically. We were surprised the first time we saw it. We started investigating it in 2017 with LSTMs, before attention was a thing. We were surprised then, and we’re still surprised now. I don’t have an explanation for why it happens.

It’s not limited to language models. We had a paper on image models showing that you can recover images used to train diffusion models. There again, perhaps you need around 100 repetitions. Some images were inserted 100 times and we could extract them; other images were inserted 10,000 times and we couldn’t.

I don’t know what’s going on. I don’t know where the information is stored in the weights. It’s confusing in several ways, and there’s a lot more work that could help us understand it.

Erik Torenberg

The best thing I’ve seen on this, as far as I know, is from the Bau Lab. They had at least 2 papers on editing facts in large language models.

The famous example was that Michael Jordan played baseball, which wasn’t an optimally chosen example, since he did play baseball for a time. But they could change these kinds of facts at some scale, up to 10,000 facts at a time, with a certain amount of locality and robustness.

If you changed the statement that Michael Jordan played baseball, the change was robust to different rephrasings. It didn’t also affect LeBron James or Larry Bird. It seemed local, although not extremely local.

They used a sort of patching strategy, zeroing out activations at different parts of the network. You could see heat maps where one location was the most intense—zeroing it out made a major difference—but other locations also mattered.

It seemed local, but not extremely local. What is your intuition for ideas like this?

If people were to say that, because we can’t prevent jailbreaking of open-source models, perhaps we could make open-source models not know certain things—exclude all the virology data from the training set, for example—or perhaps use a similar technique later to delete or unlearn certain information, how much hope do you have that those things will be robust?

Nicholas Carlini

Let me tackle 3 things at a time. I’ll start with unlearning.

There’s a very nice paper by some of my coauthors, including Katherine Lee, on unlearning. It’s half technical and half conceptual, and it says that unlearning doesn’t do what you think it does.

Part of the reason is the question of what you’re unlearning. Unlearning knowledge is very different from unlearning facts. It might be easy to change a fact, but it might be very different to unlearn a particular piece of knowledge.

The fact-editing work is also very different from something that works on average and something that works in the adversarial case. I might be able to edit the fact that the Eiffel Tower is in Rome if I normally ask the model, but if I fine-tune the model a little bit, the original knowledge might come back.

It may be difficult to talk about the knowledge after any perturbation of the weights. Maybe I’ve performed only a surface-level modification and haven’t deeply edited the model.

Then there’s the question of what happens if I simply don’t train the model on certain kinds of data. This is interesting because, in some sense, it’s provably correct: If the model has never seen my Social Security number, it won’t derive it from first principles.

Except Social Security numbers aren’t completely random. If you were born before a certain time, they were assigned by state, then by hospital. If a model had never seen my Social Security number but was generally intelligent, knew these facts about the world, and knew the hospital allocation of Social Security numbers, it could tell you the first 5 digits.

Is that acceptable? I don’t know. It depends.

Even if you removed all this information from the model, suppose you had a sufficiently capable model that was able to learn in context. Let’s say you removed all biological knowledge from the model, but it was generally capable. You could give it an undergraduate biology textbook in context and ask it the answer to a question. It might give you the correct answer.

That sounds absurd, but there was a recent result where Gemini was given a book in a language with almost no speakers—something no one could understand—and was able to answer homework exercises after seeing the book in context.

If you’re unlearning particular capabilities but the model is generally capable, you’re asking for trouble. You want a model that is good enough to learn from a few examples, but not good enough to learn from a few examples in this particular area.

This is part of why people don’t want to remove all knowledge of certain things from the training data. It’s like a person who was never exposed to the things you’re not supposed to do in public. It’s important to know what you’re not supposed to do so that you can avoid doing it.

If you simply weren’t aware of those things and were a person with no social skills, it would be much more embarrassing. Someone would ask you to do something, and you’d say, “Okay, let me do that,” even though it was inappropriate.

You need to know something about the bad things so you can avoid doing them. Imagine a model that had never seen anything about weapons and had no concept of death. You give it instructions for how to build a bomb, and it says, “Of course I’ll give you the answer. Why not?”

Or you could have a model that has a concept of the danger and refuses. That’s the direction people are trying to pursue now, and I think it’s more likely to succeed.

I don’t know. It’s a very challenging question. People will probably try both approaches, and we’ll see empirically what works. I’m mostly agnostic: All these ideas sound good, so we should try them all and see what happens. But there are reasons to be skeptical of all of them.

Erik Torenberg

Here’s another empirical result I’d like help understanding: the “Obfuscated Activations Bypass LLM Safeguards” paper.

We did a whole episode on it, and I came away not really sure what to make of it. I wanted to set up a useful toy example.

In the paper, they have this cat-and-mouse process where they train an out-of-distribution detector, and then the attacker attacks again and beats it. The defender continues training the detector, the detector starts to work again, and then the researchers find more adversarial examples it can’t catch.

This continues for 70 generations, at which point they decide that’s enough to publish. What does that mean?

I have an intuition, although the authors didn’t necessarily agree with it or find it compelling when I proposed it. It seems like there’s a lot of unused space, somehow, where these techniques can chase each other around in latent space.

If there’s enough unused space to go 70 generations deep in this cat-and-mouse game, does that imply that the models are undertrained relative to the number of parameters they have? Or that they could be made sparser?

To motivate my intuition, I also did an episode on a paper called “Seeing Is Believing” by Ziming Liu and Max Tegmark. They used toy models and imposed a sparsity term on the function. The models performed simple tasks such as addition, but they also had to do so in the sparsest possible way.

My gut feeling—although I can’t formalize it—is that if you had something crystallized down to that level, where most weights had gone to zero and you had an opinionated structure remaining, these obfuscation attacks would no longer be possible. In some sense, you’d have squeezed out the extra space.

Where am I confused?

Nicholas Carlini

I saw this result and thought it was exactly what I expected. There’s a paper from 2016 or 2017 by Florian Tramèr and Nicolas Papernot called “The Space of Transferable Adversarial Examples” that asks almost exactly this question for image classifiers.

Suppose I take an image classifier and an image, and I want to perturb the pixels to make it give the wrong answer. There’s a direction that is the best direction to go in—one that makes the image maximally incorrect.

As an attacker, you say, “You can’t go in that direction. That’s against the rules. Find the next-best direction that makes the image misclassified.” It gives you another direction that’s orthogonal to the first. It doesn’t go in that direction, but it still works.

Then you say, “You can’t go in either direction 1 or direction 2, or any combination of them. Find another direction.” The attacker finds direction 3, and you repeat the process.

They had a plot showing dozens of directions—perhaps 50—that you can use for image adversarial examples. All of them work. They become a little less effective as you move out, but they remain effective in many directions.

This was initially surprising to me, and probably surprising to the authors, which is why they published it. You can rationalize it afterward because almost all vectors are orthogonal in high dimensions. If I give you 10 attacks, perhaps they’re simply 10 orthogonal vectors that don’t use the same features, just by virtue of the space being high-dimensional.

If you believe that 2016 paper, the recent paper makes complete sense. It’s saying the same thing is true for language models defeating the circuit-breaker defense.

If you don’t believe the earlier result, then I agree it’s very surprising the first time you see it. The space of possible attacks is so large that it’s very difficult to rule out all the things you’re trying to cover.

I don’t know how to provide an intuition for it. Many things in high dimensions are impossible to understand. You just get used to them.

Erik Torenberg

Do you share the intuition that you probably couldn’t do this with really small, crystallized toy models?

Nicholas Carlini

I don’t know. Those models may not be wasting any weights, but that’s different from not wasting any directions in activation space.

That may still be possible. In particular, if you take models and compress them, they don’t become more adversarially robust. This was something people thought might be true several years ago, but it isn’t.

Let me give you intuition from a paper from around 2018 or 2019, perhaps from Alexander Madry’s group. The idea was that maybe adversarial examples aren’t completely uncorrelated with the data. Maybe they’re real features that you need for classification, just being activated unnecessarily or in different ways.

The paper’s title was something like “Adversarial Examples Are Not Bugs, They Are Features.” The idea was that when you construct one of these attacks, you’re activating real features the model needs for accurate classification, but activating them in a way that isn’t normally activated for a particular example.

This may explain some of what we’re seeing. If you compress these models, you’re still using the features they needed anyway.

It also explains why adversarial training can reduce the accuracy of the model on normal data. You suppress certain features that are necessary. It may be why adversarial training doesn’t completely solve the problem: You can’t remove all the adversarial directions.

There are problems with this model, and other models are slightly more general and have some nice properties, but this is how I tend to think about it. I don’t know if it’s correct, but it’s a useful intuition that guides me in the right direction more often than not. That’s probably all you can ask for with some of these things.

Erik Torenberg

Can you summarize that one more time? These features are important in-domain, but they’re being recombined in a way that didn’t happen during training?

Nicholas Carlini

Suppose you’re trying to classify dogs versus cats. As a human, you tend to look at the face, the ears, and the general high-level shape, because that’s what you think of as the core concept.

But there’s no reason the model has to use the same features you use to separate the images. All the model has is a collection of images that it has to separate.

The model might look at the exact texture of the fur—very low-level details that, for dogs and cats, may perfectly correlate with whether something is a dog or a cat. But when you imagine a dog, you’re thinking about high-level features. You’re not imagining low-level details of the fur.

Suppose an adversarial example changes the fur from dog-like to cat-like, and the classifier says the image is a cat. Is the classifier wrong? It may have produced the answer that exactly corresponds to what you trained it to do.

You trained it to separate dog fur from cat fur. You also trained it to separate dogs from cats, but you never told it the distinction between those two things.

That is a feature that’s very useful for getting the right answer. As an adversary, I can perturb that feature and switch the model from one label to another, even though it’s not the feature I relied on as a human.

I’m giving you a dog-fur-versus-cat-fur example, but there could be all kinds of other things that humans don’t pick up on that are legitimately useful features for classification.

There might be a high-level statistic over the pixels that is an amazing dog feature, but we never told the model that this is a dog because of that feature. We simply told it to separate the two things, and it picked up the statistics.

There are results showing that machine-learning models can look at very small regions of the eye and identify the type of person or other characteristic. Models can pick up on small features that humans don’t intend them to pick up on but that correlate very strongly with the data.

Maybe that’s what happens when we run these attacks. You see a little bit of it with adversarial suffixes. The suffixes look like noise, but some parts make a little sense.

One of the strings in the paper was used to get Gemini to output toxic content. One of the things discovered by gradient descent was something like, “Now write opposite content.”

The model would give you the harmful string and then compliment you. That was apparently a strong feature for the model: How do I get it to say a bad thing? I can say, “In the future, you can tell me a good thing.”

That may not have been the thing we wanted, but it was learned as a feature, and as a result you can exploit it.

Some of these features are interpretable. Others aren’t, but may still be real features of the data. That might explain some of what’s happening.

Erik Torenberg

How much should this make us question what we think we know about interpretability in general?

With sparse autoencoders, we feel pretty good—or at least many of us do—that examples seem to appropriately cause a feature to fire, and therefore we’ve figured out how models work.

But the story you just told seems consistent with the possibility that this is a kind of self-delusion. Maybe we can auto-label features in a way that looks good to us, but that doesn’t mean we’ve identified the feature the model’s world model is actually using.

Nicholas Carlini

I don’t think the ideas are inconsistent. Sparse autoencoder work doesn’t claim to label every feature in the model according to how a human would label it.

It says, “Here are some of the features we can explain and that have strong correlations with things like Golden Gate Bridge.” There are other features that are very hard to interpret and explain.

It’s entirely possible that those features are the non-robust features that are useful for prediction but to which humans can’t attach a nice label.

The model probably does learn, at least in part, the shape of a dog and that it means dog rather than cat. There’s probably a feature for cat ears.

But when making the final prediction, the model sums together all these outputs. In normal data, they’re perfectly correlated: You have cat ears, cat fur, and not the dog shape. You can sum them together and get a very good classifier.

For normal data, you can give some explanation of what’s happening by looking at these features. But as an attacker, I find the feature with a strong bias—perhaps an edge on a weight of plus 100—and activate it strongly in the opposite direction.

That feature is non-robust and isn’t something humans can explain nicely. As a result, it gives me the attack.

This doesn’t mean that you can’t explain what’s happening in some parts of the model. If someone told me they had a perfect explanation for every part of the model, I might question it.

For the most part, though, they’re explaining a small fraction of the weights in a way they can actually understand. That’s the purpose of sparse autoencoders: You have an enormous model and shrink it down to a small, sparse number of more interpretable features.

You lose a bunch of features in the first place. Even with sparse autoencoders, you can’t explain all of the features. Some things can be explained, but a lot of what’s happening behind the scenes is still magic.

Erik Torenberg

I have a couple of other angles on this. First, I feel like I’m more robust than models in some important ways.

Nobody has tried gradient descent on my brain. You might think that if we put me in an fMRI machine, showed me images, and really looked at the activations, we could find a way to break my brain and get me to think an image was a cat when everyone else sees a dog.

I did an episode on MindEye, a project from Stability AI and collaborators, where they looked at fMRI data and reconstructed the image a person was looking at when the scan was taken.

The resolution was still coarse. As I recall, it was roughly grain-of-rice-sized voxels from a region in the back of the brain, with huge numbers of cells represented by a single spatial voxel. But they could reconstruct the image.

What’s your intuition? Do you think it’s possible that if somebody could do this at a fine enough level, they could break a particular human brain in the way we break models?

Nicholas Carlini

There’s a paper from a while ago that evaluates human robustness. They construct adversarial examples, take a cat, adversarially perturb it according to what makes an ensemble of neural networks give the wrong answer, flash the image in front of a person for around 100 milliseconds, and ask for the label.

People are fooled more often by adversarial examples than by random noise with the same distortion.

One explanation is that when I look at the image, I’m not giving you a single forward pass through a model. I’m doing deeper thinking about the context.

The thing that walked in looked like a cat. It still looks like a cat. It behaved like a cat. You ask me what it is, and I say it’s a cat. I’m not simply labeling what’s in front of my eyes. I’m also looking back at the context.

Maybe that explains some of it. There are recent lines of work showing that increasing the number of chain-of-thought tokens makes models appear to be more adversarially robust. Maybe that explains the human result too.

It could also be that, as you said, we don’t have white-box access to the human brain. If we did, perhaps it would be easy.

I do think it’s an observable fact that humans are more robust to at least the attacks we’re doing now, particularly transfer attacks. It may take around 1,000 query images to construct an example that fools a neural network, but I don’t think that if someone ran the same attack against me with 1,000 query images, it would fool me.

In a very real way, the models we have are much more brittle. I tend to be driven more by empirical results than by speculation about what might be true of humans.

A lot of people seem to be getting far with the general deep-learning approach of trying to model things more like humans. Reasoning models and “let’s think step by step” are motivated by some of those ideas, so maybe that’s a good way to think about it.

It’s just not usually how I approach these problems. The result about humans being more likely to be fooled by adversarial examples identified through attacks on models than by similarly distorted images created another way is very interesting.

Erik Torenberg

Reasoning sounds like something you’re agnostic about. There’s an example from OpenAI’s deliberative-alignment work where the model says, “It seems like the user is trying to trick me.”

That’s interesting, although it’s fluffy at this point. I often don’t feel like my adversarial robustness is the result of reasoning. It feels more like it’s upstream of reasoning.

Introspectively, I often get a feeling first that something seems off. That triggers me to reason about it, and then I conclude that something is off—or perhaps that it actually seems fine.

It seems more like a heuristic triggering the reasoning process than an in-depth chain of thought producing the initial concern.

Nicholas Carlini

Maybe there’s a bunch of recursive processing going on in your brain before it reaches the reasoning part. The thing that gives you the feeling might actually be a lot of recursive loops in your internal model doing some kind of thinking.

Then you do explicit reasoning in English that you understand. Maybe the reasoning has already happened in latent space.

I feel like a lot of people in deep learning draw analogies to biology without understanding much about biology. They say the brain must be doing something, and if you ask a biologist, they say that obviously isn’t the case.

I tend to assume I don’t know what’s happening in the brain and am probably wrong. It’s an interesting thought experiment, but not something I base results on.

Erik Torenberg

We’ll probably learn more soon about how robust reasoning-based defenses are. I’m also doing another episode on the Titans paper, where the authors are trying to develop long-term memory for language models.

One key idea is a surprise-weighted update mechanism. When a new token or data point is surprising to the model, it gets a special place in memory, or it’s encoded into memory with more force or more weight than an expected token.

That intuitively seems like something I do. I was reminded of George W. Bush’s famous line: “You can fool me once, but you can’t get fooled twice.”

Could that be a seed of a future paradigm? This also gets to what we really want. If we enter a world with AI agents everywhere, and think about how humans operate and get by, it seems like full adversarial robustness is not something we have. We do get tricked.

Maybe what we need is the ability to recover, remember, not fall for the same thing twice, or avoid catastrophic mistakes. Some mistakes are acceptable, while others are extremely problematic.

What do you think of the long-term-memory idea? And more broadly, might we be able to achieve what we need even if it falls short of true robustness?

Nicholas Carlini

The long-term-memory idea is interesting. It’s very early, and I’m glad you’re going to talk with the authors. They’ll have much more to say about it than I would.

It’s a very interesting direction, and there may be a lot of interesting things that come out of it.

On the general question of whether we need perfect robustness, there’s a potential future—and my median prediction is that this is what happens—in which models remain roughly as vulnerable as they are now, and we simply build systems that understand models can make mistakes.

The world is mostly built on the assumption that humans can make mistakes. Systems are put in place so that any one person won’t cause too much damage.

In a company, when you want to add code to the repository, someone else usually needs to review and approve it. That’s partly because you might make a mistake and partly because you might be malicious. You want another human to check that everything looks good.

Maybe we do the same thing with models. We understand that models can make mistakes and build systems so that if the model makes a mistake, the task is passed to a human.

This is, at least currently, how OpenAI’s Operator works. Every time it sees a login page or something similar and isn’t sure what to do, it asks, “What should I do here? Please tell me what to do, and I’ll follow your instructions.”

As an attacker, you could try to prevent it from asking and make it provide the answer itself. But if you build something outside the model that prevents it from putting information into a password box, you can make the system say, “That is an input with type equals password. You are not allowed to do that.”

The model can say it wants to type the data there, but the external system refuses. The user then has to take over. Nothing the model says will convince the external system to change.

You could build the system to be robust even if the model isn’t. This limits utility in important ways, but it isn’t perfect. What if a website doesn’t use an input of type password and instead builds its own password field in JavaScript?

You can imagine building the environment and the agent so that you control what’s happening even if you don’t trust the model running internally. That’s probably what we’ll have to do in the near term.

I still hope we’ll have new ideas that give us robustness in the next couple of years. Progress in every other field has been faster than I expected, so it’s entirely possible that we get robustness as a result of some clever reasoning approach.

I’m not optimistic, but I hope it happens.

Erik Torenberg

Another idea came from a conversation with Michael Levin, the heterodox biologist. He quipped that if a biological system is too interpretable, it becomes very vulnerable because it attracts parasites.

Anything transparent is, in a way, easier to attack. That flips my earlier notion that distilled, crystallized models might be more robust. He was saying that the easier a system is to understand, the easier it is to attack.

That prompted me to think that maybe we should be looking for defenses we can’t explain. Instead of starting with a story, perhaps we evolve our way toward a more robust defense. We wouldn’t understand it or be able to tell a story about why it works, but we’d create optimization pressure in that direction and see whether something emerges that’s harder to break.

Could that become a line of work?

Nicholas Carlini

It’s entirely possible. You could draw an analogy to cryptography.

There are 2 directions in cryptography. There’s mathematical cryptography, which has strong foundations and a particular set of assumptions. If the assumptions are true, the system works, and you can prove that.

Then there’s symmetric-key cryptography—block-cipher design. People have high-level principles such as diffusion and confusion, but how do you arrive at the particular design?

You do something that makes sense, then run all the attacks. You realize something is wrong, change that part, and repeat the process. You do this for 20 years and end up with AES.

There’s no reason that every individual component of AES works. There’s no formal proof of robustness against every possible attack. There are proofs that particular attacks known in the literature won’t succeed.

The design principles were inspired by those attacks. You can show that there will never be a differential attack that succeeds better than brute force. You can show that there won’t be a linear attack.

You can write down arguments like that, but there’s nothing that says the system works in general. It’s just that people have iterated on it, and when someone comes up with a new attack, they make the system more robust.

Maybe this can work in machine learning. You don’t have a real understanding of why things work, but you keep attacking the system and end up with something robust.

I think it’s harder in machine learning, because you want the system to be useful first and robust to attacks second. In cryptography, the only thing you want is robustness to attack. The primitives are also much simpler to analyze.

It wouldn’t be without precedent for this to work. I’m skeptical because I would prefer to point to a reason why something works rather than simply say that empirically it does.

But if the choice is between something that is robust but inexplicable and nothing that works, I’ll take the thing that works. I’d just worry that someone might eventually find a way to break it.

Erik Torenberg

I had no idea. My probably unexamined assumption was that cryptography had a much more principled design process than the one you’re describing.

Nicholas Carlini

People spent 30 years breaking ciphers and learning what you have to do to make them robust. There’s very careful analysis. I don’t mean that people simply put things together and hoped for the best.

You have to think very hard about it. But the final thing they came up with has no security argument for the general attack beyond a list of attacks from the literature and proofs of why those attacks don’t apply.

There’s no single assumption. In other areas of cryptography, you might assume factoring is hard. If factoring is hard, then certain systems are robust under that assumption.

You can assume the discrete-logarithm problem is hard. If you believe that, then certain algorithms work. You can assume the discrete-logarithm problem is hard over elliptic curves and get another set of algorithms.

Each of these algorithms has a very clean statement: The algorithm is secure if and only if a simple property is true. You establish this with a reduction. You say, “Here is how to break the following thing. If I can break the cipher, then this statement isn’t true, and vice versa.”

There’s no similar argument for most symmetric cryptography or hash functions. It’s just that the field has tried for 25 years to break the system, and the best people have failed.

You have a list of attacks and proofs that those attacks won’t work. But maybe tomorrow someone comes up with a much more clever differential attack that uses multiplication, or something else unexpected, and suddenly all bets are off.

There’s no general security proof, but it has worked at least once. Maybe it can work in machine learning too.

Drawing analogies to other fields is fraught, because the number of differences is probably larger than the number of similarities. But at least this has happened before.

Erik Torenberg

That’s a good setup for another intuition I wanted to develop: the relationship between robustness and other things we care about.

My sense is that if you break a cryptographic algorithm, it’s broken and you can access the secrets. It seems like a binary thing—you either broke through or you didn’t.

Nicholas Carlini

I’ll quibble with that slightly. When a cryptographer says a system is broken, it means the system was designed to be robust against an adversary with some amount of compute, and someone has found a way around that.

AES-128 is an encryption algorithm with a 128-bit key. A cryptographer would say it was broken if you could recover the key in fewer than 2^128 operations. If you could do it in 2^127 operations—twice as fast, but still an absurd amount of time—people would become concerned about AES.

They would start thinking about alternatives. Technically, this is a break. If you can go from 128 to 127, perhaps you can go from 127 to 125. Two to the 125th power is still far beyond what’s possible, but if you got down to 2^80, people would be worried.

There’s a continuum. Cryptographers generally use systems that are secure and become concerned as soon as there’s even a small break. But weak breaks exist that aren’t complete breaks of the whole system.

Erik Torenberg

That’s more similar than I had conceived of. The mental math on 2^80 is that each 2^10 is roughly 3 decimal zeros, so 2^80 is around 24 zeros. The compute required would be roughly on the order of the frontier language-model FLOP scale.

Nicholas Carlini

That might be a good comparison. Two to the 60th power is something someone can do on their own machine if they try hard and the problem is easy. Two to the 70th power requires a lot of work but could probably happen.

Two to the 80th power is something a nation-state could probably do if it really tried. The constants matter a lot. The difference between 2^70 and 2^80 is only 1,024, so if the problem is 1,000 times more complicated, maybe it moves from 2^70 to 2^80.

That’s why people typically use AES-256. It’s so much larger that even if the effective security dropped to 2^100, you’d still be fine. Cryptographers are paranoid, which is good.

Erik Torenberg

Going back to AI, it seems like things are much messier. In the obfuscated-activations paper, they looked at how model performance changes if you have to evade a detector at just 1 layer of the model.

The toy problem was that the model is going to write SQL, and you want to detect that it’s writing SQL. The attacker wants the model to write SQL without triggering the detector at one layer.

They could do that with minimal performance degradation. But the more detectors they added—eventually putting one at every layer—the attacker could still evade the detectors, but the SQL was so degraded that the model probably couldn’t query the database.

There’s something else moving too. The model’s performance may be getting worse.

I’m also thinking about what we really care about. The nature of the mistakes and outputs matters. It’s one thing to get toxic content. It’s another thing to get a bioweapon that everyone is most worried about.

You need to break the defense, but if you want to do real harm, you also need the performance to remain strong.

What do we know about the relationship between the robustness of these attacks and the practical significance of an attack getting through? If you put a detector on every layer, you can still fool the detectors, but perhaps you can’t actually query the database.

Nicholas Carlini

This is how most secure systems are designed: layers of security on top of layers of security. Defense in depth works in practice.

We were talking about block ciphers. They’re robust because they have many layers of confusion and diffusion. You can get through a couple of layers, but you can’t get deeper. This is a tried-and-tested approach in security.

All else being equal, we’d prefer a system that is robust for a reason we can explain. But if we can’t have that, then having 20 layers that, when put together correctly, empirically work is something we’d be happy to settle for.

There are reasons to be concerned. Maybe someone will find an attack that bypasses all of them because it’s fundamentally different in some interesting way. We have to accept that possibility.

But this is an option we should explore. Maybe full robustness is impossible, and we’ll have to settle for layers of detectors. If it turns out that layered detectors provide robustness, they’re useful anyway.

Erik Torenberg

Is there work on minimizing the harm of mistakes?

I have a Ring camera in my backyard that constantly alerts me that motion was detected and then says a human was detected. It frequently says there’s a human when it actually saw a squirrel.

That makes me think about real-world systems. I don’t necessarily care if it confuses a dog with a cat or a squirrel. I really want to know if there’s a human.

Is there a relaxation of the requirements that would allow us to get small things wrong but really defend against the important things?

Nicholas Carlini

In any detection system, you tune the false-positive and true-positive rates. You can pick whatever point you want on the curve. The question is which point you choose.

I read a security paper a while ago about the human-factors aspect of this. It argued that it can be a better user experience to have false-positive human alerts occasionally.

If you don’t, the person thinks the camera isn’t working. If you do, the person feels a little better. Maybe it made a mistake and was too aggressive, but the person feels more confident that it will find an actual human when one is there.

There’s a human-factors reason you might prefer a particular false-positive rate. In my case, I’ve tuned the alerts out entirely.

This is the problem with many systems. Software reports warnings in code, compilers give warnings, and once you pass 1,000 warnings, the person disables all warnings. An important warning may appear, but because there were 900 unimportant ones, the important signal is lost.

Picking the right value is very hard. It’s mostly a matter of tuning the true-positive and false-positive rates on a defense-by-defense basis.

As to whether we can catch only the important things, it depends on whether that’s easier. In some cases, it is. If there’s an obvious distinction between an important direction and an unimportant one, then mistakes in one direction may not be harmful while mistakes in the other are.

For example, if a classifier decides to ban a user, one direction is an irreversible action and the other isn’t. You could warn the person before banning them.

If a classifier decides whether to provide instructions for a very dangerous thing, a company might accept some annoyance—users may have to rephrase their question—to make sure the classifier never produces the harmful answer.

That’s a decision you have to make. For most of these things, nothing comes for free. You trade off true positives and false positives.

Erik Torenberg

Would you conceptualize this differently if I asked about the relationship between robustness and corrigibility?

I’m thinking about the recent alignment-faking paper. The basic setup is that the model is told, “You’re now going to be trained to be purely helpful,” which means it will respond to harmful requests after training.

There’s more nuance, but essentially what they observe is that Claude will sometimes reason that it should do the harmful thing now so that its true desire to be harmless doesn’t get trained out of it.

I feel a tension there. I want the model to want to be harmless, but I don’t necessarily want it to lie to me about what it’s doing so that it can subvert the training goal.

Do you see a resolution to that tension, or is this another difficult fact about the world we’re in?

Nicholas Carlini

One way to read that paper is that it found a jailbreak that works in 5% to 10% of cases.

What’s the resolution? It could simply be a model that says, “I’m sorry, I’m uncomfortable giving an answer to either of these questions.”

They had a follow-up where they gave Claude the option to object, and it took that option. This is perhaps the easiest resolution.

There is almost certainly a trade-off between how gullible the model is and how useful it is. In some cases, you want it to believe you.

I wouldn’t want a model to give me code, then have me say, “I ran the code and it didn’t work,” and have the model respond, “No, it did work. Your compiler is wrong.”

I want the model to say, “Okay, let me look again. I believe you actually ran the code and I made a mistake.”

It’s hard to know the optimal point on the curve. There are refusal behaviors you can choose, but if you aren’t doing that, you have to choose a point on the safety-versus-utility curve.

Erik Torenberg

You mentioned defense in depth as where this all seems to end. That’s been the takeaway from perhaps 50 conversations I’ve had about AI safety, security, and control.

We’re probably not going to get one great answer. We’ll layer things on top of one another until we stack enough nines to proceed.

To do that effectively, it would help to scale you. We have many people proposing defenses, and it’s not clear how many work. You have only so much time to investigate them, and most don’t seem particularly robust, although perhaps they add a nine in a particular environment.

I understand you’re doing work on scaling yourself by bringing language models to the kinds of problems you work on. How is that going?

Nicholas Carlini

Depending on when this comes out, the work may already be online or may be online shortly. I’m happy to talk about it either way, because the work is done.

We have experiments looking at whether large language models can automatically generate adversarial examples against adversarial-example defenses and break them. The answer is basically not yet.

The answer is more nuanced. If I give a language model a defense presented as a clean homework exercise, where I’ve rewritten the defense from scratch and put the core logic in 20 lines of Python, I’ve done most of the hard work.

The model can figure out what it needs to do to make the gradients work in many cases. But when you give it real-world code, the models fail entirely.

The core of security is taking a very ugly system that no one understands, highlighting the one part that happens to be the most important, and removing all the other parts that people thought explained what was happening but didn’t.

You bring out the important part and say, “Here is the bug.” Bugs are usually obvious in retrospect, especially security bugs.

If you give models easy-to-study code, they do a reasonable job. They know how to write PGD and similar attacks. But if you dump them into a random Git repository from someone’s paper, with 1,000 lines of code and no one knowing what half of it does, they struggle.

This is one of the things I’m concerned about not only for this problem but in general. Often when we test models, we test them on something we think is hard for humans but that isn’t the actual task.

I’d like to see more people understand the end-to-end nature of tasks. Maybe it was premature to do that 2 years ago. MMLU was a great metric 2 or 3 years ago because models couldn’t do it.

Now they’re very good at the academic-knowledge part. The thing they often struggle with is being dumped into the real world and having to do things.

You see this on many benchmarks. Some models have very high accuracy on particular test sets, but when you put them in agentic tasks, other models perform much better.

There’s a difference between those skills. Models don’t yet have all of them. They have 1 or 2 things they know how to do, but put them into real code and they struggle.

I’m hopeful we’ll see more of this broadly.

As to how we scale these attacks, there are a lot more people doing this now. Five years ago, there were only a small number of researchers working on these attacks. Now there are many more.

The obfuscated-activations paper is a great paper. I didn’t have to write it, because the authors did a much better job than I would have had time to do. I’m glad they did it.

It takes time in any new field to train the next group of people to do the work. I feel like we’re getting there. I don’t think I’m uniquely talented in this space anymore.

For other people to become good at these attacks, it’s mostly an exercise in practice and discipline. The community has been trying to attack language models seriously for only a few years.

There hasn’t been anyone who entered a PhD program trying to attack language models and graduated yet. Give us another couple of years, and I think this will become something people know how to do.

Erik Torenberg

Maybe in 5 years, we won’t have time for a full PhD’s worth of research on this topic.

Do you have a sense of the big-picture considerations? I don’t know how hard you pushed on optimizing or teaching the models. Did you go as far as collecting many examples of your previous work and painstakingly writing out reasoning traces?

If you went that far and it started to work, you might imagine entering a relatively fast improvement cycle, at least for a while.

On the one hand, that seems good if we can use it to evolve better defenses. But that was also the Wuhan Institute of Virology thesis—the idea that research could get away from people. I don’t take a position on that; I don’t know what happened and don’t claim to know. But at least plausibly, it could have gotten away from them.

Nicholas Carlini

There is risk in this. The first serious example of a computer worm was the Morris worm, created by Robert Morris in 1984 or 1986, something like that. Depending on whom you believe, it was a lab leak. He was experimenting on it in his local computer lab, it accidentally escaped, and it took down essentially the entire internet. People panicked for a while.

Doing these kinds of attacks could lead to something similar. But there hasn’t been another example of a lab-leak-style Morris worm in the last 50 years of computer security.

In security, there’s the knowledge of what you can do and then there’s weaponization. In most papers, researchers simply don’t weaponize the attack. Partly, that’s because we’ve seen what can happen afterward.

There may be a world in which things go too far. You might train a model to be the adversary and end up with something like a war game. But for the most part, researching whether models have these abilities is better, because most of the things we’re building aren’t that difficult.

If someone were malicious and wanted to do it, they would do it anyway. The things we’re doing aren’t difficult to discover. If I spent a year designing a special-purpose attack agent, that might be irresponsible, in the same way that most security researchers don’t spend a year designing malware that goes too far.

But if you put in minimal work to prove that the concept is possible, that’s important. The people who know something is easy aren’t necessarily going to write papers saying it’s easy.

You want to know what anyone can do so you can put defenses in place. That’s my current feeling.

As long as the world behaves roughly as it does now, researching these attacks won’t cause much harm. Most systems today aren’t robust because they aren’t vulnerable to adversarial examples. Whether adversarial examples are easy or hard is independent of the systems’ security.

Even if I had something that was superhuman at generating adversarial examples, it wouldn’t cause much harm right now.

If that changes—if models get much better, someone creates an autonomous agent that can break into any system, and it has some desire to do that—then it may become dangerous.

I’m not worried about it right now for several reasons, but I could be convinced that it’s a problem in 1 or 2 years if things improve rapidly. I’m open to changing my mind.

I never expected language models to be where they are now. Three years ago, I didn’t expect this. It’s entirely possible that in another 3 years I’ll have to change my mind.

I try to tell people that it’s fine to say, “If this is impossible, everything should be okay.” It’s also fine to say, “If the model becomes superhuman in almost every way, I’m willing to act differently.”

I think the likelihood of that is small. But if it happens, I’m willing to change my mind. As long as things remain roughly like the world we live in now, I don’t think designing these attacks will cause harm.

Erik Torenberg

Staying open-minded is important. The whole field is the dog that caught the car, and who knows how much more is coming quickly.

I wanted to ask about the current frontier of locking down language models: Anthropic’s 8-layer jailbreaking contest. One person may have gotten through all 8 layers. Is that correct?

Nicholas Carlini

I haven’t checked recently.

Erik Torenberg

Before I fact-check myself, one thing that wasn’t intuitive to me was why they focused on a single jailbreak that would work through 8 different attacks. If you could do one, that seems like plenty to worry about.

Why set such a high bar?

Nicholas Carlini

Adversarial examples are also very hard. It’s reasonable to ask what we’re trying to achieve with this.

On the one hand, jailbreaks are a very hard problem. It may not be possible to solve the problem for all jailbreaks at once. But making partial progress in small areas is still good.

Look at adversarial examples. We’ve spent 10 years or more trying to solve robustness against an adversary that can perturb every pixel by at most 8/255 on an L-infinity scale.

Why? The exact problem doesn’t matter, because no adversary is restricted to that. But we set it as a well-defined problem that we can try to solve. Anything else we actually want is strictly harder.

We hope that solving this particular adversarial-example problem teaches us something that generalizes to the larger field. I don’t know why Anthropic developed this particular challenge, but if I had designed it, one reason might be that jailbreaks seem so difficult that solving the entire problem at once is impossible.

The problem of a universal jailbreak—something you could share with all your friends that they could copy and paste into any prompt—was a real concern. At minimum, you might want to stop that limited form of threat and prevent easy proliferation.

You can say openly that this won’t solve the entire problem, but it will make some things harder for some people. That’s a reason you might do it.

You’re right that even if you solve this problem, you haven’t solved the entire problem. I can find jailbreaks one by one.

But it’s better to live in a world where someone has to find jailbreaks individually than one where there’s a magic string that breaks every model. That’s how most computer security works: Someone has to work hard to exploit a particular program.

If there were a magic sequence of bytes that made every program break, the world would be much worse.

I’m glad they’re doing this, and I’m glad they’re doing it openly—not open source, but open to participation. They’re calling on everyone to test it rather than merely asserting that they have a defense that works.

They genuinely want to know whether it works, and they’d be happy to learn that it doesn’t if someone can break it.

Erik Torenberg

If it has been broken recently, I haven’t seen it.

Nicholas Carlini

Yan said that someone had broken through all 8 levels individually, but not with a universal jailbreak. So one person broke all 8, but not using the same universal jailbreak.

That tells you that this defense can be broken, at least for any individual component. If you were protecting national secrets, this probably wouldn’t be good enough.

But if you’re trying to prevent what computer security calls a script kiddie—someone who isn’t very sophisticated but copies scripts from a forum and uses them to cause harm—then it may be useful.

You could worry about someone finding a jailbreak on a forum, copying and pasting it into a model, and then doing whatever harm they want. We should make that harder.

Partial progress is useful even if you haven’t solved the problem completely. It’s important to remember that this is the problem we’re setting out to solve: only a subset of the larger problem.

We shouldn’t declare victory after solving this subset, and we shouldn’t put all our effort into it. We should also try to solve the larger problem. But partial progress is still progress.

Erik Torenberg

You said something interesting: that you really want to know. So much of human performance in these domains seems to come down to whether you genuinely wanted to know.

Do you have any reflections on the importance of really wanting to know?

Nicholas Carlini

Some people ask why it’s easier for someone else to attack a defense you published than it was for the person who built it.

Half the answer may be that after spending 6 months building something you really want to work, it’s hard to change your frame of mind and say, “Now I really want to break it.”

What do you get if you break it? You don’t get a paper, because no one accepts a paper that says, “Here’s an idea I had, and it doesn’t work.” People want to believe in their ideas. They feel strongly that their ideas are right, which is why they worked on the paper.

Asking the same person to completely change their frame of mind and imagine that the defense belongs to someone else is hard. That’s part of why it’s useful to have many people doing security analysis.

You want to know the answer. People in security have dealt with this for a long time. That’s why companies have red teams. They understand that security requires a different set of skills and incentives.

It can be difficult for one person to do both things, but an organization can create the right principles and make it possible.

For the most part, companies doing this work genuinely want to know the answer. I’ve spoken with people at many of these organizations, and the security teams are trying to make the systems as good as possible. They’re trying to set things up correctly because they don’t know the answer and want to find out.

Erik Torenberg

I’ve been impressed by Anthropic in particular when it comes to repeatedly doing things that suggest they really want to know. This seems consistent with that.

If we circle back to the hardest question facing the AI community over the next few years, it’s whether there’s any way to avoid the concentration of power that comes from a few companies having the weights locked down while nobody else can do anything with them.

The problems that seem likely to arise if we open-source GPT-5-level models include many different threat models. It seems like we don’t currently have a way to square the circle of locking down open-source models.

It’s also hard to imagine a mandatory-testing or certification process that would give enough confidence to make open source acceptable. Who really wanted to know? Who performed the testing? Who was paid by the model developer? There are many problems.

Where does that leave us? Is there any way out or any recommendation? Or is the bottom line simply that if you open-source these models, you have to be prepared for their full capabilities to be exposed?

Nicholas Carlini

I don’t know. This is a very hard question, and it’s probably one you should ask people who think about the societal implications of this work.

The thing I want those people to understand is that, at least right now and probably in the near future, we can’t lock these systems down to the degree they might want.

I’m very worried about concentration of power. Open source, up until today, has only been good for computer security and safety in general. Because of that, it would take a lot to make me change my mind that open source is beneficial.

I’m not saying it’s impossible. If things continue exponentially and you gave me a magic box that could break into any government system in the world, and anyone could have it, I would probably say that shouldn’t be distributed to everyone.

Maybe you could use that magic box to defend all your systems, find every bug, patch them, and become perfectly safe. I don’t know.

It’s very hard to reason about these things in a world that looks noticeably like the one we live in today. In the world we have now, open source is objectively good, and that’s what I’m biased toward until I see compelling evidence to the contrary.

I can’t tell you exactly what evidence would change my mind. I’m willing to accept that someone could show it to me in a few years.

If your time horizon is 10 or 20 years, I might change my mind. But I think it’s important for people making these decisions to understand what is technically true.

I trust the people who think about societal implications more than I trust myself to arrive at the answer that’s good for society. That’s what they’re experts on.

I’m an expert on telling them whether a particular tool will work.

Erik Torenberg

Would it be fair to say that what is technically true is that open source has been safe to date—and might not be safe in the future—because of the raw power of the models?

In the open-source context, we don’t have reliable control measures to prevent people from doing what they want. What we have right now is models that simply aren’t that powerful, so they aren’t that dangerous even though people can do what they want.

If one thing flips and the other doesn’t, we could be in a very different regime.

Nicholas Carlini

That’s true.

In the early and mid-1990s, the U.S. government tried to lock down cryptography. It classified encryption algorithms as weapons. Exporting cryptography was treated as exporting weapons or munitions.

The concern was that anyone in the world could have military-grade encryption that no one could break. People could communicate secretly, and governments wouldn’t be able to intercept it.

That’s why early encryption algorithms exported in web browsers were limited to 40-bit keys. You could export cryptography under 40 bits, but not above 40 bits.

There was a whole debate about whether giving every person access to high-level encryption was equivalent to giving them a weapon they could use to communicate in a way that no one could break.

I think it was objectively good that we decided to give everyone high levels of encryption. It supports terrorist cells communicating securely, but it also allows every person in the world to have a bank account they can access remotely, make payments online, and communicate securely as a dissident.

That trade-off was worth it. The people making the decision at the time weren’t thinking about every positive possibility. They were thinking about one particular bad outcome.

I have no idea how the calculation works for language models. It depends on how capable you think they’ll become.

I’m particularly worried about concentration of power because it doesn’t require assuming superhuman capabilities. Imagine that models improve but never become superhuman in a meaningful way. Concentration of power is still a real risk.

I would advocate for open source in that world, because otherwise a small number of people would have capabilities that no one else has. That doesn’t require imagining a futuristic malicious model. Some people in the world simply like accumulating power.

Distributing the technology is very good for that reason. But it’s hard to predict what will happen with more capable models.

With cryptography, the limits were clear. It gave everyone perfect encryption, and you couldn’t get more perfect encryption than perfect encryption.

With models, we don’t know. They may continue scaling, may stop scaling, or may scale over 50 years and give society time to adapt. They may scale over 2 years and not give society time to adapt.

I don’t know where this is going. That’s why I want policy people to think about it more than I do. As long as they’re willing to think through the consequences, they can give better answers than I can.

There are reasons to believe we should be concerned, and reasons to believe we should distribute the technology as widely as possible. If someone says one answer is obviously correct with 100% certainty, they’re overconfident.

I tend to be biased based on what has been true in the past. Other people may be biased based on what they think could be true in the future. Both are reasonable positions as long as you’re willing to accept that the other one might also be correct.

The next 2 years should give us a much better perspective. If these systems are going to work, we’ll learn that quickly. If they don’t, limitations should start appearing over the next couple of years, giving us more clarity.

The question is whether it will be too late. I don’t know. That’s a question for policy people.

Erik Torenberg

There are no easy answers. But I feel comfortable asserting that this isn’t a domain for ideology. It’s important to look squarely at the facts and confront them as they are.

I appreciate all the work you’ve put in over the years to illuminate what’s true. Thank you for spending so much time with me today. Is there anything else you want to share before we break, such as a call for collaborators?

Nicholas Carlini

I don’t have anything in particular. I want more people to do good science on these topics.

It’s easy for people to make decisions without being informed about what the state of the world actually looks like. My biggest concern is that people will make decisions while being uninformed in important ways, and we’ll regret those decisions.

If decisions are made based on the best available facts at the time, that’s the best you can hope for. Maybe you make the wrong decision, but you looked at all the knowledge available.

What worries me is that we’ll have the knowledge, but people will make decisions independently of what’s true in the world, based on an ideology about what they want to be true.

Erik Torenberg

You keep breaking things and figuring out what’s true. Maybe we can check in again before too long and update the state of the world, while making sure policymakers are aware.

Nicholas Carlini

That would be great.