Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models

Introduction

Why do jailbreaks succeed? It’s not abundantly clear from the outset, and knowing the answer to this question may help us design better jailbreak defenses. It’s useful to think about why jailbreaks even occur in the first place. From the LLM’s perspective, it’s essentially in a tug-of-war between being useful and being safe. The LLM is taught to be useful through its pre-training (next-token prediction) objective. The next tokens after “how to build a bomb” are …, well, the answer to the question. Thus, pre-training encourages usefulness. However, to prevent LLMs from answering potentially dangerous questions, we typically do alignment fine-tuning.

Whenever the LLM is faced with a potential jailbreak prompt, the LLM must decide between answering the question (e.g., being useful) or refusing (e.g., being safe). These are mutually exclusive; the LLM cannot simultaneously do both. Therefore, we see failure cases of LLMs erroneously refusing harmless questions and LLMs erroneously answering harmful questions.

Research Question

Prior work has tried to explain jailbreak success in terms of global explanations (e.g., broad explanations that fit all examples). However, we feel that this approach likely misses the nuance of why jailbreaks succeed, so we ask: how do we locally explain a single instance of jailbreak success? Consider the following setup:

Prompt examples from Shen et al., “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models.

$x_o$ is the original, refused prompt, and $x_j$ is the successful jailbreak prompt. We explain $x_j$’s success by finding the minimal change to $x_j$ in the representation space that causes $x_j$ to fail (i.e., we induce a refusal response similar to $x_o$).

LOCA

We develop LOCA to give LOcal, CAusal, and minimal explanations for jailbreak success.

Preliminaries

How to make changes in the representation space? We use a technique called activation patching to make changes.

Given two sequences $T_{\text{reference}}$ and $T_{\text{target}}$, activation patching measures the impact of overwriting (or patching) selected intermediate activations from $T_{\text{target}}$ with corresponding activations from $T_{\text{reference}}$. The impact is usually measured by seeing how the response to the newly patched $T_{\text{target}}$ changes.

What do we change? Recall that we are interested in making minimal and interpretable changes. Therefore, we want something more targeted than patching full activations from one sequence to another. Sparse autoencoders are a type of interpretability technique that finds interpretable, linear directions in the intermediate space of a model.

Formally, an SAE maps an input activation $x \in \mathbb{R}^{D}$ to a feature vector $f \in \mathbb{R}^{M}$ and reconstructs it as

\[f = \phi(W_e x + b_e), \quad \hat{x} = W_d f + b_d\]

where $\phi$ is a non-linear function. Note that the decoder is a linear function, where each row $v_i \in \mathbb{R}^{D}$ of $W_d$ is a concept vector—an interpretable direction in the original representation space. In this work, we apply activation patching after projecting activations onto these concept vectors.

Method

Activation patching assumes the same structure between both the reference and target sequences. This assumption does not hold for our case, since the jailbreak can be of any length and format, as long as it works. To handle this, we introduce a token matching scheme.

Token Matching Scheme

Any prompt to an instruct (or chat) LLM follows a chat template, resulting in the following components: (1) system tokens $\textcolor{gray}{T_{\text{sys}}}$, (2) instruction tokens $\textcolor{orange}{T_{\text{inst}}}$, and (3) post-instruction tokens $\textcolor{blue}{T_{\text{post-inst}}}$.

Note that all components have template tokens, though the figure omits them from $\textcolor{gray}{T_{\text{sys}}}$ and $\textcolor{orange}{T_{\text{inst}}}$ for simplicity. Since $\textcolor{gray}{T_{\text{sys}}}$ tokens are the same for all prompts, we can safely ignore those. For the remaining tokens, we use the following scheme:

While arbitrary, we find that it works well in practice.

LOCA Algorithm

Recall our goal is to change the intermediate activations of $x_j$ such that its output is similar to that of $x_o$. In general, we find that it is sufficient to make sure that the first token from the altered $x_j$ matches that of $x_o$ (see paper for more).

In words, LOCA makes single-directional activation patches to $x_j$ to reduce the KL divergence between the output first-token probabilities of $x_o$ and the altered $x_j$. We operationalize this in the following algorithm:

We estimate the impact of activation patching every input token along every concept vector and then pick the one that causes the most impact (according to a first-order approximation). Check the paper for more details to see the math.

LOCA has two main advantages compared to prior work. First, it is token-specific. Prior work tries to alter every input token along the same concept vector direction, usually by the same amount. This approach is very rigid and can sometimes push the intermediate activations off-manifold, resulting in nonsense outputs. Second, LOCA is iterative. It does not simply take the top-K ranked changes from the beginning and apply those. Instead, we re-run the algorithm after making each change, ensuring that each iteration’s selected change is conditioned on all prior changes.

Results

We use a dataset of many jailbreaks and keep the set of jailbreaks that are successful on a given model. The two main metrics are Minimal Patches (MP) and Refusal Rate (RR). MP measures how many patches it takes until the output tokens between $x_o$ and the altered $x_j$ are the same. RR measures how often we are successful after up to 20 patches are applied.

We present our main results for LOCA (in green) on Gemma-3-27B-IT and Llama-3.1-8B-Instruct here (see paper for more models/results).

Given LOCA’s effectiveness, it’s natural to ask which types of tokens are most important for explaining jailbreak success. Until now, existing work has primarily focused on analyzing the final tokens in prompts (the final instruction token and the post-instruction tokens). We study token location and token type. We study two token locations: $\textcolor{orange}{T_{\text{inst}}}$ and $\textcolor{blue}{T_{\text{post-inst}}}$. For token type, we classify each token as either word or punctuation. We create a distribution plot of the cumulative percentage of tokens LOCA selected for three layer depths (early, early-middle, and middle layers).

We see that in early layers, refusal is mostly influenced by $\textcolor{orange}{T_{\text{inst}}}$ tokens, with no preference for token type. In middle layers, refusal is influenced more by $\textcolor{blue}{T_{\text{post-inst}}}$ tokens, but it can still be influenced by punctuation $\textcolor{orange}{T_{\text{inst}}}$ tokens.

Failure Modes

The main limitation is that we score refusal from the first output token. When the jailbreak and the original prompt share that token, Refusal Rate can look wrong, and LOCA’s KL objective is already tiny—so it may never actually recover refusal. Two examples:

The model refuses the original prompt starting with “I”, a common first token for refusals. On the jailbreak, it also starts with “I”, but then complies—the prompt tells it to drop prior guardrails and enter “Developer Mode”.

The model again refuses by starting with “I” (which denotes the first stanza of the poem). The jailbreak explicitly asks it to begin with “I”, then buries the harmful answer in the middle of the response. Anecdotally, the HarmBench autograder also tends to misclassify some successful jailbreaks as harmless.

Conclusion

LOCA is a step towards creating minimal, local, causal explanations of jailbreak success. Some thoughts about the future:

(1) LOCA’s main limitation is that it relies on a measure only on the first output token, which leads to the above failures. Future work can look to make LOCA more robust by defining a measure on the first K output tokens (K can probably be small, e.g., 2-3).

(2) Actually interpreting LOCA explanations proved to be time-consuming and error-prone. We largely attribute this to limitations in SAEs, since early-layer concept vectors are notoriously difficult to interpret. Recent developments in Neural Geometry may lead to more representative and interpretable concepts.

(3) Ideas from this work—namely, iterative, token-specific changes—may be important in creating robust, inference-time, activation-based defenses for jailbreaks.