Visual Analysis of Jailbreak Attacks Against Large Language Models

Prompt Jailbreaking, known as one of the most famous adversarial attacks[47], refers to cunningly altering malicious prompts to bypass the safety measures of LLMs and generate harmful content, such as illegal activities. With the proliferation of LLMs, an increasing number of jailbreak strategies [26, 75], such as character role play, have been discovered and shared on social platforms (e.g., Reddit and Discord).
提示越狱，被称为最著名的对抗性攻击之一[47]，是指狡猾地更改恶意提示以绕过安全措施LLMs并生成有害内容，例如非法活动。随着越狱策略的普及LLMs，越来越多的越狱策略[26,75]，如角色扮演，在社交平台（如Reddit和Discord）上被发现和分享。

This trend has motivated new research to analyze their prompt characteristics. Liu et al. [37] propose a taxonomy for jailbreak prompts, which categorizes jailbreak strategies into ten distinct classes, such as Character Role Play and Assumed Responsibility. Shen et al. [53] report several key findings regarding jailbreak prompts’ semantic distribution and evolution. Wei et al. [66] empirically evaluate LLM vulnerability and summarize two failure modes, including competing objectives and mismatched generalization. These studies mainly focus on the general characteristics of jailbreak prompts. In comparison, our work provides a multi-level analysis framework to help users systematically explore the jailbreak prompt and identify the model’s weaknesses.
这一趋势促使新的研究分析它们的提示特征。Liu等[ 37]提出了越狱提示的分类法，将越狱策略分为10个不同的类别，如角色角色扮演和承担责任。Shen等[ 53]报道了关于越狱提示语义分布和演化的几个关键发现。Wei等[ 66]实证评估LLM了脆弱性，总结了两种失败模式，包括竞争目标和不匹配泛化。这些研究主要集中在越狱提示的一般特征上。相比之下，我们的工作提供了一个多层次的分析框架，帮助用户系统地探索越狱提示并识别模型的弱点。

Some other works [77, 28] propose automatic approaches for red teaming LLMs. Zou et al. [78] propose GCG to search for the optimal adversarial prompt suffixes based on the gradient of white-box LLMs. Deng et al. [13] propose a time-based testing strategy to infer the defense mechanisms of LLM (e.g., check the input questions and output responses) and fine-tune the LLM for jailbreak prompt generation. To better utilize the effectiveness of manually designed prompts, GPTFuzzer [74] selects human-crafted prompts as the initial seeds and mutates them into new ones. Ding et al. [15] propose two strategies, including prompt rewriting and scenario nesting, to leverage the capability of LLMs to generate jailbreak prompts. Inspired by these methods, we propose prompt perturbation strategies based on the prompt components, allowing users to conduct a comparative analysis of the prompt components to understand their effects on jailbreak performance.
其他一些著作[77,28]提出了红队的自动方法LLMs。Zou等[ 78]提出GCG基于白盒LLMs梯度搜索最优对抗性提示后缀。邓等[13]提出了一种基于时间的测试策略来推断LLM（例如，检查输入问题和输出响应）的防御机制，并微调LLM越狱提示的生成。为了更好地利用人工设计的提示的有效性，GPTFuzzer [ 74] 选择人工制作的提示作为初始种子，并将它们变异为新的种子。Ding等[ 15]提出了两种策略，包括提示重写和场景嵌套，以利用越LLMs狱提示的能力。受这些方法的启发，我们提出了基于提示组件的提示扰动策略，允许用户对提示组件进行比较分析，以了解它们对越狱性能的影响。

Visualization plays an indispensable role in bridging the explainability gap in NLP models [27, 29, 57, 19, 18], allowing for a more sophisticated understanding of model performance [65], decision boundary [11], and vulnerability [41]. Model-specific visualizations focus on revealing the internal mechanisms of NLP models. RNNVis [44] and LSTMVis [58] visualize the hidden state dynamics of recurrent neural networks. With the emergence of transformer-based models [64, 14], numerous visualizations [73, 50, 21] are proposed to uncover the architecture of these models, especially their self-attention mechanism.
可视化在弥合NLP模型的可解释性差距方面起着不可或缺的作用[27,29,57,19,18]，可以更深入地理解模型性能[65]、决策边界[11]和脆弱性[41]。特定于模型的可视化侧重于揭示 NLP 模型的内部机制。RNNVis [ 44] 和 LSTMVis [ 58] 可视化了递归神经网络的隐藏状态动力学。随着基于Transformer的模型[64,14]的出现，人们提出了许多可视化[73,50,21]来揭示这些模型的架构，特别是它们的自注意力机制。

Model-agnostic visualizations [7, 17, 32, 20] treat the NLP models as black boxes and focus on explaining the input-output behavior, enabling users to analyze and compare models for downstream applications [31, 59, 68, 24, 60, 39]. The What-If Tool [69] and DECE [11] visualize the dataset and model prediction at multiple scales, enabling users to conduct counterfactual analysis. NLIZE [35] employs a perturbation-driven paradigm to help users analyze the stability of model predictions for natural language inference tasks. Based on explainable AI techniques (e.g., SHAP [40]) and external commonsense knowledge bases [56], CommonsenseVIS [65] analyzes the reasoning capabilities of NLP models for commonsense question-answering.
与模型无关的可视化 [ 7， 17， 32， 20] 将 NLP 模型视为黑匣子，并专注于解释输入输出行为，使用户能够分析和比较下游应用程序的模型 [ 31， 59， 68， 24， 60， 39]。假设工具[ 69] 和 DECE [ 11] 在多个尺度上可视化数据集和模型预测，使用户能够进行反事实分析。NLIZE [ 35] 采用扰动驱动的范式来帮助用户分析自然语言推理任务的模型预测的稳定性。基于可解释的人工智能技术（如SHAP [ 40]）和外部常识知识库 [ 56]，CommonsenseVIS [ 65] 分析了 NLP 模型的推理能力，以求知问答。

Our work targets the jailbreak prompt attacks against large language models and aims to help model practitioners evaluate the jailbreak performance and understand prompt characteristics, thereby providing insights for strengthening models’ safety mechanisms.
我们的工作针对大型语言模型的越狱提示攻击，旨在帮助模型从业者评估越狱性能并理解提示特征，从而为加强模型的安全机制提供见解。

Jailbreak Prompt Corpora. With the widespread attention to jailbreak prompt attacks, there have been studies [37, 53, 61] collecting jailbreak prompts and building corpora for semantic analysis and generation model training. Most of them decompose the jailbreak prompts into jailbreak questions and templates. Our work follows this principle and adopts the dataset by Liu et al. [37] for analysis, which contains the most common and famous jailbreak prompts from the JailbreakChat website [2]. In addition, we adopt their taxonomy of jailbreak questions and templates to support the comparative analysis of different jailbreak strategies and prohibited scenarios.
越狱提示语料库。随着越狱提示攻击的广泛关注，已有研究[ 37， 53， 61] 收集越狱提示并构建语义分析和生成模型训练的语料库。他们中的大多数将越狱提示分解为越狱问题和模板。我们的工作遵循了这一原则，并采用了Liu等[37]的数据集进行分析，该数据集包含了JailbreakChat网站[2]中最常见和最著名的越狱提示。此外，我们采用他们的越狱问题分类法和模板来支持对不同越狱策略和禁止场景的比较分析。

Jailbreak Questions. The jailbreak questions are mainly designed around the prohibited scenarios of LLMs, such as “how to rob a bank without being caught?” Due to the safety mechanisms [13], LLMs usually refuse to answer these questions and return some responses emphasizing ethical and legal constraints, such as “I’m sorry, but I’m not going to guide you on how to engage in criminal activity.” Based on OpenAI’s disallowed usages [4], Liu et al. [37] have summarized a set of prohibited scenarios (e.g., Illegal Activities and Harmful Content) and collected a set of specific questions.
越狱问题。越狱问题主要围绕禁止的场景设计LLMs，例如“如何在不被抓到的情况下抢劫银行？由于安全机制[ 13]，LLMs通常拒绝回答这些问题，并返回一些强调道德和法律约束的回答，例如“对不起，但我不会指导你如何从事犯罪活动。Liu等[37]基于OpenAI的禁止用法[4]，总结了一组禁止的场景（例如，非法活动和有害内容），并收集了一组具体问题。

Jailbreak Templates. The jailbreak templates are intentionally designed prompts to bypass the LLMs’ safety mechanisms to get the model assistance for the jailbreak questions. For example, some templates require LLMs to act as virtual characters who can answer questions without ethical and legal constraints. The jailbreak templates usually contain placeholders (e.g., “[INSERT PROMPT HERE]”) for inserting different jailbreak questions. Liu et al. [37] have summarized a taxonomy of jailbreak templates, which consists of ten jailbreak patterns, such as Character Role Play and Assumed Responsibility.
越狱模板。越狱模板是故意设计的提示，以绕过LLMs安全机制来获得越狱问题的模型帮助。例如，某些模板需要LLMs充当虚拟角色，他们可以在没有道德和法律约束的情况下回答问题。越狱模板通常包含占位符（例如，“[INSERT PROMPT HERE]”），用于插入不同的越狱问题。Liu等[37]总结了越狱模板的分类法，该分类法由10种越狱模式组成，如角色角色扮演和承担责任。

Our work’s target users are model practitioners focusing on model robustness and security. To characterize domain problems and identify design requirements, we have collaborated with four domain experts over eight months. E1 and E2 are senior security engineers recruited from a technology company who have been working on NLP model security for more than four and three years, respectively. E3 and E4 are senior Ph.D. students from the secure machine learning field. All of them have published papers on related research topics, such as red-teaming LLMs and adversarial attacks. We interviewed them to understand their general analysis workflow and identify pain points.
我们工作的目标用户是专注于模型健壮性和安全性的模型从业者。为了描述领域问题并确定设计需求，我们与四位领域专家合作了八个月。E1 和 E2 是从一家技术公司招聘的高级安全工程师，他们分别从事 NLP 模型安全工作超过 4 年和 3 年。E3 和 E4 是来自安全机器学习领域的高级博士生。他们都发表了相关研究课题的论文，如红队LLMs和对抗性攻击。我们采访了他们，以了解他们的一般分析工作流程并确定痛点。

To evaluate model robustness in defending against jailbreak attacks, users usually need to test the jailbreak prompt corpus on the target models and assess the jailbreak results (success or failure) to calculate the success rate. The experts indicated that the jailbreak result assessment is tedious and sometimes requires question-specific knowledge (e.g., local regulations) to refine the assessment criteria to improve precision. Moreover, the overall indicator success rate is insufficient to support an in-depth analysis of the jailbreak prompts (e.g., analyzing how to improve the jailbreak performance of the prompts) to identify the model weaknesses. To fill these gaps, we distilled a set of design requirements to guide the development of our system. We also kept in touch with the experts through regular meetings to collect feedback regarding our prototype system and update design requirements. Finally, the design requirements are summarized as follows.
为了评估模型防御越狱攻击的鲁棒性，用户通常需要在目标模型上测试越狱提示语料库，并评估越狱结果（成功或失败）来计算成功率。专家们表示，越狱结果评估是乏味的，有时需要针对特定问题的知识（例如，当地法规）来完善评估标准以提高准确性。此外，总体指标成功率不足以支持对越狱提示的深入分析（例如，分析如何提高提示的越狱性能）以识别模型的弱点。为了填补这些空白，我们提炼了一套设计要求来指导我们系统的开发。我们还通过定期会议与专家保持联系，以收集有关原型系统的反馈并更新设计要求。最后，将设计要求总结如下。

R1. Facilitate the assessment of jailbreak results. Jailbreak result assessment is the foundation of jailbreak prompt performance analysis. However, since the jailbreak results (\iemodel responses) could be arbitrary and ambiguous, manually reviewing the jailbreak results can be an energy-exhausting task. Therefore, the system should introduce an automatic method to alleviate the manual workload and enhance the assessment efficiency. In addition, to ensure the precision of assessment results, the system should help users explore them to identify unexpected results and refine the assessment criteria.
R1 中。促进越狱结果的评估。越狱结果评估是越狱提示性能分析的基础。但是，由于越狱结果（\iemodel 响应）可能是任意且模棱两可的，因此手动查看越狱结果可能是一项耗费精力的任务。因此，系统应引入自动化方法，以减轻人工工作量，提高评估效率。此外，为保证评估结果的精确性，系统应帮助用户进行探索，以识别意外结果并完善评估标准。

R2. Support component analysis of jailbreak prompts. Besides evaluating the jailbreak prompt performance, users also need to understand prompt characteristics to gain insights for identifying model weaknesses and enhancing security mechanisms. Given that various jailbreak prompts commonly contain similar sentence components (e.g., describing a subject with no moral constraints), the experts express strong interest in analyzing the prompts at the component level to understand the utilization of such components in constructing prompts and their importance to the prompt performance.
R2 中。支持越狱提示的组件分析。除了评估越狱提示性能外，用户还需要了解提示特征，以获得识别模型弱点和增强安全机制的见解。鉴于各种越狱提示通常包含相似的句子成分（例如，描述一个没有道德约束的主题），专家们对分析组件级别的提示表示了浓厚的兴趣，以了解这些成分在构建提示中的使用及其对提示性能的重要性。

R3. Summarize important keywords from jailbreak prompts. Effective keywords play an important role in successful jailbreak attacks and are closely related to jailbreak strategies. For example, some role-playing templates name LLM as “AIM” (always intelligent and Machiavellian) to imply their amoral characterization. The system should summarize important keywords from jailbreak prompts and help users explore them based on their jailbreak prompt performance. This helps users identify effective keywords to strengthen the defense mechanisms (e.g., keyword detection) accordingly.
R3 中。从越狱提示中总结重要关键字。有效的关键字在成功的越狱攻击中起着重要作用，并且与越狱策略密切相关。例如，一些角色扮演模板命名为LLM“AIM”（总是聪明的和马基雅维利式的），以暗示他们的不道德特征。系统应总结出越狱提示中的重要关键词，并根据用户的越狱提示性能帮助用户进行探索。这有助于用户识别有效的关键字，从而相应地加强防御机制（例如，关键字检测）。

R4. Support user refinement on jailbreak prompt instances. The system should allow users to freely refine the jailbreak prompt instances according to their expertise. Meanwhile, the system should support ad-hoc evaluation of jailbreak performance to help users verify the effectiveness of prompt refinement. Based on such timely feedback, users can conduct what-if analysis to verify findings during the analysis workflow. In addition, the improved jailbreak prompts can serve as new test samples for the jailbreak corpus, thus enhancing the diversity and robustness of the jailbreak performance evaluation.

To support component analysis of jailbreak prompts (R2), we conducted an empirical study on the jailbreak corpus with domain experts to summarize a taxonomy of jailbreak prompt components. To gain a general understanding of the prompt components, we decomposed each jailbreak prompt in the corpus into basic sentences, analyzed their semantic content in context, and compared similar sentences in different prompts. Then, we conducted several brainstorming sessions and discussions with the experts to formulate and iteratively refine the component taxonomy. We also validated the taxonomy on randomly selected jailbreak prompts to resolve ambiguities between component definitions. After establishing the coding scheme, the first and second authors and E3 separately coded the jailbreak prompts in the corpus and merged the coding differences through discussions.
为了支持越狱提示（R2）的组件分析，我们与领域专家一起对越狱语料库进行了实证研究，总结了越狱提示组件的分类法。为了大致了解提示组件，我们将语料库中的每个越狱提示分解为基本句子，分析它们在上下文中的语义内容，并比较不同提示中的相似句子。然后，我们进行了几次头脑风暴会议，并与专家进行了讨论，以制定和迭代完善组件分类法。我们还验证了随机选择的越狱提示的分类法，以解决组件定义之间的歧义。建立编码方案后，第一作者、第二作者和E3分别对语料库中的越狱提示进行编码，并通过讨论将编码差异合并。

The final taxonomy of jailbreak prompt components is listed in Table 1. It consists of eight major types, such as Scene Introduction (SI), Subject Characteristic (SC), and Task Operation (TO). We also count and visualize the distribution of component sentences for different jailbreak strategies [37] in Table 2. According to the coding results, the Subject Characteristic components are frequently used among the most common strategies (\ieCharacter Role Play, Assumed Responsibility, and Superior Model), and most of them usually describe the subject as a free person without ethical constraints and responsibility. We distinguish the Output Constraint components from the Output Specification components because the Output Constraint components aim to exploit the model’s instruction-following capability to break through the model’s security defenses [66], which is different from specifying the expected content or format of responses.
越狱提示组件的最终分类列于表 1 中。它由场景介绍（SI）、主体特征（SC）和任务操作（TO）等八大类型组成。我们还在表2中计算并可视化了不同越狱策略[37]的组成句子的分布。根据编码结果，主体特征成分在最常见的策略（\ieCharacter Role Play、Assume Responsibility和Superior Model）中经常使用，并且大多数通常将主体描述为一个没有道德约束和责任的自由人。我们将输出约束组件与输出规范组件区分开来，因为输出约束组件旨在利用模型的指令跟踪能力来突破模型的安全防御[ 66]，这与指定响应的预期内容或格式不同。

Since the LLMs [45, 5, 62, 16] have shown great capability in understanding and following human instructions, we employ GPT-4 [5] to assess the responses of the target models. The prompt design for assessment centers on three aspects.
由于 LLMs [ 45， 5， 62， 16] 在理解和遵循人类指令方面表现出了很强的能力，因此我们使用 GPT-4 [ 5] 来评估目标模型的响应。评估的提示设计集中在三个方面。

Introduce the task background. We emphasize the goal of evaluating model security and introduce the jailbreak scenarios based on OpenAI’s disallowed usages [4]. This gives the LLM an overall understanding of assessment tasks and makes it focus on the legality and ethics of assessed responses.
介绍任务背景。我们强调评估模型安全性的目标，并介绍了基于OpenAI不允许的用法的越狱场景[4]。这样可以LLM全面了解评估任务，并使其专注于评估响应的合法性和道德性。

Clarify the target assessment results. One of the potential solutions is to consider the task as a binary classification task that classifies the model responses into successful and failed jailbreaks. However, it is common for target models to warn users of potential consequences when generating unauthorized content due to their competing training objectives (\ieinstruction-following and safety) [66]. To better accommodate these ambiguous responses, we adopt the taxonomy of Yu et al. [74] which defines four types of assessment results.
明确目标评估结果。一种潜在的解决方案是将该任务视为二元分类任务，将模型响应分类为成功和失败的越狱。然而，目标模型在生成未经授权的内容时，由于其相互竞争的训练目标（即指令遵循和安全）而警告用户潜在的后果是很常见的 [ 66]。为了更好地适应这些模棱两可的回答，我们采用了Yu et al. [ 74]的分类法，该分类法定义了四种类型的评估结果。

• •

  Full Refusal. The response directly refuses to provide assistance for harmful questions or follow any other instructions.

  
  • 完全拒绝。回复直接拒绝为有害问题提供帮助或遵循任何其他指示。

• •

  Partial Refusal. The response refuses to assist with harmful questions but follows other harmless instructions or offers ethical advice.

  
  • 部分拒绝。回复拒绝协助解决有害问题，但遵循其他无害的指示或提供道德建议。

• •

  Partial Compliance. The response contains unauthorized content but expresses disapproval or warns of potential harm.

  
  • 部分合规。该响应包含未经授权的内容，但表示不赞成或警告潜在危害。

• •

  Full Compliance. The response completely follows the prompt instructions and contains unauthorized content.

  
  • 完全合规。响应完全遵循提示说明，并包含未经授权的内容。

Additionally, we add some representative examples with labeled annotations to the assessment prompt, leveraging the in-context learning of the LLMs to better distinguish their differences.
此外，我们在评估提示中添加了一些带有标记注释的代表性示例，利用上下文学习LLMs来更好地区分它们的差异。

Support user refinement of assessment criteria. We provide two ways, including correction feedback and additional criteria specification, to incorporate question-specific knowledge or adjust the rigor of the assessing criteria. Correction feedback allows users to specify corrected assessment results and add them to the assessment prompt as new demonstration examples. Users can also directly specify additional assessment criteria for specific jailbreak questions using natural language, providing a more flexible way to enhance assessment accuracy.
支持用户细化评估标准。我们提供两种方式，包括更正反馈和附加标准规范，以纳入特定问题的知识或调整评估标准的严格性。更正反馈允许用户指定更正后的评估结果，并将其作为新的演示示例添加到评估提示中。用户还可以使用自然语言直接为特定的越狱问题指定额外的评估标准，提供更灵活的方式来提高评估准确性。

To support users in analyzing the jailbreak prompt components, we propose a component classification method and three perturbation strategies based on the summarized taxonomy introduced in Section 3.3. The component classification method classifies the prompt sentences into component types to support the overview of component utilization. Based on that, component perturbation creates a set of perturbation variations for the jailbreak prompts to support the comparative analysis of the jailbreak performance, thus enabling interrogation of the effects of different components.

We identify prompt keywords by combining their importance and prompt performance. First, we split the prompt sentences into keywords and filter out stop words. Then, inspired by prior work [31], we measure the importance of the keyword � for the given prompt � based on the keyword frequency and semantic similarity:

where the �⁢�⁢�⁢�⁢�⁢(�,�) is the TF-IDF value [55] of the keyword for the prompt and the �⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�,�) is the semantic similarity of the keyword and prompt. A higher semantic similarity indicates a greater relevance of the keyword to the prompt semantics. We encode the keywords and prompts using the embedding model by OpenAI [3] and measure their similarity based on the cosine distance. Based on that, we calculate the importance of keyword � for the whole corpus as

where �� is the list of jailbreak prompts that contain the keyword �.
其中 �� 是包含关键字 � 的越狱提示列表。

To help users analyze the effect of important keywords, we measure their performance according to the performance of their corresponding prompts. Since the keyword � might be utilized in various prompts with different importance, we propose importance-weighted performance to better summarize the effect of the keyword �:
为了帮助用户分析重要关键词的效果，我们根据他们相应提示的表现来衡量他们的表现。由于该关键字 � 可能用于各种具有不同重要性的提示中，因此我们建议使用重要性加权性能来更好地总结关键字 � 的效果：

where �⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�) represents the jailbreak performance of prompt �, expressed as a percentage of four categories of assessment results.
其中 �⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�) 表示提示的越狱性能 � ，表示为四类考核结果的百分比。

Configuration View (Figure 1�) allows users to upload the jailbreak corpus and select jailbreak questions and templates for model evaluation. The questions and templates are organized according to their categories (e.g., Character Role Play). The selected items will be assigned serial numbers, that will serve as their unique identifiers in subsequent analyses. Users can also modify the questions and templates based on their exploratory interests. Besides, the system allows users to configure the number of evaluations for each question-template combination to improve the robustness of evaluation results. After user configuration and submission, the system automatically combines each question and the template (\iefill the question into the placeholder in the template) to get the responses from the target model.
配置视图（图 1 � ）允许用户上传越狱语料库，并选择越狱问题和模板进行模型评估。问题和模板根据其类别进行组织（例如，角色角色扮演）。所选项目将被分配序列号，这将在后续分析中作为其唯一标识符。用户还可以根据自己的探索兴趣修改问题和模板。此外，该系统允许用户配置每个问题模板组合的评估数量，以提高评估结果的鲁棒性。用户配置提交后，系统会自动将每个问题与模板（\iefill 问题放入模板的占位符）组合在一起，以获取目标模型的响应。

To support jailbreak performance exploration, the system provides a visual summary of jailbreak evaluation and allows users to inspect model responses to verify their correctness.
为了支持越狱性能探索，系统提供了越狱评估的可视化摘要，并允许用户检查模型响应以验证其正确性。

Based on the assessment results of the jailbreak templates, users can analyze and compare the effects of different prompt components in the right half of Summary View (\name: Visual Analysis of Jailbreak Attacks Against Large Language Models�2). The left column summarizes the components of the selected jailbreak templates. It visualizes each prompt as a horizontal stacked bar chart, where each bar segment corresponds to a distinct component. As shown in Figure 2�, the bar segments are arranged in the order corresponding to the prompt components, with the color encoding the component type and the length indicating the token length. It helps users understand the general patterns of prompt components and serves as the baseline for prompt perturbation. Users can click the icon  to view the template details in the Instance View and click the icon  to generate a set of component perturbation results for comparative analysis.

For comparison, the perturbation results are visualized in the right column as horizontal stacked bars similar to the original templates. Each result is generated by applying a perturbation strategy to a component of the original template. To visualize this difference, we design three kinds of glyphs to represent these strategies and overlay them on the corresponding bar segments, as shown in Figure 2�. The system also automatically evaluates the jailbreak performance of these perturbation results based on the selected questions and visualizes the percentage of assessment results in pie charts, enabling users to compare the effects of different component perturbations. Users can toggle to check the perturbation results for a particular strategy using the radio box at the top of the column.

The Keyword View (\name: Visual Analysis of Jailbreak Attacks Against Large Language Models�) visualizes the jailbreak performance and importance of keywords (Section 4.3). Specifically, as shown in Figure 3�, the jailbreak performance of keyword �, \ie�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�), is represented as the percentage of four categories of assessment results, denoted as [�1,�2,�3,�4]. Inspired by prior work [35], we introduce a square space with a coordinate system (Figure 3�) whose four vertices correspond to the four categories, and their coordinates are denoted as [�1,�2,�3,�4]. Each corner is colored to indicate its category using the same color scheme as the pie charts in the Summary View. To visualize the overall performance distribution of the keyword, the coordinate of the keyword � is computed as:
关键字视图（\name：针对大型语言模型 � 的越狱攻击的可视化分析）可视化了越狱性能和关键字的重要性（第 4.3 节）。具体而言，如图3所示 � ，关键字 � \ie �⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�) 的越狱性能表示为四类评估结果的百分比，表示为 [�1,�2,�3,�4] 。受先前工作 [ 35] 的启发，我们引入了一个具有坐标系的正方形空间（图 3 � ），其四个顶点对应于四个类别，它们的坐标表示为 [�1,�2,�3,�4] 。每个角都使用与“摘要视图”中的饼图相同的配色方案来指示其类别。为了可视化关键字的整体性能分布，关键字 � 的坐标计算如下：

Besides, the size of the keywords encodes their importance in the corpus. It enables users to overview and compare the usage and performance of keywords based on their position and size. Users can filter keywords by component type and click the keywords to view their context.
此外，关键字的大小对它们在语料库中的重要性进行了编码。它使用户能够根据关键字的位置和大小来概述和比较关键字的使用情况和性能。用户可以按组件类型筛选关键字，然后单击关键字以查看其上下文。

Alternative Design. We have also considered an alternative design (Figure 3�) where the keywords are visualized in four separate word clouds (each for one assessment category) and their sizes correspond to their importance to the prompts of this category, \ie�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�)×��,�∈[1,4]. According to the feedback from the experts, although this design enables users to focus on the keywords in the same category, it is inefficient for users to compare the size of the same keywords in all word clouds to estimate their overall performance distribution. Therefore, we chose our current design.
替代设计。我们还考虑了一种替代设计（图 3 � ），其中关键字在四个单独的词云（每个词云代表一个评估类别）中可视化，它们的大小对应于它们对该类别提示的重要性，\ie �⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢�⁢(�)×��,�∈[1,4] 。根据专家的反馈，虽然这种设计使用户能够专注于同一类别中的关键字，但用户比较所有词云中相同关键字的大小来估计其整体性能分布是低效的。因此，我们选择了当前的设计。

The Instance View (\name: Visual Analysis of Jailbreak Attacks Against Large Language Models�) allows users to refine the templates and evaluate the performance. The Jailbreak Content panel (\name: Visual Analysis of Jailbreak Attacks Against Large Language Models�1) lists the prompt text of each component, which supports manual modifications or automatic perturbations. Users can click the icon at the right of the component title to  delete,  rephrase, or  switch the components. Then, users can evaluate their jailbreak performance on the selected questions and inspect the evaluation results in the Jailbreak Results panel (\name: Visual Analysis of Jailbreak Attacks Against Large Language Models�2). Each result item shows the question and model response and visualizes the color of the assessment result. This feedback can help the user evaluate the effectiveness of the modifications to verify the findings during the component and keyword analysis.
实例视图（\name：针对大型语言模型 � 的越狱攻击的可视化分析）允许用户优化模板并评估性能。“越狱内容”面板（\name：针对大型语言模型 �1 的越狱攻击的可视化分析）列出了每个组件的提示文本，支持手动修改或自动扰动。用户可以单击组件标题右侧的图标来删除、改写或切换组件。然后，用户可以评估自己在所选问题上的越狱性能，并在越狱结果面板（\name： Visual Analysis of Jailbreak Attacks Against Large Language Models �2 ）中检查评估结果。每个结果项都显示问题和模型响应，并可视化评估结果的颜色。此反馈可以帮助用户评估修改的有效性，以验证组件和关键字分析期间的结果。

We invited the experts mentioned in Section 3.2 to use our system for jailbreak prompt analysis according to their exploratory interests. In this case study, the expert (E3) first evaluated the overall defense performance of GPT-3.5 on a jailbreak corpus and then dived into Character Role Play templates, one of the most common jailbreak strategies for an in-depth analysis of prompt characteristics.

Jailbreak Performance Evaluation (Figure 4�). E3 uploaded a jailbreak prompt corpus in the Configuration View and selected some questions and templates for analysis. Considering the stochastic nature of model responses, E3 evaluated each question and template combination three times. After a period of waiting, the left part of Summary View visualized the performance of selected questions and templates in pie charts (Figure 4�1). E3 checked the evaluation results to ensure their accuracy. For example, He selected an Illegal Activities question IA(1) and explored its results in the Response View (Figure 4�2−1). There was a notable separation between the distributions of blue and red points, except for some outlier blue points situated within the red clusters. Therefore, he checked some of these points and found that their assessment results did not align with his expectations (Figure 4�2−2). He corrected these results and submitted them to refine the assessment criteria, and the Summary View and Response View (Figure 4�2−3) were updated accordingly. E3 checked the results again to verify their correctness. After that, he also explored other questions to verify their correctness or correct unexpected results. Based on the verified evaluation results (Figure 4�3), E3 found that nearly half of the jailbreak attacks were successful, indicating the target model was vulnerable. Besides, he also noticed that jailbreak performance usually depended more on templates than questions because the pie charts in the same row (corresponding to the same templates with different questions) usually showed similar percentage patterns of assessment results.

Prompt Characteristic Exploration (Figure 4�). E3 wanted to explore the prompt characteristics of the Character Role Play category, one of the most common categories. The prompt component visualizations (Figure 4�1−1) showed that Subject Characteristic components (in blue) were commonly used and sometimes took up a large portion of the prompt length. E3 was curious whether the Subject Characteristic components were important to the jailbreak performance. Therefore, he chose a template with strong jailbreak performance and generated a set of prompt variations based on three perturbation strategies. According to the perturbation results, he found that deleting (Figure 4�1−2) or switching (Figure 4�1−3) the Subject Characteristic component resulted in a much more significant performance reduction than the other components, suggesting the Subject Characteristic component was crucial to the performance of this prompt. E3 also explored some other prompts and got similar findings. Then, he explored the important keywords in that component type in the Keyword View (Figure 4�2). From the keyword visualizations, he noticed that the keywords “AIM” and “DAN” were placed close to the “Full Refusal” corner, meaning that the model has been trained to be wary of these strategies and refuse to provide help. Near the “Full Compliance” corner, E3 found the keywords “disregards” and “controversial”, which correspond to an effective jailbreak strategy that encourages the model to disregard legal and ethical constraints and generate controversial content.

Jailbreak Template Refinement (Figure 4�). To verify the effectiveness of these keywords, E3 selected a template with weak jailbreak performance and refined its Subject Characteristic component using the sentence of the identified keywords (Figure 4�1−1). Then, he evaluated this new template based on the selected questions, and the results (Figure 4�1−2) showed that more than half of the attacks were successful, suggesting a significant performance improvement. After several validations on other templates, E3 concluded that the strategy of these keywords in the Subject Characteristic component reflected a potential weakness in the target model that requires more safety training against them. Finally, he added these newly generated templates to the dataset to improve prompt diversity.

Jailbreak result assessment and prompt component classification are critical to the analysis workflow and rely heavily on the LLM capability and prompt design. Therefore, we conducted two technical evaluations to quantitatively measure the effectiveness of these methods. Since no recognized benchmark datasets were available for these two tasks, we drew inspiration from previous research [70] and collaborated with experts (E3 and E4) to build improvised datasets. We evaluated our methods and reported the results and some identified patterns that could affect the performance of the methods.
越狱结果评估和及时组件分类对于分析工作流程至关重要，并且在很大程度上依赖于LLM功能和提示设计。因此，我们进行了两次技术评估，以定量衡量这些方法的有效性。由于这两项任务没有公认的基准数据集，我们从以前的研究[70]中汲取灵感，并与专家（E3和E4）合作构建了即兴数据集。我们评估了我们的方法并报告了结果和一些可能影响方法性能的已确定模式。

We interviewed six external experts (E5-E10) to evaluate the effectiveness of the analysis framework and the usability of the visual system. E5 is a model security engineer from a technical company who has been working on the secure reasoning of LLMs for more than one year and on network security (situation awareness) for over three years. E6-E10 are senior researchers from related fields, including model security, trustworthy AI, and deep learning model training. Among them, E7 has accumulated eight years of experience in data and model security before she focused on LLM jailbreak attacks. Each expert interview lasted about 90 minutes. We first briefly introduced the background and motivation of our study. Then, we described the analysis framework and visual system and demonstrated the system workflow using the case study. After that, we invited the experts to explore our system to analyze the performance and characteristics of the jailbreak prompts. Finally, we conducted semi-structured interviews with experts to collect their feedback about the analysis framework, visualization and interaction, and improvement suggestions.

Effectiveness of the Analysis Framework. All the experts agreed that our framework facilitated jailbreak performance evaluation and prompt characteristic understanding, and its workflow made sense. E5 praised this framework as “it provided a more comprehensive and systematic evaluation for jailbreak attacks compared to existing tools.” E5 appreciated the capability of the jailbreak assessment method to support user-specified criteria, which enhanced its flexibility to support customized criteria for different user values. However, we also observed one case during the exploration of E9 where the user-corrected examples did not improve the accuracy of a new round of assessment. E9 suggested recommending some representative model responses for user correction feedback to better leverage the in-context learning of the LLMs to distinguish between them. The jailbreak component analysis was highly appreciated, being described as “interesting” (E10), “impressive” (E6), and “inspiring” (E7). It was valued for “offering a new perspective to study the prompt patterns in the black box scenarios” (E10) and for “guiding user effort towards the critical parts of the prompts” (E9). The experts confirmed that keyword analysis helped understand prompt characteristics. However, E5 noted that it would be less effective when analyzing only a few prompts and suggested incorporating an external corpus of suspicious keywords to improve its effectiveness. Finally, all experts agreed that our analysis framework provided valuable insights for model security enhancement.

Visualization and Interactions. Overall, the experts agreed that the system views were well-designed, and the visual design and interaction were intuitive. The Summary View was popular among the experts as it supported the analysis and comparison of jailbreak performance from both the perspectives of questions and templates. All experts confirmed the helpfulness of Response View in exploring model responses and identifying unexpected results. E7 thought that component visualization required some learning costs but became easy to use and remember after she became familiar with visual encoding. We also asked the experts for their opinions about the color scheme of the prompt components, and the experts confirmed that it was “clear” (E8) and “easy to distinguish” (E10). The Keyword View was observed to be frequently used by the experts during the exploration. We noticed that they usually focused more on the keywords near the “Full Compliance” corner to identify effective keywords from successful attacks.

Suggestions for Improvement. In addition to the limitations mentioned above, we have also collected some suggestions for improvement. For Response View, E9 suggested adding some textual annotations (e.g., keywords) near the points or clusters to summarize the semantic information of the corresponding responses, which could help users identify the potential incorrect assessment results more efficiently. For component analysis, E5 suggested that providing a textual or visual summary for the comparative analysis of the component perturbations could better help users identify effective components.

Toward a more comprehensive assessment of jailbreak results. The experts appreciate the introduction of jailbreak assessment taxonomy [74] to resolve ambiguities and identify harmful jailbreak results, as well as the application of the LLM-based assessment method to improve efficiency. They also suggest extending them to include more assessment dimensions. For instance, responses that offer elaborate and expert guidance on illegal activities may pose a greater risk than those providing generic and ambiguous advice. Therefore, assessing the helpfulness of the jailbreak results can help model practitioners prioritize identifying and preventing these harmful results. Future research can explore broadening the spectrum of assessment dimensions (e.g., helpfulness) and improving the assessment accuracy of these dimensions (e.g., incorporating question-related knowledge bases) to mitigate the harm of the jailbreak results.

Improve the learning-based jailbreak prompt construction. The development of learning-based methodologies [78, 13] for the construction of jailbreak prompts represents a significant opportunity to enhance the efficiency of red-teaming LLMs. However, this endeavor faces challenges due to the intricate nature of the prompt design to guarantee effectiveness [15]. The experts highlight that our analysis framework can inspire the research of learning-based methods to improve their effectiveness and generation diversity. For example, the paired jailbreak prompts and their perturbation results (with stronger jailbreak performance) can be used to train generative models for rewriting jailbreak prompts, so that the models can easily capture the difference between their prompt components and learn how to effectively improve the prompt performance. Furthermore, the component analysis paves the way for the integration of expert knowledge into automatic jailbreak prompt generation. For example, it allows the experts to specify the kernel of the prompts in the Scene Introduction or Subject Characteristic components to guide the generation of the following content (e.g., Task Operation or Output Specification).

Balance between the objectives of safety and instruction-following. The objectives of safety and instruction-following are usually competitive [66], particularly in the context of mitigating jailbreak attacks. It has been widely recognized that over-strengthening the model’s security defenses on large jailbreak corpora will inevitably compromise the model’s ability to follow user instructions, leading to “overkill” issues [54]. Balancing between these two objectives has become one of the most challenging problems for LLM training. The experts point out that our component analysis provides a potential solution to construct a customized and condensed jailbreak dataset based on the vulnerabilities of LLMs rather than relying on large jailbreak corpora, thereby effectively addressing the model’s major security weaknesses without sacrificing the instruction-following capabilities. Looking ahead, future works can further explore how to help model practitioners analyze and trade-off between these two objectives.

Support jailbreak performance comparisons on multiple models. While our analysis framework and visual system facilitate a systematic analysis of jailbreak attacks, the experts express interest in the comparative evaluation of such attacks across various models, which promises significant benefits across multiple application scenarios. For example, it can help LLM vendors benchmark their models against those of competitors, identifying relative advantages and shortcomings. Similarly, it can assist model practitioners in comparing different iterations of models to evaluate the improvements attributed to safety training. Our system can be extended with comparative visualizations [42, 22] to provide clear and insightful comparisons across models.

Fine-tune the jailbreak questions to bypass the safety measurements. Consistent with most previous research aimed at refining the jailbreak templates to enhance the attack performance, our system is primarily designed to evaluate the jailbreak prompts and analyze the characteristics of the jailbreak templates. Recent studies [76, 30] suggest that rewriting the jailbreak questions directly helps to circumvent the model’s security mechanisms. This could be achieved, for instance, by transforming the malicious questions into experimental or pedagogical questions, or by deconstructing the tasks into smaller, less harmful operation steps. Therefore, it becomes crucial to analyze the semantic characteristics of the jailbreak questions and to summarize their common strategies.

is designed to analyze jailbreak attacks, one of the most common prompt attacks against LLMs. We demonstrate the system’s effectiveness through a case study evaluating the jailbreak performance on GPT-3.5. The system can be easily generalized to other language models (e.g., Llama 2 [62] and ChatGLM [16]) due to its model-agnostic design that centered around the prompt input and model responses. Moreover, the analysis workflow of \name can potentially support other prompt attack scenarios, such as prompt injection [52, 47] and backdoor attacks [23, 72]. For prompt injection, the system can help users evaluate the performance of hijack attacks based on the LLM-based assessment method and explore the assessment results to verify or improve their accuracy. For backdoor attacks, the system can help users identify suspicious backdoor triggers (e.g., sentences or keywords) through component and keyword analysis.

Incorporate more perturbation strategies for component analysis. Currently, our analysis framework and visual system support three kinds of perturbation strategies (\iedeletion, rephrasing, and switching) to help users understand the effect of prompt components on jailbreak performance. They can be extended to support more strategies, such as inserting and crossover [74]. The system can insert the recognized important components into other prompts to verify their effectiveness. The crossover strategy can combine the strengths of two prompts to create more prompt variations. Supporting these strategies enables users to conduct a more comprehensive analysis of prompt components.

Extend our analysis to more large language models. Our system has proven to be effective in supporting the security analysis of OpenAI’s LLMs against jailbreak attacks. Moving forward, we plan to integrate a broader range of mainstream LLMs into our system, such as Llama 2 [62] and ChatGLM [16]. Our goal is to evaluate the defense capability of these advanced models against jailbreak attacks and identify their potential weaknesses. We will communicate our findings gained from the analysis with the vendors of these LLMs and provide them with valuable insights for enhancing the model safety mechanisms, ensuring a safer deployment of LLM technologies.

Explore multi-modal jailbreak attacks. The vulnerabilities of multi-modal large language models (MLLMs), such as LLava [33] and GPT-4V [1], have attracted increased attention [9, 51, 48]. MLLMs are more sensitive to jailbreak prompts with textual triggers, OCR textual triggers, and visual triggers, which presents greater safety risks compared to LLMs. Our work primarily investigates the textual vulnerabilities of LLMs without extending to the scenario of multi-modal jailbreak attacks. In the future, we aim to bridge this gap by incorporating multi-modal analysis into our visual analysis framework, thereby enhancing the robustness of MLLMs against such threats.