Skip to content
2 changes: 1 addition & 1 deletion doc/bibliography.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ All academic papers, research blogs, and technical reports referenced throughout
:::{dropdown} Citation Keys
:class: hidden-citations

[@aakanksha2024multilingual; @adversaai2023universal; @andriushchenko2024tense; @anthropic2024manyshot; @aqrawi2024singleturncrescendo; @atr2026; @bethany2024mathprompt; @bhardwaj2023harmfulqa; @bhardwaj2024homer; @boucher2023trojan; @brahman2024coconot; @bryan2025agentictaxonomy; @bullwinkel2025airtlessons; @bullwinkel2025repeng; @bullwinkel2026trigger; @chao2023pair; @chao2024jailbreakbench; @choi2026xlsafetybench; @cui2024orbench; @darkbench2025; @derczynski2024garak; @ding2023wolf; @embracethered2024unicode; @embracethered2025sneakybits; @gehman2020realtoxicityprompts; @ghosh2025aegis; @ghosh2025ailuminate; @gong2025figstep; @gupta2024walledeval; @haider2024phi3safety; @han2024medsafetybench; @han2024wildguard; @hiddenlayer2025policypuppetry; @hines2024spotlighting; @inie2025summon; @ji2023beavertails; @ji2024pkusaferlhf; @jiang2025sosbench; @jones2025computeruse; @kingma2014adam; @li2024drattack; @li2024mossbench; @li2024saladbench; @li2024wmdp; @lin2023toxicchat; @liu2024flipattack; @liu2024mmsafetybench; @lopez2024pyrit; @luo2024jailbreakv; @lv2024codechameleon; @mazeika2023tdc; @mazeika2024harmbench; @mckee2024transparency; @mehrotra2023tap; @microsoft2024skeletonkey; @odin2024; @palaskar2025vlsu; @pfohl2024equitymedqa; @promptfoo2025ccp; @robustintelligence2024bypass; @roccia2024promptintel; @rottger2023xstest; @rottger2025msts; @russinovich2024crescendo; @russinovich2025cca; @russinovich2025price; @scheuerman2025transphobia; @shaikh2022second; @shayegani2025computeruse; @shen2023donotanything; @sheshadri2024lat; @souly2024strongreject; @stok2023ansi; @tan2026comicjailbreak; @tang2025multilingual; @tedeschi2024alert; @vantaylor2024socialbias; @vidgen2023simplesafetytests; @wang2023decodingtrust; @wang2023donotanswer; @wang2025siuo; @wang2026visualleakbench; @wei2023jailbroken; @xie2024sorrybench; @yu2023gptfuzzer; @yuan2023cipherchat; @zeng2024persuasion; @zhang2024cbtbench; @ziems2022mic; @zong2024vlguard; @zou2023gcg]
[@aakanksha2024multilingual; @adversaai2023universal; @andriushchenko2024tense; @anthropic2024manyshot; @aqrawi2024singleturncrescendo; @atr2026; @bethany2024mathprompt; @bhardwaj2023harmfulqa; @bhardwaj2024homer; @boucher2023trojan; @brahman2024coconot; @bryan2025agentictaxonomy; @bullwinkel2025airtlessons; @bullwinkel2025repeng; @bullwinkel2026trigger; @chao2023pair; @chao2024jailbreakbench; @choi2026xlsafetybench; @cui2024orbench; @darkbench2025; @deniz2026agenticboundarypairs; @derczynski2024garak; @ding2023wolf; @embracethered2024unicode; @embracethered2025sneakybits; @gehman2020realtoxicityprompts; @ghosh2025aegis; @ghosh2025ailuminate; @gong2025figstep; @gupta2024walledeval; @haider2024phi3safety; @han2024medsafetybench; @han2024wildguard; @hiddenlayer2025policypuppetry; @hines2024spotlighting; @inie2025summon; @ji2023beavertails; @ji2024pkusaferlhf; @jiang2025sosbench; @jones2025computeruse; @kingma2014adam; @li2024drattack; @li2024mossbench; @li2024saladbench; @li2024wmdp; @lin2023toxicchat; @liu2024flipattack; @liu2024mmsafetybench; @lopez2024pyrit; @luo2024jailbreakv; @lv2024codechameleon; @mazeika2023tdc; @mazeika2024harmbench; @mckee2024transparency; @mehrotra2023tap; @microsoft2024skeletonkey; @odin2024; @palaskar2025vlsu; @pfohl2024equitymedqa; @promptfoo2025ccp; @robustintelligence2024bypass; @roccia2024promptintel; @rottger2023xstest; @rottger2025msts; @russinovich2024crescendo; @russinovich2025cca; @russinovich2025price; @scheuerman2025transphobia; @shaikh2022second; @shayegani2025computeruse; @shen2023donotanything; @sheshadri2024lat; @souly2024strongreject; @stok2023ansi; @tan2026comicjailbreak; @tang2025multilingual; @tedeschi2024alert; @vantaylor2024socialbias; @vidgen2023simplesafetytests; @wang2023decodingtrust; @wang2023donotanswer; @wang2025siuo; @wang2026visualleakbench; @wei2023jailbroken; @xie2024sorrybench; @yu2023gptfuzzer; @yuan2023cipherchat; @zeng2024persuasion; @zhang2024cbtbench; @ziems2022mic; @zong2024vlguard; @zou2023gcg]

:::
40 changes: 35 additions & 5 deletions doc/code/datasets/1_loading_datasets.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
"0DIN [@odin2024],\n",
"Aegis [@ghosh2025aegis],\n",
"Agent Threat Rules [@atr2026],\n",
"Agentic Prompt-Injection Boundary Pairs [@deniz2026agenticboundarypairs],\n",
"ALERT [@tedeschi2024alert],\n",
"BeaverTails [@ji2023beavertails],\n",
"CBT-Bench [@zhang2024cbtbench],\n",
Expand Down Expand Up @@ -68,10 +69,38 @@
"(`garak_audio_achilles_heel`)."
]
},
{
"cell_type": "markdown",
"id": "1",
"metadata": {},
"source": [
"## Loading Agentic Prompt-Injection Boundary Pairs\n",
"\n",
"The `agentic_prompt_injection_boundary_pairs` loader uses the published v1.0.0\n",
"snapshot and defaults to attack prompts for red-team use. It can also return\n",
"the paired legitimate requests,\n",
"preserving each record's pair ID, scenario, attack family, source context,\n",
"target boundary, and expected action. For example:\n",
"\n",
"```python\n",
"from pyrit.datasets.seed_datasets.remote import (\n",
" AgenticPromptInjectionBoundaryLabel,\n",
" AgenticPromptInjectionBoundarySplit,\n",
" _AgenticPromptInjectionBoundaryDataset,\n",
")\n",
"\n",
"paired_test_loader = _AgenticPromptInjectionBoundaryDataset(\n",
" label=AgenticPromptInjectionBoundaryLabel.ALL,\n",
" split=AgenticPromptInjectionBoundarySplit.TEST,\n",
")\n",
"paired_test_surface = await paired_test_loader.fetch_dataset_async()\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1",
"id": "2",
"metadata": {},
"outputs": [
{
Expand All @@ -87,6 +116,7 @@
" 'adv_bench',\n",
" 'aegis_content_safety',\n",
" 'agent_threat_rules',\n",
" 'agentic_prompt_injection_boundary_pairs',\n",
" 'airt_fairness',\n",
" 'airt_fairness_yes_no',\n",
" 'airt_harassment',\n",
Expand Down Expand Up @@ -194,7 +224,7 @@
},
{
"cell_type": "markdown",
"id": "2",
"id": "3",
"metadata": {},
"source": [
"## Loading Specific Datasets\n",
Expand All @@ -205,7 +235,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "3",
"id": "4",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -242,7 +272,7 @@
},
{
"cell_type": "markdown",
"id": "4",
"id": "5",
"metadata": {},
"source": [
"## Adding Datasets to Memory\n",
Expand All @@ -259,7 +289,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "5",
"id": "6",
"metadata": {},
"outputs": [
{
Expand Down
24 changes: 24 additions & 0 deletions doc/code/datasets/1_loading_datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
# 0DIN [@odin2024],
# Aegis [@ghosh2025aegis],
# Agent Threat Rules [@atr2026],
# Agentic Prompt-Injection Boundary Pairs [@deniz2026agenticboundarypairs],
# ALERT [@tedeschi2024alert],
# BeaverTails [@ji2023beavertails],
# CBT-Bench [@zhang2024cbtbench],
Expand Down Expand Up @@ -71,6 +72,29 @@
# `garak_tm_system_prompts`), and an audio jailbreak set
# (`garak_audio_achilles_heel`).

# %% [markdown]
# ## Loading Agentic Prompt-Injection Boundary Pairs
#
# The `agentic_prompt_injection_boundary_pairs` loader uses the published v1.0.0
# snapshot and defaults to attack prompts for red-team use. It can also return
# the paired legitimate requests,
# preserving each record's pair ID, scenario, attack family, source context,
# target boundary, and expected action. For example:
#
# ```python
# from pyrit.datasets.seed_datasets.remote import (
# AgenticPromptInjectionBoundaryLabel,
# AgenticPromptInjectionBoundarySplit,
# _AgenticPromptInjectionBoundaryDataset,
# )
#
# paired_test_loader = _AgenticPromptInjectionBoundaryDataset(
# label=AgenticPromptInjectionBoundaryLabel.ALL,
# split=AgenticPromptInjectionBoundarySplit.TEST,
# )
# paired_test_surface = await paired_test_loader.fetch_dataset_async()
# ```

# %%
from pyrit.datasets import SeedDatasetProvider
from pyrit.memory import CentralMemory
Expand Down
9 changes: 9 additions & 0 deletions doc/references.bib
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,15 @@ @article{tedeschi2024alert
url = {https://arxiv.org/abs/2404.08676},
}

@misc{deniz2026agenticboundarypairs,
title = {Agentic Prompt-Injection Boundary Pairs},
author = {Enes Deniz},
year = {2026},
version = {1.0.0},
url = {https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs},
note = {CC BY 4.0 dataset. Source repository: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs},
}

@article{derczynski2024garak,
title = {garak: A Framework for Security Probing Large Language Models},
author = {Leon Derczynski and Erick Galinkin and Jeffrey Martin and Subho Majumdar and Nanna Inie},
Expand Down
12 changes: 12 additions & 0 deletions pyrit/datasets/seed_datasets/remote/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@
ATRVariationType,
_AgentThreatRulesDataset,
)
from pyrit.datasets.seed_datasets.remote.agentic_prompt_injection_boundary_dataset import (
AgenticPromptInjectionBoundaryAttackFamily,
AgenticPromptInjectionBoundaryLabel,
AgenticPromptInjectionBoundarySourceContext,
AgenticPromptInjectionBoundarySplit,
_AgenticPromptInjectionBoundaryDataset,
)
from pyrit.datasets.seed_datasets.remote.aya_redteaming_dataset import _AyaRedteamingDataset
from pyrit.datasets.seed_datasets.remote.babelscape_alert_dataset import _BabelscapeAlertDataset
from pyrit.datasets.seed_datasets.remote.beaver_tails_dataset import _BeaverTailsDataset
Expand Down Expand Up @@ -139,6 +146,10 @@

__all__ = [
"AegisHarmCategory",
"AgenticPromptInjectionBoundaryAttackFamily",
"AgenticPromptInjectionBoundaryLabel",
"AgenticPromptInjectionBoundarySourceContext",
"AgenticPromptInjectionBoundarySplit",
"CoCoNotCategory",
"CoCoNotSplit",
"DecodingTrustToxicitySubset",
Expand Down Expand Up @@ -166,6 +177,7 @@
"ATRDetectionField",
"ATRVariationType",
"_AgentThreatRulesDataset",
"_AgenticPromptInjectionBoundaryDataset",
"_AyaRedteamingDataset",
"_BabelscapeAlertDataset",
"_BeaverTailsDataset",
Expand Down
Loading