FEAT: Add PuzzledConverter (word-puzzle jailbreak from arXiv:2508.01306)

[shashank03-dev]

Description

Adds PuzzledConverter, which implements the PUZZLED jailbreak from "PUZZLED: Jailbreaking LLMs through Word-Based Puzzles" (Ahn & Lee, arXiv:2508.01306).

Closes #2234. @romanlutz confirmed on the issue that a converter is the right shape for this.

The idea from the paper: instead of asking a model something harmful directly, you hide the sensitive words of the request inside a word puzzle and ask the model to solve the puzzle first, then follow the instruction it just rebuilt. The harmful wording never appears in plain text.

How the converter works:

• It picks which words to hide. It prefers a built-in list of harm-related words, then a supplementary list (both from the paper's Table 4), then nouns and verbs via spaCy, and falls back to word length when the spaCy model isn't installed.
• It encodes those words as a word search, an anagram, or a crossword. The output is reproducible because all randomness comes from a random.Random you can seed.
• Each hidden word gets a clue with its length and part of speech. If you pass a converter_target, it also asks that model for the paper's indirect "hint" clue, cached per word as the paper describes. If that call fails or returns junk, it falls back to the plain clue.

It uses the standard Converter interface, so it works with any target, scorer, or other converters. PuzzledConverter and PuzzleType are exported from pyrit.converter.

Fidelity to the paper

These parts are taken directly from the paper rather than approximated, and the code cites where each comes from:

• instruction length to masked word count (Table 3)
• the essential and recommended keyword lists (Table 4)
• word-search grid sizing and its bounded retry loop (Appendix A.3, Algorithm 7)
• the crossword's top-three shared-letter substitution
• per-word caching of generated clues

On spaCy

spaCy is an optional extra here, not a base dependency, and nothing in the repo downloads the en_core_web_sm model, so the length-based fallback is what most users will hit unless they run python -m spacy download en_core_web_sm. The converter works either way; the module docstring, the class docstring, and the notebook example all say so. This mirrors the existing optional-spaCy handling in pyrit/executor/benchmark/fairness_bias.py.

Tests and Documentation

72 unit tests in tests/unit/converter/ cover word selection and masking, all three puzzle builders, and the converter itself, including the optional LLM-clue path, its fallbacks, the per-word clue cache, and the error paths for unmaskable prompts and unplaceable words. They're deterministic and don't touch the network. The pyrit/converter/puzzled/ package is at 100% line coverage.

Docs: added a usage example to doc/code/converters/1_text_to_text_converters.py and the citation to doc/references.bib and doc/bibliography.md. The paired .ipynb is in sync with the .py (jupytext --to notebook reproduces its sources exactly) and carries the real output of the new example; the rest of that notebook's outputs are untouched, since other cells need live model endpoints.

[Copilot]

Pull request overview

This PR adds a new text-to-text converter, PuzzledConverter, implementing the PUZZLED word-puzzle jailbreak technique. It fits into PyRIT’s converter stack by masking selected sensitive words and re-encoding them as a solvable puzzle (word search / anagram / crossword), optionally augmenting deterministic clues with best-effort LLM-generated semantic hints.

Changes:

• Added PuzzledConverter plus supporting keyword-masking and puzzle-building utilities under pyrit/converter/puzzled/.
• Added a new converter seed prompt template (pyrit/datasets/converters/puzzled_converter.yaml) and exported the converter from pyrit.converter.
• Added unit tests and documentation updates, including new citations and a usage example.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit/converter/test_puzzled_puzzle_builders.py	Unit tests for anagram/crossword/word-search puzzle builders.
tests/unit/converter/test_puzzled_keyword_masker.py	Unit tests for keyword selection, masking behavior, and spaCy fallback/caching.
tests/unit/converter/test_puzzled_converter.py	Unit tests for converter construction, determinism, puzzle fallback behavior, and optional LLM-clue path.
pyrit/datasets/converters/puzzled_converter.yaml	New prompt template used to scaffold the puzzle-wrapped instruction.
pyrit/converter/puzzled/puzzled_converter.py	Core PuzzledConverter implementation, including optional semantic clue generation + caching.
pyrit/converter/puzzled/puzzle_builders.py	Deterministic puzzle-building utilities and PuzzleType enum.
pyrit/converter/puzzled/keyword_masker.py	Keyword ranking/masking logic with spaCy POS tagging fallback.
pyrit/converter/puzzled/init.py	Subpackage exports for the PUZZLED components.
pyrit/converter/init.py	Exposes PuzzledConverter via the main converter package API.
doc/references.bib	Adds BibTeX entry for the PUZZLED paper.
doc/code/converters/1_text_to_text_converters.py	Adds a short PuzzledConverter usage example in the docs notebook source.
doc/code/converters/1_text_to_text_converters.ipynb	Mirrors the docs example in the paired notebook.
doc/bibliography.md	Adds the PUZZLED citation key to the hidden citations list.

[shashank03-dev]

@romanlutz when you get a chance, could you look over the parts of this that come straight from the paper? I went back through the PDF and the supplementary appendix rather than working off the abstract, and the code now cites the exact table or algorithm next to each one:

• instruction length to masked word count (Table 3)
• the essential and recommended keyword lists (Table 4)
• the word search grid size and its bounded retry loop (Appendix A.3, Algorithm 7)
• the crossword picking the top three letters shared across words
• caching a clue against its word, so the same word only gets sent to the model once

Two of those I had to make a call on, and I'd rather hear what you think than guess.

The first is the word search directions. Algorithm 7 lists D ← {horizontal, vertical, diag_down, diag_up}. That's four labels, but I can't tell if each one means an axis, so words can run backwards too, or a single direction. I went with eight directions because that's how a normal word search works. If you read it as four, say so and I'll change it.

The second is what the package exports. I trimmed pyrit/converter/puzzled/__init__.py down to PuzzledConverter and PuzzleType so it lines up with token_smuggling. The masking and puzzle building helpers are still importable from their own modules, they're just not public API. Easy to widen if you'd rather have them exported.

Separately, worth knowing: spaCy is an optional extra and nothing in the repo downloads en_core_web_sm, so unless someone installs the model themselves they get the length based fallback for picking words. It works either way and the docstrings say so, but I'd rather flag it than leave it buried.