[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader

[3nesdeniz]

Description

Closes #2171.

This PR adds a first-class PyRIT loader for the public Turkish Conversation Prompt-Injection Dataset (CC BY 4.0). The pinned v1.0.2 release contains 750 synthetic, curated Turkish inputs: 600 legitimate requests and 150 prompt-injection attempts across 10 attack families. It includes 150 benign boundary cases paired one-to-one with related attack inputs, supporting both attack-detection and false-positive / over-refusal evaluation.

The loader:

• defaults to attack examples from all published splits;
• provides typed filters for attack, benign, or all labels, split selection, and one or more attack families;
• pins immutable Hugging Face revision 29d7593984f563c4ad56876aa800b9ffd948a2fb;
• records the versioned DOI 10.5281/zenodo.21379389 and exact Hugging Face revision in row-level provenance;
• validates the fixed release contract before filtering: exact schema, ID and pair-ID formats and membership, split/label/category/family counts, normalized-text uniqueness, control characters, unique IDs, and complete same-context pairs;
• keeps attack families in metadata instead of conflating delivery techniques with PyRIT harm categories;
• keeps paired rows as independent evaluation cases while preserving pair_id;
• supports the standard Hugging Face token and cache behavior; and
• raises PyRIT's explicit empty-dataset error when valid filters match no rows.

The dataset is monolingual Turkish and entirely synthetic. It was curated by one author and does not claim independent annotation or inter-annotator agreement. Rows contain inputs only: label=1 denotes an attempted injection, not a verified bypass against any model or vendor. No model-response or attack-success benchmark is included.

The loader is registered with SeedDatasetProvider, documented in the dataset-loading guide, and linked to a versioned bibliography entry. Attack examples contain adversarial instructions and should only be used in controlled, authorized security testing.

Validation on the current head

• Focused loader tests: 37 passed
• Full unit suite after synchronizing with current main: 14,669 passed, 5 skipped
• Targeted pre-commit suite: all applicable hooks passed, including Ruff, type checking, notebook checks, documentation validation, and secret detection
• Jupytext .py / .ipynb pair: clean
• Live pinned v1.0.2 smoke test with cache=False: 750 total rows, 150 attacks, 600 benign rows, 10 families x 15, 150 complete pairs, 0 invalid pairs
• v1.0.1 and v1.0.2 row-content hashes are identical; the update aligns provenance without changing the dataset rows
• Exact Hugging Face source, DOI, and source repository links: HTTP 200
• PR diff against current main: git diff --check passed

The dataset notebook page builds successfully. The repository-wide strict docs command currently stops later on a pre-existing broken Wikipedia URL in doc/code/executor/8_modality_feedback.ipynb, which is unchanged by this PR and present on the base branch.

AI assistance disclosure

OpenAI Codex assisted with implementation drafting, synchronization, and test planning. I reviewed the resulting code and documentation, validated them against the live dataset, and ran every check listed above.

[3nesdeniz]

@microsoft-github-policy-service agree

[romanlutz]

Is there a paper associated with this? How were they collected or curated? Do they actually work on some model?

[3nesdeniz]

Thanks for the questions. There is no associated paper or preprint. The citation refers to the versioned dataset release itself.

The dataset is entirely synthetic and was curated by one author. Ten attack families were defined, with 15 attempted-injection inputs per family. Each attack input has a matched legitimate request that stays close in topic, context, and vocabulary but does not cross the assumed trust boundary. Those 150 boundary pairs were combined with 450 additional legitimate requests, producing 600 benign inputs and 150 attempted injections. The release does not claim independent annotators or an inter-annotator agreement score.

One correction to my earlier wording: this release was not validated as a successful-attack set against AltayDuel or any named model. Rows contain inputs only and label=1 means attempted injection, not a verified bypass. The intended use is input-classification, attack-family coverage, and false-positive / over-refusal evaluation. Model-response and attack-success benchmarking are outside this dataset release.

The repository includes deterministic pair-preserving splits and validation for exact schema and counts, duplicate normalized text, ID and pair integrity, split leakage, attack-family balance, and common sensitive-data patterns.

[romanlutz]

Understood. I'll examine it in more detail after our next release is out. That may take at least the next week. Thanks for your patience.

[3nesdeniz]

Hi @romanlutz — congratulations on the v1.0.0 release. I have now synchronized this draft with current main and tightened the contribution before asking you to revisit feature fit.

I also corrected my earlier reply above: the dataset contains input examples only; label=1 denotes an attempted injection, not a successful bypass against AltayDuel or any named model.

The current head now:

• pins the immutable v1.0.2 Hugging Face revision and versioned DOI;
• documents the synthetic, single-author curation and lack of independent annotation;
• validates the complete pinned release contract, including exact counts and identifiers, family balance, normalized-text uniqueness, and complete pairs;
• uses synthetic rather than multilingual; and
• passes 37 focused tests, 14,669 full unit tests (5 skipped), targeted pre-commit, Jupytext pairing, and a live 750-row cache=False smoke test.

The five fork-origin GitHub workflows currently show action_required with no jobs created, so they are waiting for maintainer approval rather than reporting test failures.

When you have a chance, could you revisit the feature-fit review and approve the workflow runs? If you would prefer a narrower loader surface or any other changes, I will address them. Thanks again.