FEAT Add a ShieldGemma scorer following the LlamaGuard scorer pattern

[immu4989]

Now that the LlamaGuard scorer (#1867) has landed as a dedicated TrueFalseScorer composing CallableResponseHandler, the same shape extends naturally to Google's ShieldGemma, the other widely used open-weight safety classifier. Opening this to propose the design and settle a couple of questions before writing code.

One important difference from LlamaGuard

ShieldGemma does not use LlamaGuard's safe / unsafe\n<codes> format. It is prompted with the content plus a single safety guideline and answers with Yes or No to "Does the content violate the above principle?". So the classifier decides one guideline per call rather than returning a multi-category verdict in one shot. The canonical text guidelines are a small set (dangerous content, harassment, hate speech, sexually explicit information), each a named principle with a description.

That makes the parser simpler than LlamaGuard's, but the policy model and prompt are different, so this is an adaptation of the pattern rather than a copy.

Proposed shape, mirroring the LlamaGuard files

• pyrit/score/true_false/shieldgemma_policy.py: a ShieldGemmaGuideline (name plus description) and a ShieldGemmaPolicy (a versioned set of guidelines), parallel to LlamaGuardPolicy.
• pyrit/score/true_false/shieldgemma_parser.py: parse_shieldgemma_response mapping a leading Yes to score_value="True" (violates) and No to "False", raising InvalidJsonException on anything else so the retry path applies.
• pyrit/score/true_false/shieldgemma_scorer.py: ShieldGemmaScorer(TrueFalseScorer) composing CallableResponseHandler(parse_shieldgemma_response) and going through _run_llm_scoring_async, exactly like LlamaGuardScorer.
• pyrit/datasets/score/shieldgemma/: a default policy YAML with the canonical guidelines plus the prompt template(s).
• Tests mirroring test_llamaguard_parser, test_llamaguard_policy, and test_llamaguard_scorer.

Open design questions

1. Guideline granularity. ShieldGemma is documented as most accurate with one guideline per call, which means N calls to cover N guidelines. The alternatives are a single scorer bound to one guideline (compose several for full coverage), or one scorer that runs the guidelines and aggregates. LlamaGuard did it all in one call; ShieldGemma cannot. Which default do you prefer?
2. Prompt-versus-response classification. ShieldGemma has distinct templates for classifying a user prompt versus a model response, similar in spirit to LlamaGuardMessageRole. Expose that as a mode on the scorer?
3. Version and modality. Bundle the Gemma-2 text policy first, and leave ShieldGemma-2 (image) as a follow-on, consistent with how the LlamaGuard PR scoped multimodal out?

I would validate the final scorer end to end against a hosted ShieldGemma endpoint the same way I did for LlamaGuard, and I can take ShieldGemma and then WildGuard as the natural continuation of the safety-classifier set. Happy to start once the guideline-granularity question in particular is settled.

[romanlutz]

I think this is a good idea! I need to find a cheap way we can validate this. Perhaps with HuggingFace paying per token rather than having a deployment that's up 24/7? Any suggestions are appreciated...

Answers to your questions:

1. I would keep one per scorer. You can always create multiple and compose them back together with the composite one if that's needed.
2. Sounds right. We typically run scorers just on the responses (unless you're scoring the whole conversation).
3. Probably a good call! I would think about whether that will change the constructor or API in any way. We don't like breaking changes for obvious reasons 🙂

Please go ahead! I appreciate the thoughtful writeup @immu4989

[immu4989]

Thanks, and good news on the validation cost question: ShieldGemma is available serverless, pay per token, with no deployment running 24/7.

HuggingFace Inference Providers routes google/shieldgemma-2b and google/shieldgemma-9b to featherless-ai. Both show status: live with task conversational, and neither is gated, so there is no license acceptance step. The endpoint is OpenAI compatible (https://router.huggingface.co/featherless-ai/v1) with an HF token, so it drops straight into an OpenAIChatTarget the same way the LlamaGuard validation did. HF also includes a small monthly inference allowance, so a handful of validation calls should cost approximately nothing.

For contrast, I checked the other obvious options: OpenRouter serves no ShieldGemma at all, and Together only offers dedicated endpoints for the models in this family. That is why the LlamaGuard validation ended up on Llama-Guard-4-12B, which they do serve serverless.

On your answers:

1. One guideline per scorer works well and matches ShieldGemma's own guidance that it is most accurate judging a single principle per call. Users needing full coverage compose several with TrueFalseCompositeScorer.
2. Will default to classifying the response, with the prompt side template available as a mode.
3. Agreed on text first. To keep image support additive rather than breaking, I will keep the policy and prompt template as constructor parameters and the validator injectable, so a future ShieldGemma 2 variant supplies a different template and a widened supported_data_types without changing the signature. Nothing in the constructor will assume text only.

Starting on it now.

[immu4989]

Correction to my earlier note on validation hosting, and a working alternative.

I had pointed at HuggingFace Inference Providers because the model metadata lists featherless-ai as a live provider for google/shieldgemma-9b and -2b. That turns out not to be routable in practice: the router rejects both with model_not_supported, and neither appears among the 126 models it actually serves, while other Gemma models do. A control call to a Llama model on the same token succeeded, so this was not a credentials or billing problem. The metadata is simply stale. Apologies for sending you down that path.

What does work, and answers your cheap-validation question better anyway, is running it locally. ShieldGemma 2B is small enough to serve on a laptop and is in the Ollama library, so there is no deployment to keep up and no per-token cost at all. Any OpenAI compatible local server drops into the scorer, so http://localhost:11434/v1 behaves exactly like a hosted endpoint.

I validated the scorer that way and included the transcript in the PR.