PR-16 (hand-off): Benchmark methodology + announcement drafts

dsgai_scanner_tool/CHANGES_v0.3.md

@@ -9,6 +9,12 @@ dates are ISO-8601. The previous line is recorded in [`CHANGES_v0.2.md`](CHANGES
 ## [Unreleased]
 
 ### Added
+- **Benchmark methodology + announcement drafts** (PR-16 hand-off). `docs/BENCHMARK.md`
+  (corpus selection, deterministic run steps, a labeling-sheet template, a per-rule
+  precision snippet, and the responsible-disclosure rule for live findings) and
+  `docs/ANNOUNCEMENT-DRAFTS.md` (Slack / mailing-list / call-for-reports / lightning-talk).
+  The benchmark **execution + labeling** and the `scanner-v0.4.0` release remain
+  maintainer tasks.
 - **Ecosystem expansion + rule-pack export** (PR-15).
   - **C# / Rust / Ruby**: detection signals (Semantic Kernel/Azure.AI.OpenAI, async-openai,
     ruby-openai); CVE manifest parsing for **NuGet** (`*.csproj`), **crates.io**

dsgai_scanner_tool/ROADMAP.md

@@ -3,6 +3,8 @@
 This is the public roadmap for the DSGAI scanner. It turns the internal improvement
 plan into work anyone can pick up.
 
+> **Status (v0.3 shipped):** Phases 1–3 are complete and released as `scanner-v0.3.0`. Phase 4 is in progress — templating, ecosystems, and the Semgrep export have landed; the **license split** (needs OWASP leadership sign-off) and the **precision benchmark** (needs maintainer labeling) are the remaining gated items.
+
 > **Want one of these? Comment on the tracking issue and claim it.** Each unstarted
 > item below links to a GitHub issue. Rule and precision work is labelled
 > `blocked-on-phase-1` until the test corpus and self-test CI land — you can draft it
@@ -16,49 +18,49 @@ A compliance report that changes run-to-run on identical input is an opinion, no
 evidence. This phase makes pattern matching deterministic and testable, so everything
 after it is verifiable.
 
-- ⬜ **Rules as data** — extract every detection pattern into `rules/dsgai-rules.yaml`
+- ✅ **Rules as data** — extract every detection pattern into `rules/dsgai-rules.yaml`
   with a JSON Schema, so rules are reviewable data instead of prose.
-- ⬜ **Fixture app + known-answer sheet** — a small, intentionally vulnerable
+- ✅ **Fixture app + known-answer sheet** — a small, intentionally vulnerable
   multi-language GenAI app that is the test corpus, the demo, and the contributor
   on-ramp.
-- ⬜ **Deterministic runner** — a stdlib Python CLI that runs the rules via ripgrep and
+- ✅ **Deterministic runner** — a stdlib Python CLI that runs the rules via ripgrep and
   emits findings JSON + SARIF, identically every run.
-- ⬜ **Checkpoint schema + self-test CI** — a formal, redaction-checkable output schema
+- ✅ **Checkpoint schema + self-test CI** — a formal, redaction-checkable output schema
   and the CI gate that makes external rule PRs safe to merge.
 
 ## Phase 2 — Secure the pipeline itself
 
 A security scanner that is itself a prompt-injection vector or a secret-leak channel is
 a liability. This phase closes those.
 
-- ⬜ **Skill rewrite** — the LLM becomes the orchestrator; the deterministic engine owns
+- ✅ **Skill rewrite** — the LLM becomes the orchestrator; the deterministic engine owns
   pattern matching. Adds an untrusted-content trust preamble, structural secret
   redaction, stable file IDs, and honest language about what the report guarantees.
-- ⬜ **Harden the GitHub Action** — split scanning (no secrets, runs on forks) from
+- ✅ **Harden the GitHub Action** — split scanning (no secrets, runs on forks) from
   narration (restricted tools), pin actions by SHA, and fix the push-gate behaviour.
-- ⬜ **README truth pass + lighter sample image** — every claim matches reality; the
+- ✅ **README truth pass + lighter sample image** — every claim matches reality; the
   ~5 MB sample screenshot is replaced with a small one from the public fixture app.
-- ⬜ **Pre-commit: gitleaks rule pack** — ship a battle-tested gitleaks pack, keep a
+- ✅ **Pre-commit: gitleaks rule pack** — ship a battle-tested gitleaks pack, keep a
   portable no-dependency fallback script.
 
 ## Phase 3 — Precision
 
-- ⬜ **Pattern precision wave 1** — fix confirmed false positives/negatives (unquoted
+- ✅ **Pattern precision wave 1** — fix confirmed false positives/negatives (unquoted
   `.env` keys, the innocent-webhook SQL false positive), add per-rule confidence levels.
-- ⬜ **CVE pipeline rework** — move CVE fetching into the CLI (no hallucinated CVEs),
+- ✅ **CVE pipeline rework** — move CVE fetching into the CLI (no hallucinated CVEs),
   add caching, inline suppressions with reasons, a baseline for CI gating, and
   incremental `--diff` scans.
 
 ## Phase 4 — Professional polish
 
-- ⬜ **License split + SPDX headers** — content stays CC BY-SA 4.0; executable code
+- 🚧 **License split + SPDX headers** — content stays CC BY-SA 4.0; executable code
   moves to Apache-2.0 (pending OWASP leadership sign-off).
-- ⬜ **Single-source variants + templated report + static ATLAS map** — generate the
+- ✅ **Single-source variants + templated report + static ATLAS map** — generate the
   tool-neutral prompt variant from the skill, render reports deterministically from a
   template, and ship a static MITRE ATLAS technique map.
-- ⬜ **Ecosystem expansion** — C#/NuGet, Rust, Ruby coverage, plus Semgrep and gitleaks
+- ✅ **Ecosystem expansion** — C#/NuGet, Rust, Ruby coverage, plus Semgrep and gitleaks
   rule-pack exports so incumbent toolchains carry the DSGAI framework.
-- ⬜ **Benchmark + published precision report** — run against public GenAI repos,
+- 🚧 **Benchmark + published precision report** — run against public GenAI repos,
   hand-label, and publish per-rule precision. This is what separates "interesting
   project" from "tool AppSec teams adopt".
 

dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md

@@ -0,0 +1,57 @@
+# DSGAI Scanner v0.3 — announcement drafts
+
+> **Hand-off: the agent drafts, the maintainer posts.** Review, adjust the links
+> (fill the `<...>` placeholders), and post. All four items are optional but the
+> "call for scan reports" doubles as contributor recruitment + the benchmark
+> dataset.
+
+## 1. OWASP GenAI Slack post
+
+> 🛡️ **DSGAI Scanner v0.3 is out.** The OWASP DSGAI compliance scanner now has a
+> **deterministic engine** — a stdlib Python CLI runs the 21-control ruleset via
+> ripgrep and emits reproducible findings + SARIF (GitHub Code Scanning), while
+> the Claude Code skill orchestrates and writes the report. Highlights: rules as
+> data (107 patterns), a public vulnerable-fixture test corpus with CI, secrets
+> that never leave ripgrep (structural redaction), a hardened two-job Action
+> (safe on fork PRs), a gitleaks pre-commit pack, and a Semgrep export so your
+> existing toolchain can carry the framework. **Found a wrong result? That's a
+> contribution** — file an FP/FN issue and it becomes a permanent test case.
+> Repo: <link to dsgai_scanner_tool/> · Release: <scanner-v0.3.0 link>
+
+## 2. Initiative mailing list
+
+> Subject: DSGAI Scanner v0.3 — deterministic engine, CI-gated, ecosystem support
+>
+> The DSGAI scanner reached v0.3. The headline change is architectural: pattern
+> matching is now deterministic and reproducible (a compliance report that
+> changes run-to-run is an opinion, not evidence), owned by a single-file
+> stdlib CLI; the LLM's job is orchestration, judgment, and prose. We added a
+> public test corpus + self-test CI (so external rule PRs are safe to merge),
+> structural secret redaction, a hardened GitHub Action, a gitleaks pack, CVE
+> enrichment with no hallucination risk, and C#/Rust/Ruby coverage plus a Semgrep
+> export. Full changelog: <CHANGES_v0.3 link>. We'd love scan reports (below).
+
+## 3. Call for scan reports (recruits contributors + builds the benchmark)
+
+> **Run the DSGAI scanner on your GenAI repo and tell us what it got wrong.**
+> `python cli/dsgai_scan.py scan .` ($0, no LLM) or the Claude Code skill. Every
+> false-positive / false-negative you file with the
+> [issue templates](../../.github/ISSUE_TEMPLATE/) becomes a permanent, credited test case —
+> and feeds the precision benchmark. Recurring contributors are invited as
+> co-maintainers. No code required to help.
+
+## 4. Lightning-talk abstract (AppSec Global / regional CFP, ~10 min)
+
+> **Title:** A deterministic compliance scanner for GenAI apps — and why the LLM
+> shouldn't do the matching
+>
+> **Abstract:** GenAI apps leak secrets, execute model-generated SQL, and skip
+> tenant isolation in ways generic SAST misses. We built an OWASP DSGAI 2026
+> compliance scanner and learned the hard way that letting an LLM do the pattern
+> matching produces reports that change run-to-run. v0.3 splits the concern: a
+> tiny deterministic engine owns matching (reproducible, redaction-guaranteed by
+> construction — secrets never leave ripgrep); the model orchestrates and writes
+> prose, always citing evidence. Live demo: we scan an intentionally-vulnerable
+> fixture app, show the SARIF land in Code Scanning, and watch a prompt-injection
+> file in the repo have exactly zero effect on the result. 10 minutes, one scan,
+> a lot of opinions about trusting AI with security evidence.

dsgai_scanner_tool/docs/BENCHMARK.md

@@ -0,0 +1,88 @@
+# DSGAI Scanner — Precision Benchmark (methodology + labeling kit)
+
+> **Status: HAND-OFF.** The finding sheets are prepared by tooling; the
+> **hand-labeling and the per-rule precision numbers are a maintainer task**
+> (they require human judgment and a responsible-disclosure decision). This file
+> is the methodology and the template — fill the `label:` and `notes:` columns,
+> then compute precision with the snippet below. **This document is what
+> separates "interesting project" from "tool AppSec teams adopt."**
+
+## Responsible-handling rule (read first)
+
+If the benchmark surfaces a **real, live vulnerability in a public repo**, do
+**not** publish that finding here. Follow the monorepo's
+[`SECURITY.md`](../../SECURITY.md) disclosure process first, and exclude the
+finding from the public sheet until it is resolved. The benchmark measures the
+*scanner's precision*, not a list of who is vulnerable.
+
+## Candidate corpus (record exact commit SHAs)
+
+Pick 5–10 public GenAI repos spanning frameworks and maturity. Suggested mix
+(confirm each is still representative before running):
+
+- 2–3 popular **LangChain** example/starter apps.
+- 1–2 **LlamaIndex** example apps.
+- 1–2 production-grade OSS **agents** (e.g. an MCP server, a RAG service).
+- 1 **non-Python** GenAI service (JS/Go/C#) to exercise the ecosystem work.
+
+For each: record `repo`, `commit_sha`, `scanned_at`, and the `ruleset_version`
+(from `DSGAI-scan.json`) so the run is reproducible.
+
+## How to run (deterministic, $0)
+
+```bash
+for repo in <list>; do
+  git -C "$repo" rev-parse HEAD                          # record the SHA
+  python cli/dsgai_scan.py scan "$repo" --no-cve \
+    --json-out "bench/$(basename "$repo").json" --format none
+done
+```
+
+Use `--no-cve` for the precision benchmark (CVE precision is measured separately
+against OSV ground truth). Merge the `findings` arrays into the labeling sheet.
+
+## Labeling sheet template
+
+One row per finding. Fill `label` (TP / FP) and `notes` by inspecting the file.
+
+```yaml
+# bench/labels.yaml
+- repo: owner/name
+  commit: <sha>
+  rule_id: P02.1
+  control: DSGAI02
+  path: <path>
+  line: <n>
+  status: fail
+  label: TP        # TP (true positive) | FP (false positive)
+  notes: ""        # why; for value-bearing, DO NOT paste the secret
+```
+
+## Per-rule precision (run after labeling)
+
+```python
+import yaml, collections
+rows = yaml.safe_load(open("bench/labels.yaml"))
+by_rule = collections.defaultdict(lambda: [0, 0])   # rule -> [TP, FP]
+for r in rows:
+    by_rule[r["rule_id"]][0 if r["label"] == "TP" else 1] += 1
+print(f"{'rule':8} {'TP':>4} {'FP':>4} {'precision':>10}")
+for rule, (tp, fp) in sorted(by_rule.items()):
+    prec = tp / (tp + fp) if (tp + fp) else float("nan")
+    print(f"{rule:8} {tp:>4} {fp:>4} {prec:>10.2f}")
+```
+
+## Feeding results back
+
+- For rules with low precision, **lower `confidence`** (or tighten the PCRE with a
+  fixture case) in `rules/dsgai-rules.yaml` and note the change here.
+- Publish the aggregate table (per-rule TP/FP/precision) and the methodology.
+  **Never** publish an unresolved live finding (see the responsible-handling rule).
+
+## Results
+
+_To be filled after labeling._
+
+| Rule | TP | FP | Precision |
+|---|---|---|---|
+| _pending_ | | | |