From 3c088209a9bc521ae5391b42726e5fcc1db0143c Mon Sep 17 00:00:00 2001 From: emmanuelgjr Date: Sat, 18 Jul 2026 10:23:54 -0400 Subject: [PATCH 1/2] docs(scanner): benchmark methodology + announcement drafts (PR-16 hand-off) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prepares the agent-authored parts of PR-16; benchmark execution/labeling and the scanner-v0.4.0 release remain maintainer tasks (hard gate). - docs/BENCHMARK.md: corpus selection, deterministic run steps, a labeling-sheet template, a per-rule precision snippet, and the responsible-disclosure rule (never publish an unresolved live finding — follow SECURITY.md first). - docs/ANNOUNCEMENT-DRAFTS.md: Slack, mailing-list, call-for-scan-reports, lightning-talk abstract. - ROADMAP: Phases 1-3 complete (scanner-v0.3.0 shipped); Phase 4 gated items flagged. --- dsgai_scanner_tool/CHANGES_v0.3.md | 6 ++ dsgai_scanner_tool/ROADMAP.md | 30 ++++--- .../docs/ANNOUNCEMENT-DRAFTS.md | 57 ++++++++++++ dsgai_scanner_tool/docs/BENCHMARK.md | 88 +++++++++++++++++++ 4 files changed, 167 insertions(+), 14 deletions(-) create mode 100644 dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md create mode 100644 dsgai_scanner_tool/docs/BENCHMARK.md diff --git a/dsgai_scanner_tool/CHANGES_v0.3.md b/dsgai_scanner_tool/CHANGES_v0.3.md index 520f9d0..375d64d 100644 --- a/dsgai_scanner_tool/CHANGES_v0.3.md +++ b/dsgai_scanner_tool/CHANGES_v0.3.md @@ -9,6 +9,12 @@ dates are ISO-8601. The previous line is recorded in [`CHANGES_v0.2.md`](CHANGES ## [Unreleased] ### Added +- **Benchmark methodology + announcement drafts** (PR-16 hand-off). `docs/BENCHMARK.md` + (corpus selection, deterministic run steps, a labeling-sheet template, a per-rule + precision snippet, and the responsible-disclosure rule for live findings) and + `docs/ANNOUNCEMENT-DRAFTS.md` (Slack / mailing-list / call-for-reports / lightning-talk). + The benchmark **execution + labeling** and the `scanner-v0.4.0` release remain + maintainer tasks. - **Ecosystem expansion + rule-pack export** (PR-15). - **C# / Rust / Ruby**: detection signals (Semantic Kernel/Azure.AI.OpenAI, async-openai, ruby-openai); CVE manifest parsing for **NuGet** (`*.csproj`), **crates.io** diff --git a/dsgai_scanner_tool/ROADMAP.md b/dsgai_scanner_tool/ROADMAP.md index 1632360..40300b1 100644 --- a/dsgai_scanner_tool/ROADMAP.md +++ b/dsgai_scanner_tool/ROADMAP.md @@ -3,6 +3,8 @@ This is the public roadmap for the DSGAI scanner. It turns the internal improvement plan into work anyone can pick up. +> **Status (v0.3 shipped):** Phases 1–3 are complete and released as `scanner-v0.3.0`. Phase 4 is in progress — templating, ecosystems, and the Semgrep export have landed; the **license split** (needs OWASP leadership sign-off) and the **precision benchmark** (needs maintainer labeling) are the remaining gated items. + > **Want one of these? Comment on the tracking issue and claim it.** Each unstarted > item below links to a GitHub issue. Rule and precision work is labelled > `blocked-on-phase-1` until the test corpus and self-test CI land — you can draft it @@ -16,14 +18,14 @@ A compliance report that changes run-to-run on identical input is an opinion, no evidence. This phase makes pattern matching deterministic and testable, so everything after it is verifiable. -- ⬜ **Rules as data** — extract every detection pattern into `rules/dsgai-rules.yaml` +- ✅ **Rules as data** — extract every detection pattern into `rules/dsgai-rules.yaml` with a JSON Schema, so rules are reviewable data instead of prose. -- ⬜ **Fixture app + known-answer sheet** — a small, intentionally vulnerable +- ✅ **Fixture app + known-answer sheet** — a small, intentionally vulnerable multi-language GenAI app that is the test corpus, the demo, and the contributor on-ramp. -- ⬜ **Deterministic runner** — a stdlib Python CLI that runs the rules via ripgrep and +- ✅ **Deterministic runner** — a stdlib Python CLI that runs the rules via ripgrep and emits findings JSON + SARIF, identically every run. -- ⬜ **Checkpoint schema + self-test CI** — a formal, redaction-checkable output schema +- ✅ **Checkpoint schema + self-test CI** — a formal, redaction-checkable output schema and the CI gate that makes external rule PRs safe to merge. ## Phase 2 — Secure the pipeline itself @@ -31,34 +33,34 @@ after it is verifiable. A security scanner that is itself a prompt-injection vector or a secret-leak channel is a liability. This phase closes those. -- ⬜ **Skill rewrite** — the LLM becomes the orchestrator; the deterministic engine owns +- ✅ **Skill rewrite** — the LLM becomes the orchestrator; the deterministic engine owns pattern matching. Adds an untrusted-content trust preamble, structural secret redaction, stable file IDs, and honest language about what the report guarantees. -- ⬜ **Harden the GitHub Action** — split scanning (no secrets, runs on forks) from +- ✅ **Harden the GitHub Action** — split scanning (no secrets, runs on forks) from narration (restricted tools), pin actions by SHA, and fix the push-gate behaviour. -- ⬜ **README truth pass + lighter sample image** — every claim matches reality; the +- ✅ **README truth pass + lighter sample image** — every claim matches reality; the ~5 MB sample screenshot is replaced with a small one from the public fixture app. -- ⬜ **Pre-commit: gitleaks rule pack** — ship a battle-tested gitleaks pack, keep a +- ✅ **Pre-commit: gitleaks rule pack** — ship a battle-tested gitleaks pack, keep a portable no-dependency fallback script. ## Phase 3 — Precision -- ⬜ **Pattern precision wave 1** — fix confirmed false positives/negatives (unquoted +- ✅ **Pattern precision wave 1** — fix confirmed false positives/negatives (unquoted `.env` keys, the innocent-webhook SQL false positive), add per-rule confidence levels. -- ⬜ **CVE pipeline rework** — move CVE fetching into the CLI (no hallucinated CVEs), +- ✅ **CVE pipeline rework** — move CVE fetching into the CLI (no hallucinated CVEs), add caching, inline suppressions with reasons, a baseline for CI gating, and incremental `--diff` scans. ## Phase 4 — Professional polish -- ⬜ **License split + SPDX headers** — content stays CC BY-SA 4.0; executable code +- 🚧 **License split + SPDX headers** — content stays CC BY-SA 4.0; executable code moves to Apache-2.0 (pending OWASP leadership sign-off). -- ⬜ **Single-source variants + templated report + static ATLAS map** — generate the +- ✅ **Single-source variants + templated report + static ATLAS map** — generate the tool-neutral prompt variant from the skill, render reports deterministically from a template, and ship a static MITRE ATLAS technique map. -- ⬜ **Ecosystem expansion** — C#/NuGet, Rust, Ruby coverage, plus Semgrep and gitleaks +- ✅ **Ecosystem expansion** — C#/NuGet, Rust, Ruby coverage, plus Semgrep and gitleaks rule-pack exports so incumbent toolchains carry the DSGAI framework. -- ⬜ **Benchmark + published precision report** — run against public GenAI repos, +- 🚧 **Benchmark + published precision report** — run against public GenAI repos, hand-label, and publish per-rule precision. This is what separates "interesting project" from "tool AppSec teams adopt". diff --git a/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md b/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md new file mode 100644 index 0000000..852a858 --- /dev/null +++ b/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md @@ -0,0 +1,57 @@ +# DSGAI Scanner v0.3 — announcement drafts + +> **Hand-off: the agent drafts, the maintainer posts.** Review, adjust the links +> (fill the `<...>` placeholders), and post. All four items are optional but the +> "call for scan reports" doubles as contributor recruitment + the benchmark +> dataset. + +## 1. OWASP GenAI Slack post + +> 🛡️ **DSGAI Scanner v0.3 is out.** The OWASP DSGAI compliance scanner now has a +> **deterministic engine** — a stdlib Python CLI runs the 21-control ruleset via +> ripgrep and emits reproducible findings + SARIF (GitHub Code Scanning), while +> the Claude Code skill orchestrates and writes the report. Highlights: rules as +> data (107 patterns), a public vulnerable-fixture test corpus with CI, secrets +> that never leave ripgrep (structural redaction), a hardened two-job Action +> (safe on fork PRs), a gitleaks pre-commit pack, and a Semgrep export so your +> existing toolchain can carry the framework. **Found a wrong result? That's a +> contribution** — file an FP/FN issue and it becomes a permanent test case. +> Repo: · Release: + +## 2. Initiative mailing list + +> Subject: DSGAI Scanner v0.3 — deterministic engine, CI-gated, ecosystem support +> +> The DSGAI scanner reached v0.3. The headline change is architectural: pattern +> matching is now deterministic and reproducible (a compliance report that +> changes run-to-run is an opinion, not evidence), owned by a single-file +> stdlib CLI; the LLM's job is orchestration, judgment, and prose. We added a +> public test corpus + self-test CI (so external rule PRs are safe to merge), +> structural secret redaction, a hardened GitHub Action, a gitleaks pack, CVE +> enrichment with no hallucination risk, and C#/Rust/Ruby coverage plus a Semgrep +> export. Full changelog: . We'd love scan reports (below). + +## 3. Call for scan reports (recruits contributors + builds the benchmark) + +> **Run the DSGAI scanner on your GenAI repo and tell us what it got wrong.** +> `python cli/dsgai_scan.py scan .` ($0, no LLM) or the Claude Code skill. Every +> false-positive / false-negative you file with the +> [templates]() becomes a permanent, credited test case — +> and feeds the precision benchmark. Recurring contributors are invited as +> co-maintainers. No code required to help. + +## 4. Lightning-talk abstract (AppSec Global / regional CFP, ~10 min) + +> **Title:** A deterministic compliance scanner for GenAI apps — and why the LLM +> shouldn't do the matching +> +> **Abstract:** GenAI apps leak secrets, execute model-generated SQL, and skip +> tenant isolation in ways generic SAST misses. We built an OWASP DSGAI 2026 +> compliance scanner and learned the hard way that letting an LLM do the pattern +> matching produces reports that change run-to-run. v0.3 splits the concern: a +> tiny deterministic engine owns matching (reproducible, redaction-guaranteed by +> construction — secrets never leave ripgrep); the model orchestrates and writes +> prose, always citing evidence. Live demo: we scan an intentionally-vulnerable +> fixture app, show the SARIF land in Code Scanning, and watch a prompt-injection +> file in the repo have exactly zero effect on the result. 10 minutes, one scan, +> a lot of opinions about trusting AI with security evidence. diff --git a/dsgai_scanner_tool/docs/BENCHMARK.md b/dsgai_scanner_tool/docs/BENCHMARK.md new file mode 100644 index 0000000..5955c40 --- /dev/null +++ b/dsgai_scanner_tool/docs/BENCHMARK.md @@ -0,0 +1,88 @@ +# DSGAI Scanner — Precision Benchmark (methodology + labeling kit) + +> **Status: HAND-OFF.** The finding sheets are prepared by tooling; the +> **hand-labeling and the per-rule precision numbers are a maintainer task** +> (they require human judgment and a responsible-disclosure decision). This file +> is the methodology and the template — fill the `label:` and `notes:` columns, +> then compute precision with the snippet below. **This document is what +> separates "interesting project" from "tool AppSec teams adopt."** + +## Responsible-handling rule (read first) + +If the benchmark surfaces a **real, live vulnerability in a public repo**, do +**not** publish that finding here. Follow the monorepo's +[`SECURITY.md`](../../SECURITY.md) disclosure process first, and exclude the +finding from the public sheet until it is resolved. The benchmark measures the +*scanner's precision*, not a list of who is vulnerable. + +## Candidate corpus (record exact commit SHAs) + +Pick 5–10 public GenAI repos spanning frameworks and maturity. Suggested mix +(confirm each is still representative before running): + +- 2–3 popular **LangChain** example/starter apps. +- 1–2 **LlamaIndex** example apps. +- 1–2 production-grade OSS **agents** (e.g. an MCP server, a RAG service). +- 1 **non-Python** GenAI service (JS/Go/C#) to exercise the ecosystem work. + +For each: record `repo`, `commit_sha`, `scanned_at`, and the `ruleset_version` +(from `DSGAI-scan.json`) so the run is reproducible. + +## How to run (deterministic, $0) + +```bash +for repo in ; do + git -C "$repo" rev-parse HEAD # record the SHA + python cli/dsgai_scan.py scan "$repo" --no-cve \ + --json-out "bench/$(basename "$repo").json" --format none +done +``` + +Use `--no-cve` for the precision benchmark (CVE precision is measured separately +against OSV ground truth). Merge the `findings` arrays into the labeling sheet. + +## Labeling sheet template + +One row per finding. Fill `label` (TP / FP) and `notes` by inspecting the file. + +```yaml +# bench/labels.yaml +- repo: owner/name + commit: + rule_id: P02.1 + control: DSGAI02 + path: + line: + status: fail + label: TP # TP (true positive) | FP (false positive) + notes: "" # why; for value-bearing, DO NOT paste the secret +``` + +## Per-rule precision (run after labeling) + +```python +import yaml, collections +rows = yaml.safe_load(open("bench/labels.yaml")) +by_rule = collections.defaultdict(lambda: [0, 0]) # rule -> [TP, FP] +for r in rows: + by_rule[r["rule_id"]][0 if r["label"] == "TP" else 1] += 1 +print(f"{'rule':8} {'TP':>4} {'FP':>4} {'precision':>10}") +for rule, (tp, fp) in sorted(by_rule.items()): + prec = tp / (tp + fp) if (tp + fp) else float("nan") + print(f"{rule:8} {tp:>4} {fp:>4} {prec:>10.2f}") +``` + +## Feeding results back + +- For rules with low precision, **lower `confidence`** (or tighten the PCRE with a + fixture case) in `rules/dsgai-rules.yaml` and note the change here. +- Publish the aggregate table (per-rule TP/FP/precision) and the methodology. + **Never** publish an unresolved live finding (see the responsible-handling rule). + +## Results + +_To be filled after labeling._ + +| Rule | TP | FP | Precision | +|---|---|---|---| +| _pending_ | | | | From 22e56945602271194f7b93156788293733b08ae8 Mon Sep 17 00:00:00 2001 From: emmanuelgjr Date: Sat, 18 Jul 2026 10:26:04 -0400 Subject: [PATCH 2/2] fix(scanner): use a real path for the issue-templates link (PR-16) --- dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md b/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md index 852a858..828f478 100644 --- a/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md +++ b/dsgai_scanner_tool/docs/ANNOUNCEMENT-DRAFTS.md @@ -36,7 +36,7 @@ > **Run the DSGAI scanner on your GenAI repo and tell us what it got wrong.** > `python cli/dsgai_scan.py scan .` ($0, no LLM) or the Claude Code skill. Every > false-positive / false-negative you file with the -> [templates]() becomes a permanent, credited test case — +> [issue templates](../../.github/ISSUE_TEMPLATE/) becomes a permanent, credited test case — > and feeds the precision benchmark. Recurring contributors are invited as > co-maintainers. No code required to help.