docs(scanner): PR-16 benchmark prep — labeling sheets over 8 public repos#56
Merged
Merged
Conversation
Scanned 8 permissively-licensed public GenAI repos (LangChain/LlamaIndex example apps + production agents + Semantic Kernel) at pinned SHAs with the deterministic CLI only, CVE stage enabled. Untrusted content — the CLI only greps. - benchmark/METHODOLOGY.md: corpus (repos + SHAs + licenses + counts), scanner 0.3.0 / ruleset 0.3.0, exact commands, and the labeling rubric (TP/FP per rule class incl. absence-based low-confidence handling). Flags high-volume rules to spot-check (P04.7 ~1253x on npm lockfile integrity hashes; P03.1 ~2613x on crewAI doc/test endpoints) and recommends stratified sampling for large repos. - benchmark/labeling-sheets/<repo>.csv: one row per public finding (rule_id, path, line, status, confidence, empty human_verdict + notes). - benchmark/missed-candidates.md: suspected false negatives with evidence — TypeScript/C#/notebook coverage gaps (patterns are Python-shaped). - Responsible disclosure: 8 value-bearing FAIL findings (potential secrets, values redacted) are held in benchmark/SENSITIVE-holdback.md, which is GITIGNORED and NOT committed — for maintainer review before any disclosure. No precision numbers computed; no rules modified; draft PR #44 untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ground-truth labeling prep for the scanner benchmark. Eight permissively-licensed public GenAI repos (LangChain/LlamaIndex example apps, production agents, Semantic Kernel) were scanned at pinned SHAs with the deterministic CLI only (CVE stage enabled), and the raw findings were laid out as human-labeling sheets. Docs/data only — no code, rules, or schema changes.
Contents
benchmark/METHODOLOGY.md— corpus (repos + SHAs + licenses + counts), scanner 0.3.0 / ruleset 0.3.0, exact commands, and the TP/FP labeling rubric. Flags high-volume rules to spot-check (P04.7 ~1253× on npm-lockfile integrity hashes; P03.1 ~2613× on crewAI doc/test endpoints) and recommends stratified sampling for the large repos.benchmark/labeling-sheets/<repo>.csv— one row per public finding:rule_id, path, line, status, confidence+ emptyhuman_verdict, notesfor the labeler.benchmark/missed-candidates.md— suspected false negatives with evidence (TypeScript / C# / notebook coverage gaps).Responsible disclosure
8 value-bearing FAIL findings (potential live secrets) are redacted and held in
benchmark/SENSITIVE-holdback.md, which is gitignored and NOT part of this PR — for maintainer review before any disclosure. Verified: the holdback file is not in the tree, and no secret-shaped values appear in any committed sheet.Deliberately out of scope
No precision/recall numbers computed yet (that's the labeling step, a maintainer task). No rules modified. Draft PR #44 untouched.
Note
Everything under
benchmark/is untrusted third-party content — it was only ever grepped, never executed.