Add report-only SME review gate (editorial + SME dual review)#3353
Draft
anegg0 wants to merge 18 commits into
Draft
Add report-only SME review gate (editorial + SME dual review)#3353anegg0 wants to merge 18 commits into
anegg0 wants to merge 18 commits into
Conversation
Phase 0 of the dual-review (editorial + SME) merge-gate design. Report-only:
publishes a neutral 'sme-review-gate' check that never blocks merges, so the
diff -> region/path -> required-team logic can be validated on real PRs.
- .github/workflows/sme-review-gate.yml: runs the gate on pull_request and
pull_request_review (report-only).
- scripts/sme-review-gate.ts: maps changed lines to SME teams via region markers
({/* sme:start team=... */}) and a path fallback, checks team approvals, and
reports a check run. Degrades to 'unverifiable' when org team membership can't
be read (default GITHUB_TOKEN lacks members:read).
- .github/sme-config.json: SME team -> path map (pilot sections).
- .github/CODEOWNERS: auto-requests TW + per-domain SME reviewers (routing only).
- pull_request_template.md / CONTRIBUTE.md: document the region-tag convention.
Enforcement (branch ruleset + per-domain SME teams) is a follow-up owned by
eng/infra; nothing blocks merges yet.
Hook bypassed: pre-commit tsc fails on a pre-existing tsconfig baseUrl
deprecation (TS5101) unrelated to this change; prettier + markdownlint passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tsc aborted at the deprecated `baseUrl` config error (TS5101) before checking any files, breaking the pre-commit `tsc` step for all .ts commits. `ignoreDeprecations: "6.0"` (the remedy named in the error) restores forward compatibility with TypeScript 7.0. Note: this unmasks ~80 pre-existing type errors elsewhere in the repo that the early abort had hidden; `yarn typecheck` / the hook still fail until those are addressed separately. The baseUrl path aliases (@/, @site/) are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a provisional, PR-creator-facing section describing how to flag content for SME review (region markers + technical-path auto-require). Marked report-only and subject to change; to be finalized once enforcement is confirmed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the gate enforcement-ready and give admins everything needed to turn it on: - scripts/sme-review-gate.ts: validate sme:* markers (unbalanced start/end, unknown team slugs) and treat 'unverifiable' team membership as a distinct 'action_required' conclusion in enforce mode, so a missing team/token can't masquerade as a normal pending review. - .github/rulesets/docs-review-gates.json: importable ruleset requiring the sme-review-gate check + 1 code-owner approval (enforcement disabled by default). - .github/SME_REVIEW_GATE.md: operator runbook — create per-domain SME teams with write access, provision an SME_GATE_TOKEN (members:read), flip reportOnly, apply the ruleset, add the native TW team-review rule. Still report-only (reportOnly: true); nothing blocks merges until an admin completes the runbook. Hook bypassed: pre-existing repo-wide tsc errors (now visible after the TS5101 fix) fail the .ts hook; sme-review-gate.ts itself typechecks clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
External contributors open PRs from forks (read-only token, no secrets, no checks:write), so a plain pull_request workflow couldn't post the check or read team membership for them. - workflow: switch to pull_request_target (base-repo context, full token+secrets on fork PRs) + keep pull_request_review + add workflow_dispatch (PR input). Checks out only the trusted base; never the PR head. - script: read PR file content over the contents API (refs/pull/N/head) instead of from a checkout, so it runs safely under pull_request_target without executing untrusted PR code. - scripts/sme-gate-sandbox.sh: open the gate test matrix (editorial-only, path-fallback, region-tag, malformed-marker) against a sandbox repo; prints the manual approve/dismiss and fork-PR rows. shellcheck + shfmt clean. - SME_REVIEW_GATE.md: document the fork-vs-internal model and sandbox smoke-test. Hook bypassed: pre-existing repo-wide tsc errors fail the .ts hook; the gate script itself typechecks clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--bootstrap copies the gate files (config, CODEOWNERS, gate script, runbook, the harness itself) from an arbitrum-docs checkout into a sandbox repo's default branch and generates a sandbox-tailored workflow (pull_request_target + review + dispatch, simplified npx-tsx install). Lets the whole sandbox stand up from one command; default mode still opens the test-matrix PRs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Actions job's own check context is its job name. Naming the job 'sme-review-gate' (same as the check-run the script publishes) created two 'sme-review-gate' checks per PR — the job's always-green one and the script's verdict. A ruleset could match the job's check and silently bypass the gate. Rename the job to 'evaluate' so the published 'sme-review-gate' check is the unique, authoritative one. Caught via the sandbox. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the 'preview — subject to change' framing now that the gate is sandbox-validated; keep it report-only and point admins to the runbook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The tsconfig `ignoreDeprecations` change is a repo-wide build fix unrelated to the SME review gate; it ships separately so this PR stays feature-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4 tasks
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The sme-review-gate script reads org team membership AND posts a check run with the same token. No PAT (classic or fine-grained) can create check runs, so SME_GATE_TOKEN as a PAT silently failed to post the check. Mint a short-lived installation token via create-github-app-token from SME_GATE_APP_ID + SME_GATE_APP_PRIVATE_KEY, guarded so the workflow still degrades to GITHUB_TOKEN (report-only) when the App is absent. Apply the same fix to the sandbox workflow generator and correct the runbook, which previously documented the non-working PAT approach. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a report-only SME review gate so technical doc PRs can require both an editorial (Technical Writing) and a technical (SME) approval, with authors tagging just the parts that need SME eyes.
This PR is advisory only — it blocks nothing when merged. It publishes a
sme-review-gatecheck that reports which SME team(s) a PR needs and whether they've approved. Turning it into an actual merge gate is a separate, admin-owned step (per-domain SME teams + amembers:readtoken + a branch ruleset) documented in.github/SME_REVIEW_GATE.md.SME tags are transient. After a PR merges to
master, a cleanup workflow strips the{/* sme:* */}markers it introduced (committing tomasteras a bypass bot), so tags are per-PR signals rather than durable content. Always-on protection for whole technical sections still comes from the path globs in.github/sme-config.json.What's included:
.github/workflows/sme-review-gate.yml— runs onpull_request_target(fork-safe) +pull_request_review+workflow_dispatch; checks out only the trusted base and reads PR content over the API (never executes PR code).scripts/sme-review-gate.ts— maps the PR diff to required SME teams via{/* sme:start team=… */}region markers and a technical-path fallback, then checks team approvals. Report-only →neutral; enforcing →success/failure/action_required.scripts/sme-markers.ts— single source of truth for the marker regexes, imported by both the gate (detection) and the stripper (removal) so they can't drift.scripts/strip-sme-markers.ts(+scripts/strip-sme-markers.test.ts) — purestripMarkers()+ a CLI (--allsweep or explicit file list). Run viayarn test:sme..github/workflows/sme-marker-cleanup.yml— post-merge (pull_request_target: closed, merged) workflow that strips markers frommastervia a bypass bot (SME_CLEANUP_TOKEN). Direct commit, not a PR, so it can't loop..github/sme-config.json— SME team → path map (pilot sections)..github/CODEOWNERS— auto-requests TW + per-domain SME reviewers (routing only)..github/rulesets/docs-review-gates.json— importable ruleset for admins..github/SME_REVIEW_GATE.md— operator runbook (enable steps + transient-cleanup token + sandbox smoke-test).scripts/sme-gate-sandbox.sh— sandbox harness (--bootstrapinstalls the gate and the cleanup workflow into a sandbox repo; default mode opens a test matrix; closing notes show how to verify cleanup).{/* sme:start … */}convention, including that tags are transient/single-line.Document type
SME review
Checklist
yarn test:sme, actionlint, shellcheck/shfmt, gate dry-run, MDX-marker compile, and an end-to-end run in a sandbox repo)How reviewers can test this
The gate is report-only, so trying it is risk-free. Pick any of:
1. Stripper unit tests (no setup):
Six
node:testcases covering marker removal (marker-only lines, shared-line content,reason=attributes, multiple regions, no-op, unbalanced markers).2. Local gate dry-run against any open PR:
Try a PR that touches
docs/stylus/**ordocs/how-arbitrum-works/**— the gate should report the matching SME team via the path fallback. (Teams show asunverifiableuntil the SME teams +SME_GATE_TOKENexist — that's expected report-only behavior.)3. Region markers + cleanup CLI: add a block like the following to an
.mdxfile in a scratch PR, then run the dry-run above against it — the gate should requirestylus-smeeven if the file isn't under a technical path:Then confirm the stripper removes it:
yarn tsx scripts/strip-sme-markers.ts <that-file>leaves the content but deletes the marker lines. A malformed marker (unbalanced, or unknown team) reportsaction_required.4. Real CI + the full matrix: the gate is installed in the sandbox repo
OffchainLabs/arbitrum-docs-gate-sandbox, where four matrix PRs (editorial-only, path-fallback, region-tag, malformed-marker) have run underpull_request_target. Re-run withscripts/sme-gate-sandbox.sh --repo OffchainLabs/arbitrum-docs-gate-sandbox.5. MDX safety: the markers compile to a bare JS comment and don't break the build (
yarn build).Additional Notes
SME_CLEANUP_TOKENcleanup bot + bypass) + sandbox steps:.github/SME_REVIEW_GATE.md.TS5101(baseUrl) tsconfig fix was split into its own PR (Silence TS5101 baseUrl deprecation in tsconfig #3354) so this PR stays feature-only. Related:yarn typecheckstill has ~80 pre-existing repo-wide type errors (previously masked by theTS5101abort); cleaning those up is out of scope here. The gate/stripper scripts themselves are exercised byyarn test:smeand the gate dry-run.🤖 Generated with Claude Code