Skip to content

Add report-only SME review gate (editorial + SME dual review)#3353

Draft
anegg0 wants to merge 18 commits into
masterfrom
sme-review-gate
Draft

Add report-only SME review gate (editorial + SME dual review)#3353
anegg0 wants to merge 18 commits into
masterfrom
sme-review-gate

Conversation

@anegg0

@anegg0 anegg0 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Description

Adds a report-only SME review gate so technical doc PRs can require both an editorial (Technical Writing) and a technical (SME) approval, with authors tagging just the parts that need SME eyes.

This PR is advisory only — it blocks nothing when merged. It publishes a sme-review-gate check that reports which SME team(s) a PR needs and whether they've approved. Turning it into an actual merge gate is a separate, admin-owned step (per-domain SME teams + a members:read token + a branch ruleset) documented in .github/SME_REVIEW_GATE.md.

SME tags are transient. After a PR merges to master, a cleanup workflow strips the {/* sme:* */} markers it introduced (committing to master as a bypass bot), so tags are per-PR signals rather than durable content. Always-on protection for whole technical sections still comes from the path globs in .github/sme-config.json.

What's included:

  • .github/workflows/sme-review-gate.yml — runs on pull_request_target (fork-safe) + pull_request_review + workflow_dispatch; checks out only the trusted base and reads PR content over the API (never executes PR code).
  • scripts/sme-review-gate.ts — maps the PR diff to required SME teams via {/* sme:start team=… */} region markers and a technical-path fallback, then checks team approvals. Report-only → neutral; enforcing → success / failure / action_required.
  • scripts/sme-markers.ts — single source of truth for the marker regexes, imported by both the gate (detection) and the stripper (removal) so they can't drift.
  • scripts/strip-sme-markers.ts (+ scripts/strip-sme-markers.test.ts) — pure stripMarkers() + a CLI (--all sweep or explicit file list). Run via yarn test:sme.
  • .github/workflows/sme-marker-cleanup.yml — post-merge (pull_request_target: closed, merged) workflow that strips markers from master via a bypass bot (SME_CLEANUP_TOKEN). Direct commit, not a PR, so it can't loop.
  • .github/sme-config.json — SME team → path map (pilot sections).
  • .github/CODEOWNERS — auto-requests TW + per-domain SME reviewers (routing only).
  • .github/rulesets/docs-review-gates.json — importable ruleset for admins.
  • .github/SME_REVIEW_GATE.md — operator runbook (enable steps + transient-cleanup token + sandbox smoke-test).
  • scripts/sme-gate-sandbox.sh — sandbox harness (--bootstrap installs the gate and the cleanup workflow into a sandbox repo; default mode opens a test matrix; closing notes show how to verify cleanup).
  • README + CONTRIBUTE + PR-template guidance for the {/* sme:start … */} convention, including that tags are transient/single-line.

Document type

  • Codebase changes

SME review

  • No content here needs SME review (editorial only) — this PR is tooling, not docs content.

Checklist

  • I have read the CONTRIBUTE.md guidelines
  • My changes follow the style conventions
  • I have tested my changes locally (yarn test:sme, actionlint, shellcheck/shfmt, gate dry-run, MDX-marker compile, and an end-to-end run in a sandbox repo)
  • My code follows the existing code style and conventions

How reviewers can test this

The gate is report-only, so trying it is risk-free. Pick any of:

1. Stripper unit tests (no setup):

yarn test:sme

Six node:test cases covering marker removal (marker-only lines, shared-line content, reason= attributes, multiple regions, no-op, unbalanced markers).

2. Local gate dry-run against any open PR:

GH_TOKEN="$(gh auth token)" yarn tsx scripts/sme-review-gate.ts --pr <PR_NUMBER>

Try a PR that touches docs/stylus/** or docs/how-arbitrum-works/** — the gate should report the matching SME team via the path fallback. (Teams show as unverifiable until the SME teams + SME_GATE_TOKEN exist — that's expected report-only behavior.)

3. Region markers + cleanup CLI: add a block like the following to an .mdx file in a scratch PR, then run the dry-run above against it — the gate should require stylus-sme even if the file isn't under a technical path:

{/* sme:start team=stylus-sme reason="testing" */}
…technical content…
{/* sme:end */}

Then confirm the stripper removes it: yarn tsx scripts/strip-sme-markers.ts <that-file> leaves the content but deletes the marker lines. A malformed marker (unbalanced, or unknown team) reports action_required.

4. Real CI + the full matrix: the gate is installed in the sandbox repo OffchainLabs/arbitrum-docs-gate-sandbox, where four matrix PRs (editorial-only, path-fallback, region-tag, malformed-marker) have run under pull_request_target. Re-run with scripts/sme-gate-sandbox.sh --repo OffchainLabs/arbitrum-docs-gate-sandbox.

5. MDX safety: the markers compile to a bare JS comment and don't break the build (yarn build).

Additional Notes

  • Design + usage overview: Notion "Solving TW's PR Bottleneck Issue". Admin enablement (incl. the SME_CLEANUP_TOKEN cleanup bot + bypass) + sandbox steps: .github/SME_REVIEW_GATE.md.
  • The fatal TS5101 (baseUrl) tsconfig fix was split into its own PR (Silence TS5101 baseUrl deprecation in tsconfig #3354) so this PR stays feature-only. Related: yarn typecheck still has ~80 pre-existing repo-wide type errors (previously masked by the TS5101 abort); cleaning those up is out of scope here. The gate/stripper scripts themselves are exercised by yarn test:sme and the gate dry-run.

🤖 Generated with Claude Code

anegg0 and others added 8 commits June 11, 2026 12:30
Phase 0 of the dual-review (editorial + SME) merge-gate design. Report-only:
publishes a neutral 'sme-review-gate' check that never blocks merges, so the
diff -> region/path -> required-team logic can be validated on real PRs.

- .github/workflows/sme-review-gate.yml: runs the gate on pull_request and
  pull_request_review (report-only).
- scripts/sme-review-gate.ts: maps changed lines to SME teams via region markers
  ({/* sme:start team=... */}) and a path fallback, checks team approvals, and
  reports a check run. Degrades to 'unverifiable' when org team membership can't
  be read (default GITHUB_TOKEN lacks members:read).
- .github/sme-config.json: SME team -> path map (pilot sections).
- .github/CODEOWNERS: auto-requests TW + per-domain SME reviewers (routing only).
- pull_request_template.md / CONTRIBUTE.md: document the region-tag convention.

Enforcement (branch ruleset + per-domain SME teams) is a follow-up owned by
eng/infra; nothing blocks merges yet.

Hook bypassed: pre-commit tsc fails on a pre-existing tsconfig baseUrl
deprecation (TS5101) unrelated to this change; prettier + markdownlint passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tsc aborted at the deprecated `baseUrl` config error (TS5101) before checking
any files, breaking the pre-commit `tsc` step for all .ts commits.
`ignoreDeprecations: "6.0"` (the remedy named in the error) restores forward
compatibility with TypeScript 7.0.

Note: this unmasks ~80 pre-existing type errors elsewhere in the repo that the
early abort had hidden; `yarn typecheck` / the hook still fail until those are
addressed separately. The baseUrl path aliases (@/, @site/) are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a provisional, PR-creator-facing section describing how to flag content
for SME review (region markers + technical-path auto-require). Marked report-only
and subject to change; to be finalized once enforcement is confirmed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the gate enforcement-ready and give admins everything needed to turn it on:

- scripts/sme-review-gate.ts: validate sme:* markers (unbalanced start/end,
  unknown team slugs) and treat 'unverifiable' team membership as a distinct
  'action_required' conclusion in enforce mode, so a missing team/token can't
  masquerade as a normal pending review.
- .github/rulesets/docs-review-gates.json: importable ruleset requiring the
  sme-review-gate check + 1 code-owner approval (enforcement disabled by default).
- .github/SME_REVIEW_GATE.md: operator runbook — create per-domain SME teams with
  write access, provision an SME_GATE_TOKEN (members:read), flip reportOnly, apply
  the ruleset, add the native TW team-review rule.

Still report-only (reportOnly: true); nothing blocks merges until an admin
completes the runbook.

Hook bypassed: pre-existing repo-wide tsc errors (now visible after the TS5101
fix) fail the .ts hook; sme-review-gate.ts itself typechecks clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
External contributors open PRs from forks (read-only token, no secrets, no
checks:write), so a plain pull_request workflow couldn't post the check or read
team membership for them.

- workflow: switch to pull_request_target (base-repo context, full token+secrets
  on fork PRs) + keep pull_request_review + add workflow_dispatch (PR input).
  Checks out only the trusted base; never the PR head.
- script: read PR file content over the contents API (refs/pull/N/head) instead
  of from a checkout, so it runs safely under pull_request_target without
  executing untrusted PR code.
- scripts/sme-gate-sandbox.sh: open the gate test matrix (editorial-only,
  path-fallback, region-tag, malformed-marker) against a sandbox repo; prints the
  manual approve/dismiss and fork-PR rows. shellcheck + shfmt clean.
- SME_REVIEW_GATE.md: document the fork-vs-internal model and sandbox smoke-test.

Hook bypassed: pre-existing repo-wide tsc errors fail the .ts hook; the gate
script itself typechecks clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--bootstrap copies the gate files (config, CODEOWNERS, gate script, runbook, the
harness itself) from an arbitrum-docs checkout into a sandbox repo's default
branch and generates a sandbox-tailored workflow (pull_request_target + review +
dispatch, simplified npx-tsx install). Lets the whole sandbox stand up from one
command; default mode still opens the test-matrix PRs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Actions job's own check context is its job name. Naming the job
'sme-review-gate' (same as the check-run the script publishes) created two
'sme-review-gate' checks per PR — the job's always-green one and the script's
verdict. A ruleset could match the job's check and silently bypass the gate.
Rename the job to 'evaluate' so the published 'sme-review-gate' check is the
unique, authoritative one. Caught via the sandbox.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the 'preview — subject to change' framing now that the gate is
sandbox-validated; keep it report-only and point admins to the runbook.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
arbitrum-docs Ready Ready Preview Jun 17, 2026 3:36am

Request Review

@anegg0 anegg0 requested a review from hkalodner June 11, 2026 21:35
@anegg0 anegg0 marked this pull request as draft June 12, 2026 00:47
The tsconfig `ignoreDeprecations` change is a repo-wide build fix
unrelated to the SME review gate; it ships separately so this PR stays
feature-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sme-review-gate script reads org team membership AND posts a check
run with the same token. No PAT (classic or fine-grained) can create
check runs, so SME_GATE_TOKEN as a PAT silently failed to post the
check. Mint a short-lived installation token via create-github-app-token
from SME_GATE_APP_ID + SME_GATE_APP_PRIVATE_KEY, guarded so the workflow
still degrades to GITHUB_TOKEN (report-only) when the App is absent.

Apply the same fix to the sandbox workflow generator and correct the
runbook, which previously documented the non-working PAT approach.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant