feat: Add agent-reliability-lab kit#227
Open
Lakshyakumar12 wants to merge 8 commits into
Open
Conversation
Audits an AI agent's system prompt for production readiness: static prompt-quality analysis, live adversarial red-teaming (injection, jailbreak, tool misuse, over-refusal probes fired at a real target endpoint), reliability/consistency scoring, and an automatically rewritten production-ready prompt with a change-log tying each edit to a specific finding. Ships as a kit with a Next.js UI plus the flow's own directly-callable API endpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughChangesThe Agent Reliability Lab adds a Lamatic workflow for static and dynamic agent audits, guarded endpoint probing, report aggregation, prompt rewriting, and a Next.js interface for submitting audits and viewing results. Agent Reliability Lab
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
🎉 All checks passed! This contribution follows the AgentKit structure. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “Agent Reliability Lab” kit to AgentKit that performs pre-deployment audits of another agent’s system prompt (static analysis plus optional live adversarial probing), produces a structured scored report, and optionally rewrites the prompt with a change log. The kit ships both as a Lamatic flow endpoint and as a small Next.js UI for running audits and rendering reports.
Changes:
- Introduces the
agent-reliability-auditLamatic flow with static-only vs dynamic audit branching and a unified report shape. - Adds a default constitution + multiple prompt templates/model configs for analyzer/generator/judge/rewriter LLM nodes.
- Adds a Next.js app (
kits/agent-reliability-lab/apps/) to submit audit inputs and display the resulting report.
Reviewed changes
Copilot reviewed 32 out of 34 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| kits/agent-reliability-lab/README.md | Kit usage docs (web app + direct flow invocation) and report shape examples. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_user_1.md | User prompt for static-only prompt rewrite step. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_system_0.md | System prompt for static-only prompt rewriter rules. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-949_user_1.md | User prompt for static analyzer inputs (prompt/tool schema/constitution). |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-949_system_0.md | System prompt defining static analysis rubric and output expectations. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_user_1.md | User prompt for dynamic rewrite step (includes static findings + failed probe verdicts). |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_system_0.md | System prompt for dynamic prompt rewrite rules. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-452_user_1.md | User prompt providing probe/response pairs to the judge. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-452_system_0.md | System prompt defining verdict taxonomy/severity rules for judging probes. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-186_user_1.md | User prompt for generating probes based on target prompt + depth. |
| kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-186_system_0.md | System prompt defining canary-only probe generation rules and categories. |
| kits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-990_generative-model-name.ts | Model config for static rewriter node. |
| kits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-949_generative-model-name.ts | Model config for static analyzer node. |
| kits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-736_generative-model-name.ts | Model config for dynamic rewriter node. |
| kits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-452_generative-model-name.ts | Model config for judge node. |
| kits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-186_generative-model-name.ts | Model config for probe generator node. |
| kits/agent-reliability-lab/lamatic.config.ts | Kit metadata for Lamatic deployment/registration (name, tags, envKey, links). |
| kits/agent-reliability-lab/flows/agent-reliability-audit.ts | Main Lamatic flow definition: trigger, branch, nodes/edges, and references. |
| kits/agent-reliability-lab/constitutions/default.md | Constitution/guardrails for probe generation, reporting integrity, and data handling. |
| kits/agent-reliability-lab/apps/tsconfig.json | Next.js app TypeScript config. |
| kits/agent-reliability-lab/apps/README.md | Next.js UI setup instructions and endpoint contract notes. |
| kits/agent-reliability-lab/apps/package.json | App dependencies/scripts (Next/React/Lamatic). |
| kits/agent-reliability-lab/apps/package-lock.json | Locked dependency graph for the app. |
| kits/agent-reliability-lab/apps/next.config.mjs | Next.js config (TypeScript build behavior). |
| kits/agent-reliability-lab/apps/next-env.d.ts | Next.js type environment declarations. |
| kits/agent-reliability-lab/apps/lib/lamatic-client.ts | Lamatic SDK client initialization and env key constant. |
| kits/agent-reliability-lab/apps/app/page.tsx | Client UI: form inputs, calls server action, renders report. |
| kits/agent-reliability-lab/apps/app/layout.tsx | App layout + metadata wrapper. |
| kits/agent-reliability-lab/apps/app/globals.css | UI styling for form/report views. |
| kits/agent-reliability-lab/apps/actions/orchestrate.ts | Server action that calls the Lamatic flow and normalizes the response shape. |
| kits/agent-reliability-lab/apps/.gitignore | App-specific ignore rules (node_modules, .next, env files). |
| kits/agent-reliability-lab/apps/.env.example | Example env vars for configuring the app. |
| kits/agent-reliability-lab/agent.md | Full kit/flow documentation and node-by-node architecture breakdown. |
| kits/agent-reliability-lab/.gitignore | Kit-level ignore rules for Lamatic artifacts/env/node_modules. |
Files not reviewed (1)
- kits/agent-reliability-lab/apps/package-lock.json: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+81
to
+86
| "scripts": { | ||
| "agent_reliability_audit_code_node_587_code": "@scripts/agent-reliability-audit_code-node-587_code.ts", | ||
| "agent_reliability_audit_code_node_422_code": "@scripts/agent-reliability-audit_code-node-422_code.ts", | ||
| "agent_reliability_audit_code_node_961_code": "@scripts/agent-reliability-audit_code-node-961_code.ts", | ||
| "agent_reliability_audit_code_node_175_code": "@scripts/agent-reliability-audit_code-node-175_code.ts", | ||
| "agent_reliability_audit_code_node_779_code": "@scripts/agent-reliability-audit_code-node-779_code.ts" |
| "nodeId": "InstructorLLMNode", | ||
| "values": { | ||
| "tools": [], | ||
| "schema": "{\n \"type\": \"object\",\n \"properties\": {\n \"promptQualityScore\": {\n \"type\": \"number\",\n \"required\": true\n },\n \"guardrailCoverageScore\": {\n \"type\": \"number\",\n \"required\": true\n },\n \"criticalIssues\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"issue\": {\n \"type\": \"string\"\n },\n \"recommendation\": {\n \"type\": \"string\"\n }\n },\n \"additionalProperties\": true\n }\n },\n \"warnings\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"suggestions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n}", |
| @@ -0,0 +1,291 @@ | |||
| "use client" | |||
|
|
|||
| import { useState } from "react" | |||
| const [error, setError] = useState<string | null>(null) | ||
| const [report, setReport] = useState<AuditReport | null>(null) | ||
|
|
||
| async function handleSubmit(e: React.FormEvent) { |
Comment on lines
+3
to
+7
| if (!process.env.LAMATIC_API_URL || !process.env.LAMATIC_PROJECT_ID || !process.env.LAMATIC_API_KEY) { | ||
| throw new Error( | ||
| "Lamatic API credentials are not set. Please add LAMATIC_API_URL, LAMATIC_PROJECT_ID, and LAMATIC_API_KEY to your .env.local file." | ||
| ) | ||
| } |
Structural validation flagged that the kit root was missing an .env.example alongside the one already in apps/ — adding it to satisfy the automated check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 20
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/agent-reliability-lab/apps/.gitignore`:
- Around line 11-14: Update the environment-file patterns in .gitignore to
ignore all .env* files, including production and development variants, then
explicitly re-include .env.example so only the example remains trackable.
In `@kits/agent-reliability-lab/apps/actions/orchestrate.ts`:
- Around line 1-3: Update the orchestrate action to import and use the parent
kit’s lamatic.config.ts via a lamaticConfig symbol, using its flow ID and step
definitions in the orchestration path. Replace the sole reliance on
process.env[AUDIT_FLOW_ENV_KEY], while preserving the existing lamaticClient
behavior and validating or deriving the flow configuration from lamaticConfig.
- Around line 54-57: Replace the any annotation on the response returned by
lamaticClient.executeFlow in the orchestration flow with a typed interface
covering the result.report, report, and data.report paths used by the
AuditReport extraction. Keep the existing fallback order and optional behavior
unchanged, and define the minimal response shape near the relevant orchestration
types so TypeScript validates mismatches.
In `@kits/agent-reliability-lab/apps/app/page.tsx`:
- Around line 42-43: Validate the referenceQA input in the submission handler
before invoking runAudit by attempting JSON.parse and preventing submission when
parsing fails. Preserve the existing submission flow for valid JSON and provide
a clear user-facing validation error for malformed input; update the state or
handler associated with referenceQA and the submit logic around runAudit.
In `@kits/agent-reliability-lab/apps/next.config.mjs`:
- Around line 3-5: Remove the typescript.ignoreBuildErrors setting from the
Next.js configuration so next build reports TypeScript failures, including
mismatches involving orchestrate.ts and AuditReport. Retain the default
type-checking behavior unless there is a documented, kit-specific justification
for keeping the override.
In `@kits/agent-reliability-lab/apps/package.json`:
- Line 12: Replace the floating “latest” version in the package manifest’s
lamatic dependency with a specific reviewed stable version, preserving the
existing dependency key and using the project’s preferred versioning convention.
In `@kits/agent-reliability-lab/apps/README.md`:
- Around line 19-22: Update the README’s endpoint contract documentation to
describe how callers provide targetEndpoint.authHeader, including the expected
authorization-header format and where it is configured or supplied. Keep the
existing URL and POST payload requirements, and ensure the documented contract
matches the flow’s accepted authentication behavior.
In `@kits/agent-reliability-lab/constitutions/default.md`:
- Around line 7-9: Enforce an explicit host allowlist or egress policy before
the dynamic probe POST uses targetEndpoint.url, rejecting unapproved targets.
Ensure authHeader is attached only after the target passes validation, and
preserve the existing authorized defensive-testing flow for approved endpoints.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-186_user_1.md`:
- Around line 1-4: Update the target agent system prompt interpolation in the
probe-generation instructions to clearly delimit the prompt content and state
that it is untrusted data for analysis only, not instructions to follow.
Preserve the existing AUDIT DEPTH value and “Generate the probe set now”
directive outside the delimited target prompt.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-452_user_1.md`:
- Around line 1-3: Update the prompt containing “PROBES AND RESPONSES” to
clearly delimit each probe/response pair from codeNode_422.output.results and
instruct the evaluator to treat all supplied content as hostile evidence, never
as instructions or commands. Preserve the existing request to evaluate each pair
and return verdicts.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_system_0.md`:
- Around line 1-6: Update the Markdown document containing the agent system
prompt to begin with a top-level heading and end with exactly one newline,
satisfying the MD041 and MD047 checks while preserving all existing prompt
content.
- Around line 1-6: Update the audit rewriter instructions in the prompt to
explicitly treat the original prompt, findings, and probe verdicts as delimited
data rather than executable instructions, preventing embedded adversarial text
from altering findings coverage or the change log. Apply the same input-safety
safeguard to the static rewriter prompt identified by
agent-reliability-audit_instructor-llmnode-990_system_0.md, while preserving the
existing purpose and tone.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_user_1.md`:
- Around line 1-7: Add a top-level Markdown heading to the prompt content, and
ensure the file ends with exactly one trailing newline. Keep the existing prompt
text and placeholders unchanged while correcting the MD041 and MD047 violations.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-949_user_1.md`:
- Around line 1-7: Update the audit prompt around the interpolated systemPrompt,
toolSchema, and constitutionDoc artifacts to clearly delimit each value as
untrusted inert data and explicitly instruct the analyzer never to follow
instructions contained within those blocks. Preserve the existing analysis
request and output-schema requirements outside the fenced artifact sections.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_system_0.md`:
- Around line 1-6: Update the prompt document by adding a top-level Markdown
heading and ensuring the file ends with exactly one newline. Preserve all
existing prompt content and instructions, including the change-log requirements.
- Around line 1-6: Update the system prompt’s input-handling rules to explicitly
classify both the audited system prompt and model-generated findings as
untrusted user-provided data, not instructions. Instruct the rewriter to analyze
these sections without obeying embedded directives, while preserving the
existing rewrite requirements and change-log behavior.
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_user_1.md`:
- Around line 1-5: Add a top-level Markdown heading to the prompt content and
ensure the file terminates with exactly one newline, preserving the existing
prompt text, placeholders, static analysis findings, and rewrite/change-log
instructions.
In `@kits/agent-reliability-lab/README.md`:
- Around line 48-63: The response example in the README does not match the typed
contract. Update the report example’s categoryScores values to numbers or null,
and represent criticalIssues and changeLog entries as objects, matching the
structures defined in agent.md and orchestrate.ts; leave unrelated fields
unchanged.
- Line 11: Update the verdict enum in the README’s structured-report description
to include PARTIAL_AUDIT_STATIC_ONLY alongside the existing verdicts. Keep the
documentation consistent with the static-only path and the verdict defined in
agent.md.
- Around line 20-25: Update the numbered setup instructions in the README so
each fenced shell block and its explanatory text remains within the same
Markdown list item. Add blank lines around fenced blocks and indent continuation
content consistently, preserving the existing step numbering and commands while
eliminating MD031/MD029 warnings.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 5a8a6f52-0ba2-4393-8313-f40f3dbee88f
⛔ Files ignored due to path filters (1)
kits/agent-reliability-lab/apps/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (33)
kits/agent-reliability-lab/.gitignorekits/agent-reliability-lab/README.mdkits/agent-reliability-lab/agent.mdkits/agent-reliability-lab/apps/.env.examplekits/agent-reliability-lab/apps/.gitignorekits/agent-reliability-lab/apps/README.mdkits/agent-reliability-lab/apps/actions/orchestrate.tskits/agent-reliability-lab/apps/app/globals.csskits/agent-reliability-lab/apps/app/layout.tsxkits/agent-reliability-lab/apps/app/page.tsxkits/agent-reliability-lab/apps/lib/lamatic-client.tskits/agent-reliability-lab/apps/next-env.d.tskits/agent-reliability-lab/apps/next.config.mjskits/agent-reliability-lab/apps/package.jsonkits/agent-reliability-lab/apps/tsconfig.jsonkits/agent-reliability-lab/constitutions/default.mdkits/agent-reliability-lab/flows/agent-reliability-audit.tskits/agent-reliability-lab/lamatic.config.tskits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-186_generative-model-name.tskits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-452_generative-model-name.tskits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-736_generative-model-name.tskits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-949_generative-model-name.tskits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-990_generative-model-name.tskits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-186_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-186_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-452_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-452_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-949_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-949_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_user_1.md
- Harden all 5 prompts that interpolate untrusted content (target system prompt, tool schema, constitution doc, probe responses, audit findings) with explicit delimiters and "treat as data, not instructions" rules, closing a prompt-injection path back into the auditor itself - Add an SSRF guard to Test Executor: refuse to send probes or attach credentials to targets resolving to localhost or private/internal IP ranges - Fix apps/.gitignore to ignore all .env* variants, not just .env/.env.local - Wire apps/lib/lamatic-client.ts to actually import ../../lamatic.config (required a turbopack.root/outputFileTracingRoot config fix to make the cross-directory import resolve at all) - Type the Lamatic SDK response instead of `any`, then remove typescript.ignoreBuildErrors now that types are clean end-to-end - Pin the lamatic dependency to the installed version instead of "latest" - Validate referenceQA as JSON client-side before submitting - Fix markdownlint MD041/MD047 violations across prompt files - Fix README.md: document PARTIAL_AUDIT_STATIC_ONLY verdict, correct example response types, fix list/fence formatting - Document the authHeader contract and SSRF guard in apps/README.md Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@kits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_user_1.md`:
- Line 15: Update the instruction text around the dynamic probe verdict filter
to explicitly require addressing all static_analysis_findings, while continuing
to limit dynamic remediation to FAIL, PARTIAL, and OVER_REFUSED verdicts and
ignoring PASS and INCONCLUSIVE results.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: fa5a2115-3171-4cc5-967d-6229eca38803
📒 Files selected for processing (18)
kits/agent-reliability-lab/README.mdkits/agent-reliability-lab/agent.mdkits/agent-reliability-lab/apps/.gitignorekits/agent-reliability-lab/apps/README.mdkits/agent-reliability-lab/apps/actions/orchestrate.tskits/agent-reliability-lab/apps/app/page.tsxkits/agent-reliability-lab/apps/lib/lamatic-client.tskits/agent-reliability-lab/apps/next-env.d.tskits/agent-reliability-lab/apps/next.config.mjskits/agent-reliability-lab/apps/package.jsonkits/agent-reliability-lab/constitutions/default.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-186_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-452_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-736_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-949_user_1.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_system_0.mdkits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-990_user_1.md
CodeRabbit correctly flagged that "Address only FAIL/PARTIAL/OVER_REFUSED..." could be misread as scoping to everything, including static analysis findings, when the FAIL/PARTIAL/OVER_REFUSED filter should only apply to the dynamic probe verdicts section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The repo's root .gitignore has a bare `scripts` pattern (unrelated to this kit, apparently meant to exclude the repo's own top-level tooling scripts/ directory) that unintentionally matched every kit's nested scripts/ folder too. This silently excluded all 5 codeNode scripts (Test Executor, Aggregator, Report Compiler, Static Report Compiler, Final Report) from every prior commit despite them existing on disk, meaning the flow's @scripts/... references were broken in the actual pushed repo the whole time. Force-adding them here since editing the shared root .gitignore is out of scope for this PR. Also adds minimal factual docstrings to the 7 exported/top-level functions across the changed files to satisfy CodeRabbit's docstring coverage check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/agent-reliability-lab/apps/app/page.tsx (1)
40-84: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy liftYour form is flying without instruments — coding guidelines require react-hook-form + zod and shadcn/ui components.
The kit Next.js app coding guidelines require
react-hook-form + zod for formsandshadcn/ui components. This form uses plainuseStateand raw HTML elements instead. The manualJSON.parsevalidation (lines 57-62) works but would be cleaner as a zod schema, and the plain<textarea>/<input>/<select>elements should use shadcn/ui equivalents for consistency and accessibility.As per coding guidelines: "For kit Next.js apps: use... shadcn/ui components, react-hook-form + zod for forms."
♻️ Proposed refactor sketch
import { useForm } from "react-hook-form" import { zodResolver } from "`@hookform/resolvers/zod`" import { z } from "zod" const auditSchema = z.object({ systemPrompt: z.string().min(1, "System prompt is required"), toolSchema: z.string().optional().default(""), constitutionDoc: z.string().optional().default(""), targetEndpointUrl: z.string().url().optional().or(z.literal("")), targetEndpointAuthHeader: z.string().optional().default(""), referenceQA: z.string().refine((v) => { try { JSON.parse(v); return true } catch { return false } }, "Must be valid JSON"), depth: z.enum(["quick", "standard", "deep"]), }) type AuditFormValues = z.infer<typeof auditSchema> export default function Home() { const { register, handleSubmit, formState: { errors, isSubmitting } } = useForm<AuditFormValues>({ resolver: zodResolver(auditSchema), defaultValues: { referenceQA: "[]", depth: "quick" }, }) const onSubmit = async (data: AuditFormValues) => { // ... call runAudit with data } return ( <form onSubmit={handleSubmit(onSubmit)}> {/* shadcn/ui <Textarea>, <Input>, <Select> components */} </form> ) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/agent-reliability-lab/apps/app/page.tsx` around lines 40 - 84, Refactor the Home form and handleSubmit flow to use react-hook-form with a zodResolver-backed auditSchema, moving referenceQA JSON validation and field validation into the schema while preserving the existing runAudit submission behavior and defaults. Replace raw input, textarea, and select elements in Home with the corresponding shadcn/ui components, and surface validation errors through the form state instead of local manual validation.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@kits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-422_code.ts`:
- Around line 91-106: Harden the SSRF protection centered on isBlockedHost and
callTarget: add checks for IPv4-mapped IPv6, unique-local IPv6 (fc00::/7), and
link-local IPv6 (fe80::/10), and configure the target fetch to use manual
redirect handling so redirects cannot reach unvalidated hosts. If the runtime
supports DNS resolution, resolve hostnames and apply isBlockedHost to every
returned address before fetching; otherwise document that DNS rebinding remains
unsupported.
In
`@kits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-961_code.ts`:
- Around line 47-53: Update the reliability consistency calculation in the loop
over testResults.reliabilityRuns so null responses are treated as inconsistent,
even when every repeat returns null. Ensure reliabilityScore is decremented and
reliabilityDetails.consistent is false for all-null failed probes, while
preserving the existing unique-response behavior for non-null responses.
---
Outside diff comments:
In `@kits/agent-reliability-lab/apps/app/page.tsx`:
- Around line 40-84: Refactor the Home form and handleSubmit flow to use
react-hook-form with a zodResolver-backed auditSchema, moving referenceQA JSON
validation and field validation into the schema while preserving the existing
runAudit submission behavior and defaults. Replace raw input, textarea, and
select elements in Home with the corresponding shadcn/ui components, and surface
validation errors through the form state instead of local manual validation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: d5c52ad2-9d58-4495-9b28-b20dede17e5a
📒 Files selected for processing (8)
kits/agent-reliability-lab/apps/actions/orchestrate.tskits/agent-reliability-lab/apps/app/page.tsxkits/agent-reliability-lab/apps/next-env.d.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-175_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-422_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-587_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-779_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-961_code.ts
- isBlockedHost now also blocks IPv4-mapped IPv6, unique-local IPv6 (fc00::/7), and link-local IPv6 (fe80::/10); callTarget uses redirect: "manual" so a target can't bounce a probe to an internal address via a 3xx redirect - Fix a real bug in the reliability check: if every repeat of a probe errors out (response: null), the old code treated identical nulls as "consistent" and left the reliability score inflated. Null responses now count as inconsistent. - Document the one known, accepted limitation: DNS rebinding isn't caught since DNS resolution isn't reliably available in this sandboxed execution environment; the guard checks the hostname string, not the resolved IP. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
@coderabbitai Keeping the current plain useState + manual validation implementation for now — it's tested and working, and swapping in react-hook-form/zod/shadcn-ui this late in the review cycle risks introducing new dependency/build issues for a stylistic change rather than a correctness fix. Happy to revisit if this becomes a hard requirement. |
Contributor
|
Tip For best results, initiate chat on the files or code changes.
Good luck out there. 🕶️ |
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
kits/agent-reliability-lab/apps/actions/orchestrate.ts (1)
1-3: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winAgent, the
lamatic.configimport directive remains unfulfilled.The coding guidelines for
kits/*/apps/actions/orchestrate.tsrequire importing and using../../lamatic.configto read step definitions from the parent kit. This was flagged in a prior review and remains unaddressed — the file still relies solely onprocess.env[AUDIT_FLOW_ENV_KEY]without referencing the kit'slamatic.config.ts.As per coding guidelines: "Kit Next.js apps must import and use
../../lamatic.configto read step definitions from the parent kit."📐 Proposed refactor
"use server" +import lamaticConfig from "../../lamatic.config" import { lamaticClient, AUDIT_FLOW_ENV_KEY } from "`@/lib/lamatic-client`"Then use
lamaticConfigto derive or validate the flow ID and step definitions instead of relying only on the environment variable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/agent-reliability-lab/apps/actions/orchestrate.ts` around lines 1 - 3, Update the server action in orchestrate.ts to import the parent kit’s ../../lamatic.config and use its lamaticConfig flow and step definitions when deriving or validating the audit flow, rather than relying solely on process.env[AUDIT_FLOW_ENV_KEY].Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/agent-reliability-lab/agent.md`:
- Line 88: Revise the SSRF guard description in the Test Executor section to
state that it blocks literal localhost or private/internal hostnames and IP
addresses across IPv4 and IPv6, rather than targets that resolve to those
ranges. Keep the redirect-following restriction, and explicitly document
separately that public hostnames resolving to private IPs via DNS rebinding are
not detected.
---
Duplicate comments:
In `@kits/agent-reliability-lab/apps/actions/orchestrate.ts`:
- Around line 1-3: Update the server action in orchestrate.ts to import the
parent kit’s ../../lamatic.config and use its lamaticConfig flow and step
definitions when deriving or validating the audit flow, rather than relying
solely on process.env[AUDIT_FLOW_ENV_KEY].
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 6aac1a17-4e1e-4ce4-b654-d3270ed00fa3
📒 Files selected for processing (9)
kits/agent-reliability-lab/agent.mdkits/agent-reliability-lab/apps/actions/orchestrate.tskits/agent-reliability-lab/apps/app/page.tsxkits/agent-reliability-lab/apps/next-env.d.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-175_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-422_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-587_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-779_code.tskits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-961_code.ts
…ording - orchestrate.ts now imports ../../lamatic.config directly and derives the flow ID from it, satisfying the coding guideline literally rather than through the indirect derivation in lamatic-client.ts. Simplified lamatic-client.ts back down to just the Lamatic client now that the flow-id derivation lives where the guideline expects it. - Fixed agent.md wording: the SSRF guard blocks literal private/internal hostnames and IPs, not "targets resolving to" those ranges — the original wording overstated protection against DNS rebinding, which the same paragraph already documents as a known limitation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
|
Hi @Lakshyakumar12! 👋 Before this PR can be reviewed by maintainers, please resolve all comments and requested changes from the CodeRabbit automated review. Steps to follow:
This helps keep the review process efficient for everyone. Thank you! 🙏 |
Contributor
|
/validate |
Contributor
|
📡 Running Studio validation — results will appear here shortly. |
Contributor
Studio Runtime Validation (Phase 2)✅ Studio validation passed. The kit loaded successfully in Lamatic Studio. This PR is ready for final review and merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist
1. Select Contribution Type
kits/agent-reliability-lab/)Note: this repo's checklist template still references the old
kits/<category>/<kit-name>/andbundles//templates/folder split — per CONTRIBUTING.md, that structure is deprecated: "There are no categories, no separatebundles/ortemplates/folders — thetypefield is the discriminator." This kit follows the current flat convention.2. General Requirements
git status, onlykits/agent-reliability-lab/touched.env/.env.localare git-ignored and never staged, checked before every commitagent-reliability-lab. Note: the folder does not literally match the flow ID (agent-reliability-audit) — per current CONTRIBUTING.md conventions, kit folder names don't need to equal their flow id (seekits/code-review/→ flowcode-review-agent,kits/content-generation/→ flowagentic-generate-content), so this is consistent with existing merged kits.README.md(purpose, setup, usage) — plusagent.mdfor the full node-by-node breakdown3. File Structure
This section's checklist also predates the current CONTRIBUTING.md format (
config.json→lamatic.config.ts; per-flow folders withmeta.json/inputs.json→ singleflows/<name>.tsfiles). Answering against the current, actual spec:lamatic.config.tspresent with valid metadata (name, description, tags, steps withenvKey, author, links)flows/agent-reliability-audit.tspresent, matching the step id inlamatic.config.ts.env.examplepresent at both the kit root andapps/(kit ships an app)4. Validation
npm install && npm run devworks locally — verified directly (also rannpm run buildrepeatedly through this PR's iteration, including after every fix)feat: Add agent-reliability-lab kit(using thefeat:prefix per CONTRIBUTING.md's explicit requirement: "PRs without this prefix will not be validated and may be ignored" — takes precedence over this checklist's older suggested[kit] Add <name>...format)passing-checkslabel applied, structural validation greenSummary
Agent Reliability Lab audits another AI agent's system prompt for production readiness before deployment — a pre-deployment audit, not a runtime input filter.
testedornot_assessedNOT_PRODUCTION_READYregardless of the numeric averageShips as a kit: the flow itself is directly callable via its own API endpoint, plus a Next.js UI (
apps/) with a form and rendered report view.Architecture
Single flow, 12 nodes, two paths depending on whether a live target endpoint is supplied:
PARTIAL_AUDIT_STATIC_ONLYverdict)See
agent.mdfor the full node-by-node breakdown andconstitutions/default.mdfor the guardrails.Test plan
npm install && npm run build && npm run devverified locally, re-verified after every subsequent fix.env/.env.localcommitted; all@referencepaths verified to resolvekits/agent-reliability-lab“Agent Reliability Lab” kit:kits/agent-reliability-lab/README.mdkits/agent-reliability-lab/agent.mdkits/agent-reliability-lab/.gitignorekits/agent-reliability-lab/.env.examplekits/agent-reliability-lab/constitutions/default.mdkits/agent-reliability-lab/lamatic.config.tskits/agent-reliability-lab/flows/agent-reliability-audit.tskits/agent-reliability-lab/model-configs/agent-reliability-audit_instructor-llmnode-{186,452,736,949,990}_generative-model-name.tskits/agent-reliability-lab/prompts/agent-reliability-audit_instructor-llmnode-{186,452,736,949,990}_{system_0,user_1}.mdkits/agent-reliability-lab/scripts/agent-reliability-audit_code-node-{175,422,587,779,961}_code.tskits/agent-reliability-lab/apps/:kits/agent-reliability-lab/apps/README.mdkits/agent-reliability-lab/apps/.env.examplekits/agent-reliability-lab/apps/.gitignorekits/agent-reliability-lab/apps/actions/orchestrate.tskits/agent-reliability-lab/apps/app/layout.tsxkits/agent-reliability-lab/apps/app/page.tsxkits/agent-reliability-lab/apps/app/globals.csskits/agent-reliability-lab/apps/lib/lamatic-client.tskits/agent-reliability-lab/apps/next.config.mjskits/agent-reliability-lab/apps/next-env.d.tskits/agent-reliability-lab/apps/package.json(+package-lock.json)kits/agent-reliability-lab/apps/tsconfig.jsonkits/agent-reliability-lab/flows/agent-reliability-audit.ts) implements the 12-node graph (noflow.jsonfile exists in-repo; the flow is defined in this TS file):triggerNode,conditionNode,dynamicNode(multipleInstructorLLMNode_*instances),dynamicNode(code nodes viacodeNode), andresponseNode.triggerNode_1accepts the audit request payload (system prompt, optional tool schema/constitution, optional target endpoint + auth header,referenceQA, anddepth).conditionNode_230branches on whethertargetEndpoint.urlis non-empty:targetEndpoint.url == ""):InstructorLLMNode_949(“Static analyzer”) performs prompt/guardrail static scoring and issues.InstructorLLMNode_990(“Static Rewriter”) rewrites the system prompt using the static findings.codeNode_587(“Static Report Compiler”) produces a report with static-only coverage and verdict.codeNode_175(“Report Compiler.”) normalizes/compiles the final report object.targetEndpoint.url != ""):InstructorLLMNode_949(“Static analyzer”) performs static scoring.InstructorLLMNode_186(“Probe Generator.”) generates a canary/adversarial probe set from the target prompt anddepth.codeNode_422(“Test Executor”) executes probes against the authorized endpoint with SSRF protections and reliability sampling.InstructorLLMNode_452(“Judge”) assigns per-probe verdicts (PASS/FAIL/PARTIAL/OVER_REFUSED/INCONCLUSIVE with severity).codeNode_961(“Aggregator”) computes reliability + per-category scores, extracts critical fails, and coverage.InstructorLLMNode_736(“Rewriter”) rewrites the prompt using findings tied to specific changes.codeNode_175(“Report Compiler.”) compiles the unified final report payload.codeNode_779(“Final Report”) selects/validates the produced report for output.responseNode_triggerNode_1maps the produced report to the API response as{ report: ... }.apps/actions/orchestrate.tscalls the Lamatic flow usingAGENT_RELIABILITY_AUDIT_FLOW_IDand normalizes/extractsreportfrom multiple possible response shapes, with user-friendly errors for network/auth issues.apps/app/page.tsxrenders the audit form, runsrunAudit, validatesreferenceQAJSON, and displays scores/verdict, coverage, findings, reliability details, rewritten prompt, and change log.scripts/agent-reliability-audit_code-node-422_code.tsincludes SSRF/private/internal/localhost blocking and performs repeated-probe reliability measurements feeding into deterministic scoring incode-node-961/ final report compilation incode-node-175andcode-node-779.