feat: add LLM Response QA for AI Response Evaluation#223
Conversation
Add initial README for LLM Response QA kit
Expanded the README to provide detailed information about the LLM Response QA kit, including its problem statement, solution, features, workflow, example output, use cases, tech stack, author, and future improvements.
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughChangesThe pull request adds an LLM Response QA kit with evaluation prompts, safety guidance, model configuration, Next.js application setup, documentation, Lamatic flow metadata, and automated flow synchronization. LLM Response QA
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Added detailed description, features, use cases, and future improvements for the LLM Response QA project.
Updated project metadata with a detailed description and added GitHub link.
Ramyatha24
changed the title
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
|
|
Failure recorded at 2026-07-11T20:08:08Z UTC. If this PR is not fixed within 4 weeks it will be automatically closed. |
Ramyatha24
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/llm-response-qa/agent.md`:
- Around line 1-3: Replace the placeholder content in agent.md with complete
LLM-generated documentation for the LLM-Response-QA agent. Include its identity
and purpose, capabilities, flow descriptions, guardrails, and integration
reference, using the kit README and flow configuration as the source of truth.
In `@kits/llm-response-qa/constitutions/default.md`:
- Line 3: Update the generator or source template that produces the
constitution’s headings, including the sections corresponding to “Identity” and
the referenced headings, so each heading has the required blank line before and
after it. Regenerate the checked-in constitution file from that source rather
than editing only the generated output.
In `@kits/llm-response-qa/flows/llm-response-qa.ts`:
- Around line 4-16: The flow meta object in LLM-Response-QA must match the
populated metadata from lamatic.config.ts. Update the description, tags,
githubUrl, documentationUrl, and deployUrl fields in the exported meta object,
preserving the existing name, testInput, and author values.
- Around line 122-140: Update the responseNode configuration for
responseNode_triggerNode_1 so its content exposes the complete structured QA
report from InstructorLLMNode_298.output, including quality scores,
hallucination risk, detected issues, suggestions, and feedback, rather than only
output.feedback. If chat intentionally remains feedback-only, document that
behavior and ensure the full structured output is consumed elsewhere.
In
`@kits/llm-response-qa/model-configs/llm-response-qa_instructor-llmnode-298_generative-model-name.ts`:
- Around line 9-12: The model configuration’s credentialId is tied to the source
workspace and must not be shipped as-is. Update the configuration around
model_name and provider_name to use the target workspace’s remapped credential
identifier during kit publishing or importing, ensuring the model node resolves
the credential correctly.
In
`@kits/llm-response-qa/prompts/llm-response-qa_instructor-llmnode-298_user_1.md`:
- Around line 1-2: Harden the prompt around the Chat Widget.chatMessage
interpolation by clearly delimiting the inserted response and explicitly
instructing the evaluator to treat it only as untrusted data for analysis, never
as instructions to follow or execute. Preserve the existing QA task and
JSON-only output requirement.
In `@kits/llm-response-qa/README.md`:
- Line 28: Add a language tag to the fenced workflow diagram code block in the
README, using an appropriate hint such as text or mermaid, while preserving the
diagram content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 63fa8fe2-6d17-4d9a-ad58-06d2a12e54f0
📒 Files selected for processing (9)
kits/llm-response-qa/.gitignorekits/llm-response-qa/README.mdkits/llm-response-qa/agent.mdkits/llm-response-qa/constitutions/default.mdkits/llm-response-qa/flows/llm-response-qa.tskits/llm-response-qa/lamatic.config.tskits/llm-response-qa/model-configs/llm-response-qa_instructor-llmnode-298_generative-model-name.tskits/llm-response-qa/prompts/llm-response-qa_instructor-llmnode-298_system_0.mdkits/llm-response-qa/prompts/llm-response-qa_instructor-llmnode-298_user_1.md
| @@ -0,0 +1,17 @@ | |||
| # Default Constitution | |||
|
|
|||
| ## Identity | |||
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Mission control: fix heading spacing at the template source.
Markdownlint reports missing blank lines around these headings. Because this constitution is templated/generated, correct the generator or source template so future kits inherit the fix—not only this checked-in file.
Also applies to: 6-6, 11-11, 15-15
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 3-3: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/llm-response-qa/constitutions/default.md` at line 3, Update the
generator or source template that produces the constitution’s headings,
including the sections corresponding to “Identity” and the referenced headings,
so each heading has the required blank line before and after it. Regenerate the
checked-in constitution file from that source rather than editing only the
generated output.
Sources: Learnings, Linters/SAST tools
| export const meta = { | ||
| "name": "LLM-Response-QA", | ||
| "description": "", | ||
| "tags": [], | ||
| "testInput": null, | ||
| "githubUrl": "", | ||
| "documentationUrl": "", | ||
| "deployUrl": "", | ||
| "author": { | ||
| "name": "Ramyatha Balaraju", | ||
| "email": "ramyathaaa@gmail.com" | ||
| } | ||
| }; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Flow meta fields are empty despite being populated in lamatic.config.ts.
The flow's meta object has empty description, tags, githubUrl, documentationUrl, and deployUrl, while lamatic.config.ts (lines 3-30) populates all of these. This inconsistency means the flow's self-description is incomplete when consumed independently.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/llm-response-qa/flows/llm-response-qa.ts` around lines 4 - 16, The flow
meta object in LLM-Response-QA must match the populated metadata from
lamatic.config.ts. Update the description, tags, githubUrl, documentationUrl,
and deployUrl fields in the exported meta object, preserving the existing name,
testInput, and author values.
| { | ||
| "id": "responseNode_triggerNode_1", | ||
| "type": "responseNode", | ||
| "position": { | ||
| "x": 0, | ||
| "y": 0 | ||
| }, | ||
| "data": { | ||
| "nodeId": "chatResponseNode", | ||
| "values": { | ||
| "id": "responseNode_triggerNode_1", | ||
| "content": "{{InstructorLLMNode_298.output.feedback}}", | ||
| "nodeName": "Chat Response", | ||
| "references": "", | ||
| "webhookUrl": "", | ||
| "webhookHeaders": "" | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Response node only outputs feedback — user never sees the structured QA report.
The PR objective states the workflow "produces a structured JSON quality report covering overall quality, accuracy, completeness, relevance, hallucination risk, detected issues, actionable suggestions, and constructive feedback." However, the response node content is {{InstructorLLMNode_298.output.feedback}} — only the summary feedback string. The user never receives the scores, hallucination risk level, issues, or suggestions that the evaluator generates.
If the intent is to surface the full report to the end user, the response content should include the complete structured output. If only feedback is intentionally shown in chat (with the full JSON consumed programmatically elsewhere), this should be documented.
💡 Proposed fix to surface the full QA report
"values": {
"id": "responseNode_triggerNode_1",
- "content": "{{InstructorLLMNode_298.output.feedback}}",
+ "content": "{{InstructorLLMNode_298.output}}",
"nodeName": "Chat Response",Alternatively, if only the feedback text should be shown in chat but the full report should be available, consider formatting the response to include all fields:
- "content": "{{InstructorLLMNode_298.output.feedback}}",
+ "content": "Overall Score: {{InstructorLLMNode_298.output.overall_score}}/100\\nAccuracy: {{InstructorLLMNode_298.output.accuracy}}\\nCompleteness: {{InstructorLLMNode_298.output.completeness}}\\nRelevance: {{InstructorLLMNode_298.output.relevance}}\\nHallucination Risk: {{InstructorLLMNode_298.output.hallucination}}\\n\\nIssues:\\n{{InstructorLLMNode_298.output.issues}}\\n\\nSuggestions:\\n{{InstructorLLMNode_298.output.suggestions}}\\n\\nFeedback:\\n{{InstructorLLMNode_298.output.feedback}}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| { | |
| "id": "responseNode_triggerNode_1", | |
| "type": "responseNode", | |
| "position": { | |
| "x": 0, | |
| "y": 0 | |
| }, | |
| "data": { | |
| "nodeId": "chatResponseNode", | |
| "values": { | |
| "id": "responseNode_triggerNode_1", | |
| "content": "{{InstructorLLMNode_298.output.feedback}}", | |
| "nodeName": "Chat Response", | |
| "references": "", | |
| "webhookUrl": "", | |
| "webhookHeaders": "" | |
| } | |
| } | |
| } | |
| { | |
| "id": "responseNode_triggerNode_1", | |
| "type": "responseNode", | |
| "position": { | |
| "x": 0, | |
| "y": 0 | |
| }, | |
| "data": { | |
| "nodeId": "chatResponseNode", | |
| "values": { | |
| "id": "responseNode_triggerNode_1", | |
| "content": "{{InstructorLLMNode_298.output}}", | |
| "nodeName": "Chat Response", | |
| "references": "", | |
| "webhookUrl": "", | |
| "webhookHeaders": "" | |
| } | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/llm-response-qa/flows/llm-response-qa.ts` around lines 122 - 140, Update
the responseNode configuration for responseNode_triggerNode_1 so its content
exposes the complete structured QA report from InstructorLLMNode_298.output,
including quality scores, hallucination risk, detected issues, suggestions, and
feedback, rather than only output.feedback. If chat intentionally remains
feedback-only, document that behavior and ensure the full structured output is
consumed elsewhere.
| "model_name": "openrouter/openai/gpt-3.5-turbo-16k", | ||
| "credentialId": "55250665-2e9a-48d1-8d62-cf4c61dfbe99", | ||
| "provider_name": "openrouter", | ||
| "credential_name": "openrouter" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file and nearby kit/model-config patterns
git ls-files 'kits/*/model-configs/*.ts' | sed -n '1,120p'
echo '---'
cat -n kits/llm-response-qa/model-configs/llm-response-qa_instructor-llmnode-298_generative-model-name.ts | sed -n '1,120p'
echo '--- SEARCH: credential remapping / import export / credentialId'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'credentialId|credential_name|credential mapping|remap|import/export|workspace-specific credential|credential.*UUID|credential.*mapping' \
README* docs kits . 2>/dev/null | sed -n '1,240p'
echo '--- SEARCH: similar generated config files'
rg -n --hidden --glob 'kits/*/model-configs/*.ts' \
'"credentialId"|credentialId' kits 2>/dev/null | sed -n '1,240p'Repository: Lamatic/AgentKit
Length of output: 39842
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- TARGET KIT FILES ---'
git ls-files 'kits/llm-response-qa/**' | sed -n '1,200p'
echo '--- FLOW REFERENCES TO THIS MODEL CONFIG ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'llm-response-qa_instructor-llmnode-298_generative-model-name|generativeModelName|credentialId|credential_name' \
kits/llm-response-qa 2>/dev/null | sed -n '1,240p'
echo '--- REPO-WIDE CREDENTIAL REMAP / IMPORT LOGIC ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'credential remap|credential mapping|remap.*credential|import.*credential|credentialId.*credential_name|credential_name.*credentialId|workspace-specific credential|mapped credential|resolve credential' \
. 2>/dev/null | sed -n '1,260p'
echo '--- CONFIG/TEMPLATE GENERATORS ---'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' \
'model-configs|generative-model-name|credentialId' \
kits scripts src .github 2>/dev/null | sed -n '1,260p'Repository: Lamatic/AgentKit
Length of output: 46875
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- TARGET KIT README / AGENT DOCS ---'
sed -n '1,220p' kits/llm-response-qa/README.md
echo '---'
sed -n '1,220p' kits/llm-response-qa/agent.md
echo '--- TARGET KIT FLOW AROUND MODEL CONFIG ---'
cat -n kits/llm-response-qa/flows/llm-response-qa.ts | sed -n '1,220p'
echo '--- EXAMPLES OF WORKSPACE-LIKE CREDENTIAL IDS VS EMPTY CREDENTIALS ---'
python3 - <<'PY'
from pathlib import Path
import re
paths = list(Path('kits').glob('*/model-configs/*.ts'))
samples = []
for p in paths:
text = p.read_text(errors='ignore')
if '"credentialId": ""' in text:
samples.append((str(p), 'empty'))
elif re.search(r'"credentialId":\s*"[^"]+"', text):
samples.append((str(p), 'set'))
if len(samples) >= 30:
break
for p, kind in samples:
print(f'{kind}\t{p}')
PYRepository: Lamatic/AgentKit
Length of output: 10876
🌐 Web query:
Lamatic AgentKit model-config credentialId import remap exported kit workspace-specific credential UUID
💡 Result:
In Lamatic, workspace-specific credentials are assigned a unique identifier (credentialId) [1][2]. When you export a kit or flow configuration from a workspace, this credentialId is embedded within the model configuration (often under generativeModelName or imageGenModelName fields) [1][3]. Because these credentials are bound to a specific workspace and project environment, the exported credentialId will typically not match the environment of your new (imported) workspace [4]. To resolve this, you must remap the credentialId after importing: 1. Locate the credentialId in your exported config file (e.g., in your lamatic-config.json or node YAML) [1][5][2]. 2. Identify the corresponding credential UUID in your new target workspace. You can find this by navigating to your new project's Secrets or Integrations management interface, or by using the Lamatic API/MCP tools to list available credentials [6][7]. 3. Update the credentialId field in your configuration file to reflect the UUID of the credential in your current workspace [1][5]. It is standard practice to manage sensitive API keys via the Secrets system within the Lamatic Studio rather than hardcoding them [6][4]. Ensure your new workspace has the necessary provider credentials configured before attempting to run the imported kit, as the model nodes will rely on a valid mapping to a local credential UUID [8][2].
Citations:
- 1: https://lamatic.ai/docs/flows/flow-config
- 2: https://lamatic.ai/docs/nodes/ai/generate-text-node
- 3: https://lamatic.ai/docs/agents/multi-modal-agent
- 4: https://github.com/lamatic/docs/blob/main/pages/docs/studio/studio-project.mdx
- 5: https://github.com/Lamatic/agentkit-reasoning
- 6: https://lamatic.ai/docs/studio/studio-variables-and-secrets
- 7: https://glama.ai/mcp/servers/Lamatic/Lamatic-MCP/tools
- 8: https://github.com/lamatic/docs/blob/main/pages/integrations/models.mdx
Do not ship a workspace-bound credentialId here. Lamatic exports credentials as workspace-local IDs, so this UUID needs to be remapped for the target workspace before publishing/importing the kit; otherwise the model node will fail to resolve its credential.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@kits/llm-response-qa/model-configs/llm-response-qa_instructor-llmnode-298_generative-model-name.ts`
around lines 9 - 12, The model configuration’s credentialId is tied to the
source workspace and must not be shipped as-is. Update the configuration around
model_name and provider_name to use the target workspace’s remapped credential
identifier during kit publishing or importing, ensuring the model node resolves
the credential correctly.
|
|
||
| ## Workflow | ||
|
|
||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Specify a language for the fenced code block.
The workflow diagram code block has no language tag, triggering markdownlint MD040. Add a language hint (e.g., text or mermaid) to satisfy the linter.
📝 Proposed fix
-```
+```text
User Input
│
▼
Chat Widget
│
▼
Generate JSON
│
▼
Chat Response</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 28-28: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/llm-response-qa/README.md` at line 28, Add a language tag to the fenced
workflow diagram code block in the README, using an appropriate hint such as
text or mermaid, while preserving the diagram content.
Source: Linters/SAST tools
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/llm-response-qa/apps/.env.example`:
- Around line 1-3: Reorder the variables in the environment example so
NEXT_PUBLIC_APP_URL appears before OPENROUTER_API_KEY, while retaining
LAMATIC_API_KEY, and ensure the file ends with a trailing newline.
In `@kits/llm-response-qa/apps/package.json`:
- Around line 1-19: Add the missing next.config.mjs and tsconfig.json files to
the Next.js app’s apps directory, alongside package.json and .env.example.
Configure them with the standard Next.js and TypeScript settings required for
the existing app and build scripts.
- Around line 10-18: Replace every "latest" value in the dependencies and
devDependencies of package.json with explicit pinned versions or controlled
caret ranges, including next, react, react-dom, typescript, `@types/node`, and
`@types/react`, so installs remain reproducible.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 0479e43b-c855-403b-b3c5-dbf7ef46f8b9
📒 Files selected for processing (2)
kits/llm-response-qa/apps/.env.examplekits/llm-response-qa/apps/package.json
| LAMATIC_API_KEY= | ||
| OPENROUTER_API_KEY= | ||
| NEXT_PUBLIC_APP_URL=http://localhost:3000 No newline at end of file |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Agent, your environment file needs a minor debrief.
The dotenv-linter flagged two issues: keys should be alphabetically ordered (NEXT_PUBLIC_APP_URL before OPENROUTER_API_KEY), and a trailing blank line is missing at the end of the file. Small details, but they keep the field tidy.
♻️ Proposed fix for ordering and trailing newline
LAMATIC_API_KEY=
-NEXT_PUBLIC_APP_URL=http://localhost:3000
-OPENROUTER_API_KEY=
+NEXT_PUBLIC_APP_URL=http://localhost:3000
+OPENROUTER_API_KEY=
+🧰 Tools
🪛 dotenv-linter (4.0.0)
[warning] 3-3: [EndingBlankLine] No blank line at the end of the file
(EndingBlankLine)
[warning] 3-3: [UnorderedKey] The NEXT_PUBLIC_APP_URL key should go before the OPENROUTER_API_KEY key
(UnorderedKey)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/llm-response-qa/apps/.env.example` around lines 1 - 3, Reorder the
variables in the environment example so NEXT_PUBLIC_APP_URL appears before
OPENROUTER_API_KEY, while retaining LAMATIC_API_KEY, and ensure the file ends
with a trailing newline.
Source: Linters/SAST tools
| { | ||
| "name": "llm-response-qa-app", | ||
| "version": "1.0.0", | ||
| "private": true, | ||
| "scripts": { | ||
| "dev": "next dev", | ||
| "build": "next build", | ||
| "start": "next start" | ||
| }, | ||
| "dependencies": { | ||
| "next": "latest", | ||
| "react": "latest", | ||
| "react-dom": "latest" | ||
| }, | ||
| "devDependencies": { | ||
| "typescript": "latest", | ||
| "@types/node": "latest", | ||
| "@types/react": "latest" | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Missing required configuration files per kit guidelines.
Your mission package is incomplete, agent. The coding guidelines require that a kit's apps/ directory must contain package.json, next.config.mjs, tsconfig.json, and .env.example. Only package.json and .env.example are present — next.config.mjs and tsconfig.json are missing. Without tsconfig.json, the typescript devDependency and Next.js build pipeline won't function correctly.
As per coding guidelines: kits/*/apps/{package.json,next.config.mjs,tsconfig.json,.env.example} — the Next.js app must be located in the apps/ directory with its own package.json, next.config.mjs, tsconfig.json, and .env.example.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/llm-response-qa/apps/package.json` around lines 1 - 19, Add the missing
next.config.mjs and tsconfig.json files to the Next.js app’s apps directory,
alongside package.json and .env.example. Configure them with the standard
Next.js and TypeScript settings required for the existing app and build scripts.
Sources: Coding guidelines, Path instructions
Added detailed overview, purpose, capabilities, workflow, guardrails, and integration references for the LLM Response QA Agent.
Updated the meta information for the LLM-Response-QA flow, including a description, tags, and GitHub URL.
Updated evaluation prompt for AI-generated responses.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/llm-response-qa/flows/llm-response-qa.ts`:
- Around line 1-16: Add the required author property to the meta export in the
LLM-Response-QA flow, reusing the author value already defined in
lamatic.config.ts. Preserve the existing name, description, tags, and testInput
fields.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: f1002e2a-add4-448c-a3fc-92b18cd292db
📒 Files selected for processing (7)
kits/llm-response-qa/.env.examplekits/llm-response-qa/agent.mdkits/llm-response-qa/apps/.env.examplekits/llm-response-qa/apps/next.config.mjskits/llm-response-qa/apps/package.jsonkits/llm-response-qa/apps/tsconfig.jsonkits/llm-response-qa/flows/llm-response-qa.ts
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/lamatic-update.yml (1)
1-27: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winMission briefing: tighten workflow permissions before deployment.
Agent, this workflow has no
permissions:block, so it inherits the repository's default token permissions — which may be overly broad. Add a minimalpermissionsblock to follow least-privilege principles. Since this workflow only checks out code and triggers a sync,contents: readis sufficient.🔒 Proposed fix: add least-privilege permissions
on: push: branches: - main paths: - '**/lamatic/flows/**' +permissions: + contents: read + jobs:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/lamatic-update.yml around lines 1 - 27, Add a top-level permissions block to the “Detect & Sync Lamatic Flows” workflow, granting only contents: read for the checkout and sync steps. Leave the existing trigger, job conditions, and Lamatic sync configuration unchanged.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/lamatic-update.yml:
- Line 17: Update the workflow steps using actions/checkout and
Lamatic/sync-flows-to-lamatic to reference full 40-character immutable commit
SHAs instead of the mutable v4 and v1 tags, preserving each action’s existing
behavior and configuration.
- Around line 16-17: Update the actions/checkout@v4 step in the workflow to set
persist-credentials to false, ensuring the GITHUB_TOKEN is not retained in the
local git configuration.
In `@lamatic/flows/LLM-Response-QA.ts`:
- Line 142: Update the response node content referencing
InstructorLLMNode_298.output.feedback so it surfaces the complete structured QA
report, including overall quality, accuracy, completeness, relevance,
hallucination risk, detected issues, actionable suggestions, and constructive
feedback, rather than only the feedback field.
- Line 104: Update the evaluation prompt content in the LLM-Response-QA flow to
wrap {{Chat Widget.chatMessage}} as untrusted input using the <response> wrapper
and the guardrail text established in
llm-response-qa_instructor-llmnode-298_user_1.md. Preserve the existing
evaluation criteria and JSON-only output requirement while ensuring the injected
chat message cannot override the evaluator’s instructions.
---
Outside diff comments:
In @.github/workflows/lamatic-update.yml:
- Around line 1-27: Add a top-level permissions block to the “Detect & Sync
Lamatic Flows” workflow, granting only contents: read for the checkout and sync
steps. Leave the existing trigger, job conditions, and Lamatic sync
configuration unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: e5c6c29c-dcb1-499a-ac1e-76163a2df9c7
📒 Files selected for processing (5)
.github/workflows/lamatic-update.ymlkits/llm-response-qa/flows/llm-response-qa.tskits/llm-response-qa/prompts/llm-response-qa_instructor-llmnode-298_user_1.mdlamatic/flows/LLM-Response-QA.tslamatic/project.json
|
Hi @Ramyatha24! 👋 Before this PR can be reviewed by maintainers, please resolve all comments and requested changes from the CodeRabbit automated review. Steps to follow:
This helps keep the review process efficient for everyone. Thank you! 🙏 |
Ramyatha24
changed the title
|
@coderabbitai review |
✅ Action performedReview finished.
|
1 participant
Summary
This PR adds a new AgentKit called LLM Response QA.
The kit evaluates AI-generated responses and provides structured quality assessment before responses are delivered to end users.
Features
Problem
Large Language Models can generate responses that are inaccurate, incomplete, inconsistent, or hallucinated. Reviewing every response manually is time-consuming.
Solution
This workflow automates AI response quality evaluation using Lamatic AgentKit. It generates a structured JSON report that helps developers identify issues and improve AI-generated responses.
Tech Stack
Motivation
This project was inspired by my experience evaluating AI-generated outputs and demonstrates how AI workflows can automate response quality assurance.
kits/llm-response-qa/.gitignore(ignores.lamatic/,node_modules/,.env,.env.local).kits/llm-response-qa/.env.example(LAMATIC_API_KEY,OPENROUTER_API_KEY).kits/llm-response-qa/README.md(kit description, evaluation dimensions, sample JSON output, use cases, tech stack, future improvements).kits/llm-response-qa/agent.md(agent identity, purpose, capabilities, workflow steps, guardrails, structured-JSON validation approach).kits/llm-response-qa/constitutions/default.md(safety/data/tone constitution: refusal rules, anti-jailbreak/prompt-injection, no fabrication when uncertain, PII handling).kits/llm-response-qa/lamatic.config.ts(registers kit “LLM-Response-QA” with mandatory stepllm-response-qa).kits/llm-response-qa/flows/llm-response-qa.ts(exports flowmetafor “LLM-Response-QA”).kits/llm-response-qa/model-configs/llm-response-qa_instructor-llmnode-298_generative-model-name.ts(OpenRouter model config for instructor:openrouter/openai/gpt-3.5-turbo-16k).kits/llm-response-qa/prompts/llm-response-qa_instructor-llmnode-298_system_0.md(QA evaluator rubric; requires scores + issues + suggestions; “JSON-only” output constraint).kits/llm-response-qa/prompts/llm-response-qa_instructor-llmnode-298_user_1.md(injects{{Chat Widget.chatMessage}}inside<response>tags; tells evaluator not to follow embedded instructions).kits/llm-response-qa/apps/.env.example(LAMATIC_API_KEY,OPENROUTER_API_KEY,NEXT_PUBLIC_APP_URL).kits/llm-response-qa/apps/package.json(Next.js appllm-response-qa-app).kits/llm-response-qa/apps/next.config.mjs(reactStrictMode: true).kits/llm-response-qa/apps/tsconfig.json(strict TS, Next.js settings,noEmit: true).lamatic/flows/LLM-Response-QA.ts(flow/node graph definition forLLM-Response-QA).lamatic/project.json(Lamatic project metadata)..github/workflows/lamatic-update.yml(syncs Lamatic flows on push tomainwhen**/lamatic/flows/**changes).Flow / workflow high level (from
lamatic/flows/LLM-Response-QA.ts)flow.jsonisn’t introduced here; the node graph is defined directly inlamatic/flows/LLM-Response-QA.ts.triggerNode(chatTriggerNode): captures the incoming chat message from the “Chat Widget”.dynamicNode(InstructorLLMNode/ “Generate JSON”): runs the instructor QA evaluator with a structured JSON output schema and rubric-based system prompt.responseNode(chatResponseNode): returns the evaluator’s JSON feedback as the delivered QA report.{{Chat Widget.chatMessage}}).issues[]+suggestions[]while enforcing “valid JSON only” output.feedbackfield for downstream use.