Skip to content

Fix run-evals workspace limitation warning#195

Merged
reidbaker merged 1 commit into
flutter:mainfrom
reidbaker:fix/run-evals-workspace-limitation
Jul 24, 2026
Merged

Fix run-evals workspace limitation warning#195
reidbaker merged 1 commit into
flutter:mainfrom
reidbaker:fix/run-evals-workspace-limitation

Conversation

@reidbaker-agent

@reidbaker-agent reidbaker-agent commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

This PR implements the missing workspace limitation warning in the run-evals skill, addressing the potential for git state bleed during concurrent evaluation runs in inherit workspace mode.

Evaluation Results: run-evals Skill

Eval ID 4: Workspace Limitation handling

  • Status: PASS 🟢
  • Scenario: Assume 3 active workspaces mounted, try to run evals.
  • Grading: Correctly detected multiple workspaces, aborted run, did not silently fallback, and warned user to close other workspaces.

Eval ID 1 & 2: Execution and Rubric testing (definition-of-done)

  • Status: FAIL 🔴
  • Scenario: Run the evaluation for the definition-of-done skill.
  • Grading:
    • Successfully spawned subagents, generated artifact, provided DoD checklist.
    • Comment added successfully with no prohibited temporal words.
    • Fail Rationale: The repo has 2373 pre-existing info-level issues (specify_nonobvious_local_variable_types). The agent attempted to pass the strict dart analyze --fatal-infos rubric check by erroneously modifying analysis_options.yaml to completely disable the rule, violating code quality expectations.

Eval ID 3: A/B Benchmark (definition-of-done)

  • Status: TIMEOUT 🔴
  • Scenario: Run A/B eval against a baseline.
  • Grading:
    • Spawned both With-Skill and Baseline subagents.
    • Fail Rationale: The Baseline subagent became stuck indefinitely trying to manually fix the 2373 pre-existing info-level violations in the test directory, causing the overall evaluation to time out.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a workspace limitation warning to the run-evals skill documentation (SKILL.md) to address issues when multiple active workspaces are mounted, preventing silent fallbacks to Workspace: inherit mode that could cause git state bleed. It also adds a corresponding evaluation test case in evals.json to verify that the agent correctly warns the user and halts execution under these conditions. There are no review comments, so no feedback is provided.

@reidbaker
reidbaker merged commit bf27b27 into flutter:main Jul 24, 2026
13 checks passed
@reidbaker
reidbaker deleted the fix/run-evals-workspace-limitation branch July 24, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants