Skip to content

Make scheduled outerloop builds succeed when only Helix tests fail #129049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mmitche wants to merge 4 commits into
base: release/8.0
Choose a base branch
from
+18 −0
Open

Make scheduled outerloop builds succeed when only Helix tests fail #129049

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
69af95a
Don't fail scheduled outerloop builds on Helix work item failures
mmitche Jun 5, 2026
ab6a89e
Make scheduled outerloop Helix step succeed instead of continueOnError
mmitche Jun 5, 2026
b36307c
TEMP: also apply failOnTestFailures on Manual runs (validation only, …
mmitche Jun 5, 2026
522f1e2
Revert temporary Manual-gate validation change
mmitche Jun 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions eng/pipelines/libraries/helix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ parameters:
extraHelixArguments: ''
shouldContinueOnError: false
scenarios: ''
# When false, Helix work item and test failures do not fail the build, so the "Send to Helix"
# step still succeeds. Unlike shouldContinueOnError (which only marks the build as
# partiallySucceeded), this produces a fully successful build. Scheduled builds with
# always:false set this so that flaky tests don't cause AzDO to re-queue the same commit daily.
failOnTestFailures: true

steps:
- script: $(_msbuildCommand) $(_warnAsErrorParamHelixOverride) -restore
Expand All @@ -26,6 +31,8 @@ steps:
/p:TestScope=${{ parameters.testScope }}
/p:TestRunNamePrefixSuffix=${{ parameters.testRunNamePrefixSuffix }}
/p:HelixBuild=$(Build.BuildNumber)
/p:FailOnWorkItemFailure=${{ parameters.failOnTestFailures }}
/p:FailOnTestFailure=${{ parameters.failOnTestFailures }}
${{ parameters.extraHelixArguments }}
/bl:$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)/SendToHelix.binlog
displayName: Send to Helix
Expand Down
11 changes: 11 additions & 0 deletions eng/pipelines/libraries/outerloop.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ extends:
testScope: outerloop
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
# On scheduled runs (always:false) don't fail the build on Helix work item or
# test failures, so flaky outerloop tests don't keep AzDO re-queueing the same
# commit. The Send to Helix step fully succeeds (not partiallySucceeded).
${{ if eq(variables['Build.Reason'], 'Schedule') }}:
failOnTestFailures: false

- ${{ if eq(variables['isRollingBuild'], false) }}:
- template: /eng/pipelines/common/platform-matrix.yml
Expand Down Expand Up @@ -81,6 +86,9 @@ extends:
testScope: outerloop
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
# Don't fail scheduled builds on Helix work item/test failures (see above).
${{ if eq(variables['Build.Reason'], 'Schedule') }}:
failOnTestFailures: false

- ${{ if eq(variables['includeWindowsOuterloop'], true) }}:
- template: /eng/pipelines/common/platform-matrix.yml
Expand All @@ -106,3 +114,6 @@ extends:
testScope: outerloop
creator: dotnet-bot
extraHelixArguments: /p:BuildTargetFramework=net48
# Don't fail scheduled builds on Helix work item/test failures (see above).
${{ if eq(variables['Build.Reason'], 'Schedule') }}:
failOnTestFailures: false
Loading