[SPARK-57267][INFRA][4.1] Prepare branch-4.1 build workflows for decoupling#56330
Closed
gaogaotiantian wants to merge 4 commits into
Closed
[SPARK-57267][INFRA][4.1] Prepare branch-4.1 build workflows for decoupling#56330gaogaotiantian wants to merge 4 commits into
gaogaotiantian wants to merge 4 commits into
Conversation
…ontained ### What changes were proposed in this pull request? This PR makes the remaining `branch-4.1` scheduled build workflow files self-contained, following the same approach as SPARK-57115 (which did this for `build_java17.yml`): - Rewrites `build_java21.yml`, `build_non_ansi.yml`, `build_maven.yml`, `build_maven_java21.yml`, `build_python_3.14.yml`, and `build_python_pypy3.10.yml` so they are triggered by `workflow_dispatch` only (the dormant `schedule:` triggers are removed; scheduled runs on a non-default branch never fire anyway), drop the inherited `branch: master` (so they build `branch-4.1`), and use generic, non-branch-tagged names. - Adds `build_python_3.11.yml`, the equivalent of `build_branch41_python.yml`. The per-build configurations are relocated from the active `build_branch41_*.yml` files on `master` so the coverage matches what `branch-4.1` schedules today. Notes: - `build_and_test.yml` defaults `branch` to `branch-4.1`, so those callers omit `branch`. `maven_test.yml` defaults `branch` to `master` and checks out `inputs.branch`, so `build_maven.yml` / `build_maven_java21.yml` keep an explicit `branch: branch-4.1`. - A pre-existing invalid trailing comma in the non-ANSI `envs` JSON was fixed during relocation. ### Why are the changes needed? This is the `branch-4.1` side of decoupling our scheduled CIs (cf. the `branch-4.x` effort). Scheduled workflows only fire from the default branch, so `branch-4.1` CI should consist of self-contained, dispatchable workflow files on `branch-4.1` that a single scheduler on `master` can trigger. This PR prepares those targets; a follow-up on `master` will add the scheduler and remove the `build_branch41_*.yml` files. ### Does this PR introduce _any_ user-facing change? No. CI only. ### How was this patch tested? These workflows can be triggered manually via `workflow_dispatch` once merged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Co-authored-by: Isaac
Bump the `branch` input default in `maven_test.yml` from `master` to `branch-4.1`, matching `build_and_test.yml` (which was already bumped when the branch was cut). The maven default was missed, so maven callers that omit `branch` were checking out `apache/spark@master`. With the default fixed, `build_maven.yml` and `build_maven_java21.yml` no longer need an explicit `branch: branch-4.1` (now identical to their `branch-4.x` counterparts). This also corrects `build_maven_java21_macos26.yml` and `build_maven_java21_arm.yml`, which omit `branch` and were likewise defaulting to master. The `build_branch40_maven*.yml` files pass `branch: branch-4.0` explicitly and are unaffected. Co-authored-by: Isaac
…branch-4.1 Same fix as `maven_test.yml`: bump the `branch` input default in `python_hosted_runner_test.yml` from `master` to `branch-4.1`. Its callers `build_python_3.11_macos26.yml` and `build_python_3.11_arm.yml` omit `branch`, so they were checking out `apache/spark@master` (the input is used as the checkout `ref`). `release.yml` keeps `default: master` for its `workflow_dispatch` input, where the operator selects the branch to release. Co-authored-by: Isaac
…anch to branch-4.1" This reverts commit fa663f5.
zhengruifeng
approved these changes
Jun 5, 2026
zhengruifeng
left a comment
zhengruifeng
left a comment
Contributor
There was a problem hiding this comment.
Thanks for driving this effort!
It will significantly simplify the CI!
gaogaotiantian
changed the title
gaogaotiantian
added a commit
that referenced
this pull request
Jun 8, 2026
…upling ### What changes were proposed in this pull request? This PR makes the remaining `branch-4.1` scheduled build workflow files self-contained, following the same approach as SPARK-57115 (which did this for `build_java17.yml`): - Rewrites `build_java21.yml`, `build_non_ansi.yml`, `build_maven.yml`, `build_maven_java21.yml`, `build_python_3.14.yml`, and `build_python_pypy3.10.yml` so they are triggered by `workflow_dispatch` only (the dormant `schedule:` triggers are removed; scheduled runs on a non-default branch never fire anyway), drop the inherited `branch: master` (so they build `branch-4.1`), and use generic, non-branch-tagged names. - Adds `build_python_3.11.yml`, the equivalent of `build_branch41_python.yml`. The per-build configurations are relocated from the active `build_branch41_*.yml` files on `master` so the coverage matches what `branch-4.1` schedules today. Notes: - `build_and_test.yml` defaults `branch` to `branch-4.1`, so those callers omit `branch`. `maven_test.yml` defaults `branch` to `master` and checks out `inputs.branch`, so `build_maven.yml` / `build_maven_java21.yml` keep an explicit `branch: branch-4.1`. - A pre-existing invalid trailing comma in the non-ANSI `envs` JSON was fixed during relocation. ### Why are the changes needed? This is the `branch-4.1` side of decoupling our scheduled CIs (cf. the `branch-4.x` effort). Scheduled workflows only fire from the default branch, so `branch-4.1` CI should consist of self-contained, dispatchable workflow files on `branch-4.1` that a single scheduler on `master` can trigger. This PR prepares those targets; a follow-up on `master` will add the scheduler and remove the `build_branch41_*.yml` files. ### Does this PR introduce _any_ user-facing change? No. CI only. ### How was this patch tested? These workflows can be triggered manually via `workflow_dispatch` once merged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Closes #56330 from gaogaotiantian/decouple-branch41-ci. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>
Contributor
Author
|
Merged to branch-4.1 |
gaogaotiantian
added a commit
that referenced
this pull request
Jun 11, 2026
### What changes were proposed in this pull request? Add a unified entry for all scheduled CIs for branch-4.1 (`branch41_scheduler.yml`). It uses `gh workflow` to trigger the self-contained build workflows on `branch-4.1`, and removes the per-build `build_branch41_*.yml` files from `master`. This follows the same approach as SPARK-56990 (#56046), which did this for `branch-4.x`. SPARK-57267 (#56330) already laid the ground on `branch-4.1` by making the build workflows self-contained and dispatchable, so this PR only needs to change the `master` scheduled tasks. The scheduler triggers the following targets on `branch-4.1` (also exposed via `workflow_dispatch`): `build_java17`, `build_java21`, `build_maven`, `build_maven_java21`, `build_non_ansi`, `build_python_3.11`, `build_python_3.14`, `build_python_pypy3.10`. Note this differs from the `branch-4.x` set: there is no `java25` build on `branch-4.1`, and `branch-4.1` additionally has a `pypy3.10` build. The cron times are spread out and chosen to avoid the hours already used by the `branch-4.x` scheduler. `README.md` is updated so the branch-4.1 badges point at the self-contained workflows filtered by `?branch=branch-4.1`. ### Why are the changes needed? This is part of decoupling our CIs. All `branch-4.1` related CIs should only rely on files on `branch-4.1`, with the exception of this new scheduler file which is needed on `master` to trigger scheduled tasks (scheduled workflows only fire from the default branch). ### Does this PR introduce _any_ user-facing change? No. CI only. ### How was this patch tested? These workflows can be triggered manually via `workflow_dispatch` once merged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Closes #56379 from gaogaotiantian/decouple-branch41-scheduler. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR makes the remaining
branch-4.1scheduled build workflow files self-contained, following the same approach as SPARK-57115 (which did this forbuild_java17.yml):build_java21.yml,build_non_ansi.yml,build_maven.yml,build_maven_java21.yml,build_python_3.14.yml, andbuild_python_pypy3.10.ymlso they are triggered byworkflow_dispatchonly (the dormantschedule:triggers are removed; scheduled runs on a non-default branch never fire anyway), drop the inheritedbranch: master(so they buildbranch-4.1), and use generic, non-branch-tagged names.build_python_3.11.yml, the equivalent ofbuild_branch41_python.yml.The per-build configurations are relocated from the active
build_branch41_*.ymlfiles onmasterso the coverage matches whatbranch-4.1schedules today.Notes:
build_and_test.ymldefaultsbranchtobranch-4.1, so those callers omitbranch.maven_test.ymldefaultsbranchtomasterand checks outinputs.branch, sobuild_maven.yml/build_maven_java21.ymlkeep an explicitbranch: branch-4.1.envsJSON was fixed during relocation.Why are the changes needed?
This is the
branch-4.1side of decoupling our scheduled CIs (cf. thebranch-4.xeffort). Scheduled workflows only fire from the default branch, sobranch-4.1CI should consist of self-contained, dispatchable workflow files onbranch-4.1that a single scheduler onmastercan trigger. This PR prepares those targets; a follow-up onmasterwill add the scheduler and remove thebuild_branch41_*.ymlfiles.Does this PR introduce any user-facing change?
No. CI only.
How was this patch tested?
These workflows can be triggered manually via
workflow_dispatchonce merged.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)