Split ci-ubuntu workflow into multiple jobs and add caching

[silas-hw]

ci-ubuntu can be split up into multiple jobs which run in parallel. This also potentially leads the way to further CI improvements. Currently, html generation and standard unit tests run in parallel. Furthermore, the golden testing step would benefit significantly from caching build files. Some rather non-scientific benchmarking showed running golden tests a second time shortened each test by often over a minute, backed up by Test.Golden calling a shell script that itself invokes cabal build for each test. Caching all of dist-newstyle would consume about 500MB, so it should be done sparingly; currently, it will only be cached on the master branch and if dist-newstyle changes after running the golden tests (by checking the sha1sum of dist-newstyle before and after running the tests).

GitHub Actions is a bit of a pain in that there's no way to have some way of setting up an initial workspace for each job to run in. This means that the init steps (set up env, install things, etc...) have to run on every job. To reduce redundant code I split this out into a composite workflow (see https://docs.github.com/en/actions/tutorials/create-actions/create-a-composite-action). This action will be given all the env variables set up by the calling workflow and can likewise set them with the usual write to GITHUB_ENV (see https://github.com/orgs/community/discussions/51280#discussioncomment-8726096).

I'm still not fully confident on this though.

Edit:

Data is now shared between jobs via upload- and download-artifact, as well as output variables if appropriate.

After this is merged, I plan on working more to resolve #2248, starting with more granularity over when jobs run.

[silas-hw]

Seems checkout needs to run first

[gallais]

That summary looks concerning: https://github.com/agda/agda-stdlib/actions/runs/28607124313?pr=3049
In case of cache miss, it looks like you'll rebuild Agda twice (in parallel but that's still wasteful).

I would recommend looking at the structure of the Idris2 CI pipeline:
https://github.com/idris-lang/Idris2/blob/main/.github/workflows/ci-idris2-and-libs.yml

You can see from the summary https://github.com/idris-lang/Idris2/actions/runs/28065704913?pr=3775
that we have a clear dependency graph & reuse artifacts produced in one stage in the later ones
(via actions/upload-artifact@v4).

So you could have a first stage checking whether we have agda in the cache. If not, build it and upload it.
Then have testing and upload the agdai files for the lib.
Then have in parallel golden testing & html generation.
Then deploy the html only if we are in the right branch and everything succeeded.

[silas-hw]

Seems a much better approach indeed. Initially dismissed upload-artificact since it seemed more tedious than needed, but given more thought it seems significantly more reasonable.

[silas-hw]

GITHUB_PATH is only for writing by the looks of it, fixing that now

[silas-hw]

Right now I don't like how sudo is before 'chmod' and that it's done on entire folders. Unsure if it should be changed, but chmod without sudo results in 'operation not permitted' due to how upload- and download- artifact handles permissions.

I imagine it would also be best to delete the artifacts after the workflow run?

Sorry for the large number of 'fix' commits. It's been a while since I've handled GitHub Actions.

[silas-hw]

So you could have a first stage checking whether we have agda in the cache. If not, build it and upload it. Then have testing and upload the agdai files for the lib. Then have in parallel golden testing & html generation. Then deploy the html only if we are in the right branch and everything succeeded.

@gallais It can't seem to find the agdai files. Am I doing something obviously wrong? I presume wildcards with upload-artifact work differently to how I think they do.

HTML is all in one job still, but I'll split out deploying it tomorrow.

[gallais]

I think they should all be in the _build/ directory.

Edit:

Sorry for the large number of 'fix' commits

No worries, we all do that. CI is hell

[gallais]

It would be worth investigating why we have so much rechecking in the golden testing because
it does feel like we ought to be able to take advantage of the already computed .agdai files.

If the issue is that we are checking the same files over and over again because every invocation
is with slightly different, and incompatible, OPTIONS then the principled solution may be to
change Agda itself. In order to maximise sharing we may need to have _build have subdirectories
corresponding not only to the Agda versions but also the sets of options.

[silas-hw]

It would be worth investigating why we have so much rechecking in the golden testing because it does feel like we ought to be able to take advantage of the already computed .agdai files.

If the issue is that we are checking the same files over and over again because every invocation is with slightly different, and incompatible, OPTIONS then the principled solution may be to change Agda itself. In order to maximise sharing we may need to have _build have subdirectories corresponding not only to the Agda versions but also the sets of options.

If best, because it seems this may be a larger task, I can revert the caching steps and move it to a separate PR once this has been merged.

[silas-hw]

My understanding was wrong: the agdai files from test-stdlib were being used. It's just that it only has a marginal difference per test, and the cabal build step is much more significant (which is what would need to be cached)

[silas-hw]

I'm currently not crazy confident on how caching on test-stdlib-golden is done (i.e. sha1sum and how the key is named)

[gallais]

This is already a win IMO even if you cannot figure out how to speed up
the golden testing. If that step is really not being sped up by running after
the library build then we should consider running it in parallel with test-stdlib
(and not bother with caching for now).

We could even run every golden-testing group separately (passing only=XXX
to make testsuite will only run the test cases whose name contain XXX as a
substring so for instance make testsuite only=data/ only runs the ones in the
data/ subdirectory!)

[silas-hw]

My instincts are telling me running golden tests in parallel will be faster than the speedup from reusingg agdai files, so I'm testing that out now.

It's almost certainly a good idea to shard/split the golden tests as well. Given they're pretty well structured into directories it shouldn't be too hard to split the groups evenly between a set number of runners instead of having them hard-coded (some bash wizardry should work). I might do that in a separate PR as to not bloat out this one too much (although I'm well aware it has already gotten quite big).

[silas-hw]

Well that shaved off ~10 minutes from the average CI run for this PR!

After forcing a cache save by editing the workflow and then reverting it to make sure it does in fact work I'm going to leave this and make new changes in fresh PRs.

[silas-hw]

Wow! That's a whole lot faster (14 minutes), and I believe it was only using 1-2 runners across it (test-stdlib and test-stdlib-golden started a few minutes after html-generate).

Golden testing itself only took 5 minutes!

The only thing I'm still not sure of is if the if for saving a new cache will work properly.

[silas-hw]

For some reason the hashes of dist-newstyle are always different. Probably something to do with hidden files.