Make TestPackage work if the package is read-only

[ThomasBreuer]

We want to run the package tests in the following situations.

• In CI tests for the package, the package manual has not been built. In order to run also the manual examples, we read the package's makedoc.g. This works because the doc and tstdirectories are writable.

• When the package has been installed, its doc and tst directories are in general not writable. In this case, we expect that test files containing the manual examples are available, and we do not read makedoc.g. The files with manual examples have names utils*.tst, if utils01.tst does not exist, testall.g reports a test failure.

This is intended to resolve a problem discussed in oscar-system/GAP.jl/pull/1379.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.21%. Comparing base (e388180) to head (b37b28c).

Additional details and impacted files

@@           Coverage Diff           @@
##           master      #94   +/-   ##
=======================================
  Coverage   85.21%   85.21%           
=======================================
  Files          32       32           
  Lines        1867     1867           
=======================================
  Hits         1591     1591           
  Misses        276      276           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ThomasBreuer]

@fingolfin Is this proposal an acceptable solution for the abovementioned problem in GAP.jl,
in the sense that a new utils release will fix it?

(Also the GAP packages in the rsync distribution of GAP are read-only, thus the problem will occur also in GAP installations via rsync.)

[fingolfin]

Seems good to me. @cdwensley ?

[cdwensley]

Of course I am happy to merge this, but I do not understand why these utils0*.tst files are being created.
Is the idea that the manual examples should no longer be reproduced as test files (because there is no guarantee that differences might arise over time)? What is the point of creating the utils0*.tst if the test files do exist?
Is it now suggested that test files containing the manual examples should be deleted? Or be renamed as utils0*.tst?

[fingolfin]

The file makedoc.g creates the utils0*.tst files. This is because of the extract_examples := true setting in it.

Those .tst files contain the examples given in your package manual.

As a result, after running makedoc.g, running the test suite will also run all examples in your manual.

In order to ensue that when running the package tests (by calling TestPackage("utils") or reading tst/testall.g directly), those utils*.tst files are always present, someone added the code which reads makedoc.g to tst/testall.g, which is now causing trouble.

@ThomasBreuer proposes a fix which I think would work.

But just to say, this is the only package I am aware of doing this. All other packages using extract_examples := true do one of these two things:

1. simply add the *.tst files extracted from the manual to git
   • while this is not the "pure" thing to do, it is conceptually simple and works well in practice for "mature" packages that rarely change their manual examples.
   • if one modifies, adds or removes examples, one has to run makedoc.g and then commit the changes to the *.tst files.
2. the other approach I've seen is to just run makedoc.g inside CI before running the tests
   • that avoids the "impure" addition of generated files to the repository
   • downside is that TestPackage may look as if it passes locally, but then fails in CI: namely if one forgets to run makedoc.g once, locally, before testing the package

Anyway, all three approaches seems fine to me. Personally I usually go with 1 because it is simplest.