Add check-clang-unit, check-llvm-unit with LLVM_WINDOWS_PREFER_FORWARD_SLASH enabled on llvm-clang-x86_64-win-fast bot

[Jwata]

This PR enables running check-clang-unit, check-llvm-unit tests twice on llvm-clang-x86_64-win-fast bot: once with default settings (backslash path behavior), and once with the forward slash override enabled (LLVM_WINDOWS_PREFER_FORWARD_SLASH=1).

Context & Rationale

There is no test coverage for LLVM_WINDOWS_PREFER_FORWARD_SLASH feature. Adding LLVM_WINDOWS_PREFER_FORWARD_SLASH enabled tests will improve the coverage, so that the change owners can notice test breakages.
It was also discussed to add tests with the flag to an existing bot, rather than adding a new builder.

Key Changes

• llvm-clang-x86_64-win-fast will run the following tests:

  1. check-llvm-unit (Default)
  2. check-clang-unit (Default)
  3. check-llvm-unit-forward-slashes (with LLVM_WINDOWS_PREFER_FORWARD_SLASH=1 override)
  4. check-clang-unit-forward-slashes (with LLVM_WINDOWS_PREFER_FORWARD_SLASH=1 override)

• Introduces a new check type with target, env_override, name_suffix to support overriding an environment variable.

Questions for reviewers

• Is there a better way than the env_override approach?
• Should llvm-clang-x86_64-expensive-checks-win also duplicate the checks, too?

[zmodem]

Can you provide a non-AI-generated description?

[Jwata]

Can you provide a non-AI-generated description?

Hmm, it might be too verbose. I made it concise. PTAL

[Jwata]

@boomanaiden154
Can you take a look or help find someone can review this PR?

[boomanaiden154]

Why exactly does this need to be passed as an env variable?

It seems like this should just be a CMake option. https://github.com/llvm/llvm-project/blob/56ccbc25315030088b04435e4828efbc22f0e926/llvm/CMakeLists.txt#L638

[Jwata]

This allows us to build only once and run tests twice: the first ones with default, and the second ones with LLVM_WINDOWS_PREFER_FORWARD_SLASH feature, rather than building twice or duplicating the bot.

[boomanaiden154]

Can you take a look or help find someone can review this PR?

You need to get approval from the bot's owner.

[Jwata]

Can you take a look or help find someone can review this PR?

You need to get approval from the bot's owner.

I see. Thanks.

@gkistanova Could you take a look?

[rnk]

Instead of configuring the buildbot level to build the check*unit* targets twice in different environments, I would configure this at the lit test suite level. You can look at the sanitizer tests, which do some matrix testing (re-running the test suite in multiple configurations). They do this in a way that allows you to run all the tests in a single lit invocation, reducing bottlenecks, improving utilization, and reducing cycle time.

You could control that via a cmake option.

[vvereschaka]

Thank you for PR @Jwata , but there are few moments:

• a goal of the llvm-clang-x86_64-win-fast builder is a building of each commit for the main components -- LLVM/Clang -- as fast as it possible and adding two additional steps of the testing increases an average build time for 30-35%. It seems to much for me and I wouldn't like to slowdown this builder that much at least for now.

• actually, you don't need to run both of those checks on the same builder. There are few more Windows bots running the LLVM/Clang unit tests (check-clang/check-llvm or check-all) and they already testing the build config without LLVM_WINDOWS_PREFER_FORWARD_SLASH (-win-x- as example). You'll need just a builder that checks a build with LLVM_WINDOWS_PREFER_FORWARD_SLASH=ON. We can add -DLLVM_WINDOWS_PREFER_FORWARD_SLASH=ON as CMake parameter of the build configuration for this builder and check it here without any additional checks.

• I see a failed LLVM-Unit test when I built with -DLLVM_WINDOWS_PREFER_FORWARD_SLASH=ON on my local computer:

********************
Failed Tests (1):
  LLVM-Unit :: Support/./SupportTests.exe/FileSystemTest/CreateRelativeDirectorySymlink


Testing Time: 13.42s

Total Discovered Tests: 11096
  Skipped:   644 (5.80%)
  Passed : 10451 (94.19%)
  Failed :     1 (0.01%)

This test must be fixed before applying that changes on the buildbot configuration.

• Also, I don't see this failure when build without CMake's LLVM_WINDOWS_PREFER_FORWARD_SLASH, but run them with set LLVM_WINDOWS_PREFER_FORWARD_SLASH=1 environment variable. Looks like the environment variable is not supported at all (or does not work as expected).

PS. You can find the owners of the builders on https://lab.llvm.org/buildbot/#/workers page at Admin column.