SOLR-17508 Port BATS tests to Powershell (Pester)

[janhoy]

https://issues.apache.org/jira/browse/SOLR-17508

WIP Feel free to try and fix issues :)

Toy project to try to replicate the BATS tests for Windows, using the Pester test framework for Powershell (https://pester.dev). With aid of Claude Code, this branch was written in one go in literally 3 minutes.
It includes a github PR workflow test running windows, so we'll have a way to run it. Here's the prompt used:

You'll now get a greenfield task. In Solr we use BATS test framework for automated testing of start/stop 
solr and CLI tools. We do not have similar support for testing on Windows commands. You will integrate 
Pester (test/test_version.bats) tests.

The approach we'll take is to "port" each BATS test to windows, as the CLI commands are supposed to 
be identical, except the obvious differences between bash and Powershell.

As a start, I want you to port the solr/packaging/test/test_version.bats test.
- First, create a folder 'powershell-tests' under solr/packaging.
- Then, integrate pester in our unit test framework by peeking at how it's done in solr:packaging:integrationTests task
- Create a new task for this, it should only run when on windows.
- Next, port the bats test 'test_version.bats' over.

I don't have windows, only Mac, so give me a hint on how I can test this. I can spin up a Windows VM 
and check out this branch there and test, but is there a smarter way like Wine or something that runs on 
mac? 

[epugh]

One thought is, and I think you maybe are thinking this way, is to make sure that we only port over tests that we think have a windows specific challenges. There are bats tests like the Tika extraction stuff or the OpenNLP that probably (!) shouldn't have any windows specific nuances that would suggest a pester equivalent...

[malliaridis]

When I tried to run

.\gradlew.bat --no-daemon -p solr/packaging pesterTests

I got multiple exceptions because pester was pre-installed with an older version (3.4.0). Pester getting started said to run:

Install-Module Pester -Force

as admin, and it required me to provide the parameter -SkipPublisherCheck as it refused to upgrade it.

After confirming the pester installation with Import-Module Pester -PassThru it did not throw errors anymore.

A hint of setting up and running tests may be useful, and we may also need to run pester installation on the CI before running the tests?

I also noticed that running the gradle task pesterTests it sometimes does not execute the tests. I believe it is due to some gradle caching, because when I update a line in Gradle, it always runs through the first time, but any follow-up execution skips the pester tests. Maybe this is a problem in CI too?

[malliaridis]

This output (and a similar one in Zk.Tests.ps1) adds some verbosity tot he console during execution which I would prefer to enable manually if needed. Perhaps we could use the pester verbosity here and do something like:

if ($PesterPreference.Show -contains 'Detailed') { # gradle script sets verbosity to Normal
  # ...
}

This would also require an update in the gradle task to allow changing the verbosity via -Dpester.verbosity. Something like

def pesterVerbosity = (project.findProperty("pester.verbosity")
  ?: System.getProperty("pester.verbosity", "Normal"))

and in the pester config

// ...
"\$config.Output.Verbosity = '${pesterVerbosity}'; " +
// ...

[malliaridis]

This block always fails, leading to the 9 tests being skipped. Is Solr ever started during the test?

[malliaridis]

Could this line be configured in a BeforeEach block and reduce duplication maybe?

[malliaridis]

Oh, and I forgot to leave a comment content-wise. It is a good start for introducing tests. I believe in the future we will add more windows-specific cases that will matter more. But for now, I believe it is fine with the included tests. Well done! 👍

[janhoy]

One thought is, and I think you maybe are thinking this way, is to make sure that we only port over tests that we think have a windows specific challenges. There are bats tests like the Tika extraction stuff or the OpenNLP that probably (!) shouldn't have any windows specific nuances that would suggest a pester equivalent...

Yea, that could be an option. I did the small isolated "help", "version" and "zk" tests, and they are all happy so far, hopefully proving that the Windows scripts quack the same way as linux one (if the tests actually verifies enough of the output). We probably need some of the run-example ones.

Feel free to suggest a list of prioritized tools that gives best value, where we know there is/have been issues.

But I see no reason we could not add many tests on the windoes side as well.

@malliaridis I won't have time to persue this the coming days, so I hereby give permission to any committer who wants to hijack the PR and just commit directly to the PR branch. It's a POC and if you want to polish it to the point of a first merge, that would be great.