Test framework detection in Hanami

[katafrakt]

I have a Hanami project using RSpec for tests, but Ruby LSP does not detect it as a test framework. Upon examining the code I figured this is because the detection code looks for gems directly mentioned in Gemfile starting with "rspec" string (so rspec-rails etc. are caught), but in Hanami it's hanami-rspec.

I understand that the ideal solution would be a Hanami addon that takes care of that (and I think some folks are working on it), but at the same time I was wondering if maybe relaxing the detection conditions could not immediately solve the issue here. I was thinking about something like:

Any gem contains "rspec" in direct dependencies and there's an exact match for "rspec" in all dependencies

Do you think it would lead to some false positives?

EDIT: After some more code spelunking, I think it would not be so easy for an addon to change that, which makes this perhaps even more worth discussing.

[vinistock]

Thanks for the feedback.

Regarding what you mentioned, unfortunately, checking for the string rspec alone does lead to false positives as we received reports of Minitest users who like the RSpec expectations and so they do have a direct dependency to a gem with rspec in its name.

That said, we're no longer using that strategy to detect test frameworks since introducing the new explorer implementation. Add-ons now have much more flexibility to introduce test groups and examples and the gem names are not really involved anymore.

Do you happen to have the new experience turned off in your settings?

{
  // This is how you would turn it off
  "rubyLsp.featureFlags": {
     "fullTestDiscovery": false,
  }
}

And are you on the latest versions of both the VS Code extension and the server gem?

[katafrakt]

Thanks for the answer. I understand the issue with Minitest with RSpec matchers. It would be hard to come around, I think. Maybe Minitest detection should go first in this case - if Minitest is detected, not check for RSpec?

As for the fullTestDiscovery, I don't use VSCode, but I certainly did not (knowingly) disable anything. However, examining the code again, I see that the discovery is made here in global_state.rb. The actual method just looks at the dependencies. I'm using Ruby LSP from main branch. What am I missing wrt using the new detection strategy?

[vinistock]

Those methods aren't used as part of the new test discovery logic. They're there for compatibility for now, but they'll be removed soon.

What editor are you using? You can enable feature flags by passing initialization options:

{
  "initializationOptions": {
    "enabledFeatureFlags": {
      "fullTestDiscovery": true,
    }
  }
}

That said, all of this test explorer related functionality is has not been standardized as part of the LSP spec, so we built all of this custom for VS Code to be used alongside our extension. How are you currently using this in other editors?

[katafrakt]

I'm using Emacs with Eglot. I think with the feature flag enabled it detects correctly, I was thrown off by still receiving the debug message from the old test discovery logic:

[jsonrpc] e[21:05:51.899] <-- window/logMessage {"method":"window/logMessage","params":{"type":4,"message":"Detected test library: unknown"},"jsonrpc":"2.0"}

However, I understand now this is mostyl for VSCode mostly. It is used in Code Lenses only, which are not supported in eglot anyway. Maybe I would try with different LSP client some day. For now, thanks for guiding me through the configuration.