Test Explorer not listing Cucumber-JVM tests

[evilC]

Repro:

1. Create Java project
2. Maven
3. Cucumber.io archetype (7.11.1)
4. everything default

Installed plugins (EVERYTHING else disabled):

Test runner for java version: v0.42.0
Also tried v0.42.2024080609 (pre-release)

Issue:

• Test explorer shows no tests.
• Clicking run briefly shows 0/1 tests passed
• Then switches to 0/0 tests

Generated project:
demo.zip

Using Maven to create a normal JUnit5 test seems to work fine, this only appears to affect cucumber tests

[evilC]

This appears to be a JUnit5 specific issue. If I switch to JUnit4, the Test Explorer works just fine

[GuillaumeSavin]

i also have this problem, i think it is not really the version of junit the problem, because when i run only pure junit5 test the test explorer work just fine but when i run cucumber test with junit5 then i think vs code is unable to parse the raw test output because of cucumber + junit5.

Here is what i get as raw test output for junit5 pure test:
%TESTC 3 v2
%TSTTREE2,fr.gsavin.vscode_test.VscodeTestApplicationTests,true,3,false,1,VscodeTestApplicationTests,,[engine:junit-jupiter]/[class:fr.gsavin.vscode_test.VscodeTestApplicationTests]
%TSTTREE3,test(fr.gsavin.vscode_test.VscodeTestApplicationTests),false,1,false,2,test(),,[engine:junit-jupiter]/[class:fr.gsavin.vscode_test.VscodeTestApplicationTests]/[method:test()]
%TSTTREE4,test2(fr.gsavin.vscode_test.VscodeTestApplicationTests),false,1,false,2,Example #1.5,,[engine:junit-jupiter]/[class:fr.gsavin.vscode_test.VscodeTestApplicationTests]/[method:test2()]
%TSTTREE5,contextLoads(fr.gsavin.vscode_test.VscodeTestApplicationTests),false,1,false,2,contextLoads(),,[engine:junit-jupiter]/[class:fr.gsavin.vscode_test.VscodeTestApplicationTests]/[method:contextLoads()]
%TESTS 3,test(fr.gsavin.vscode_test.VscodeTestApplicationTests)

%FAILED 3,test(fr.gsavin.vscode_test.VscodeTestApplicationTests)
%EXPECTS
true
%EXPECTE
%ACTUALS
false
%ACTUALE
%TRACES
org.opentest4j.AssertionFailedError: expected: but was:
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
at fr.gsavin.vscode_test.VscodeTestApplicationTests.test(VscodeTestApplicationTests.java:18)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

%TRACEE
%TESTE 3,test(fr.gsavin.vscode_test.VscodeTestApplicationTests)

%TESTS 4,test2(fr.gsavin.vscode_test.VscodeTestApplicationTests)

%TESTE 4,test2(fr.gsavin.vscode_test.VscodeTestApplicationTests)

%TESTS 5,contextLoads(fr.gsavin.vscode_test.VscodeTestApplicationTests)

%TESTE 5,contextLoads(fr.gsavin.vscode_test.VscodeTestApplicationTests)

%RUNTIME7524

Here is what i got with junit4 + cucumber:

%TESTC 5 v2
%TSTTREE1,fr.gsavin.vscode_test.Cucumber4Tests,true,1,false,-1,fr.gsavin.vscode_test.Cucumber4Tests,,
%TSTTREE2,Test,true,5,false,-1,Test,,
%TSTTREE3,Sunday is not friday #1(Test),false,1,false,-1,Sunday is not friday #1(Test),,
%TSTTREE4,Sunday is not friday #2(Test),false,1,false,-1,Sunday is not friday #2(Test),,
%TSTTREE5,Sunday is not friday #3(Test),false,1,false,-1,Sunday is not friday #3(Test),,
%TSTTREE6,Sunday is not friday #4(Test),false,1,false,-1,Sunday is not friday #4(Test),,
%TSTTREE7,Sunday is not friday #5(Test),false,1,false,-1,Sunday is not friday #5(Test),,
%TESTS 3,Sunday is not friday #1(Test)

%TESTE 3,Sunday is not friday #1(Test)
%TESTS 4,Sunday is not friday #2(Test)

%TESTE 4,Sunday is not friday #2(Test)
%TESTS 5,Sunday is not friday #3(Test)

%FAILED 5,Sunday is not friday #3(Test)
%TRACES
org.opentest4j.AssertionFailedError: expected: but was:
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
at fr.gsavin.vscode_test.steps.testSteps.i_should_be_told(testSteps.java:24)
at ✽.i should be told false(file:///C:/Users/Guisa/Documents/projects/java/test_vscode/vscode_test/src/test/resources/features/test.feature:5)

%TRACEE

%TESTE 5,Sunday is not friday #3(Test)
%TESTS 6,Sunday is not friday #4(Test)

%FAILED 6,Sunday is not friday #4(Test)
%TRACES
org.opentest4j.AssertionFailedError: expected: <false_test> but was:
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
at fr.gsavin.vscode_test.steps.testSteps.i_shouldnt_be_told_nothing(testSteps.java:29)
at ✽.i shouldnt be told nothing false_test(file:///C:/Users/Guisa/Documents/projects/java/test_vscode/vscode_test/src/test/resources/features/test.feature:6)

%TRACEE

%TESTE 6,Sunday is not friday #4(Test)
%TESTS 7,Sunday is not friday #5(Test)

%FAILED 7,Sunday is not friday #5(Test)
%TRACES
org.opentest4j.AssertionFailedError: expected: but was:
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
at fr.gsavin.vscode_test.steps.testSteps.i_should_be_told(testSteps.java:24)
at ✽.i should be told false(file:///C:/Users/Guisa/Documents/projects/java/test_vscode/vscode_test/src/test/resources/features/test.feature:5)

%TRACEE

%TESTE 7,Sunday is not friday #5(Test)

%RUNTIME7476

And here is what i got with junit5 + cucumber:
%TESTC 5 v2
%TSTTREE2,fr.gsavin.vscode_test.CucumberTests,true,1,false,1,CucumberTests,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]
%TSTTREE3,Cucumber,true,1,false,2,Cucumber,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]
%TSTTREE4,Test,true,1,false,3,Test,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]
%TSTTREE5,Sunday is not friday,true,1,false,4,Sunday is not friday,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]
%TSTTREE6,Examples,true,5,false,5,Examples,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]
%TSTTREE7,Example #1.1,false,1,false,6,Example #1.1,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]/[example:10]
%TSTTREE8,Example #1.2,false,1,false,6,Example #1.2,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]/[example:11]
%TSTTREE9,Example #1.3,false,1,false,6,Example #1.3,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]/[example:12]
%TSTTREE10,Example #1.4,false,1,false,6,Example #1.4,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]/[example:13]
%TSTTREE11,Example #1.5,false,1,false,6,Example #1.5,,[engine:junit-platform-suite]/[suite:fr.gsavin.vscode_test.CucumberTests]/[engine:cucumber]/[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]/[example:14]
%TESTS 7,Example #1.1

%TESTE 7,Example #1.1

%TESTS 8,Example #1.2

%TESTE 8,Example #1.2

%TESTS 9,Example #1.3

%FAILED 9,Example #1.3
%EXPECTS
true
%EXPECTE
%ACTUALS
false
%ACTUALE
%TRACES
org.opentest4j.AssertionFailedError: expected: but was:
at fr.gsavin.vscode_test.steps.testSteps.i_should_be_told(testSteps.java:24)
at ✽.i should be told false(classpath:features/test.feature:5)

%TRACEE
%TESTE 9,Example #1.3

%TESTS 10,Example #1.4

%FAILED 10,Example #1.4
%EXPECTS
false_test
%EXPECTE
%ACTUALS
test
%ACTUALE
%TRACES
org.opentest4j.AssertionFailedError: expected: <false_test> but was:
at fr.gsavin.vscode_test.steps.testSteps.i_shouldnt_be_told_nothing(testSteps.java:29)
at ✽.i shouldnt be told nothing false_test(classpath:features/test.feature:6)

%TRACEE
%TESTE 10,Example #1.4

%TESTS 11,Example #1.5

%FAILED 11,Example #1.5
%EXPECTS
true
%EXPECTE
%ACTUALS
false
%ACTUALE
%TRACES
org.opentest4j.AssertionFailedError: expected: but was:
at fr.gsavin.vscode_test.steps.testSteps.i_should_be_told(testSteps.java:24)
at ✽.i should be told false(classpath:features/test.feature:5)

%TRACEE
%TESTE 11,Example #1.5

%RUNTIME7578

From general the raw output seem correct and similar the only difference when it don't work i think it is in %TSTREE when junit5 + cucumber are much more "complicated" and i think it is when vs code extension don't seems to follow up

And on the following image we can just saw that vs code is stuck to CucumberTests(cucumber + junit5 test) as the icon on the left mean pending (maybe i am not sure about the icon signification), in all case the icon is different from the other one (Test: Cucumber + junit4, VscodeTestApplicationTest are junit5 pure test)

[mpkorstanje]

From general the raw output seem correct and similar the only difference when it don't work i think it is in %TSTREE when junit5 + cucumber are much more "complicated" and i think it is when vs code extension don't seems to follow up

If you are using the cucumber-junit-platform-engine, then Cucumber presents itself as an engine on the JUnit Platform. It can not be "more complicated".

However unlike the JUnit Jupiter Engine, Cucumber doesn't have class sources, but rather provides file or classpath sources. This has been tripping up various build tools and IDEs already junit-team/junit-framework#2849.

[mpkorstanje]

As expected, the the plugin is parsing JUnits UniqueId here:

vscode-java-test/src/runners/junitRunner/JUnitRunnerResultAnalyzer.ts

Lines 187 to 193 in e3eb3bd

	protected getTestId(message: string): string {
	if (message.includes('engine:junit5') || message.includes('engine:junit-jupiter') || message.includes('engine:jqwik')) {
	return this.getTestIdForJunit5Method(message);
	} else {
	return this.getTestIdForNonJunit5Method(message);
	}
	}

As this does not include the Suite Engine (engine:junit-platform-suite), we end up here:

vscode-java-test/src/runners/junitRunner/JUnitRunnerResultAnalyzer.ts

Lines 252 to 259 in e3eb3bd

	protected getTestIdForNonJunit5Method(message: string): string {
	/**
	* The following regex expression is used to parse the test runner's output, which match the following components:
	* '(?:@AssumptionFailure: |@Ignore: )?' - indicate if the case is ignored due to assumption failure or disabled
	* '(.*?)' - test method name
	* '\(([^)]*)\)[^(]*$' - class fully qualified name which wrapped by the last paired brackets, see:
	* https://github.com/microsoft/vscode-java-test/issues/1075
	*/

Which assumes there is a class name which Cucumber doesn't have obviously.

[mpkorstanje]

Also worth noting the current test uses Cucumbers JUnit implementation 4 through the JUnit Vintage Engine, not the Cucumber JUnit Platform Engine for JUnit 5.

vscode-java-test/test/test-projects/junit/src/test/java/junit5/cucumber/CucumberTest.java

Lines 7 to 9 in e3eb3bd

	@RunWith(Cucumber.class)
	@CucumberOptions(features = "src/test/resources")
	public class CucumberTest {}

For JUnit 5 this would look something like:

@Suite
@IncludeEngines("cucumber")
@SelectPackages("features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "junit5.cucumber")
public class RunCucumberTest {
}

[GuillaumeSavin]

What i mean by "more complicated" is the presence of special character like in this part "[feature:classpath%3Afeatures%2Ftest.feature]/[scenario:2]/[examples:8]" and it is a little longer than when using junit4 or just pure junit5 test.

For the test runner (junit5 + cucmber ) i use for the output i gave earlier i use pretty much what you gave:

Matt (@suite)
@IncludeEngines("cucumber")
@SelectPackages("features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "fr.gsavin.vscode_test.steps" )
public class CucumberTests {

}

[GuillaumeSavin]

i can also say that even with junit4 + cucumber, the test result don't give much more information (i don't have any trace when a test failed for exemple) i just know if the test succeed or not but it is better than with junit5 + cucumber anyway

[mpkorstanje]

Ah. No, from my comments you should take that the vscode-java-test plugin is written with the wrong assumptions about JUnit 5. And because of that it only supports engine:junit-jupiter and engine:jqwik.

[GuillaumeSavin]

ah ok thanks for the information so right now, cucumber fall in failsafe generic regex that was what you mean ?

[mpkorstanje]

Yes. And further more both getTestIdForJunit5Method and getTestIdForNonJunit5Method are trying to extract class and method information out of JUnits UniqueId. However these are merely incidental details.

On the JUnit Platform tests are described by the TestIdentifier which has separate attributes for displayName, source, tags, type, id, ect. Ideally vscode-java-test would use the source to determine the class and method name (where applicable) and treat the id as an opaque element.

Anyway, I hope that information is helpful to someone working on a fix.

[github-actions]

Hi Clive Galway (@evilC), I'm an AI Support assistant here to help with your issue. While the team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

The Java Test Runner extension doesn’t natively pick up Cucumber-JVM “RunWith(Cucumber.class)” tests. To enable discovery you can switch to the JUnit Platform engine support for Cucumber:

• Add the JUnit Platform engine dependency in your pom.xml:

<dependency>
  <groupId>io.cucumber</groupId>
  <artifactId>cucumber-junit-platform-engine</artifactId>
  <version>7.11.1</version>
  <scope>test</scope>
</dependency>

• Create a JUnit Platform suite class (e.g. RunCucumberTest.java):

import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.Suite;
import static io.cucumber.junit.platform.engine.Constants.*;

@Suite
@SelectClasspathResource("features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "com.example")
public class RunCucumberTest { }

• (Optional) Add a junit-platform.properties under src/test/resources:

cucumber.features=classpath:features
cucumber.glue=com.example

After this the Test Explorer should list and run your Cucumber scenarios via JUnit Platform.

No existing issue appears to be a true duplicate.

Other references with low confidence

• Issue Test report doesn't display latest test result when Cucumber + JUnit5 is used #1008: “Test report doesn’t display latest test result when Cucumber + JUnit5 is used” – similar Cucumber+JUnit 5 setup but focuses on reporting, no definitive solution.
  Test report doesn't display latest test result when Cucumber + JUnit5 is used #1008

The team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!