Skip failed tests

[mwisnicki]

I would like to be able to mark certain failed tests as skipped/ignored.
The use case is to keep executing and reporting errors from some flaky tests (until they are fixed of course) but without failing the build.

Currently it is only possible to skip/ignore tests before they are run.
Once executed I don't think there is a way to change outcome to skipped in InvocationInterceptor or TestExecutionExceptionHandler.

[sormuras]

It should work using a custom TestExecutionExceptionHandler extension. Did you try it? Here are some resources:

• https://junit.org/junit5/docs/current/user-guide/#extensions-exception-handling
• https://github.com/junit-team/junit5/tree/main/documentation/src/test/java/example/exception

If everything else doesn't work, you may always resort to a try-catch in your test method and "handle" appropriate exception calling Assumptions.abort(...).

[mwisnicki]

That will report test as successful. I'm trying to report test as failed or at least skipped.

[sormuras]

Here are two examples showing an instance of RuntimeException, that would trigger a test to be marked as failed, being transformed into an aborted test - which in turn is interpreted (by IDEA) as skipped tests:

  @Test
  void f() {
    try {
      throw new RuntimeException("might happen");
    } catch (RuntimeException exception) {
      Assumptions.abort("f");
    }
  }

  @Test
  @ExtendWith(AbortOnRuntimeException.class)
  void g() {
    throw new RuntimeException("might happen");
  }

  public static class AbortOnRuntimeException implements TestExecutionExceptionHandler {
    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable)
            throws Throwable {
      if (throwable instanceof RuntimeException exception) {
        Assumptions.abort(exception.getMessage());
      }
      throw throwable;
    }
  }

[mwisnicki]

Sorry, I missed Assumptions tip. That indeed marks test as skipped!

However the message is not added to report.
In fact event standard @Disabled("reason") is not added to report.

Is this a bug?

[mwisnicki]

Actually I'm running it through gradle and looks like gradle does its own report writing that's not quite complete - gradle/gradle#4605