Distinguish between failed and broken tests when using a 3rd party assertion library with Allure.NUnit and Allure.Xunit

[I37937]

I'm submitting a ...

• bug report
• feature request
• support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

The test case failed using fluentassertions or Playwright are shown as broken instead of failed

If the current behavior is a bug, please provide the steps to reproduce and, if possible, a minimal demo of the problem

Create a simple test using FluentAssertions or Playwright dotnet, when you run the tests the resutl is shown as broken

What is the expected behavior?

The test should be failed if I have configured the list of exceptions (failExceptions).

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

• OS Windows
• .NET framework .NET 8
• Test framework: nunit@4.0.0
• Allure adapter: allure-nunit@2.12.0

Other information

I was taking a look to the AllureNUnitHelper and I was able to see that the IsBroken function is waiting elements in result.Assertions list but when you use FluentAssertions or Playwright this list is not update so the tests are always mark as broken.

[gamerka]

Did you check this page?
https://allurereport.org/docs/nunit-configuration/#allurefailexceptions

[I37937]

Yes, I do, I have added the exceptions but I still having the same issue

[arman-avetisyan]

Hello, I have the same issue,
I am using Xunit, Allure.Net.Commons and Shouldly
When I run a test with this simple step, and generate an Allure report
the test status displayed as Broken

Also, I tried to add the Shouldly.ShouldAssertException to the allureConfig.json, the step becomes Failed but the test is still displayed as Broken

[flydyk]

Looks like this code is making a test report broken despite the settings
https://github.com/allure-framework/allure-csharp/blob/main/Allure.Xunit/AllureXunitHelper.cs#L39-L42

            var status = failure.ExceptionTypes.Any(
                exceptionType => !exceptionType.StartsWith("Xunit.Sdk.")
            ) ? Status.broken : Status.failed;

[S-Kulyk]

The same issue with Nunit & FluentAssertions
It seems that the Nunit does not track the exception type. It has a stack trace in a message but doesn't hold an exception type directly. IMO, it would be convenient to detect Failed/Broken status by Nunit's ResultState.Failure/Error respectively

Also, I wonder if there is any workaround to have a "Failed" status instead of "Broken"?

[ohlori]

Any chance there's a timeline for this?

[Dimitri-M-Dumont]

Patiently waiting on a fix for this too. 😅