ScalaTest fails to fail in Firefox and Chrome

[gshakhn]

Not sure if this is a ScalaTest bug or a scala-js-env-selenium bug, but a failing test fails to report the failure to sbt. Therefore the build passes when it shouldn't.

Minimal project is available at gshakhn/scalajs-selenium-scalatest-failure

[nicolasstucki]

When debugging the minimal case I found issue #47, which might be producing this bug. But in that case the fails are only in Chrome and they do report the failure to SBT.

[nicolasstucki]

@gshakhn have you tried using parallelExecution in Test := false?

[gshakhn]

@nicolasstucki Just tried that in gshakhn/scalajs-selenium-scalatest-failure@3e31364. It still fails to fail on my machine. Does it work on yours?

For what it's worth, the original bug I hit was only in Chrome. It reported the Firefox failures to SBT appropriately, failing the build correctly. Not sure why the simplified example fails in both browsers.

[nicolasstucki]

Found the source of the issue, it is in the ScalaTest custom StackDepthExceptionHelper for Scala.js. In the method StackDepthExceptionHelper.getStackDepth there are some accesses stackTraces: Array[StackTraceElement] that can be out of bounds.

[nicolasstucki]

@gshakhn you should report this issue on the ScalaTest issue list.
The key part of the exception is the TypeError: stackTraces.u[((depth1 + 3) | 0)] is undefined which implies that one of the stackTraces(depth1 + 3) array accesses in StackDepthExceptionHelper.getStackDepth is out of bounds.

[jatcwang]

I think I am hitting the same issue as well when one of my tests fails in Scalatest, running on node. Just commenting here so all the info is in one thread instead of two.

scala.scalajs.js.JavaScriptException:
[info]   at org.scalatest.exceptions.StackDepthExceptionHelper$.getFailedCodeFileName(/home/.../js/target/scala-2.11/https:/raw.githubusercontent.com/scalatest/scalatest/v3.0.0-M15/scalatest.js/src/main/scala/org/scalatest/exceptions/StackDepthExceptionHelper.scala:44:85)
[info]   at org.scalatest.exceptions.StackDepth$class.failedCodeFileName(/home/.../js/target/scala-2.11/https:/raw.githubusercontent.com/scalatest/scalatest/v3.0.0-M15/scalatest.js/target/scala-2.11/src_managed/main/scala/org/scalatest/exceptions/StackDepth.scala:77:91)