Compile error in test run ends up with "tests passed" message

[viktorasm]

If running tests on multiple packages at once, and one of them has compilation problem, you end up with:

Testing started at 4:55 PM ...
# nevercrashed.com/locationservice
locationservice/locationResource.go:96: undefined: asdf
PASS
ok      nevercrashed.com/currencyservice    0.009s
FAIL    nevercrashed.com/locationservice [build failed]

The UI will not report of any problem, you'll just get "tests passed" message and a green test result bar. Any way to get a visual indication that there was a problem, as well as special case of Event Log message?

[zolotov]

At the moment go test is too pure to catch such cases. Maybe golang/go#2981 will help us in future

[mdwhatcott]

I sympathize with everyone who wishes that go test had a -json flag.

Doesn't go test return a non-zero exit code when compilation fails? It seems like that would be enough to signal a failed test run and have the UI respond accordingly.

---

BTW, I'm the author of GoConvey, which parses the current output of go test and turns it into an internally defined json structure--not the nicest thing to deal with but it works. I created GoConvey because I wanted a test runner like the one built into IntelliJ (I didn't know about the golang plugin back then, and it was probably very early in its development--but I'm hooked now!).

So, the go test output could certainly be parsed in the meantime, even if the approach is somewhat problematic (and fragile by nature).

[zolotov]

Doesn't go test return a non-zero exit code when compilation fails?

Not sure. Also I'm not sure whether it's allowed by IntelliJ API to explicitly fail passed tests. Anyway it doesn't look like a proper solution. Ideally, IDEA should group tests in packages while testings, in this case IntelliJ test framework will see that package doesn't contain any tests but error message and will mark it red.

which parses the current output of go test and turns it into an internally defined json structure

How does it work? Is this streaming parser?

So, the go test output could certainly be parsed in the meantime, even if the approach is somewhat problematic (and fragile by nature).

When you have whole output – yes, when you want to parse it on the fly – no.

[mdwhatcott]

Yes, go test exits with a return code of 2 when compilation fails. It exits with a 1 when tests fail. A 0 is reserved for compilation and tests succeeding.

The approach I took with GoConvey did not try to stream the output. I run go test <package> in a subprocess, then parse the complete output of that package's tests and send them to the browser.

The question is: What's worse, not having streaming output or seeing a fase positive in the form of "Tests Passed"?

Here's a demonstration of the GoConvey UI in action:

https://www.youtube.com/watch?v=wlUKRxWEELU

[zolotov]

Yes, go test exits with a return code of 2 when compilation fails. It exits with a 1 when tests fail. A 0 is reserved for compilation and tests succeeding.

Ok, I'll investigate whether it can be used

I run go test in a subprocess

We have no problems with that. The problems started when you run go test <directory/...

The question is: What's worse, not having streaming output or seeing a fase positive in the form of "Tests Passed"?

There is no such choice, it's different issue. But yes, streaming is very important. If we have no streaming – we have no test reporting, nobody would use IDE for run test if it can't show the progress during the test. So we cannot reuse goconvey parser.

Yet another solution for both issues: make a custom test runner, that will be handle test runs on directory, collect all packages, run them one by one and wrap each of them output with some custom lines.

[mdwhatcott]

Thanks for the clarification. Good luck with this--I'm excited to see how this gets sorted out.