[Feature Request] Aggregate by exception

[3tilley]

I don't know if this is the right place to do this, but I'd love a way of aggregating failures by exception type, message, or error line. In a large codebase, a small change can generate a cascade of failures that actually are only down to errors in one or two places. It would be really helpful to be able to see that although 50 tests failed, 40 of them were from a db connectivity error and the other 10 arose from the same error in a utility function.

The alternative would be to tackle it at the CLI level, but I like the idea of being able to dynamically aggregate by different views to work out what's going on.

Happy to aid in the development if this would be a welcome feature.

[RibeiroAna]

I think it would be a great feature! You could create an option --aggregate-by-exception or something like that, so people who don't wish to have this could still see the tests work. The biggest difficulty for me is how to know if two errors are the same, we use a Test Report and I don't see there a ErrorType or ExceptionType there. Do you have any ideia on how to do it? I don't think it is a good thing to compare the longrepr, because there is things specific to the test case. If you don't have any ideia, we may ask to a more experienced pytest developer.

[3tilley]

It's a good question. Either add it upstream or use regex to pick it out? What do you think about it not being a CLI flag, but instead done entirely with JS on the page?

…

On Wed, 23 Jan 2019, 00:24 Ana Ribeiro ***@***.*** wrote: I think it would be a great feature! You could create an option --aggregate-by-exception or something like that, so people who don't wish to have this could still see the tests work. The biggest difficulty for me is how to know if two errors are the same, we use a Test Report <https://github.com/pytest-dev/pytest/blob/ba452dbcf01f16f1bdad1950c91977298588712d/src/_pytest/reports.py#L99> and I don't see there a "ErrorType" there. Do you have any ideia on how to do it? I don't think it is a good thing to compare the longrepr, because there is things specific to the test case. If you don't have any ideia, we may ask to more experienced pytest developers. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#195 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABFfSNL2hox9aeVI2aQYkAVt_tZbuNWkks5vF6vagaJpZM4aNuPs> .

[BeyondEvil]

In the pytest_runtest_makereport hook, in the call parameter you can access the exception type (and more) in call.excinfo. @3tilley @RibeiroAna

[RibeiroAna]

@3tilley as you saw, it is easier to do in the plugin because there is this hook, on the JS side is going to be much more complicated, less efficient, and people who has restriction on JS use on their browsers won't be able to see. I think you may start working on this.

[RibeiroAna]

@3tilley I think I've had a better insight on this! Maybe it can be a combination of JS and Python. For example, for errors of a certain type, you could create a HTML class "error-x" and them add it to every table row that has this kind of error and use JS to filter them, as there is now to filter according to the outcome. I think it wont change so much the way people use Pytest-HTML, so it wouldn't be necessary a parameter or something like that. I think that way would be better.

@davehunt maybe you want to give your input before @3tilley starts working on this?

[3tilley]

That sounds good to me, I think we're almost there. My concern here is that actually there are lots of different levels you might want to aggregate at. Just exception type, exception message, first half of exception message etc. Would this mean you have each of these as separate html metadata? Imagine you wanted to see all the tables you didn't have access to, and there were a whole load of failures like: * SqlError: User x cannot read column A from product_table * SqlError: User x cannot read column C from product_table * SqlError: User x cannot read column Q from meta_table Grouping by SqlError wouldn't be good enough, and grouping by full text wouldn't quite cut it either. What the developer would really want to see would be two groups, one for product_table and one for meta_table. Would there be any way they could supply a regex, or we could do some kind of clever grouping?

…

On Thu, 24 Jan 2019, 03:09 Ana Ribeiro ***@***.*** wrote: @3tilley <https://github.com/3tilley> I think I've had a better insight on this! Maybe it can be a combination of JS and Python. For example, for errors of a certain type, you could create a HTML class "error-x" and them add it to every table row that has this kind of error and use JS to filter them, as there is now to filter according to the outcome. I think it wont change so much the way people use Pytest-HTML, so it wouldn't be necessary a parameter or something like that. I think that way would be better. @davehunt <https://github.com/davehunt> maybe you want to give your input before @3tilley <https://github.com/3tilley> starts working on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#195 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABFfSCSiduP71pdtTQHgApk2pgmXFJOiks5vGSPRgaJpZM4aNuPs> .

[BeyondEvil]

I think a good approach would be to start simple, by for example just aggregate by exception. Then build from there. @3tilley