Customize junitxml Pytest report to add multiple attributes from custom marker

[utkjad]

Using Pytest==3.2.3 on Ubuntu 16.04
Suggestion, bug?, enhancement,"How-do-I-do-this-in-pytest"

Each test case in my test suite has multiple attributes associated with it that I wish to include in XML(junit-xml) report. Following code snippet gives a clear picture about it.

@data(*get_csv_data("csv/blah.csv"))
@unpack
@pytest.mark.run(order=70)
@pytest.mark.webtest.with_args(jira="QA5555", second="second_arg_add")
def test_your_stuff(self,arg1, arg2):
    # Actual Test 

While customizing reports, it is easy to add attributes in pytest-html plugin, by adding parameters in extras attribute of pytest-html Following code snippet gives clear picture about it.

# conftest.py
@pytest.mark.hookwrapper
def pytest_runtest_makereport(item, call):
  pytest_html = item.config.pluginmanager.getplugin('html')
  outcome = yield
  report = outcome.get_result()
  extra = getattr(report, 'extra', [])
  if report.when == 'call':
     extra.append(pytest_html.extras.text("<blah>")
     report.extra = extra

Also, I can easily get the attributes by following item.keywords.get('webtest').kwargs How can I do the same for junitxml?

A few findings -

1. junitxml does not have extras
2. record_xml_property - I do not want to use it as it is a feature in test phase. Also adding multiple arguments with decorator looks like a nice approach. I do not want to hamper code readability by adding multiple record_xml_property calls in the test.

def test_function(record_xml_property):
    record_xml_property("key", "value")
    assert 0

3. The following approach

if hasattr(request.config, "_xml"):
request.config._xml.add_custom_property(name, value)

I was not able to get hold of request object here in the hook
def pytest_runtest_makereport(item, call):
The item.config._xml points to LogXML in junit-xml which in turn has no method add_custom_property associated with it.
So, What would be the best way to add more attributes to junitxml, so that it would roughly look like this -

<testcase classname="test_function" file="test_function.py" line="0" 
          name="test_function" time="0.0009">
  <properties>
    <property name="jira" value="1234566" />
  </properties>
</testcase>

or like this

<testcase classname="test_function" file="test_function.py" line="0"
          name="test_function" time="0.0009" jira="2345667">
</testcase>

[RonnyPfannschmidt]

currently this is not supported but its a nice to have feature,

unfortunately i wont be able to participate in making it anytime soon

[utkjad]

@RonnyPfannschmidt
Thanks a lot for the response.
A few follow-up questions -

1. Current Workaround - I guess if I get hold of request object in def pytest_runtest_makereport(item, call): or def pytest_runtest_logreport(report): ;
   there are chances that the following code might work fine.

if hasattr(request.config, "_xml"):
request.config._xml.add_custom_property(name, value)

Any input?

Also, if we get hold of request object, we can certainly inject it in record_xml_property. This is because of the signature of the record_xml_property, which takes request as input.

What is this request object? What is the purpose of this? How can we get hold of it in a hook?

2. Why is junitxml plugin not a separate plugin like pytest-html?

[RonnyPfannschmidt]

request is the fixture reuqest for the current fixture/test it brings along tools to manage things

requests are typically not availiable in hooks

junitxml is currently not separate for historic reasons, we ought to consider changing that in future

[utkjad]

@RonnyPfannschmidt Coolio!
About the current workaround, anything that I should try to do?

First Approach
I was thinking to emulate behavior of record_xml_property LINK
I can get logXML object and and emulate it.

logxml = getattr(item.config, "_xml")
node_reporter = logxml.node_reporter(item.nodeid) 

I am assuming return node_reporter.add_property is very much internal to fixture functionality and should not be hooked/overwritten/touched ? If it is not the case, how can I connect the dots here? Should I really follow this?

Second Approach
Add as much as you can in html report and convert it to XML as post processing tool. It would be a pain for sure, but doable, Any input on this?

In conclusion, what in your opinion, would be the best way for workaround as of now?

Edit: Just adding @nicoddemus because of this

[RonnyPfannschmidt]

right now i dont have a clear oppinion and i wont find a time window in near future to take a look at the full problem again

[nicoddemus]

@utkjad I think your first approach is the best workaround that can be done at the moment.

A possible enhancement regarding junitxml would be to add the same extra support that pytest-html has been using, which I think has been successful so far.

@davehunt any comments about junitxml using the same extra attribute?

[davehunt]

@nicoddemus pytest-html was originally based on junitxml, so it feels natural for some of the features to find their way back. Using report.extra may have unexpected effects if using both junitxml and pytest-html though, so perhaps this should be namespaced? Sorry if I've misunderstood, it's been a while since I've looked at this code. 😕

[nicoddemus]

Indeed there's the namespace issue, plus what can be and cannot be supported by each output format.

[simplesomeone]

This would be lovely to have. Here's a use-case I have:

My company has a test-suite of 2000 pytests, each with markers representing the JIRA ticket they are associated to. For example:

@pytest.mark.OD-4000
def test_mytest():
...

We run this regression suite and submit our JUNIT XML results to a test-case management tool: https://www.qasymphony.com/software-testing-tools/qtest-manager/test-case-management/

The test-case management tool offers an API in which we could submit test cases and their associated JIRA tickets.

However, we currently don't have a way of getting these markers :/