JUnit report treats all severities as failures

[InvisibleSmiley]

First of all, thanks for this great tool and the JUnit report format output option. Much appreciated!

We use the JUnit report format to create reports for Jenkins so that a build fails if an OpenAPI file fails to pass linting.

Not all issues that are found are of the same importance to us, so we use a custom ruleset to disable or change/downgrade the serverity of certain rules.

However it turns out that the JUnit report creates failure tags in the output XML for all severities.
For "error" that is clear, for "warn" it is debatable, but for "info" and "hint" it is outright wrong. Those should not be treated as failures.
The failure tags include the severity as "type" attrribute but that makes no difference to applications that interpret the result XML. All they see is a failure. AFAIK, applications commonly only discriminate between errors, failed and skipped tests.

As a workaround we needed to disable all rules that we actually only wanted to downgrade in severity.

Ideally I'd like to have only "error" result in a JUnit report failure, "warn" maybe depending on an extra flag, and "info" and "hint" getting ignored for this report format. The latter two severities would then only appear in "lint" output.

Regarding the "treat warnings as errors" flag, maybe let yourself inspire by this Spectral issue: stoplightio/spectral#935

[InvisibleSmiley]

In the end what I need is a way to "sync" the exit code of the "lint" command with the presence/absencse of "failure" tags in the JUnit report XML. If the former returns 0, the latter should be absent and vice-versa.

[evilz]

As said before this tool as great and junit IS very cool for CI.
Do you think it's possible to add pass test when check is OK. Cause it looks so red 😅