Document mypy's exit code

[jp-larose]

Feature

(This is related to #6003)

Could there be an exit code convention akin to what pylint does?
https://pylint.readthedocs.io/en/latest/user_guide/usage/run.html#exit-codes

Pitch

What I like about this is that it distinguishes two different variations of pylint execution errors along with different categories of linting errors, and they can be combined by bitwise addition.

I found a project that elegantly uses these pylint exit codes to allow my CI script to pass even if there are linting issues in the code, but fail if pylint doesn't run correctly.
https://pypi.org/project/pylint-exit/
I'm not suggesting the mypy team create such a tool (but also not discouraging it), but just showing a proof of concept for how exit codes can be used more broadly.

This allows for easier control over what constitutes a failing CI job. In my case, I want the job to pass if mypy does its job and identifies static typing errors, but fail if mypy fails to run properly. If there are type-checking errors, I can look at the report later.

However, I could see a use for saying: fail if these types of type-checking errors are found, but report and pass otherwise, and fail if mypy just doesn't work (e.g. due to bad config).

[A5rocks]

As a hack, I suppose you could run it once with some set of error codes, and then rerun it (ignoring exit code) with another? That'll take twice as long of course and is more fragile.

[hauntsaninja]

mypy fails with exit code 2 if there's an issue with how you're using mypy and exit code 1 for type checking issues detected

[hauntsaninja]

I guess we don't mention this in the docs, so changing this to a docs issue

[hauntsaninja]

Oh wait, this is the exact same as #6003 that you link. Now I'm confused.

[jp-larose]

#6003 is about documenting exit codes.

This is about expanding exit codes.

I was looking for more fine grained breakdown of categories, e.g.

• untyped definitions
• incorrect typing
• type inference error

The specifics probably need further discussion.

A couple people pointed out that using exit code 1 for type errors and 2 for mypy errors doesn't work for uncaught exceptions.

[hauntsaninja]

Hm, I don't think there's a way to compress that information into a single number that really makes sense. Unlike pylint we only really have errors and they don't have a priority. You're probably better off parsing error codes from the messages and aggregating them based on the logic you want.

By uncaught exceptions do you mean crashes? Those also have exit code 2 (and are rare, fixing them is about the highest priority thing that we do).