Default "--soft-error-limit" behavior causes tons of headache for mypy newbies with large code bases

[sveinugu]

Bug Report

While not strictly a bug, I would argue that there is a "bug" in the logic behind the decision to add a default --soft-error-limit of 200 (added in #10579). The idea was to make it easier to identify missing stubs as the root cause for a flood of errors, especially useful when settings are strict. However, I would assume that strict settings are typically used by mypy experts while newbies would typically start with the default settings and gradually work their way up to strict settings (if at all). So while the default values of the strictness settings favor newbies, I think the default behavior of --soft-error-limits inconsistently (and unhelpfully) favors experts at the expense of newbies.

Furthermore, this error limit applies to all errors, regardless of whether they are caused by missing stubs or not. This leads to very confusing error outputs for large code bases that are in the process of being typed by mypy newbies. Since Python typing already is a complicated heap of (insert word here), even for devs with tons of experience in non-typed Python, I would think it is of importance for the mypy project to try to lower the threshold for adoption by newbies as much as possible. However, the default setting of --soft-error-limit does the opposite: It is counterintuitive that mypy by default has an error limit that silences errors.

The default behavior is especially confusing when mypy is used in the context of an IDE such as PyCharm (also mentioned by @ilevkivskyi in this comment #10579 (comment)). In this case, the error messages are filtered by module name so that the note (Skipping most remaining errors due to unresolved imports or missing stubs; fix these first) will appear in a random module and most probably not be read by the user. In my personal experience, trying to debug the discrepancy between the inline mypy errors and the lack of the same errors in the global view made me in the end want to pull out my last few remaining strands of hair rather than continue the hunt. This particular discrepancy was one of the worst issues I have ever tried to debug, as there were no patterns to which errors were silenced. Debugging, on the other hand, is typically done by trying to find the pattern that cause the bug to appear:

Example debugging session: In my particular debugging approach, I tried figuring out why an inline error in a particular module was silenced in a global mypy run by carrying out a binary search of including/excluding modules to locate the exact reason for the silencing, while grep-ing the output on the specific module I selected. grep-ing of course silenced the real error message (however it did appear in my module of interest once in a while, but with no apparent pattern). In the end I had a set of unrelated modules where, for some unexplained reason, removing any one of them from the mypy run would "un-silence" the error for the module of interest. In the end, I concluded that there was no pattern, there had to be a limit somewhere. Googling the note text mentioned above also did not directly lead me to the answer, that also took a while. I am sure other approaches to debugging this issue can be equally or more confusing, as exemplified in issues #10956 and #14389.

To Reproduce

As a mypy newbie, run mypy on a large code base that have been partially or fully typed, but where detailed fixing of mypy errors have been postponed due to the priority of developing actual features. To increase the pain level to the maximum, do this in PyCharm or other IDE where errors are filtered by module. Then:

1. Be motivated that there are not more than around 200 errors in the whole code base!
2. Start fixing errors in a module of choice
3. Discover that the inline errors from the module do not appear in the global view
4. Try to debug the discrepancy: choose your favorite debugging method and whatever pattern seems most likely to you as the cause
5. Discover that the more you debug, the less sense you are able to make of the issue
6. Start questioning your own sanity!
7. Alternatively, start questioning the sanity of the mypy library
8. (If you endure wasting hours of your precious time): Find the actual cause of the bug. Set --soft-error-limit=-1 and witness the total number of errors multiply
9. Despair at the real number of errors you need to fix!

Expected Behavior

1. All errors are reported by default
2. (Alternatively) All errors not caused by missing stubs are reported by default
3. If --soft-error-limit is set, a note should be added at the very end of the error output that also mentions the actual config/command-line setting to tweak
4. A config setting soft_error_limit should be added that corresponds to the command-line flag
5. Log output at verbosity=1 should, for each module, log whether any errors were silenced due to the setting

Actual Behavior

No more than 200 errors are reported (in my case only 139 errors and 41 notes were reported, for some reason). Errors are randomly silenced in global runs for no apparent reason and with no easily noticed error messages. Reading the log at verbosity=1 gives no clue to the reason why errors are silenced.

Your Environment

• Mypy version used: 1.1.1
• Mypy command-line flags: --show-column-numbers --follow-imports silent (default for PyCharm mypy plugin)
• Mypy configuration options from mypy.ini (and other config files):
  mypy_path = "src:tests"
• Python version used: 3.10

[JukkaL]

You convinced me that the soft error limit can be confusing and should not be enabled by default. Let's just disable it by default as a first step, and we can improve it later.

[toryth]

I can add to this. I'm a mypy newb. For some time prior to this comment I thought --follow-imports=silent was actually mysteriously enabling some reveal_type() usages to work. Now that I know about this setting I'm realizing it's not that --follow-imports=silent was enabling it to work, it was actually just enabling me to be under the threshold.

I had originally missed "Skipping most remaining errors due to unresolved imports or missing stubs; fix these first" in part because I was grepping for "Reveal".