B909 improvements

[Martin Imre]

Note Rule B909 originally was B038, but after some discussion, it was decided that we make it optional and it's B909 now (see #456).

I have not updated the content of this issue, so please read "B038" as "B909".

---

Description

With the introduction of B038 (#445), I initially introduced a bug causing many false positives, which should be fixed with #453.
#453 implements a check for the methods listed in Python Builit-in Mutable Sequence Types documentation, but not the operators.
However, #453 left out some mutations for the sake of getting a fix to the flood of false positives (#451) out asap.

In this issue, I will propose further changes to harder B038, to detect more (hopefully all) mutations to iterables that the python builtins allow.

Assignment Operations

This is a very obvious one that was not implemented yet, and I wonder how I could've overseen it.
This part will cover the following operations:

for _ in foo:
  foo = bar
  foo *= bar
  foo += bar
  foo[i] = bar
  foo[i:j] = bar
  foo[i:j:k] = bar
  foo |= bar
  foo &= bar
  foo -= bar
  foo ^= bar

The implementation will check if any ast.Assign and ast.AugAssign have the name or a subscript of the name of the iterable as target.

Further, to also check for annotated assignments, the plan is to inspect ast.AnnAssign nodes that are not simple (ie have simple = 0).
simple = 1 are nodes of the form foo: bar, whereas simple = 0 have any of the following forms:

for _ in foo:
  foo: t = bar
  foo: t *= bar
  foo: t += bar
  foo[i]: t = bar
  foo[i:j]: t = bar
  foo[i:j:k]: t = bar
  foo: t |= bar
  foo: t &= bar
  foo: t -= bar
  foo: t ^= bar

I'm not sure if all of the above are legal syntax, but the implementation will likely look very similar and thus would cover all those cases out of the box.

Further, everything of the above also applies to object attributes, ie, we'd also cover foo.bar = -esque assignments if foo.bar is the loop iterable.

Container-Specific Operations

#451 already implements a check for 2 container specific operations: list.sort() and dict.popitem() .
Further operations are missing:

• dict.setdefault
• dict.update
• set.update
• set.intersection_update
• set.difference_update
• set.symmetric_difference_update
• set.add
• set.discard
• object.__setrattr__
• object.__delrattr__

These all can be implemented similarly to the ones in #451 by adding them to the MUTATING_FUNCTIONS list of the B038Checker.

The nasty ones

These are functions that are not used often, or just don't fall into the scheme of the ones listed above:

• delattr
• setattr

These will need special handling in visit_Call of the B038Checker, but shouldn't be too complicated.

Discussion

I hope my proposal covers all the mutations that are doable with the python built-ins.
If not please let me know.

One special mention goes to del x - this is actually not mutating the iterable itself and only deletes the reference to the iterator.
The following snippet runs without problems

>>> a = [1,2,3]
>>> for i in a:
...   if i == 1:
...     del a
...   print(i)
... 
1
2
3

However, def x[y] does mutate x.
Should this case distinction be made? Currently both would be reported.

I also would like to refactor the code for B038 a bit to just make it far more readable, currently it's a bit messy 😅

---

Please let me know what you think about the proposed changes and whether I should go ahead and implement them 🙂

[Jelle Zijlstra]

Thanks for writing this up.

Assign and AnnAssign can be ignored, since they just rebind the name, they do not mutate the container. (foo: T += 3 and similar are syntax errors.) Similarly, I wouldn't worry about the attribute setting mechanisms (setattr/delattr/etc.), because you can't set an attribute on builtin containers like list, tuple, set, and dict. There could be user-defined subclasses where setting an attribute interferes with iteration, but that feels too speculative for a lint rule.

[Martin Imre]

A good catch with Assign and AnnAssign.

Regarding the syntax errors: you're right - AugAssign and AnnAssign cannot be mixed.

What do you think about del?
Should we also remove that, when it only targets the container itself instead of an element inside of it?

[Jelle Zijlstra]

We should catch del container[i], but not del container.

[Martin Imre]

Sounds good to me.

I'll implement these improvements some time next week 🙂