ParserSyntaxError when a module have a "from django.utils import timezone" import
luzfcb opened this issue · comments
Describe the bug
I'm migrating a medium-sized codebase, and I found this strange behavior: if the code has a "from django.utils import timezone" import, a ParserSyntaxError is raised
$ pre-commit run -a
djcodemod................................................................Failed
- hook id: djcodemod
- exit code: 1
Running codemods: AssignmentTagTransformer, CookieDateTransformer, FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, InlineHasAddPermissionsTransformer, ModelsPermalinkTransformer, OnDeleteTransformer, QuerySetPaginatorTransformer, SignalDisconnectWeakTransformer, URLResolversTransformer
Executing codemod...
Finished codemodding 62 files!
- Transformed 62 files successfully.
- Skipped 0 files.
- Failed to codemod 0 files.
- 0 warnings were generated.
Running codemods: AssignmentTagTransformer, CookieDateTransformer, FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, InlineHasAddPermissionsTransformer, ModelsPermalinkTransformer, OnDeleteTransformer, QuerySetPaginatorTransformer, SignalDisconnectWeakTransformer, URLResolversTransformer
Executing codemod...
Codemodding /home/luzfcb/projects/big_project/foobar/tests/test_managers.py
Traceback (most recent call last):
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(import)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_cli.py", line 295, in _execute_transform
output_tree = transformer.transform_module(input_tree)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_command.py", line 88, in transform_module
tree = self._instantiate_and_run(transform, tree)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_command.py", line 58, in _instantiate_and_run
return inst.transform_module(tree)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
return self.transform_module_impl(tree_with_metadata)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
return tree.visit(self)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_nodes/module.py", line 91, in visit
result = super(Module, self).visit(visitor)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_nodes/base.py", line 235, in visit
leave_result = visitor.on_leave(self, with_updated_children)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/matchers/_visitors.py", line 512, in on_leave
retval = CSTTransformer.on_leave(self, original_node, updated_node)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_visitors.py", line 72, in on_leave
updated_node = leave_func(original_node, updated_node)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/visitors/_add_imports.py", line 393, in leave_Module
*[
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/visitors/_add_imports.py", line 394, in <listcomp>
parse_statement(
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 100, in parse_statement
result = _parse(
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
result = parser.parse()
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 111, in parse
self._add_token(token)
File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 187, in _add_token
raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:7.
Incomplete input. Encountered 'import', but expected '.', '...', or 'NAME'.
from import datetime
^
Failed to codemod /home/luzfcb/projects/big_project/foobar/tests/test_managers.py
To Reproduce
pipx install "django-codemod==1.5.5"
echo "from django.utils import timezone" > foobar.py
djcodemod run --deprecated-in 1.9 --deprecated-in 1.10 --deprecated-in 1.11 --deprecated-in 2.0 --deprecated-in 2.1 --deprecated-in 2.2 foobar.py
Additional context
Hi @luzfcb! Thanks for the report. That's a stange one, I cannot reproduce with the CLI installed in my development environment for the project, however, I'm able to do so if I install django-codemod with pipx, like you do.
I've removed a few options you've used and I tracked down the bug to --deprecated-in 2.2, and then to the FixedOffsetTransformer codemodder. You'd get the bug with this:
djcodemod run --codemod FixedOffsetTransformer foobar.py
I'll take a look, but in the meantime you can avoid passing --deprecated-in 2.2...
I came here to report this bug too. Using the latest version:
❯ python --version
Python 3.7.10
❯ django-admin --version
3.0.14
❯ djcodemod run --removed-in=3.1 ./blog/models.py
Running codemods: FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, QuerySetPaginatorTransformer
Executing codemod...
Codemodding /Users/jefftriplett/.virtualenvs/trailhawks.com/src/trailhawks.com-git/blog/models.py
Traceback (most recent call last):
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(import)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_cli.py", line 295, in _execute_transform
output_tree = transformer.transform_module(input_tree)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_command.py", line 88, in transform_module
tree = self._instantiate_and_run(transform, tree)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_command.py", line 58, in _instantiate_and_run
return inst.transform_module(tree)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
return self.transform_module_impl(tree_with_metadata)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
return tree.visit(self)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_nodes/module.py", line 91, in visit
result = super(Module, self).visit(visitor)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_nodes/base.py", line 235, in visit
leave_result = visitor.on_leave(self, with_updated_children)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/matchers/_visitors.py", line 512, in on_leave
retval = CSTTransformer.on_leave(self, original_node, updated_node)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_visitors.py", line 72, in on_leave
updated_node = leave_func(original_node, updated_node)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/visitors/_add_imports.py", line 404, in leave_Module
for module, aliases in module_and_alias_mapping.items()
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/visitors/_add_imports.py", line 405, in <listcomp>
if module != "__future__"
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/entrypoints.py", line 105, in parse_statement
detect_default_newline=False,
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
result = parser.parse()
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/base_parser.py", line 111, in parse
self._add_token(token)
File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/base_parser.py", line 191, in _add_token
raw_column=token.start_pos[1],
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:7.
Incomplete input. Encountered 'import', but expected '.', '...', or 'NAME'.
from import datetime
^
Failed to codemod /Users/jefftriplett/.virtualenvs/trailhawks.com/src/trailhawks.com-git/blog/models.py
If it's helpful, I have a public project you can reproduce it on: https://github.com/trailhawks/trailhawks.com
Yes I think I've nailed down the problem to this section, it's just wrong (it replaces the import for the parent module, even if the thing being codemodded isn't used in the file):
django-codemod/django_codemod/visitors/base.py
Lines 167 to 175 in 7942036
I'll probably disable this feature and push a new release (hopefully tomorrow) until I can make a fix, this is probably breaking a lot of other things.
I think the issue can be avoided by using django-codemod<1.4 since it was most likely introduced in #347
The problem should no longer happen in 1.5.6, I've disabled the problematic feature for now. I'll try to fix it properly later, not really have time at the moment...