0.7.0 super slow

[waynebliu]

Thanks for getting out 0.7.0. Unfortunately, it's a lot slower, so much that I can't use it. Hope you guys can fix it quickly and easily!

Here's %prun the import of a file with 0.7.0 - 49s, 10s in tokenize:

     96288231 function calls (96256267 primitive calls) in 49.538 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
5481478 10.142 0.000 31.655 0.000 tokenize.py:487(_tokenize)
5482586 5.618 0.000 5.618 0.000 {method 'match' of 're.Pattern' objects}
5478461 4.262 0.000 7.726 0.000 re.py:273(_compile)
5874063 3.028 0.000 3.028 0.000 {built-in method new of type object at 0x7f16a4188820}
3305 2.488 0.001 34.606 0.010 inputtransformer.py:132(push)
503552 2.246 0.000 4.233 0.000 pathlib.py:63(parse_parts)
5477626 1.931 0.000 10.787 0.000 tokenize.py:147(_compile)
5477828 1.782 0.000 2.820 0.000 types.py:164(get)
441830 1.593 0.000 2.550 0.000 {built-in method posix.stat}
5478017 1.274 0.000 8.998 0.000 re.py:234(compile)
5478068 1.218 0.000 4.150 0.000 :1(new)
5477753 1.038 0.000 1.038 0.000 enum.py:633(value)
125888 1.025 0.000 1.826 0.000 pathlib.py:523(_select_from)
5151461 0.977 0.000 0.977 0.000 {built-in method sys.intern}
11207694 0.896 0.000 0.896 0.000 {method 'append' of 'list' objects}
6213169 0.617 0.000 0.617 0.000 {built-in method builtins.isinstance}
5477549 0.593 0.000 0.593 0.000 {method 'span' of 're.Match' objects}
125888 0.542 0.000 0.625 0.000 {built-in method posix.scandir}
251776 0.464 0.000 6.161 0.000 pathlib.py:1115(glob)
503552 0.425 0.000 0.801 0.000 pathlib.py:709(str)
55653 0.405 0.000 1.250 0.000 :1356(find_spec)
251776 0.380 0.000 4.240 0.000 pathlib.py:649(_parse_args)
3305 0.363 0.000 34.993 0.011 inputsplitter.py:729(_transform_line)
125888 0.354 0.000 0.605 0.000 finder.py:45(fuzzy_query)
17984 0.331 0.000 12.444 0.001 finder.py:52(fuzzy_file_search)
377664 0.258 0.000 0.376 0.000 pathlib.py:692(_format_parsed_parts)
503552 0.248 0.000 0.354 0.000 pathlib.py:294(splitroot)
503552 0.239 0.000 1.040 0.000 pathlib.py:719(fspath)
251776 0.238 0.000 4.785 0.000 pathlib.py:1024(new)
256918/256673 0.228 0.000 6.391 0.000 {method 'extend' of 'list' objects}
2197655 0.223 0.000 0.223 0.000 {method 'isidentifier' of 'str' objects}
1399047 0.205 0.000 0.205 0.000 {method 'endswith' of 'str' objects}
2228732/2224486 0.200 0.000 0.201 0.000 {built-in method builtins.len}
251776 0.197 0.000 4.487 0.000 pathlib.py:669(_from_parts)
252101 0.194 0.000 0.194 0.000 {method 'split' of 'str' objects}
251776 0.185 0.000 1.956 0.000 pathlib.py:483(select_from)
388704 0.182 0.000 0.472 0.000 :56(_path_join)
412356 0.177 0.000 0.177 0.000 {method 'fullmatch' of 're.Pattern' objects}
770044/769976 0.173 0.000 0.174 0.000 {method 'join' of 'str' objects}
251776 0.172 0.000 1.771 0.000 pathlib.py:1368(is_dir)
388704 0.166 0.000 0.236 0.000 :58()

Here's the import of the same file with 0.6.2 - 13s:

     23295889 function calls (23261202 primitive calls) in 13.109 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)
431616 2.003 0.000 3.711 0.000 pathlib.py:63(parse_parts)
387869 1.399 0.000 2.249 0.000 {built-in method posix.stat}
107904 0.889 0.000 1.545 0.000 pathlib.py:523(_select_from)
4415541 0.832 0.000 0.832 0.000 {built-in method sys.intern}
107904 0.450 0.000 0.515 0.000 {built-in method posix.scandir}
431616 0.416 0.000 0.730 0.000 pathlib.py:709(str)
215808 0.416 0.000 5.380 0.000 pathlib.py:1115(glob)
4938969 0.381 0.000 0.381 0.000 {method 'append' of 'list' objects}
55653 0.362 0.000 1.142 0.000 :1356(find_spec)
215808 0.335 0.000 3.697 0.000 pathlib.py:649(_parse_args)
107904 0.307 0.000 0.519 0.000 finder.py:50(fuzzy_query)
17984 0.288 0.000 10.944 0.001 finder.py:57(fuzzy_file_search)
220915/220705 0.225 0.000 5.606 0.000 {method 'extend' of 'list' objects}
431616 0.220 0.000 0.317 0.000 pathlib.py:294(splitroot)
323712 0.214 0.000 0.314 0.000 pathlib.py:692(_format_parsed_parts)
215808 0.202 0.000 4.199 0.000 pathlib.py:1024(new)
215808 0.189 0.000 3.933 0.000 pathlib.py:669(_from_parts)
431616 0.185 0.000 0.915 0.000 pathlib.py:719(fspath)
215808 0.179 0.000 1.717 0.000 pathlib.py:483(select_from)
1201203 0.172 0.000 0.172 0.000 {method 'endswith' of 'str' objects}
216162 0.169 0.000 0.169 0.000 {method 'split' of 'str' objects}
333773 0.162 0.000 0.417 0.000 :56(_path_join)
215808 0.153 0.000 1.538 0.000 pathlib.py:1368(is_dir)
664361/664293 0.149 0.000 0.150 0.000 {method 'join' of 'str' objects}
333773 0.146 0.000 0.207 0.000 :58()
353448 0.142 0.000 0.142 0.000 {method 'fullmatch' of 're.Pattern' objects}
674973 0.129 0.000 0.129 0.000 {built-in method builtins.isinstance}
35968 0.117 0.000 0.226 0.000 inspect.py:1389(getclosurevars)
107904 0.116 0.000 0.920 0.000 pathlib.py:1356(exists)
323712 0.113 0.000 2.090 0.000 pathlib.py:1178(stat)
323712 0.088 0.000 0.088 0.000 pathlib.py:1034(_init)
354551 0.082 0.000 0.082 0.000 {built-in method new of type object at 0x7f77b34f6820}
107904 0.079 0.000 0.127 0.000 pathlib.py:682(_from_parsed_parts)
107904 0.079 0.000 1.200 0.000 pathlib.py:507(_select_from)
667 0.078 0.000 0.078 0.000 {built-in method marshal.loads}
107904 0.074 0.000 0.202 0.000 pathlib.py:1044(_make_child_relpath)
765 0.072 0.000 12.284 0.016 :1240(_get_spec)
334047 0.068 0.000 0.068 0.000 :222(_verbose_message)
670175 0.062 0.000 0.062 0.000 {method 'rstrip' of 'str' objects}
55653 0.062 0.000 12.147 0.000 finder.py:69(find_spec)
17984 0.060 0.000 0.286 0.000 finder.py:104(get_loader_details)

I also have another ipynb that never finishes importing after 30min - I'm not sure why as it's actually shorter than this one. Maybe because it has one cell that is larger than the biggest cell in this file...

[waynebliu]

Awesome! Thanks