Speed up pytype using --precompiled-builtins

[Rebecca Chen]

We observed that pytype is surprisingly slow when analyzing https://github.com/google/pycnite, a small project with only 9 .py files. I profiled pytype on a few of the files and discovered that it is spending an unreasonable amount of time (~2s per file) parsing the builtins and typing stubs. We already have a way to parse and pickle these as a preprocessing step (--generate-builtins) and to load the pickle rather than re-parsing every time (--precompiled-builtins).

Perhaps we could generate the builtins in out/bin/pytype/ to speed up the CI workflow and also check them in every time a commit is made to main. I briefly tried to do this and ran into a few issues:

• --generate-builtins assumes that typeshed contains only .pyi files, stdlib/VERSIONS, tests/pytype_exclude_list.txt, and the METADATA.toml files in each stubs subdirectory. In particular, it crashes when it encounters stdlib/_typeshed/README.md and the @tests dirctories in stubs/. This wasn't an issue previously because --generate-builtins was being run with a copy of typeshed into which we'd copied only the files we wanted.
• typeshed/stubs/ causes dependency resolution issues. The stubs have dependencies on external packages (typeshed's pytype_test has some special logic for this: https://github.com/python/typeshed/blob/0ea043253e70d0304478a6d0b58bcda4cc583d08/tests/pytype_test.py#L163). Plus, for whatever reason, trying to pickle them leads to random issues with resolving class attributes. This wasn't an issue previously because we were only pickling the stdlib and the six stubs.
• Because of the above problem, I tried deleting everything under stubs/ and pickling just the stdlib. This seems to work as long as I set options.typeshed = True so stubs/ files missing from the pickle can still be loaded: http://google3/third_party/py/pytype/config.py;l=663;rcl=564564177.
• out/bin/pytype and out/bin/pytype-single set TYPESHED_HOME to the typeshed submodule. I had to manually set/unset TYPESHED_HOME depending on whether I wanted it point to a modified typeshed for pickling or the full typeshed so that stubs left out of the pickle could still be found.