Investigate autocomplete on PyCharm and stubs files

Question

Investigate autocomplete on PyCharm and stubs files

FilippoOggionni opened this issue 3 years ago · comments

Original issue was there.

Resource: https://tudat.slack.com/archives/D01GSTLQT1S/p1620138100040900
(private chat between @ggarrett13 and @FilippoOggionni)

Back to pybind auto-generated stubs
I have looked into pybind11 options, specifically the options.disable_function_signatures() that you talked about in the thread. Unfortunately, it is not something that I can (yet) try out locally, but I hope I will be able to do it soon.
In summary, you say that the option of pybind11 auto-generated is not convenient for the following two reasons:
It is superfluous with function signatures: e.g., we would ideally like BodyListSettings instead of tudatpy.kernel.simulation.environment_setup.BodyListSettings (on a side note, is the length of the string an issue only for the autogenerated docs?);
more importantly, it cannot deal with overloaded functions.
For problem 2, I think there is a solution. Perhaps it is not automatic with pybind11 autogen stubs, but it uses MyPy (see this message and the related link). You can have a look at this (5th bullet point), but also this one (I did not fully understand the differences between the two, so if you catch them please feel free to lecture me haha). Shall we give it a try?
For problem 1, I am not sure if there is a solution, but it does not seem that it is the main limiting problem. Also, are we gonna use these stubs to build the tudatpy API? In addition, something that I was thinking about is that the option options.disable_function_signatures() is set only on the highest-level pybind module (here). From pybind docs (here), it seems that this option works at a module level. Does it work hierarchically? In other words, what happens if you disable the function signature at a higher level but leave it on at a lower level? Which one of the two dominates? I assume the higher level, but if I am wrong it could be a workaround for problem 1.

Filippo Oggionni · Answer 1 · Wed Oct 20 2021 23:12:33 GMT+0800 (China Standard Time)

Slack conversation:

Geoffrey 2 months ago

Unfortunately, it is not something that I can (yet) try out locally
how so?
Geoffrey 2 months ago
(on a side note, is the length of the string an issue only for the autogenerated docs?);
Basically just appearance...

Geoffrey 2 months ago
And the fact that the function signature type hint alone would probably span more than a single line alone.

Filippo 2 months ago
how so?
I have not yet tested nor installed the tudat-bundle, after the issue with the new M1 Apple chip. I will get to it soon.
Basically just appearance...
And the fact that the function signature type hint alone would probably span more than a single line alone.
Okay, makes sense!
Looking forward to hear about the rest 🙂

Geoffrey 2 months ago
Again, appearance, but that's just a practical hindrance in legibility in the end.
more importantly, it cannot deal with overloaded functions.
I'll have a look at MyPy and get back to you.
Also, are we gonna use these stubs to build the tudatpy API?
There's a lot that can be said here. The points I'm certain about are:
tudatpy cannot used in autodoc (automatic docstring generation in sphinx) unless it is built currently. This adds extra complexity to the logistics in API documentation generation. Not a train crash though.
tudat and tudatpy currently don't have the same structuring for the API in the source, which presents some issues with my prototype system.

Geoffrey 2 months ago
Point i'm unclear on:
I think stubs can be used to generate the API, and would make a really neat place to view the docstrings when developing in an IDE, with the "jump to source" shortcut

Geoffrey 2 months ago
"I think stubs can be used to generate the API" being the key if. If it's possible, it might solve some issues in the future. I'll have a look at MyPy. Did you find any stack overflow posts on no autocomplete with pybind modules?

Filippo 2 months ago
I did not find anything specifically about autocomplete, but I think autocomplete relies on the stubs, so the two are strictly correlated (right?). Anyway, have a look at the stub autogen from MyPy and to the related links I posted - they should guide you quite well, I hope. In the meantime, I am trying to install the tudat-bundle again

Geoffrey 2 months ago
I would say with 96% confidence, from what I read, PyCharms autocomplete comes entirely from stubs. The exact process, whether they inspect and generate their own from Pybind or the process is a little different, not sure.

Geoffrey 2 months ago
But in my mind, almost one and the same. I'll have a look at MyPy a little after storms walk, arriving at library, and making a rough agenda for friday.

Filippo 2 months ago
I would say with 96% confidence, from what I read, PyCharms autocomplete comes entirely from stubs.
I agree.
The exact process, whether they inspect and generate their own from Pybind or the process is a little different, not sure.
I will look into this, but not today…
I’ll have a look at MyPy a little after storms walk, arriving at library, and making a rough agenda for friday.
Sure! Let me know.

Filippo Oggionni · Answer 2 · Wed Oct 20 2021 23:13:03 GMT+0800 (China Standard Time)

Useful material from this issue:

@bugcy013 PyCharm can investigate binary modules out of box, but there are two limitations:

PyCharm scans binary modules only at startup and builds so-called "stubs" (files with extension .pyi). So your module should be already in PYTHONPATH / site-packages / installed by pip / etc.
It can't extract actual functions signatures (all functions/methods are parsed as func(*args, **kwargs))
A workaround would be to write/generate .pyi files by yourself and mark directory with them as "Sources Root" (Right click on directory with stubs in project tree > "Mark Directory as" > "Sources Root").

Wring stubs by hand is cumbersome and error-prone. Fortunately pybind11 automatically generate descriptive docstrings. In my I have a script which generates stubs from docstrings and makes PyCharm (partially) understand my extension module.

My poor script to generate stubs from pybind11 docstrings:(https://github.com/sizmailov/pyxmolpp2/blob/master/write_type_hints.py).

While pybind11 docstrings are good, but not perfect for this particular use case, you may observe glitches in .pyi files generated by the (limited) script. Both could be sufficiently improved.

At the same time PyCharm interpretation of .pyi files is not 100% complete, so some PyCharm functionality might be disabled for this reason.

Hope you'll find this information helpful.

Filippo Oggionni · Answer 3 · Wed Oct 20 2021 23:13:59 GMT+0800 (China Standard Time)

Useful material:
this StackOverflow post.

DominicDirkx · Answer 4 · Tue Nov 02 2021 18:22:33 GMT+0800 (China Standard Time)

I had a conversation with @transferorbit yesterday, and managed to get the autocomplete working on vim! @transferorbit : could you post a summary of what you did to achieve this? A similar workflow may be feasible in other IDEs as well.

transferorbit · Answer 5 · Tue Nov 09 2021 06:45:35 GMT+0800 (China Standard Time)

Yes, I have successfully managed to get autocompletion (for a.o. TudatPy) working within vim.
Here’s a screenshot demo.
Hopefully I haven’t forgotten any details about what’s necessary to make this happen, but in the end it wasn’t even hard. And the cool thing is, in contrast to most of what you’ll see when searching the web, you can do awesome stuff with vim straight out of the box, so neat, with nothing added.

Some things you can do with vim, straight-out-of-the-box:
• context-aware autocomplete with the tab key
• view documentation (‘only’ as included in the docstrings, I believe)

WHAT TO DO TO MAKE IT WORK

Use the installation configuration of vim required for omni-completion

Use a version of vim >= 7
Ensure that your installed vim configuration has the following options set:
a. +conceal
b. +python3
- Check this at the CLI using vim --version
- Check which vim executable your CLI is running with which vim. For example, when running vim at the macOS CLI with no conda environment active, it will access /usr/bin/vim.

If necessary, get the required vim installation configuration

The good news is that this is easy to obtain: my recommendation is, install vim in any conda environment you wish to use with conda install vim (or just include it in a yml environment file). This should, by default, install vim with the vim configuration options mentioned in (2) above.

Add this to your .vimrc

“
“ manual vimrc settings:   add built-in CleverTab key mapping for autocompletion
“                       ref: help ins-completion
"
function! CleverTab()
    if strpart( getline(‘.’), 0, col(‘.’)-1 ) =~ ‘^\s*$’
        return “\<Tab>”
    else
        return “\<C-X>\<C-O>”
    endif
endfunction
inoremap <Tab> <C-R>=CleverTab()<CR>

“
“ manual vimrc settings: autocomplete/omnicompletion
"
au FileType python setl ofu=python3complete#Complete

NB: The omnicompletion works without using the CleverTab function using default key combinations, but I’m sure you’ll soon agree you don’t prefer those over the tab key. #YMMV

Things you don’t need (despite what you read on the web)

a self-compiled version of vim
jedi-vim | https://github.com/davidhalter/jedi-vim
youcompleteme

Reference
(all to be found in the built-in vim documentation)

:h ins-completion
CleverTab (under ins-completion)
:h new-omni-completion

DominicDirkx · Answer 6 · Thu Jan 13 2022 22:41:24 GMT+0800 (China Standard Time)

Somehow this issue has magically been resolved: auto-complete is now working