Python package virtualenv as a dependency
maxileith opened this issue · comments
Current Situation
Hi everyone,
first of all I want to say that I am very greatful for this amazing project. It really has changed the way I use my HomeKit home.
I am currently developing the plugin homebridge-appletv-enhanced which enables more possibilities than the vanilla integration of Apple TV into HomeKit. It is still in beta phase, but my aim is to provide a painless experience to the users from installation to initial setup to the usage. I am aiming to get this plugin verified.
The plugin is utilizing the pyatv python package because sadly there is no maintained npm packages which implements the protocol to control an Apple TV … and I know what you are thinking now … how could a plugin that is utilizing a python package provide a painless installation experience?
First, let’s talk about how easy it is for the user up until now.
- The user installs the plugin, no postinstall scripts necessary
- The user configures the plugin via the UI
- The user restarts the homebridge
- The plugin starts (PythonChecker.ts)
a. The plugin checks if a supported Python version is installed
b. The plugin checks if the python packagevirtualenv
is installed
c. The plugin creates a virtual environment in the user storage path of homebridge (if not already created)
d. The plugin installs python packages according to a requirements.txt (if requirements not already satisfied)
e. The plugin discovers devices and so on … installation complete
Since Python 3 is in the docker image, the apt package and the raspian image by default, this requirement should not be a problem.
However, now we are finally coming to the problem I currently have, in order to comply with the requirements of getting a plugin verified, the plugin is not allowed to modify the users system, which is totally understandable and right to do in my mind. So my plugin creates a virtual environment (steps c-d) in the users storage path which is allowed according to the verified requirements. The issue with that is, that many users do not have installed the python package virtualenv
. There was already an issue created in my repository regarding this problem.
Proposed Change
Include the apt package python3-venv in the docker image, the raspian image and as a dependency in the apt package.
What would be the benefit?
- Plugins that depend on python packages could rely on
virtualenv
being present. This would encourage to use virtual python environments which isolates the plugins that utilize python. - Users don‘t need to install ˋvirtualenvˋ manually
- In cases such as mine with the Apple TV plugin where there is simply no npm package which meets the needs, one can use a python package in an venv without worrying if the venv can be created or not
Wouldn‘t it be better to "just" use plain type-/javascript? Of course, but in many cases there is not the time to create complex JavaScript libraries that provide functionality that can already be provided by a well maintained python package, like in my case. When not supporting virtualenv
many developers may be deterred since in the end, there will be no way of installing a plugin painlessly and complying to the verified requirements at the same time without virtualenv
(when utilizing python).
Additional Context
No response
Are you also going to resolve the sluggishness of the pyatv package with this approach ?
FYI - My version of the same - https://github.com/NorthernMan54/homebridge-cmd-television
Which I use to be able to tell Siri and Alexa to pause and resume TV
I will be updating those packages next month, and can include. Will need assistance around testing though
Hi @NorthernMan54,
I did no experience any problems regarding pyatv besides incompatibility with Python 3.11 which I have fixed with this PR postlund/pyatv#2270.
The plugin that I have written is very fast. We are talking <<1s response time. So it is most likely an implementation problem. I cannot look at you source code since there is no source code in your repo.
However, from what I can tell from the README your installation, pairing and configuration looks largely manual in contrast to my plugin. As I said, my plugin is installing the venv, the requirements automatically and the pairing process is done without the need to go to the CLI (After autodiscovery of Apple TVs). The pairing process is happening in a web application that is hosted by the plugin when pairing. All the credentials as well as the appletv configs are stored in the userstorage path.
@NorthernMan54 I just saw the index.js, so source code is checked in. However, the points from my comment before are still valid.
What I found was that the atvremote commands were sluggish as they negotiated authentication every time ( this was about 18 months ago )
ie
time atvremote --id 08:66:98:D4:CF:79 --airplay-credentials `cat ~/.homebridge/atv_airplay.cred` --companion-credentials `cat ~/.homebridge/atv_companion.cred` pause
real 0m1.961s
user 0m1.535s
sys 0m0.249s
PS I would gladly update mine to yours, if it was faster!!!!
@NorthernMan54 not the right place to discuss that. Please only comment if you have comments on the topic that I have created this issue for.
@NorthernMan54 I didn't want to deter you away from the discussion, I just wanted to make sure this issue was about what I opened it for. We can discuss if it makes sense to ship the mentioned python3-venv package with homebridge or not. The discussion about whether pyatv for a verified plugin should then be discussed in the issue about verification. The message before was a bit harsh, sorry for that ...
I have added python3-venv to the test area for the next Homebridge-apt-pkg. Can you take a moment and test it to ensure that everything is okay for your plugin.
https://github.com/NorthernMan54/homebridge-apt-pkg/releases/tag/1.20.1
You should be able to downland the appropriate DEB file, and install it with apt-get install ./homebridge....
@NorthernMan54 That sounds awesome. I will test it on a fresh Raspberry PI OS Bookworm installation and let you know if it is working :)
So I just tried it on a Raspberry Pi 3 with Pi OS Bookworm 64-bit installed on it. Regarding the virtual environment everything works fine. python3-venv
is installed as a dependency and my plugin can create the virtual python environment for itself without any issues.
However, there are problems occuring when installing the python depdencies. Not all dependencies are built for every system architecture, e.g. arm64. Therefore pip tries to build them on itself which requires python3-dev
. I simplly did not recongnize that on my productive and development system since I have installed this package on these systems anyways. Would it be possible to include python3-dev
in the dependencies as well?
However, I would like to mention that adding python3-venv
is a benefit even without python3-dev
, especially for users of systems for which the packages are already built. The python3-dev
package is only necessary for those for whom not all python packages are prebuilt.
FYI - I moved to the package we are working with
Does installing python3-dev
then require compiling code, and a build environment as well ?
The raspbian image and docker image both leverage this package, so the change only needs to be made in one place
@NorthernMan54 yes, this does mean that there has to be code compiled. In the case of Raspberry Pi OS Lite 64-bit the build tools were already installed. Installing python3-dev
was sufficient.
Build has been kicked off, it should take about an hour, and a release will be created in the test environment
https://github.com/NorthernMan54/homebridge-apt-pkg/actions/runs/6981891690
Release target to be populated
https://github.com/NorthernMan54/homebridge-apt-pkg/releases/tag/1.20.2
I have made another fresh Pi OS install and can confirm that everything is working with v1.20.2. 👍🏻
I just merged and published this as 1.1.4. can you check that production version out, then I will release the other packages.
Everything is working fine on a fresh install with Homebridge 1.1.4 from the official repository. 🎉
Docker image was published a few minutes ago, and the homebridge-raspbian-image is in progress, it usually takes a few hours
hi all!
I am running Homebridge on Docker within unRAID.
installed container - Oznu/Homebridge:latest
Node.js Version : v20.11.0
Homebridge : v1.7.0
UI : v4.55.1
[2/10/2024, 2:49:05 PM] [Apple TV Enhanced] [I] Python check: Python 3.8.10 is installed and supported by the plugin.
[2/10/2024, 2:49:05 PM] [Apple TV Enhanced] [W] Python check: Virtual python environment is not present. Creating now ...
[2/10/2024, 2:49:05 PM] [Apple TV Enhanced] [E] Python check: virtualenv python module is not installed. If you have installed homebridge via the apt package manager, update the homebridge apt package to 1.1.4 or above (this applies for installations based on the Raspberry Pi OS iamge as well). When using the official docker image, update the image to version 2023-11-28 or above. Otherwise install the python virtualenv module manually.
when i run in terminal - python3 -m pip install virtualenv
it does download files which would suggest it was missing and when re-running that command i get:
Requirement already satisfied: virtualenv in /usr/local/lib/python3.8/dist-packages (20.25.0)
Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (0.3.8)
Requirement already satisfied: filelock<4,>=3.12.2 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (3.13.1)
Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (4.2.0)
I don't really know what to do next. Any suggestions? I just want to get the Enhanced Apple TV plugin running!
Use the latest Homebridge image: https://hub.docker.com/r/homebridge/homebridge
Use the latest Homebridge image: https://hub.docker.com/r/homebridge/homebridge
just wiped the whole thing and forced search through DockerHub and pulled homebridge/homebridge.
Oznu/Homebridge:latest = not the latest as you rightfully point out in multiple replies across multiple raised issues referring to virtualenv.
thank you for your patience!
its now working. Thank you.