Python package virtualenv as a dependency

Question

Python package virtualenv as a dependency

maxileith opened this issue 10 months ago · comments

Maximilian Leith commented 10 months ago

Current Situation

Hi everyone,

first of all I want to say that I am very greatful for this amazing project. It really has changed the way I use my HomeKit home.

I am currently developing the plugin homebridge-appletv-enhanced which enables more possibilities than the vanilla integration of Apple TV into HomeKit. It is still in beta phase, but my aim is to provide a painless experience to the users from installation to initial setup to the usage. I am aiming to get this plugin verified.

The plugin is utilizing the pyatv python package because sadly there is no maintained npm packages which implements the protocol to control an Apple TV … and I know what you are thinking now … how could a plugin that is utilizing a python package provide a painless installation experience?

First, let’s talk about how easy it is for the user up until now.

The user installs the plugin, no postinstall scripts necessary
The user configures the plugin via the UI
The user restarts the homebridge
The plugin starts (PythonChecker.ts)
a. The plugin checks if a supported Python version is installed
b. The plugin checks if the python package virtualenv is installed
c. The plugin creates a virtual environment in the user storage path of homebridge (if not already created)
d. The plugin installs python packages according to a requirements.txt (if requirements not already satisfied)
e. The plugin discovers devices and so on … installation complete

Since Python 3 is in the docker image, the apt package and the raspian image by default, this requirement should not be a problem.

However, now we are finally coming to the problem I currently have, in order to comply with the requirements of getting a plugin verified, the plugin is not allowed to modify the users system, which is totally understandable and right to do in my mind. So my plugin creates a virtual environment (steps c-d) in the users storage path which is allowed according to the verified requirements. The issue with that is, that many users do not have installed the python package virtualenv. There was already an issue created in my repository regarding this problem.

Proposed Change

Include the apt package python3-venv in the docker image, the raspian image and as a dependency in the apt package.

What would be the benefit?

Plugins that depend on python packages could rely on virtualenv being present. This would encourage to use virtual python environments which isolates the plugins that utilize python.
Users don‘t need to install ˋvirtualenvˋ manually
In cases such as mine with the Apple TV plugin where there is simply no npm package which meets the needs, one can use a python package in an venv without worrying if the venv can be created or not

Wouldn‘t it be better to "just" use plain type-/javascript? Of course, but in many cases there is not the time to create complex JavaScript libraries that provide functionality that can already be provided by a well maintained python package, like in my case. When not supporting virtualenv many developers may be deterred since in the end, there will be no way of installing a plugin painlessly and complying to the verified requirements at the same time without virtualenv (when utilizing python).

Additional Context

No response

Northern Man · Answer 1 · Mon Oct 30 2023 22:43:10 GMT+0800 (China Standard Time)

Are you also going to resolve the sluggishness of the pyatv package with this approach ?

FYI - My version of the same - https://github.com/NorthernMan54/homebridge-cmd-television
Which I use to be able to tell Siri and Alexa to pause and resume TV

I will be updating those packages next month, and can include. Will need assistance around testing though

Maximilian Leith · Answer 2 · Mon Oct 30 2023 22:55:21 GMT+0800 (China Standard Time)

Hi @NorthernMan54,

I did no experience any problems regarding pyatv besides incompatibility with Python 3.11 which I have fixed with this PR postlund/pyatv#2270.

The plugin that I have written is very fast. We are talking <<1s response time. So it is most likely an implementation problem. I cannot look at you source code since there is no source code in your repo.

However, from what I can tell from the README your installation, pairing and configuration looks largely manual in contrast to my plugin. As I said, my plugin is installing the venv, the requirements automatically and the pairing process is done without the need to go to the CLI (After autodiscovery of Apple TVs). The pairing process is happening in a web application that is hosted by the plugin when pairing. All the credentials as well as the appletv configs are stored in the userstorage path.

Maximilian Leith · Answer 3 · Mon Oct 30 2023 22:59:04 GMT+0800 (China Standard Time)

@NorthernMan54 I just saw the index.js, so source code is checked in. However, the points from my comment before are still valid.

Northern Man · Answer 4 · Mon Oct 30 2023 23:03:54 GMT+0800 (China Standard Time)

What I found was that the atvremote commands were sluggish as they negotiated authentication every time ( this was about 18 months ago )

ie

time atvremote --id 08:66:98:D4:CF:79 --airplay-credentials `cat ~/.homebridge/atv_airplay.cred` --companion-credentials `cat ~/.homebridge/atv_companion.cred` pause

real	0m1.961s
user	0m1.535s
sys	0m0.249s

Northern Man · Answer 5 · Mon Oct 30 2023 23:04:45 GMT+0800 (China Standard Time)

PS I would gladly update mine to yours, if it was faster!!!!

Maximilian Leith · Answer 6 · Mon Oct 30 2023 23:05:41 GMT+0800 (China Standard Time)

@NorthernMan54 not the right place to discuss that. Please only comment if you have comments on the topic that I have created this issue for.

Maximilian Leith · Answer 7 · Tue Oct 31 2023 03:18:06 GMT+0800 (China Standard Time)

@NorthernMan54 I didn't want to deter you away from the discussion, I just wanted to make sure this issue was about what I opened it for. We can discuss if it makes sense to ship the mentioned python3-venv package with homebridge or not. The discussion about whether pyatv for a verified plugin should then be discussed in the issue about verification. The message before was a bit harsh, sorry for that ...

Northern Man · Answer 8 · Fri Nov 24 2023 10:27:04 GMT+0800 (China Standard Time)

I have added python3-venv to the test area for the next Homebridge-apt-pkg. Can you take a moment and test it to ensure that everything is okay for your plugin.

https://github.com/NorthernMan54/homebridge-apt-pkg/releases/tag/1.20.1

You should be able to downland the appropriate DEB file, and install it with apt-get install ./homebridge....

Maximilian Leith · Answer 9 · Fri Nov 24 2023 15:53:47 GMT+0800 (China Standard Time)

@NorthernMan54 That sounds awesome. I will test it on a fresh Raspberry PI OS Bookworm installation and let you know if it is working :)

Maximilian Leith · Answer 10 · Fri Nov 24 2023 22:08:43 GMT+0800 (China Standard Time)

So I just tried it on a Raspberry Pi 3 with Pi OS Bookworm 64-bit installed on it. Regarding the virtual environment everything works fine. python3-venv is installed as a dependency and my plugin can create the virtual python environment for itself without any issues.

However, there are problems occuring when installing the python depdencies. Not all dependencies are built for every system architecture, e.g. arm64. Therefore pip tries to build them on itself which requires python3-dev. I simplly did not recongnize that on my productive and development system since I have installed this package on these systems anyways. Would it be possible to include python3-dev in the dependencies as well?

However, I would like to mention that adding python3-venv is a benefit even without python3-dev, especially for users of systems for which the packages are already built. The python3-dev package is only necessary for those for whom not all python packages are prebuilt.

Northern Man · Answer 11 · Fri Nov 24 2023 22:14:03 GMT+0800 (China Standard Time)

FYI - I moved to the package we are working with

Does installing python3-dev then require compiling code, and a build environment as well ?

The raspbian image and docker image both leverage this package, so the change only needs to be made in one place

Maximilian Leith · Answer 12 · Fri Nov 24 2023 22:20:12 GMT+0800 (China Standard Time)

@NorthernMan54 yes, this does mean that there has to be code compiled. In the case of Raspberry Pi OS Lite 64-bit the build tools were already installed. Installing python3-dev was sufficient.

Northern Man · Answer 13 · Fri Nov 24 2023 22:24:49 GMT+0800 (China Standard Time)

Build has been kicked off, it should take about an hour, and a release will be created in the test environment

https://github.com/NorthernMan54/homebridge-apt-pkg/actions/runs/6981891690

Release target to be populated

https://github.com/NorthernMan54/homebridge-apt-pkg/releases/tag/1.20.2

Maximilian Leith · Answer 14 · Fri Nov 24 2023 23:51:55 GMT+0800 (China Standard Time)

I have made another fresh Pi OS install and can confirm that everything is working with v1.20.2. 👍🏻

Northern Man · Answer 15 · Mon Nov 27 2023 23:47:46 GMT+0800 (China Standard Time)

I just merged and published this as 1.1.4. can you check that production version out, then I will release the other packages.

Maximilian Leith · Answer 16 · Wed Nov 29 2023 00:20:41 GMT+0800 (China Standard Time)

Everything is working fine on a fresh install with Homebridge 1.1.4 from the official repository. 🎉

Northern Man · Answer 17 · Wed Nov 29 2023 02:27:22 GMT+0800 (China Standard Time)

Docker image was published a few minutes ago, and the homebridge-raspbian-image is in progress, it usually takes a few hours

jammmyb · Answer 18 · Sat Feb 10 2024 22:56:41 GMT+0800 (China Standard Time)

hi all!

I am running Homebridge on Docker within unRAID.

installed container - Oznu/Homebridge:latest

Node.js Version : v20.11.0
Homebridge : v1.7.0
UI : v4.55.1

[2/10/2024, 2:49:05 PM] [Apple TV Enhanced] [I] Python check: Python 3.8.10 is installed and supported by the plugin.
[2/10/2024, 2:49:05 PM] [Apple TV Enhanced] [W] Python check: Virtual python environment is not present. Creating now ...
[2/10/2024, 2:49:05 PM] [Apple TV Enhanced] [E] Python check: virtualenv python module is not installed. If you have installed homebridge via the apt package manager, update the homebridge apt package to 1.1.4 or above (this applies for installations based on the Raspberry Pi OS iamge as well). When using the official docker image, update the image to version 2023-11-28 or above. Otherwise install the python virtualenv module manually.

when i run in terminal - python3 -m pip install virtualenv

it does download files which would suggest it was missing and when re-running that command i get:

Requirement already satisfied: virtualenv in /usr/local/lib/python3.8/dist-packages (20.25.0)
Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (0.3.8)
Requirement already satisfied: filelock<4,>=3.12.2 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (3.13.1)
Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (4.2.0)

I don't really know what to do next. Any suggestions? I just want to get the Enhanced Apple TV plugin running!

Maximilian Leith · Answer 19 · Sat Feb 10 2024 23:21:03 GMT+0800 (China Standard Time)

Use the latest Homebridge image: https://hub.docker.com/r/homebridge/homebridge

jammmyb · Answer 20 · Sat Feb 10 2024 23:26:50 GMT+0800 (China Standard Time)

Use the latest Homebridge image: https://hub.docker.com/r/homebridge/homebridge

just wiped the whole thing and forced search through DockerHub and pulled homebridge/homebridge.

Oznu/Homebridge:latest = not the latest as you rightfully point out in multiple replies across multiple raised issues referring to virtualenv.

thank you for your patience!

its now working. Thank you.