Build for homeassistant docker?
jshank opened this issue · comments
I've augmented my Dockerfile to include your script and running into problems with Selenium. Since the Home Assistant Dockerffile is based on the python2.7 image, do you have a recommendation for getting up to selenium 3.141.59?
Error Message from Home Assistant
Unable to install package selenium==3.141.59: Could not find a version that satisfies the requirement selenium==3.141.59 (...)
No matching distribution found for selenium==3.141.59
Selenium version
pip install selenium
Requirement already satisfied: selenium in /usr/local/lib/python3.7/site-packages (2.53.6)
Dockerfile
FROM homeassistant/home-assistant:latest
MAINTAINER Jim Shank <jshank@theshanks.net>
RUN set -x && wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& printf "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list \
&& apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
google-chrome-stable \
&& rm -rf /var/lib/apt/lists
@jshank first, let me say thanks for trying this out. whatever i can do to help get this working in other environments, the better.
when you say the HA dockerfile is based on 2.7 image, which dockerfile are you referring to? both of THESE GUYS appear to be using python3.
if i'm not mistaken, ubuntu (17/18+) should now be coming with 3 by default, and if you see python2 being installed, it's probably for reverse compatibility, so building/optimizing/configuring anything for python2 at this point is borderline daft.
i did look into the current version of selenium and noticed that it looks like the one via pip3 appears to have been rolled back to 3.141.0 from 3.141.59. i've made the necessary changes to my repo, so maybe give it another go and see if that was the issue. if not, we can continue troubleshooting, and if we get it working, i'd love to get my hands on the whole dockerfile and maybe include it in my repo or link to yours as reference for other users using the same environment configuration.
@shr00mie - That was a typing error, the Docker file is based on python 3.7, sorry. I was referring to this file.
FROM python:3.7
LABEL maintainer="Paulus Schoutsen <Paulus@PaulusSchoutsen.nl>"
I'll give it another go.
Also, LOL at that Selenium version number. Someone has a good sense of humor. Will 2.53.6 work?
For anyone trying to follow this, I manually git clone https://github.com/shr00mie/gmapslocsharing.git /docker/mapped/hass/config since Docker won't actually have access to the mapped /config directory during build.
@jshank i suppose that would depend on which functionality i implemented which may have only been added since then. i think i developed it with 3.12something. the way it SHOULD work is that when HA loads the custom component and reads the REQUIREMENTS section in my device_tracker.py, it should install selenium (and all other package requirements) automatically into HA's venv on demand, so you shouldn't have to worry about any manual install. if i had to guess, 2.53.6 is probably the first version available via pip3 after they ported it from pip. it MIGHT, but if my hypothesis is correct, it shouldn't be something you would need to do manually anyway.
lemme know!
btw, i didn't even think about making it available to docker that way, but now that you point it out, i'm like fucking duh...i'll throw something in the readme for that for others.
I think I placed everything in the right paths but seems like it can't find gmapslocsharing
Does the deps path need to be declared somewhere else?
Error from Home Assistant logs
Error setting up platform google_maps
Traceback (most recent call last):
File "/usr/src/app/homeassistant/components/device_tracker/__init__.py", line 184, in async_setup_platform
disc_info)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/config/custom_components/google_maps/device_tracker.py", line 49, in setup_scanner
scanner = GoogleMapsScanner(hass, config, see)
File "/config/custom_components/google_maps/device_tracker.py", line 57, in __init__
from gmapslocsharing import GoogleMaps
ModuleNotFoundError: No module named 'gmapslocsharing'
Finding the missing package
find / -name gmapslocsharing
/config/deps/lib/python3.6/site-packages/gmapslocsharing
Where packages are in the Docker image
find / -name site-packages
/usr/local/lib/python3.7/site-packages
/usr/local/lib/python2.7/site-packages
/config/deps/lib/python3.6/site-packages
/config/deps/lib/python3.7/site-packages
@jshank do me a favor and take a peek in /deps/lib/python3.6/site-packages/ and let me know if there's anything in there besides gmapslocsharing. i have a sneaking suspicion that might be the problem per HA path designation.
reason being: this isn't a proper pip package yet because i've never made one, so when you clone, you're using a static path for where the automated pip package downloads would otherwise go per component requirements.
well. i know what i'm going to try to learn to do this week...
if i'm right and gmapslocsharing is the only package under python3.6 deps, then just change the folder name in the mapped docker folder to python3.7 and i suspect it should work as that's probably the PATH that's set in HA to search for automatically downloaded modules in HA's venv.
Hmm, I thought you were on to something :/ The permissions are a bit goofy but shouldn't interfere as everyone has read/execute on the packages.
ls /config/deps/lib/python3.6/site-packages -al
total 52
drwxrwxr-x 13 1000 1000 4096 Mar 6 14:15 .
drwxrwxr-x 4 1000 1000 4096 Mar 6 14:15 ..
drwxr-xr-x 6 root root 4096 Mar 27 2018 dateutil
drwxr-xr-x 3 root root 4096 Jul 31 2018 elkm1
drwxr-xr-x 2 root root 4096 Jul 31 2018 elkm1-0.4.7.dist-info
drwxr-xr-x 3 root root 4096 Dec 12 11:51 elkm1_lib
drwxr-xr-x 2 root root 4096 Dec 12 11:51 elkm1_lib-0.7.13.dist-info
drwxrwxr-x 2 1000 1000 4096 Mar 6 14:15 gmapslocsharing
drwxr-xr-x 2 root root 4096 Mar 27 2018 pyserial-3.4.dist-info
drwxr-xr-x 2 root root 4096 Mar 27 2018 python_dateutil-2.7.2.dist-info
drwxr-xr-x 6 root root 4096 Mar 27 2018 serial
drwxr-xr-x 3 root root 4096 May 23 2018 suds_passworddigest
drwxr-xr-x 2 root root 4096 May 23 2018 suds_passworddigest-0.1.2a0.dist-info
ls /config/deps/lib/python3.7/site-packages -al
total 796
drwxr-xr-x 8 root root 4096 Mar 6 14:27 .
drwxr-xr-x 3 root root 4096 Mar 6 14:27 ..
drwxr-xr-x 2 root root 4096 Mar 6 14:27 Brotli-1.0.7.dist-info
drwxr-xr-x 2 root root 4096 Mar 6 14:27 __pycache__
-rwxr-xr-x 1 root root 774936 Mar 6 14:27 _brotli.cpython-37m-x86_64-linux-gnu.so
-rw-r--r-- 1 root root 1857 Mar 6 14:27 brotli.py
drwxr-xr-x 3 root root 4096 Mar 6 14:27 chromedriver_binary
drwxr-xr-x 2 root root 4096 Mar 6 14:27 chromedriver_binary-2.46.0.dist-info
drwxr-xr-x 5 root root 4096 Mar 6 14:27 selenium
drwxr-xr-x 2 root root 4096 Mar 6 14:27 selenium-3.141.0.dist-info
ls /config/deps/lib/python3.6/site-packages/gmapslocsharing/ -al
total 40
drwxrwxr-x 2 1000 1000 4096 Mar 6 14:15 .
drwxrwxr-x 13 1000 1000 4096 Mar 6 14:15 ..
-rw-rw-r-- 1 1000 1000 128 Mar 6 14:15 __init__.py
-rw-rw-r-- 1 1000 1000 7373 Mar 6 14:15 cookie.py
-rw-rw-r-- 1 1000 1000 577 Mar 6 14:15 core.py
-rw-rw-r-- 1 1000 1000 1325 Mar 6 14:15 localization.py
-rw-rw-r-- 1 1000 1000 7042 Mar 6 14:15 location.py
-rw-rw-r-- 1 1000 1000 2942 Mar 6 14:15 person.py
yeah. makes sense. remember, all the other components are downloaded by HA and gmapslochsaring is being mapped into the docker container from the external environment, hence the UID:GID from the user running the docker instance.
move gmapslocsharing to 3.7 and let me know where you're at then.
if the move isn't enough and then it becomes a permission issue, them maybe set the PUID & PGID running docker to that of your user and or docker group, not sure how you got your base environment set up...
The errors went away after moving it to /config/deps/lib/python3.7/site-packages but I'm not seeing any updates to the device_trackers. I had to enable full debug to see this output, do you know what I need to enable to see it just for this component? I tried custom_components.device_tracker.google_maps: debug without any luck.
2019-03-07 08:54:54 INFO (MainThread) [homeassistant.loader] Loaded google_maps.device_tracker from custom_components.google_maps.device_tracker
2019-03-07 08:54:54 WARNING (MainThread) [homeassistant.loader] You are using a custom component for google_maps.device_tracker which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 08:54:55 INFO (MainThread) [homeassistant.components.device_tracker] Setting up device_tracker.google_maps
2019-03-07 08:54:56 INFO (SyncWorker_11) [gmapslocsharing.cookie] Initiating Cookie Check.
2019-03-07 08:54:56 DEBUG (SyncWorker_11) [gmapslocsharing.cookie] Cookie Check - Opening login URL via requests.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.cookie] Cookie Check - Checking if cookie already exists.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.cookie] Cookie Check - Validating cookie.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.cookie] Cookie Check - Cookie exists and is valid.
2019-03-07 08:54:57 INFO (SyncWorker_11) [gmapslocsharing.cookie] Cookie checks out.
2019-03-07 08:54:57 INFO (SyncWorker_11) [gmapslocsharing.location] Performing data query.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Query Data - Requesting location data.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Query Data - Decompressing and decoding response.
2019-03-07 08:54:57 INFO (SyncWorker_11) [gmapslocsharing.location] Performing data cleanup.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Clearing previously clean output
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Moving data to local variable.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Clearing contents of self.data.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Splitting output by person per row.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Deleting useless/unknown content.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Creating dict for each person in output.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Data Cleanup - Successfully stored output for 0 people.
2019-03-07 08:54:57 INFO (SyncWorker_11) [gmapslocsharing.location] Creating people.
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Create People - Clearing previously stored people
2019-03-07 08:54:57 DEBUG (SyncWorker_11) [gmapslocsharing.location] Create People - Converting location data into Person object.
2019-03-07 08:55:28 INFO (SyncWorker_17) [gmapslocsharing.location] Performing data query.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Query Data - Requesting location data.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Query Data - Decompressing and decoding response.
2019-03-07 08:55:28 INFO (SyncWorker_17) [gmapslocsharing.location] Performing data cleanup.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Clearing previously clean output
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Moving data to local variable.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Clearing contents of self.data.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Splitting output by person per row.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Deleting useless/unknown content.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Creating dict for each person in output.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Data Cleanup - Successfully stored output for 0 people.
2019-03-07 08:55:28 INFO (SyncWorker_17) [gmapslocsharing.location] Creating people.
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Create People - Clearing previously stored people
2019-03-07 08:55:28 DEBUG (SyncWorker_17) [gmapslocsharing.location] Create People - Converting location data into Person object.
2019-03-07 08:55:59 INFO (SyncWorker_8) [gmapslocsharing.location] Performing data query.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Query Data - Requesting location data.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Query Data - Decompressing and decoding response.
2019-03-07 08:55:59 INFO (SyncWorker_8) [gmapslocsharing.location] Performing data cleanup.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Clearing previously clean output
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Moving data to local variable.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Clearing contents of self.data.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Splitting output by person per row.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Deleting useless/unknown content.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Creating dict for each person in output.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Data Cleanup - Successfully stored output for 0 people.
2019-03-07 08:55:59 INFO (SyncWorker_8) [gmapslocsharing.location] Creating people.
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Create People - Clearing previously stored people
2019-03-07 08:55:59 DEBUG (SyncWorker_8) [gmapslocsharing.location] Create People - Converting location data into Person object
alright. progress. so based on the debug, it looks like everything's working. that said, now i'm wondering how your gmaps setup is configured.
do you have an iphone or android?
- this device tracker will probably work better with android as maps is native to the OS and the device sends periodic updates to the account location is being shared with on an automated periodic basis regardless of the state of google maps.
how are you using google maps sharing?
- where did the cookie come from? did you use my component to generate the cookie or was it extracted from elsewhere?
- the use case for the proper implementation of this device tracker is to create an external google account and then share your location with this external account as you can't share your own location with...yourself...assumption being that YOU know where you are already...and you can't share your location with...yourself.
- this external account should have 2FA enabled via Google Prompt (that account should also be logged in on your mobile not only for 2FA but also to confirm and manage google maps sharing from your and other accounts as necessary).
fun fact: you should now have a folder called debug in the config folder which should contain screenshots from the selenium+chrome login process. that's for an additional layer of debugging.
Device
I'm using an Android device (Pixel 3) and everything was working fine on the old component. I was receiving regular updates with address, lat, lon and battery level.
Where did the cookie come from
The cookie is actually missing, I assumed you were storing it somewhere else. I had deleted .google_maps_location_sharing.cookies before I installed the new component as this was a common troubleshooting step to fix the old component (forces reauth).
Google Maps Implementation
I created a dedicated Google Apps account hass@mygappsdomain.com and then my family shared their location with the hass@ account. Logging in as that user, I can see everyones location.
2FA
Not enabled as this used to break the old component. Is it required?
Debug folder
Not there. Is there anything special I have to do to enable it?
- enable 2FA via Google Prompt. make sure you're logged into that account on your pixel and that the prompt should be sent to that pixel3 exclusively.
- in the gmapslocsharing folder, edit core.py and under init, change debug to True.
- if it exists, delete .google_maps_location_sharing.cookies in the config folder.
- restart home assistant
Enabled 2FA, it broke the hangouts and calendar integrations.
Edited the debug setting to True in config/deps/lib/python3.7/site-packages/gmapslocsharing/core.py and restarted Hass. No debug folder and no new errors
Still no .google_maps_location_sharing.cookies in config.
It did just prompt for auth via Google Prompt. Still no joy on a cookie or location updates.
FYI - setting gmapslocsharing.location: debug in the logger component at least gives me debug output from the library.
aaaah. ok. i think i know what the issue is. since i developed this on a standard deployment, it's probably a path issue. so...it's using Pathlib to pass the path around the component. so IF I HAD TO GUESS...you should probably have an error in syslog from docker or HA about the path not existing when trying to save the cookie as it's trying to store the cookie in a path on the local OS per a standard deployment instead of within docker...let me think on this for a sec.
you'd have to tweak the cookie storage path to reflect the relative config directory within the docker environment.
alright. for shits and giggles...add this to core.py under gmapslocsharing module:
config_path = '/config/'
cookie_filename = '.google_maps_location_sharing.cookies'
so it looks like this:
Edits made, container restarted. No joy on a cookie being generated or tracker updates.
cat config/deps/lib/python3.7/site-packages/gmapslocsharing/core.py
from .location import LocationSharing as location
from .cookie import CookieMonster as cookie
class GoogleMaps:
def __init__(self, username, password, config_path, cookie_filename, localization='US', debug=True):
config_path = '/config/'
cookie_filename = '.google_maps_location_sharing.cookies'
self.cookie = cookie(username, password, config_path, cookie_filename, localization=localization, debug=debug)
self.location = location(self.cookie.r, config_path, localization=localization, debug=debug)
def run(self):
# TODO: best practices in this scenario? try? if?
self.cookie.run()
self.location.update()
Log output
2019-03-07 11:38:00 INFO (SyncWorker_3) [gmapslocsharing.cookie] Initiating Cookie Check.
2019-03-07 11:38:00 DEBUG (SyncWorker_3) [gmapslocsharing.cookie] Cookie Check - Opening login URL via requests.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.cookie] Cookie Check - Checking if cookie already exists.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.cookie] Cookie Check - Validating cookie.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.cookie] Cookie Check - Cookie exists and is valid.
2019-03-07 11:38:01 INFO (SyncWorker_3) [gmapslocsharing.cookie] Cookie checks out.
2019-03-07 11:38:01 INFO (SyncWorker_3) [gmapslocsharing.location] Performing data query.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Query Data - Requesting location data.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Query Data - Decompressing and decoding response.
2019-03-07 11:38:01 INFO (SyncWorker_3) [gmapslocsharing.location] Performing data cleanup.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Clearing previously clean output
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Moving data to local variable.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Clearing contents of self.data.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Splitting output by person per row.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Deleting useless/unknown content.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Creating dict for each person in output.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Data Cleanup - Successfully stored output for 0 people.
2019-03-07 11:38:01 INFO (SyncWorker_3) [gmapslocsharing.location] Creating people.
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Create People - Clearing previously stored people
2019-03-07 11:38:01 DEBUG (SyncWorker_3) [gmapslocsharing.location] Create People - Converting location data into Person object.
well now i'm just confused, because it shouldn't be getting past cookie validation without a cookie. it is a hidden file, so you'd have to check for it by doing ls -la. it won't show up just via normal ls.
I thought the exact same thing. Really weird. Yep, I had checked for the hidden file.
root@shanknas:/config# ls -al .*
-rw-r--r-- 1 1000 1000 6 Feb 27 18:18 .HA_VERSION
-rw-r--r-- 1 1000 1000 4096 Dec 15 11:22 ._client_secret_snip.apps.googleusercontent.com.json
-rw-r--r-- 1 1000 1000 4096 Jan 14 10:05 ._locations.json
-rw------- 1 root root 1064 Mar 7 13:39 .google.token
-rw-r--r-- 1 1000 1000 44 Jan 5 2018 .uuid
are you running this in docker compose? if so, can you show me the entire entry for HA?
if not, how are you mapping config between local OS and HA?
I executed that from inside the docker container using docker exec -it hass bash. Here is the full mapping in docker-compose.yaml
version: "3"
services:
hass:
build: .
# image: "homeassistant/home-assistant"
image: jshank/home-assistant
container_name: hass
restart: always
network_mode: "host"
volumes:
- "./config:/config"
- "/etc/localtime:/etc/localtime:ro"
devices:
- "/dev/ttyACM0:/dev/ttyACM0"
- "/dev/ttyS0:/dev/ttyS0"
i mean...i doubt the relative path on the OS side would be the issue. i tend to prefer full paths, but if it's working for you, i suppose that shouldn't be an issue. config is not ro on the docker side, so that's not it...
are you in the US?
Sure am, just up north from you in San Jose.
are you on the HA discord?
hit me up there. same username.
A great collaboration was had. Good to close and working well for docker.