googlesamples / assistant-sdk-python

Samples and bindings for the Google Assistant API

Home Page:http://developer.google.com/assistant/sdk

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

"Illegal Instruction" by googlesamples-assistant-pushtotalk in Pi Zero(grpc issue)

Dimeiza opened this issue · comments

Problem:

I installed Googlesamples-assistant-pushtotalk(Google Assistant Service) in Pi Zero recently , but it displays "Illegal instruction" error message and I can't execute it.

20180419_pushtotalk_fail

This messages didn't appear when I installed before(3/25/2018).

Environment:

Raspberry Pi Zero W
RASPBIAN STRETCH WITH DESKTOP (2018-03-13)

procedure:

I executed official installation procedure.

https://developers.google.com/assistant/sdk/guides/service/python/embed/install-sample

cause:

I think that it is caused by grpcio pip package .

I found grpcio package installation with "grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl" from log of installation Google Assistant. This file name looks like to compliant armv6 architecture, but "Illegal Instruction" error was occurred.

So, I tried to install grpcio from source before Google Assistant installation.

python -m pip install --upgrade --no-binary :all: grpcio
python -m pip install --upgrade google-assistant-library
python -m pip install --upgrade google-assistant-sdk[samples]
python -m pip install --upgrade google-auth-oauthlib[tool]

...

Then, "Illegal instruction" error was not occurred and I can use pushtotalk well.

In Pi Zero, latest version of grpcio(1.11.0) is installed from binary package. but previous version(1.10.0) is installed from source.

I think this is why "Illegal Instruction" is occurred recently.

Request:

Would you check this problem please?
If I should send this issue gRPC team, please tell me. And I will do so.

Thank you for taking your time.

Just to report that I had this issue too when I installed Google Assistant on my ZeroW.

Thought it was related to the ReSpeaker sound card I used, So I reported it here:
respeaker/seeed-voicecard#62

Excellent that you have found the possible source of the error.

Not sure if this happens to be the root cause or not but just by looking at PyPI index, I can confirm that none of gRPC ARM packages are “arm6”. All of them are “arm7l” designated. https://pypi.org/project/grpcio/1.11.0/#files

I’m curious how you got the arm6 installed from a binary wheel?

according a installation log of google assistant...

Requirement not upgraded as not directly required: six>=1.9.0 in ./env/lib/python3.5/site-packages (from tenacity<5,>=4.1.0; extra == "samples"->google-assistant-sdk[samples]) (1.11.0)
Collecting grpcio>=1.3.5 (from google-assistant-grpc==0.1.0; extra == "samples"->google-assistant-sdk[samples])
  Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl (7.9MB)
    100% |????????????????????????????????| 7.9MB 317kB/s

I have confirmed this file can be downloaded now.

It seems that this file is a binary file to complient armv7 architecture, because I heard other user could run Google Assistant with this package in Raspberry Pi 3.

I don't grasp what happened because I don't know well about python package system.

I don't know who operates https://www.piwheels.org but for sure it is not an officially supported gRPC artifact. Some third party must have built it.

The official packages are all here: https://pypi.org/project/grpcio/#files

/cc @bennuttall who operate piwheels: is piwheels repo set by default as a PyPi mirror on Raspbian? Does it takes precedence over PyPi if arm6 wheels are missing are there is only arm7 wheels?

Ah, I can see what's happened here. It hasn't come up before.

Before piwheels, PyPI didn't accept Arm wheel uploads, so we automated building and hosting Arm wheels built from sdist of all packages on PyPI.

Since PyPI uploads moved to warehouse's back-end (long before the very recent front-end switch), Arm wheel uploads have been permitted.

Since Raspbian userland is 32-bit only, regardless on Pi model/CPU, all wheels built on a Pi 2/3 are identical to those built on Pi 1/Zero, despite the platform tag being armv6l/armv7l. But they're actually armv6 because of the agnostic userland.

We build "armv7l" wheels and provide duplicate files tagged as "armv6l" because they are compatible with all Pis in Raspbian. See RTIMULib for example.

When a compatible wheel is available on PyPI, all we do is download it, not build it from source. If this happens it is usually a pure python wheel which is not problem. However, in this case Armv7 wheels have been provided but not Armv6 (note: PyPI/warehouse does not allow uploading multiple identical files even with different tags like we do on piwheels).

is piwheels repo set by default as a PyPi mirror on Raspbian?

It's configured as an additional index in /etc/pip.conf. If you remove this file, your requests will only hit PyPI. pip will choose wheels over sdist, so more often than not, pip will chose files from piwheels over PyPI.

Does it takes precedence over PyPi if arm6 wheels are missing are there is only arm7 wheels?

Precedence is determined by the provider of the best matching file. Usually (e.g numpy) piwheels is preferred due to providing a wheel over PyPI's sdist, but in the case of a pure python package providing a wheel on PyPI (e.g. gpiozero), they are equal and PyPI is preferred as the primary index.

If the user is on an "armv7" platform (Pi 2 or 3), pip will only match armv7 wheels. armv6 is simply not a match, the same way x86 isn't. So in this case, the user would get the armv7 wheel from pypi.

If the user is on an armv6 platform (Pi 1/Zero) and no armv7 wheel is provided on PyPI, pip's choice is between a seemingly valid platform wheel from piwheels vs. sdist from PyPI. piwheels wins.

However, if the original wheel is true armv7 (as opposed to armv6 userland posing as armv7) we have a problem, because piwheels copied it to "armv6l" and is providing that to Pi 1/Zero users.

How was the armv7 wheel built? QEMU maybe? If it contains Armv7 instructions it will not work on Pi 1/Zero (and I'm not even sure about Pi 2/3).

Resolution:

  • If the wheel is true armv7, does it definitely work on a Pi 2/3? If it was actually built by Raspbian, I have no idea what happened.
  • If it works on a Pi 2/3 but not on Pi 1/Zero, can you rebuild on a Pi 1 and try to upload it to PyPI? If it differs from the armv7 wheel that will work fine - just let me know and I'll delete it from piwheels.
  • If you remove /etc/pip.conf and pip install from PyPI on a Pi 1/Zero, does building from source work? If so, we can just blacklist this package from piwheels and not get in the way by trying to provide wheels.
  • If building on a Pi 3 works, and a duplicate wheel for armv6 also works on Pi 1 then can you just upload sdist only in future and leave it to piwheels to build Arm wheels?

I will test and report 👍

I have successfully built an Arm wheel on a Pi 3, and tested installing it on another Pi 3. It seems to work but I'm not sure how to test it.

I have also duplicated it as armv6l.

You can test:

pip install http://bennuttall.com/files/grpcio-1.11.0-cp35-cp35m-linux_armv7l.whl  # pi 2/3
pip install http://bennuttall.com/files/grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl  # pi 1/zero

If those work ok, I'd suggest the maintainers don't bother uploading emulated platform wheels to PyPI, and let piwheels build them natively. Any reason not to?

@bennuttall I am not 100% sure who asked us (@kpayson64 should have more context), but I think we have had people pinging us a couple times we forgot to upload the ARM wheels simultaneously with the rest of our wheels. I suspect the reason might be not all ARM users are on Raspbian or Raspberry Pi? Frankly, we are not in love with building on ARM either, and we would love to defer to other people if we could, but I strongly suspect that's not an option. @proppy, do you think that is a viable solution for everyone?

Is this the right place for this issue? Should it be raised with grpc/grpc perhaps? Is gprc a google project or not?

Frankly, we are not in love with building on ARM either, and we would love to defer to other people if we could, but I strongly suspect that's not an option. @proppy, do you think that is a viable solution for everyone?

Yeah it's a complex beast. piwheels solves one problem for one set of users really well, i.e. raspbian / raspberry pi users. We don't claim or pretend to be trying to solve a wider Arm issue. But maintainers providing Arm wheels upstream (i.e. PyPI) must ensure they are compatible with other Arm boards. We (piwheels) do not recommend users of other Arm boards use piwheels as their repo! (they can replicate the effort if they want!)

If piwheels is breaking something for Pi 1/Zero users then I'll remove it, but moving forward if the authors can agree on a coherent solution, that would be great 👍

@bennuttall gRPC was started by Google and donated to and currently owned by CNCF. I don't think there's much to be discussed from the gRPC side. I just brought it up because you suggested completely outsourcing the ARM wheel production to pywheels, and I wanted to get a quick assessment on its fundamental feasibility from the community's perspective, as people on this thread are probably more familiar with the community needs around the ARM distribution than most, but beyond that there's not much we can do from gRPC's side. What I think makes sense regardless of the upstream is for pywheels to either serve a correct arm6 build or not serve one at all so users end up fetching and building from source.

Sorry, I can't test it because I'm so busy.
As I wrote above, I have already found out a solution for me.

I think Google Assistant team has a responsibility to test a solution and fix this problem.

There are many people that failed to install Google Assistant in own Pi Zero even if they followed official install procedure. It isn't user's problem and it is a problem of official guide, isn't it?

As a result, it seems that these people have had misunderstanding about "Google Assistant is not supported Pi Zero." I would appreciate if you write clearly support status of Pi Zero in official document to avoid misunderstanding of these users.

And as outside opinion from user-side third person, I think it is a good way that build from source if you can't cope with ARM complex environment properly. It may requires a lot of time, but it is better than halt caused by illegal binary.

I have already reported all information I have. Please handle this issue as you wish.

Thank you for your support.

@bennuttall is there a way the piwheels infra can always rebuild arm6 wheels, even if there is one one PyPi? That seems to be the root cause of the issue (I can imagine it can happens for other wheels than the grpc one).

I've replaced our wheels with the ones I built last night, so Pi 2/3 users will still get the armv7 wheel from PyPI, but Pi 1/Zero users will get the armv6 one from piwheels (whereas without piwheels they'd get sdist and would have to build - it took about 20 mins to build on a Pi 3, which could be more like 2 hours on a Pi 1).

I just discussed the matter with @kpayson64 who was trying to build the ARMv6 builds with QEMU before and he told me the generated artifacts for ARM6 from QEMU were broken and that's why we never published ARM6 ones on PyPI.

@bennuttall Can you modify the pywheels process such that it builds from source instead of renaming the ARMv7 version? I can imagine this might cause problems not just for gRPC but also other packages.

Today, I had the opportunity to build Google Assistant with Pi Zero.

That's OK.
I succeeded in executing pushtotalk without installing grpcio from the source explicitly.

Collecting grpcio>=1.3.5 (from google-assistant-grpc==0.1.0; extra == "samples"->google-assistant-sdk[samples])
  Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl (7.8MB)
    100% |????????????????????????????????| 7.8MB 376kB/s

Thank you for prompt response and improvement. I will tell it to Pi Zero users around.

There seems to remain discussion about the provision of the armv6 package, so I leave this issue open.

Thank you very much.

@bennuttall is there a way the piwheels infra can always rebuild arm6 wheels

There's no need to rebuild all wheels, because "armv6" and "armv7" wheels are identical when built on a Pi. It's only when there's an armv7 wheel on PyPI (this is the first one I've seen, as it's a relatively new addition that causes an issue).

It may be that we can check the wheel downloaded and force a rebuild when we see it's armv7. See piwheels/piwheels#66

I can confirm that this is still an issue. Ran through the example to set up the demo on my zero w, and got the Illegal instruction message.

A new grpcio has been released so I'll have to rebuild it again

Rebuilt and now in the index. Can you try again?

https://www.piwheels.org/simple/grpcio/

We will automate this forced rebuild when we have time to make the necessary changes.

Went through the example on my rpi zero w and got Illegal instruction message today as well

We recently shipped a new version of grpcio package so I suspect the substitution of arm7 packages to arm6 is still happening. @bennuttall is there a way to automate this?

I've updated again - will automate when I get time to work it out. See piwheels/piwheels#66

I've been following the install guide at http://wiki.seeedstudio.com/ReSpeaker_2_Mics_Pi_HAT/ and hit the same problem on my Pi Zero (v1.3).

I wasn't sure quite what the conclusion of the discussion above meant, but I tried doing "python -m pip install grpcio" on my Pi Zero to manually install the package, and that appeared to install an "armv6" version, however I still received an "Illegal Instruction" on running the demo. Last few lines of terminal below...


(env) root@speaky:/home/pi/projects/speaky# python -m pip install grpcio
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting grpcio
Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.13.0-cp35-cp35m-linux_armv6l.whl (8.1MB)
100% |████████████████████████████████| 8.2MB 395kB/s
Requirement already satisfied: six>=1.5.2 in ./env/lib/python3.5/site-packages (from grpcio) (1.11.0)
Installing collected packages: grpcio
Successfully installed grpcio-1.13.0
(env) root@speaky:/home/pi/projects/speaky# python3 env/bin/google-assistant-demo --device-model-id speaky-pi
Illegal instruction

So I guess there's either some other package with a wrong architecture, or the grpcio package is somehow still pulling in an incorrect version (maybe still installed elsewhere on the system after the first failed install?)

erk, sorry about the bold - it's my first github comment, I've clearly got the MD syntax wrong!

I have blacklisted the grpcio package from piwheels and deleted all files. This should prevent any issues for Pi 1/Zero users, although it will slow down installs for these users.

At a later date I will allow this package to be built for Armv6 as well which should resolve the issue without making people build from source.

@BrightOrangeMuppet can you try again?

I deleted the venv I had before and cleaned out my .cache/pip directory and started again with the Google Assistant section of http://wiki.seeedstudio.com/ReSpeaker_2_Mics_Pi_HAT/.

When it gave the "Illegal instruction" this time, I did a manual install of grpcio, and got the results below...

(env) root@speaky:/home/pi/projects/speaky# python -m pip install --upgrade grpcio
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting grpcio
  Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.13.0-cp35-cp35m-linux_armv6l.whl (8.1MB)
    100% |<ascii blocks snipped>| 8.2MB 342kB/s 
Requirement already satisfied, skipping upgrade: six>=1.5.2 in ./env/lib/python3.5/site-packages (from grpcio) (1.11.0)
Installing collected packages: grpcio
Successfully installed grpcio-1.13.0
(env) root@speaky:/home/pi/projects/speaky# env/bin/google-assistant-demo --device-model-id speaky-pi
Illegal instruction
(env) root@speaky:/home/pi/projects/speaky# 

This looks like it's followed the same grpcio install process as last time, the .whl is the same and the "six>=1.5.2" seems the same warning.

I'll do some poking around to see if I can find anything that looks like an odd conflicting package, but my Python-fu is quite weak, so don't keep your fingers crossed!

Sorry I forgot to delete the files, not just the record of them. I have now - please try again.

Aah, ok. Currently running "python -m pip install --upgrade grpcio" and it's taking a while so I assume it's doing the slow compiling work you mentioned above. Looks promising, fingers crossed!

tl;dr - YAY it works!
At least the googlesamples-assitant-pushtotalk works, the google-assistant-demo still gives "Ilegal instruction".

After some tedious shenanigans with reinstalling and compile troubles last night, I finally did a fairly thorough clean out and reinstall following Dimeiza's path of installing grpcio before any of the assistant stuff.

For some reason this also required me to "apt install libffi-dev libssl-dev" before the google-assistant-sdk would install. I also decided to bin the venv approach and be root as I'll be rebuilding the Zero when this works.

So the steps I ended up with were:

  1. pip uninstall grpc grpcio
  2. rm -rf ~/.cache/pip/*
  3. apt install libffi-dev libssl-dev
  4. python -m pip install --upgrade --no-binary :all: grpcio
  5. python -m pip install --upgrade google-assistant-library
  6. python -m pip install --upgrade google-assistant-sdk[samples]
  7. python -m pip install --upgrade google-auth-oauthlib[tool]
  8. google-oauthlib-tool --client-secrets ./mycredentials.json --scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless
    (follow the link, paste in the resulting secret string)
  9. googlesamples-assistant-pushtotalk --project-id speaky-pi --device-model-id speaky-pi

After some pondering I see "Press Enter to send a new request..." and yup, it works!

Thanks for the help Ben. I'll do some more poking at the google-assistant-demo script to see if there's a clue about which package is causing the problem there. I guess it may be another armv6/7 packaging issue?

I have been attempting to set up the Assistant Service on a Pi Zero, and am having this issue again -- I've tried both the fully official installation instructions, and @shivasiddharth 's cool setup over at https://github.com/shivasiddharth/GassistPi. It seems like it's crept back in somehow, and the resolution in this thread doesn't work as far as I can test.

Since I'm so far ahead now (running Buster), should this be a new issue? I can post whatever logs and such are needed, as I already have on shivasiddharth/GassistPi#1131.

commented

Also experiencing the "Illegal instruction" error on Raspberry Pi Zero Raspbian Buster 10. Tried @BrightOrangeMuppet 's suggestion building grpcio from scratch. Will update once its done. It's been building 40 minutes so far.

Update: yeah after building grpcio with the no binaries option (don't know how long it took, I let it run overnight) push to talk is detecting my voice!

I'm also stymied by the illegal instruction response to pushtotalk, I've tried to follow source @BrightOrangeMuppet but I see:
sudo python -m pip install --upgrade --no-binary :all: grpcio
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting grpcio
Downloading https://files.pythonhosted.org/packages/8c/34/7dafc9052bd9b2b41c5a8912aeeca01e179d16de17e9c275633d4b807330/grpcio-1.38.0.tar.gz (21.4MB)
99% |████████████████████████████████| 21.4MB 1.9MB/s eta 0:00:01Killed
so apparently I never get to the compile stage...

When I try to install grpcio with

python -m pip install --upgrade --no-binary :all: grpcio

It takes too long to install. I already tried upgrading pip but still. Any solution ?

Also experiencing the "Illegal instruction" error on Raspberry Pi Zero Raspbian Buster 10. Tried @BrightOrangeMuppet 's suggestion building grpcio from scratch. Will update once its done. It's been building 40 minutes so far.

Update: yeah after building grpcio with the no binaries option (don't know how long it took, I let it run overnight) push to talk is detecting my voice!

Did it take building about that long ?

commented

Also experiencing the "Illegal instruction" error on Raspberry Pi Zero Raspbian Buster 10. Tried @BrightOrangeMuppet 's suggestion building grpcio from scratch. Will update once its done. It's been building 40 minutes so far.
Update: yeah after building grpcio with the no binaries option (don't know how long it took, I let it run overnight) push to talk is detecting my voice!

Did it take building about that long ?

As I said, it took at least 40 minutes. I stepped away from it after 40 minutes and let it run overnight. I don't know exactly how long it took start-to-finish because I didn't see when it finish. If you get to the stage where you see GCC compiling the binaries, step away and cross your fingers that it finishes successfully. It likely doesn't take all night- maybe a few hours? If you do time how long it takes, please post your results whether successful or not.

I threw in the towel trying to build grpcio from scratch on the Pi Zero. Never could get it to work deterministically and the overnight feedback loop was frustrating. Installing an older armv6-compatible binary will work right away, but there is likely a tradeoff in functionality:

pip install grpcio -Iv grpcio==1.11.0

I'm building the recent grpcio wheels and I'll import them into piwheels when they're done.

They're built and imported now, there shouldn't be any problems any more. See: https://www.piwheels.org/project/grpcio/

Fabulous! Could you just give the install command, I've just read your 2017 blog, but I'd like to make sure what to do next.

Looks like you need:

Buster:

sudo apt install libatomic1 libstdc++6

Stretch:

sudo apt install libc6 libgcc1 libstdc++6

Sorry, I'm obviously missing something crucial:

sudo apt install libatomic1 libstdc++6
Reading package lists... Done
Building dependency tree
Reading state information... Done
libatomic1 is already the newest version (8.3.0-6+rpi1).
libstdc++6 is already the newest version (8.3.0-6+rpi1).
0 upgraded, 0 newly installed, 0 to remove and 249 not upgraded.

sudo pip3 install grpcio
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: grpcio in /usr/local/lib/python3.7/dist-packages (1.33.2)
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.7/dist-packages (from grpcio) (1.15.0)

And as before, not surprisingly given the above

pi@lib:~/env/bin $ ./googlesamples-assistant-pushtotalk
Illegal instruction

this is with /etc/pip.conf with

[global]
extra-index-url=https://www/piwheels.org/simple

Linux lib 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l
Build info: Fri Nov 20 09:43:06 UTC 2020 @

Uninstall grpcio first:

sudo pip3 uninstall grpcio -y
sudo pip3 install grpcio

Linux lib 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l
Build info: Fri Nov 20 09:43:06 UTC 2020 @
Last login: Fri Jul 2 16:43:28 2021 from fe80::cf6:ae3b:cbaa:c626%wlan0
pi@lib:~ $ sudo pip3 uninstall grpcio -y
Uninstalling grpcio-1.33.2:
Successfully uninstalled grpcio-1.33.2
pi@lib:~ $ sudo pip3 install grpcio
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting grpcio
Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.38.1-cp37-cp37m-linux_armv6l.whl (44.3MB)
99% |████████████████████████████████| 44.3MB 1.4MB/s eta 0:00:01Killed
pi@lib:~ $ cd env/bin
pi@lib:~/env/bin $ ./googlesamples-assistant-pushtotalk
Illegal instruction

What happened here?

99% |████████████████████████████████| 44.3MB 1.4MB/s eta 0:00:01Killed

It looks like it didn't complete?

Try:

wget https://www.piwheels.org/simple/grpcio/grpcio-1.38.1-cp37-cp37m-linux_armv6l.whl 
sudo pip3 install grpcio-1.38.1-cp37-cp37m-linux_armv6l.whl 

Right, /var/log/kern.log reports Out of memory. I increased the swap space (this is a pi 0) from 100 to 200 and now it installs

pi@lib:~ $ sudo pip3 install grpcio
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting grpcio
Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.38.1-cp37-cp37m-linux_armv6l.whl (44.3MB)
100% |████████████████████████████████| 44.3MB 1.9kB/s
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.7/dist-packages (from grpcio) (1.15.0)
Installing collected packages: grpcio
Successfully installed grpcio-1.38.1

but I still get
pi@lib:~/env/bin $ ./googlesamples-assistant-pushtotalk
Illegal instruction

Do I need the prerelease?

Seems that the prerelease version doesn't help either:

pi@lib:~ $ sudo pip3 uninstall grpcio -y
Uninstalling grpcio-1.38.1:
Successfully uninstalled grpcio-1.38.1
pi@lib:~ $ sudo pip3 install grpcio==1.39.0rc1
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting grpcio==1.39.0rc1
Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.39.0rc1-cp37-cp37m-linux_armv6l.whl (45.1MB)
100% |████████████████████████████████| 45.1MB 1.8kB/s
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.7/dist-packages (from grpcio==1.39.0rc1) (1.15.0)
Installing collected packages: grpcio
Successfully installed grpcio-1.39.0rc1
pi@lib:~ $ cd env/bin
pi@lib:~/env/bin $ ./googlesamples-assistant-pushtotalk
Illegal instruction