Assistant capability and reverse-engineering thereof

[Ilya]

I have had this idea for some time, but have very little information to do away with.

Thankfully, Pixel Buds Pro work well with GrapheneOS on Android, so I had been wondering: you know how these support AI assist via Google Assistant application? Now that there's many Mac Studio's and Ryzen DDR5-based inference environments in the market, on-premise cloud deployments have become more commonplace. Also: there has been improvements on-edge, namely whisper.cpp and llama.cpp builds via Android NDK are able to leverage local accelerators.

Is there a chance that the proprietary AI assist capability from Google isn't entirely proprietary?

I wonder if AI assist is ultimately implemented using some signalling protocol over Bluetooth that basically defers to recording and subsequent playback of arbitrary audio. Obviously, there's playback interrupt features but it's probably handled by the firmware itself. If this is the case, then we could potentially modify Pixel Buds application itself to use alternative inference providers.

Based on what you know about Bluetooth & Pixel Buds specifically, how hard could it be, and where would we want to start to learn specifically what's happening under the hood? If it should involve reverse-engineering the relevant apk's, would it also be tractable? If yes, what are the chances it would yield readable, meaningful results? I reckon they probably use NDK themselves, tapping into Android's bluetooth internals, and from the symbols & sniffing alone would it be possible to infer complete logic?

P.S. Great work, your code is quite tidy & indeed, would be worthwhile to investigate how what you do relates to what they do.

[Maximilian Luz]

Hi, sorry for the late reply. Somehow I forgot about this...

First off, I agree with you that the playback interruption is likely firmware based, I think you could probably try if that still works when it's not connected to an android device.

Regarding the AI assistant feature, I'm pretty sure it just forwards the microphone to the device somehow. There are some additional protocols beyond the maestro one (for controlling settings etc.), and I think that one of those is responsible for that. But I haven't looked into them at all yet, so I can't say which one or how exactly it works.

So for implementing something like that, I think we'd first need to figure out how the pixel app/android integration does that. Again, it's probably some kind of RFCOMM bluetooth protocol, so if we can figure that out, then I'm pretty sure we can make that work. IIRC I tried to figure out how the long-press-button event is being sent, but it's not via the maestro protocol.

The way I went for the maestro/settings stuff is by reverse engineering the pixelbuds apk, so something like that could work again. Unless it's somehow a deeper integration into android itself via some device-independent protocol.

I reckon they probably use NDK themselves, tapping into Android's bluetooth internals, and from the symbols & sniffing alone would it be possible to infer complete logic?

That should be possible as well, I think. If you can manage to make that work. I briefly tried some sniffing via wireshark but that didn't work out for me.