Restricting the user agent for Chromium does not work

Question

Restricting the user agent for Chromium does not work

BlueMax opened this issue 10 years ago · comments

BlueMax commented 10 years ago

CM11 (24.11.2013)
Galaxy SIII - I9300
XPrivacy 1.10.27
Xposed 2.4 beta 1

Marcel Bokhorst commented 10 years ago

:-(

Marcel Bokhorst commented 10 years ago

:-(

Marcel Bokhorst · Answer 1 · Tue Nov 26 2013 22:02:57 GMT+0800 (China Standard Time)

@BlueMax can you please confirm if this version can restrict the user agent on KitKat: http://d-h.st/vHJ

BlueMax · Answer 2 · Tue Nov 26 2013 22:13:57 GMT+0800 (China Standard Time)

Yep, gimme a minute...

BlueMax · Answer 3 · Tue Nov 26 2013 22:32:14 GMT+0800 (China Standard Time)

Doesn't work... Also, the user-agent permissions are not present anymore. There are only two right now. There used to be 6 or so...

Marcel Bokhorst · Answer 4 · Tue Nov 26 2013 22:38:52 GMT+0800 (China Standard Time)

About the restrictions: that is correct, getSettings has replace the previous present functions.

Did you reboot your device after installing the test version of XPrivacy?
Can you enable XPrivacy logging (settings menu) and capture a logcat like this:

adb shell "logcat | grep XWeb"

Marcel Bokhorst · Answer 5 · Tue Nov 26 2013 22:40:33 GMT+0800 (China Standard Time)

The screenshot shows getSettings was actually called.

BlueMax · Answer 6 · Tue Nov 26 2013 22:49:46 GMT+0800 (China Standard Time)

Yep, i did reboot.

I/XPrivacy/XWebView( 1927): android: hooked android.webkit.WebView.getSettings/view (1)
I/XPrivacy/XWebView( 6438): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6438): Hooking getUserAgentString
I/XPrivacy/XWebView( 6438): Hooking setUserAgentString
I/XPrivacy/XWebView( 6438): Queue usage data=10027/view/getSettings=true size=4
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): Queue usage data=10027/view/getSettings=true size=7
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (file)

Marcel Bokhorst · Answer 7 · Tue Nov 26 2013 23:13:54 GMT+0800 (China Standard Time)

@BlueMax can you please try if this version works: http://d-h.st/EvR

BlueMax · Answer 8 · Tue Nov 26 2013 23:20:25 GMT+0800 (China Standard Time)

Still no go...
Also, new permission 'WebView.constructor' has no orange triangle.

I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.constructor/view (4)
I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.getSettings/view (1)
I/XPrivacy/XWebView( 6489): Queue usage data=10027/view/WebView.constructor=true size=4
I/XPrivacy/XWebView( 6489): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6489): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6489): Hooking getUserAgentString
I/XPrivacy/XWebView( 6489): Hooking setUserAgentString
I/XPrivacy/XWebView( 6489): Queue usage data=10027/view/getSettings=true size=5
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): Queue usage data=10027/view/getSettings=true size=8
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6651): Queue usage data=10027/view/WebView.constructor=true size=4
I/XPrivacy/XWebView( 6651): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6651): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6651): Hooking getUserAgentString
I/XPrivacy/XWebView( 6651): Hooking setUserAgentString
I/XPrivacy/XWebView( 6651): Queue usage data=10027/view/getSettings=true size=5
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): Queue usage data=10027/view/getSettings=true size=8
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (file)

BlueMax · Answer 9 · Tue Nov 26 2013 23:24:47 GMT+0800 (China Standard Time)

Wait, wait, wait.... It works. something is cached. If i change the user-agent (switch to desktop version) it shows faked data.

I will try previous version v1.10.28.1.

Marcel Bokhorst · Answer 10 · Tue Nov 26 2013 23:25:17 GMT+0800 (China Standard Time)

Could you capture a log like this:

adb shell "logcat | grep XPrivacy"

start just before you run the browser, but reboot first, so I can see the initial hooking too.

BlueMax · Answer 11 · Tue Nov 26 2013 23:30:05 GMT+0800 (China Standard Time)

It just works when i switch to 'desktop' user-agent. And now even the 'WebView.constructor' permission is lit. But when i switch back to normal/mobile user-agent its back to unfaked state.

Correction: It just works for one time (right after switching user agent in browser - no matter what user agent). Reloading the page resets to original state. Switching the user-agent in the browser gives another one-time fake.

Marcel Bokhorst · Answer 12 · Tue Nov 26 2013 23:35:58 GMT+0800 (China Standard Time)

Can you provide me with a logcat for the switch?

BlueMax · Answer 13 · Tue Nov 26 2013 23:44:40 GMT+0800 (China Standard Time)

Sure...
http://pastebin.com/vfEGruYJ

What has been done:

logcat started
Browser loaded page with original user agent
Agent switched manually in browser
Page gets automatically reloaded (shows faked user agent=ggg)
logcat aborted

Marcel Bokhorst · Answer 14 · Tue Nov 26 2013 23:56:36 GMT+0800 (China Standard Time)

Another test version: http://d-h.st/rES

(sorry, I cannot test myself, since I don't have a device that can run KitKat yet)

If it not works, please make a new logcat:

adb shell "logcat | grep XWeb"

BlueMax · Answer 15 · Wed Nov 27 2013 00:04:37 GMT+0800 (China Standard Time)

Indeed.. :)
Even the switch doesn't work. Its crashing somewhere...

I/XPrivacy/XWebView( 1927): android: hooked android.webkit.WebView.constructor/view (4)
I/XPrivacy/XWebView( 1927): android: hooked android.webkit.WebView.getSettings/view (1)
I/XPrivacy/XWebView( 6563): Queue usage data=10027/view/WebView.constructor=true size=4
I/XPrivacy/XWebView( 6563): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6563): Hooking com.android.webview.chromium.ContentSettingsAdapter
W/System.err( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:72)
W/System.err( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:60)
I/Xposed ( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:72)
I/Xposed ( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:60)
I/XPrivacy/XWebView( 6563): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6563): Queue usage data=10027/view/WebView.constructor=true size=7
I/XPrivacy/XWebView( 6563): get 10027/WebView.constructor view=restricted (file)

Marcel Bokhorst · Answer 16 · Wed Nov 27 2013 00:19:30 GMT+0800 (China Standard Time)

Another test version: http://d-h.st/Pll

BlueMax · Answer 17 · Wed Nov 27 2013 00:27:56 GMT+0800 (China Standard Time)

Still only for one time...

I/XPrivacy/XWebView( 6757): Queue usage data=10027/view/WebView.constructor=true size=1
I/XPrivacy/XWebView( 6757): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6757): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6757): Hooking setUserAgent
I/XPrivacy/XWebView( 6757): Hooking setUserAgentString
I/XPrivacy/XWebView( 6757): Queue usage data=10027/view/getSettings=true size=2
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): Queue usage data=10027/view/getSettings=true size=1
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)

Marcel Bokhorst · Answer 18 · Wed Nov 27 2013 01:53:23 GMT+0800 (China Standard Time)

Unfortunately I am out of ideas how to fix this at the moment.
I will think about it, maybe I will get a good idea.

gdmzyejian · Answer 19 · Mon Dec 02 2013 01:53:00 GMT+0800 (China Standard Time)

Also does not work in version 4.12

Marcel Bokhorst · Answer 20 · Mon Dec 02 2013 02:23:10 GMT+0800 (China Standard Time)

@gdmzyejian :
I am using CM10 = Android 4.1.2 and it works perfectly for me.
Please provide a logcat.

BlueMax · Answer 21 · Mon Dec 02 2013 03:30:12 GMT+0800 (China Standard Time)

On KitKat its the following scenario:
If i start the browser with https://www.startpage.com/m, search for 'user agent' and go to www.useragentstring.com (5th link, creates new tab) the user agent is original/untouched. But when i copy the URL (www.useragentstring.com), restart the browser and paste the URL directly into the browser the user agent string is faked. Also, most sites i enter by bookmarks are faked properly.

Its probably somewhat related to the new tabs. I created several stack traces, forced all Set-/GetUserAgent, parent/child methods and linked lists/hashtables with constants but there's still an alternative runtime path. I couldn't find a proper target.

The browser uses an internal counter for custom user agents (overriding system static string from framework-res.apk). If that value is '0' system static settings are in effect. I traced it back and forth into several system parts browser.apk/framework-res.apk/WebviewChromium.jar and also stumbled upon native methods but whatever i touched the original user agent always came through. The shared/static system settings are kinda hard to trace offline. They use exceptions and abstract methods to sync the settings and i don't have a proper debug chain installed.

My theory is that the browser omits filling the user agent value on purpose so that the system fills it up with static system values. But i couldn't test it yet...

Marcel Bokhorst · Answer 22 · Mon Dec 02 2013 03:37:36 GMT+0800 (China Standard Time)

@BlueMax good work!

Maybe getUserAgent(String) needs to be hooked to fix this.

Did you find useful class/method names to look into?
Chromium is open source, so we can look things up.

BlueMax · Answer 23 · Mon Dec 02 2013 06:36:51 GMT+0800 (China Standard Time)

Didn't you already hooked getUserAgentString? It's present in the pastebin log above.
I haven't found any new methods i haven't had verified already being ineffective as well. I will poke around some more later.

Marcel Bokhorst · Answer 24 · Mon Dec 02 2013 18:16:01 GMT+0800 (China Standard Time)

@BlueMax could you please try if this version fixes your problem: http://d-h.st/4Wc

Please capture a log (whether it works or not).

Marcel Bokhorst · Answer 25 · Mon Dec 02 2013 19:56:55 GMT+0800 (China Standard Time)

Please try this newer version: http://d-h.st/IIe

BlueMax · Answer 26 · Mon Dec 02 2013 22:39:56 GMT+0800 (China Standard Time)

Still no go...

Log:
Startpage.com (probably faked) -> useragentstring.com (unfaked)
browser restart -> useragentstring.com (faked)

I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.constructor/view (4)
I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.getSettings/null (1)
I/XPrivacy/XWebView( 6541): Queue usage data=10027/view/WebView.constructor=true size=3
I/XPrivacy/XWebView( 6541): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked android.webkit.WebSettings.getDefaultUserAgent/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.getUserAgent/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.getUserAgentString/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.setUserAgent/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.setUserAgentString/view (1)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=4
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebView( 6541): Queue usage data=10027/view/WebView.constructor=true size=7
I/XPrivacy/XWebView( 6541): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=7
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebView( 6541): Queue usage data=10027/view/WebView.constructor=true size=2
I/XPrivacy/XWebView( 6541): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=3
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=3
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)

BlueMax · Answer 27 · Mon Dec 02 2013 23:33:51 GMT+0800 (China Standard Time)

I've traced the string some more. The one in framework.res (strings.xml) seems to be obsolete/outdated:
Mozilla/5.0 (Linux; U; Android %s) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 %sSafari/534.30

The real user agent string is:
Mozilla/5.0 (Linux; Android 4.4; GT-I9300 Build/KRT16S) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36

Note the 534.30 <> 537.36. Probably legacy string from old engine not needed in new chromium engine anymore.

I just started analyzing the native chromium libraries (20MB monster). It seems it does contain the real user-agent string mask (Mozilla/5.0 (%s) AppleWebKit/%d.%d (KHTML, like Gecko) %s Safari/%d.%d). I've hooked some native methods already leading to nothing (same partial leak) but there must be another bit/flag/switch/setting/whatever that leads to different runtime path (ignoring all setUserAgentString methods).

I will cross reference the string mask to see whats involved.

Marcel Bokhorst · Answer 28 · Mon Dec 02 2013 23:39:56 GMT+0800 (China Standard Time)

@BlueMax again thanks for looking into this!

I guess the old Android browser is present next to Chromium, explaining two user agent strings.

BlueMax · Answer 29 · Mon Dec 02 2013 23:45:52 GMT+0800 (China Standard Time)

They just started implementing chromium. Several things are still buggy like saving passwords, incognito mode and some dialogs/TextViews (like the one that pops up when you want to change the theme on XDA forum). Logcat shows crash due to "Not implemented yet". :-)

BlueMax · Answer 30 · Wed Dec 04 2013 09:29:57 GMT+0800 (China Standard Time)

Can you try hooking Landroid.webkit.WebView.loadUrl? Its an 'abstract method' that leads to Chromium for KitKat and into Webkit(?) for others. If this works all other user-agent related hooks for all systems should be obsolete.

The class holds two methods. Both are used in Android Browser.
.method public loadUrl(Ljava/lang/String;)V
.method public loadUrl(Ljava/lang/String;Ljava/util/Map;)V

1st argument is URL, 2nd is additional http header parameters including user-agent.

XPrivacy could force-jump 1st method into 2nd one and 2nd method could be hooked with (static) faked 2nd argument holding the user-agent.

If that's not possible, unreasonable or doesn't work we can still tackle Chromium directly by enforcing something like the below runtime path as a last resort. Its probably not as "code evolution resistant" as the loadUrl method above though.

java.lang.Throwable: stack dump
java.lang.Thread.dumpStack(Thread.java:489)
com.android.org.chromium.content.browser.LoadUrlParams.setOverrideUserAgent(LoadUrlParams.java:204)
com.android.org.chromium.android_webview.AwContents.loadUrl(AwContents.java:879)
com.android.webview.chromium.WebViewChromium.loadUrlOnUiThread(WebViewChromium.java:629)
com.android.webview.chromium.WebViewChromium.loadUrl(WebViewChromium.java:524)
android.webkit.WebView.loadUrl(WebView.java:785)
com.android.browser.Tab.loadUrl(Tab.java:1747)

setOverrideUserAgent is integer, held '2' as i logged and is only called when fake was successful. Probably involved in enabling honoring setUserAgentString internally. Somewhere there i expect the target. Haven't looked all too thoroughly into this yet. I hope we don't need it...

Marcel Bokhorst · Answer 31 · Wed Dec 04 2013 16:54:58 GMT+0800 (China Standard Time)

Test version with loadUrl hooked: http://d-h.st/j4E

BlueMax · Answer 32 · Thu Dec 05 2013 04:12:34 GMT+0800 (China Standard Time)

Still no go...
The problematic case is when a webpage opens a new tab/view by itself. If a page will be opened explicitly by 'open in new tab' the tab is stable/faked under every condition. But when it's been opened automatically by pressing a link its unfaked. Going into URL bar just pressing -enter- fakes the page for one shot. Reloading the page falls back to unfaked. Reloading also has none of the hooked methods involved (at least none show up with grep XWeb).

These two 'open new tab' scenarios show different runtime paths. The stable one shows 'setUserAgentString' and 'LoadUrl'. The unstable one only has 'setUserAgentString'.

I've disabled hooking 'LoadUrl' and it doesn't show any difference so its probably not of any use.

I will inject more stack dumps and watch variables to analyze the two scenarios. There's still enough left i haven't looked into...
UA_OVERRIDE_FALSE:I
UA_OVERRIDE_INHERIT:I
UA_OVERRIDE_TRUE:I

getUserAgentLocked
nativeUpdateUserAgentLocked
nativeGetDefaultUserAgent
setUserAgentOverride
nativeSetUseDesktopUserAgent
nativeGetUseDesktopUserAgent

Marcel Bokhorst · Answer 33 · Thu Dec 05 2013 04:18:10 GMT+0800 (China Standard Time)

Thanks for testing.
With your persistence I believe we can fix this problem!

gdmzyejian · Answer 34 · Thu Dec 05 2013 11:20:57 GMT+0800 (China Standard Time)

In the Android 4.12, MIUIv5, MIUI browser can fake UA, but others do not work.

gdmzyejian · Answer 35 · Thu Dec 05 2013 11:25:06 GMT+0800 (China Standard Time)

Xprivicy 1.10.37
MIUI V5 （4.12）
Xposed 2.4.1

BlueMax · Answer 36 · Sat Dec 07 2013 10:15:55 GMT+0800 (China Standard Time)

Its a bug in the chromium engine (native code). The chromium internal child tabs do not properly inherit parent properties delivered from Android/Java code. The built-in custom user-agents (Android browser) are affected as well. Most user actions ("open in new tab") result in a new (parent) WebView instance (thus unaffected) compared to clicked links that create new tabs automatically (as child). Unless we can hook native code i'm out of ideas at the moment.

Marcel Bokhorst · Answer 37 · Sat Dec 07 2013 15:22:17 GMT+0800 (China Standard Time)

@BlueMax thanks for your research!

I guess we have to wait until this Chromium bug is fixed.

BlueMax · Answer 38 · Mon Dec 09 2013 04:00:44 GMT+0800 (China Standard Time)

Just for reference:

The primary source (method) for the browser getting the system user-agent from the engine is 'nativeGetDefaultUserAgent'. It feeds all UA variables but hooking involves some risks since the browser compares old/new user-agent in some cases and prevents (if both equally) calling methods involved in overriding UA intentionally (thus native code will never be contacted about an intended change/override resulting in untouched/unfaked UA).
loadUrl is somewhat more reliable than setUserAgentString on child tabs. Its the only hook that remains stable on child tabs when an URL is entered manually afterwards (startpage.com->search for 'user agent'->pick useragentstring.com (opens in child tab)->Press enter in URL bar on same address->reload page). 'SetUserAgentString' brakes this behavior if hooked as well (child tab URL reload only stable for one load).
There is at least one app ("Codecheck") that doesn't like loadUrl. It loads the page but immediately blanks it out. If there are more side-effects like this 'loadUrl' might be a 'dangerous' candidate (unless there's something wrong with the hook implementation).

Marcel Bokhorst · Answer 39 · Mon Dec 09 2013 15:50:33 GMT+0800 (China Standard Time)

Thanks again for your research!

I only want to hook native/private methods as a last resort, because these hooks can break easily in a future releases
For now I think it is good to keep both hooks (setUserAgentString and loadUrl)