Android 12 scoped storage hell
hrydgard opened this issue · comments
Android Q will introduce "scoped storage", which imposes really heavy restrictions on external storage. And in the version after Q, it will be a requirement for all apps, even those who target Android-28 (which PPSSPP will do as long as possible). So we will be able to limp along for a while longer without it, but next year, new devices will start requiring it, and it'll greatly worsen the user experience. We need to find ways to make it as workable as possible.
For us, the filesystem really is part of the UX to match the real PSP with an USB connection.
Users will additionally have problems if they install the app on an older OS, upgrade to Q, and then downgrade to an older version of the app (which is done sometimes to bisect bugs, etc).
The one bright point is that the scoped storage APIs allows getting file descriptors so I/O performance will at least not be hampered much - assuming the file descriptors are unrestricted with regards to seek etc.
https://android-developers.googleblog.com/2019/04/android-q-scoped-storage-best-practices.html
EDIT: I have requested a reprieve through AndroidManifest.xml, the hammer drops with Android 12.
It seems there is a way to delay it even when targeting Q:
https://developer.android.com/preview/privacy/scoped-storage#opt-out-of-filtered-view
But presumably not for R.
-[Unknown]
Would PPSSPP be able to do something like, on API levels up to 28 (or 29 with the flag @unknownbrackets references above) use the default structure.
For 29+ with scoped storage, use ACTION_OPEN_DOCUMENT_TREE and ask the user to pick the "PSP" folder.
Set the flags in the Intent to: Intent.FLAG_GRANT_READ_URI_PERMISSION | Intent.FLAG_GRANT_WRITE_URI_PERMISSION | Intent.FLAG_GRANT_PERSISTABLE_URI_PERMISSION
This will allow PPSSPP to, via ContentResolver, read and write to the folder structure there, albeit not as easily as with the File APIs.
I recommend looking at DocumentFile though I'd recommend against using it directly. Its performance is not very great =(
The other folders, including MUSIC, PICTURE, and VIDEO seem like they could be replaced with using MediaStore when inside scoped storage, or you could use a similar method to the "PSP" directory to get persistent access to those as well.
I'm guessing you already know about ParcelFileDescriptor and ParcelFileDescriptor#detachFd(), which should hopefully work for accessing files via the Storage Access Framework from C++.
Hopefully this is a bit of help.
Well, it seems more complicated than that. Feel free to correct me if I'm misunderstanding.
For example, PSP games will try to call sceIoLseek on returned PSP file handles. They may do something like this:
- Open a save data file.
- Seek to the data position (0x100) and read data.
- Seek to the header (0x000) and read the time information.
- Seek back to the footer (0xF00) and read more data.
Those are just example offsets, but we can't deal with the concept that we "might" get a file descriptor that doesn't support seeking. So the experience probably becomes:
- User selects a document tree.
- PPSSPP tries to open a test file in that tree and determines it's unusable.
- PPSSPP says something the user will ultimately read as "guess again, idiot."
- User gets frustrated and quits app. It doesn't work.
Unless I'm missing some way to filter the "trees", this will at best be a terrible experience, and at worst be straight up unusable on some devices.
The alternative is to try to hack around the limitations in the document API. For example, if a file is opened for read and write, read the whole thing in, then open it again for writing and wait for the game to write (possibly with seeking.) There will be tons of bugs, save data corruption, and other issues. Sounds like some kind of nightmare.
As long as we get a real file descriptor, then we should be fine I guess for saving. It's the path to get there that's the problem.
Given all those problems, I assume we won't be making users select the PSP folder by default, and instead use the hidden away app-private folder. Then we need to handle "installing" gamefaqs zips of saves, and probably need a way to transfer data in out. Hopefully one that doesn't make PPSSPP require network for you to play a single-player game that autosaves. See #1019.
And that's just about saves. Haven't even gone into how people will transfer ISOs onto their device, or navigate to homebrew. It has many of the same problems, although at least ISOs are read only (they absolutely require seeking, though.)
So - yes. It can be worked around. As long as everything is device-local storage, it's not even a big technical problem. It's just going to be very confusing for anyone who doesn't understand storage really well.
-[Unknown]
Alright, so some news!
https://developer.android.com/preview/privacy/storage has been updated with the changes for Android 11.
We have a reprieve, in that as long as we only target Android 10, we can still opt out! So that pushes the issue to the day when Google Play will no longer accept apps targeting 10, or I guess when 12 comes out. So we can probably stumble along another year or so without changes.
There's also a new MANAGE_EXTERNAL_STORAGE permission, that will be very cumbersome to explain to users how to enable though, which makes things easier for us if enabled. There's also support to use raw paths to read "external media libraries" whatever that is, might possible be useful?
Either way, no longer an emergency but I should really start preparing :P
Oops, this wasn't meant to be closed by that.
-[Unknown]
Bumping this to the next version, since we still have quite some time left...
Alright, the final deadline has arrived, and it's November 2021.
https://android-developers.googleblog.com/2020/11/new-android-app-bundle-and-target-api.html
We need to find workaround and solutions for this before then, otherwise we won't be able to make new releases on Google Play. And if Android 12 is released before then, we'll also be in trouble there because they probably won't keep allowing 10-targeted apps to access storage freely.
I plan to do the work early spring.
According to this https://stackoverflow.com/questions/60360368/android-11-r-file-path-access
On Android R, the
Filerestrictions that were added in Android Q is removed. So we can once again accessFileobjects.
...
It is recommended to only useFileobjects when you need to perform "seeking", like when using FFmpeg, for example.If you want to access a
Fileor want a file path from aUrithat was returned fromMediaStore, I've created a library that handles all the exceptions you might get. This includes all files on the disk, internal and removable disk. When selecting aFilefrom Dropbox, for example, theFilewill be copied to your applications directory where you have full access, the copied file path will then be returned.
Well we'll still need to deal with Android Q
There's still the "guess again, idiot" problem where users will be able to select folders (i.e. Dropbox) that don't support seeking.
Example: a user will put an ISO on their Dropbox account. Then PPSSPP will ask where their ISOs are, and pop up a dialog with their Dropbox account in the list. However, selecting that account won't work. We won't have a File to back it, presumably.
-[Unknown]
I'm gonna use this comment as a notepad, adding findings as I find them.
High level concerns
We will eventually need to configure PPSSPP to target Android 12, which will enforce Scoped Storage all the way back to Android 10. These devices need a nice migration path.
Our use of on-device storage
There are two main things to deal with:
Picking and loading ISO files
- We will support the Android file picker. It seems that we can obtain a file descriptor to a picked file - and there is an OPENABLE flag to pass, that will filter out stuff like Google Drive that we can't get file descriptors from.
- We will probably be able to use the current file browser within \Android\data\org.ppsspp.ppsspp\files so if users put ISOs there, they'll be easily loadable
The PSP/ directory, containing saves, config, everything else
-
As far as I understand, we will have full reign over \Android\data\org.ppsspp.ppsspp\files on all storage devices including the main one. This is the path we get from
getExternalFilesDir(null).getAbsolutePath(). It has a few advantages and drawbacks:- Plus: It's always there, no need for the user to create or choose it in any way.
- Plus: It is user-accessible through USB from PC, although the user has to dive in a few directory levels
- Minus: If the user manually uninstalls the app, this directory will be deleted. I predict many complaints about lost save games
- Minus: Still more unwieldy than just /PSP in the root
- Minus: The user will not automatically understand where this is, it'll need explanation for USB access
- Plus/Minus: No longer shared with PPSSPP Gold since it'll get \Android\data\org.ppsspp.ppssppgold\files
-
Still figuring out whether a directory from OPEN_DOCUMENT_TREE will be viable as a PSP/ directory - how low a level of access we really get. If this is viable, maybe we can just ask the user to choose where the PSP/ folder should be placed. This path we can't store in PPSSPP.ini since that lives in the path, instead we'll probably unconditionally move the ini file to the getExternalFilesDir directory so that it's still user-accessible, no matter where the user chooses to put the PSP folder.
Current release plan
Release PPSSPP 1.12 which will do the following:
- Still targets Android 10, so it gets the exception, nothing needs to change for users for now
- Offer to migrate the PSP/ directory to \Android\data\org.ppsspp.ppsspp\files
Release PPSSPP 1.13 which does the following:
- Targets 12. Hopefully OPEN_DOCUMENT_TREE will work out - in that case we can ask the user to pick the PSP directory, or to migrate it to \Android\data\org.ppsspp.ppsspp\files .
Plus: It is user-accessible through USB from PC, although the user has to dive in a few directory levels
Hm, I thought I saw issues for some people - not on all devices - not having access to this specific path via USB storage. Could've misunderstood for sure. If that consistently works, that's a lot better (even if hard to find.)
This path we can't store in PPSSPP.ini since that lives in the path, instead we'll probably unconditionally move the ini file to the getExternalFilesDir directory so that it's still user-accessible, no matter where the user chooses to put the PSP folder.
We currently put "memstick_dir.txt" inside the "data" directory that is not normally accessible. Then we load the ini from that folder. I think this is better because then the ini is at least as accessible as the other files, and it's a straight-forward experience across platforms.
-[Unknown]
Oh right, I forgot we already have the memstick_dir.txt mechanism like that.
Some more research into OPEN_DOCUMENT_TREE returns a content URI that we need a mess of probably-slow java APIs to traverse, read and write. No way to hand it over to native I/O, except on a single file basis, as far as I can tell.
So I'm thinking to always or at least by default keep the PSP directory under \Android\data\org.ppsspp.ppsspp , and offer a mechanism to wholesale migrate a PSP directory from elsewhere using OPEN_DOCUMENT_TREE to read the data. That could be done in a single java function so we don't need to expose a fine grained file API through JNI.
Although if we want to use OPEN_DOCUMENT_TREE for the builtin game browser, we probably need at least directory listing.
What's even nastier is that the DocumentFile wrapper which exposes a somewhat sane Java API for traversing such directories is not actually part of the Android API, instead it's in the AndroidX library, which I think is not compatible as far back as we want it to be. So either I'll just grab the relevant parts and put them in our repo, or write my own wrapper.
Are we able to get an fd from a directory at all - even a subdirectory (the parcel thing?) If yes, we could use fchdir potentially and be home free.
If openFileDescriptor won't even ever return a native directory, I guess my "guess again, idiot" scenario above was too rosy.
-[Unknown]
Unlike OPEN_DOCUMENT by default, OPEN_DOCUMENT_TREE appears to be restricted to local folders. It won't, as far as I can tell, let you open a folder on Drive.
Still, that doesn't mean we can get an fd for a dir through it, but does solve the "guess again" problem to a large degree.
Well, if we can't actually use the folder without sending all directory traversal though Java, it does seem worse... by which I mean, you can't even guess again. I also wouldn't depend on OPEN_DOCUMENT_TREE never returning remote paths - clearly the API is intended for that, right?
I suppose if it's only directory traversal, and we can get regular fds which are seekable, we could probably get remoting working for all the directory traversal. Game code can really only open directories, list them, and list through Savedata. Iostat would be more problematic...
So I'm assuming we'll just go the locked-in-a-folder, always-wipe-on-uninstall approach.
Should import also delete the source files? Worried about a user with homebrew that are big, or confusion when they add additional files into the old PSP tree.
How does this interact with users wanting their storage on an actually external microSD card? Will they no longer be able to?
I guess we also need export also, so that you can export before uninstalling. Ugly.
-[Unknown]
Lots of questions, few clear answers. Yeah, I do think we'll end up with that, with just ISO browsing through OPEN_DOCUMENT_TREE.
As for data on SD card, there is getExternalFilesDirs(), an array-return version of getExternalFilesDir(), which does return the /Android/data/org.ppsspp.ppsspp folder on every SD card attached, with full access, as fas as I can tell. Not yet calling that, but could easily.
Yeah, it'll end up ugly. I'm not happy at all about any of this.
Unlike OPEN_DOCUMENT by default, OPEN_DOCUMENT_TREE appears to be restricted to local folders. It won't, as far as I can tell, let you open a folder on Drive.
I believe that NextCloud supports ACTION_OPEN_DOCUMENT_TREE but I think you should be able to use Intent.EXTRA_LOCAL_ONLY to restrict the selection to local files.
Still, that doesn't mean we can get an fd for a dir through it, but does solve the "guess again" problem to a large degree.
I haven't checked myself, but I asked an engineer and they think it should be possible to take the FD of a directory and use fdopendir(3) to read it. (If you try it and it doesn't work, can you let me know?)
So I have some Samba shares mounted via Android Samba Client and was testing if #14225 would finally allow me to stream my games, but unfortunately it still doesn't work (related #10384).
The file picker doesn't even allow me to pick ISOs, and chokes on CSOs:
content://com.google.android.sambadocumentsprovider/document/smb%3A%2F%2FServer%2FShare%2Fgame.cso
Could not load game. Failed initializing CPU or memory
Also Browse doesn't let me select my shares yet but you're probably aware of that.
It's exactly not going to work with remote files on Drive etc., at least not right away. Even if we get that working for ISOs naively, I would expect subpar buffering/caching performance until we do something smart there.
You're not going to be able to store your save games on samba for exactly the reasons outlined above in this issue.
-[Unknown]
Just storing some snippets for reference:
(DocumentFile is from AndroidX)
val tree = DocumentFile.fromTreeUri(applicationContext, uri) ?: return@withContext
contentResolver.openFileDescriptor(tree.uri, "r")?.let {
val useFd = it.detachFd()
openDirNative(useFd)
}struct dirent *details;
while ((details = readdir(root)) != nullptr) {
if (details->d_type != DT_DIR) {
// Do something for files...
}
//...
}It's exactly not going to work with remote files on Drive etc., at least not right away. Even if we get that working for ISOs naively, I would expect subpar buffering/caching performance until we do something smart there.
@unknownbrackets It would be unfair to compare online cloud services to local Samba shares. The throughput is way higher than necessary (almost exclusively 1 GBit/s) and the latency is not bad either. I'm using the same setup on my PC and it works great but there I use it in conjunction with Cache full ISO in RAM (for several reasons like allowing smbd to close the connection) so it doesn't really matter either way. With mobile devices nowadays having 6+ GB RAM I think it would make sense to enable that option there as well.
@nic0lette Turns out fdopendir doesn't work when scoped storage is fully on, on Android 11 at least on a Pixel 3a, it only returns subdirectories in folders from OPEN_DOCUMENT_TREE, while on Android 10 (Pocophone F1) it returns both files and subdirectories.
@unknownbrackets It would be unfair to compare online cloud services to local Samba shares.
I'm comparing to direct network communication between a wifi mobile device and a wired network storage, which I've actually tested the latency of. My tests were mostly on 600 (N) and 1900 (AC) connections. But yes, if you use cache iso in RAM it may be fine (although 1.8 GB is a big cost on a 6 GB device, especially when that 6 GB is VRAM too.)
Online services are likely even worse indeed, but I assume any online service will ultimately cache reads and buffer writes in some way. Though for ISOs, maybe that's a bad thing if it has to download the whole thing before reading any of it.
Turns out
fdopendirdoesn't work when scoped storage is fully on, on Android 11 at least on a Pixel 3a
Well, that's not good. If we can still get seekable fds from files, but only through Java, it should be doable without crazy bugs. But it'll be a ton of work. For games, a lot could live in the async io manager, I guess, but we'd probably want to abstract that out...
-[Unknown]
I wonder how this gonna affect Android TV support?
That is indeed a good question.
@nic0lette will the ACTION_OPEN_DOCUMENT_TREE dialog be fixed in Android TV? I heard before that it doesn't work (though haven't tried it lately).
From what I've found, no, it doesn't look like ACTION_ OPEN_DOCUMENT_TREE is supported on Android TV.
@nic0lette that is unfortunate and potentially a huge problem. Why? Is it going to change?
Probably only Nvidia Shield TV support it https://commonsware.com/blog/2017/12/27/storage-access-framework-missing-action.html
On the issue tracker mentioned:
Most domestic Android phones shipped in China lack the complete support of Storage Access Framework. Some (e.g. MIUI) even disabled "com.android.documentsui" completely, without proper replacement implementation.
After talking with engineer in Xiaomi, he said MIUI always respects CTS but SAF is not covered in CTS.
That post is old, and Android TV devices that old won't be affected anyway since they can just keep using the old methods of accessing storage.
What I'm concerned about is Android 10 and Android 11 TV devices, once we change our target version of Android to 11 or later which will make scoped storage / SAF mandatory for them.
I added a comment to https://issuetracker.google.com/issues/169471812 .
That post is old, and Android TV devices that old won't be affected anyway since they can just keep using the old methods of accessing storage.
What I'm concerned about is Android 10 and Android 11 TV devices, once we change our target version of Android to 11 or later which will make scoped storage / SAF mandatory for them.
Ah I thought it gonna affect ones before Android 10 (like mine which uses 9) but I it seems like its not.
I wonder if there are Android TV devices that even use 11.
I think we can preserve data on Android TV installs, and from other existing installs on other devices, by using preserveLegacyExternalStorage when targeting Android 11 or higher. This allows the previous permission model until the next uninstall, so we'll simply take the opportunity and offer the user to migrate all data to the allowed external directory (android/Data/org.ppsspp.ppsspp) if we detect that we can still access the old PSP directory.
Unfortunately this still has the problem that data will disappear if the user then uninstalls the app, which some users might not be used to. But it's just the way it will be. We can potentially offer users on non-TV devices to use a more permanent directory for memstick storage using ACTION_OPEN_DIRECTORY_TREE but not sure if worth the trouble and confusion....
Just documenting some path formats, trying to understand content URIs. We probably should not rely on the formats of these strings anyway, instead only use full strings as returned from APIs, but this is a bit impractical when browsing a directory subtree that we have full permissions for anyway, we'll need so many redundant file listings...
(notes on URL encodings: %3A = ':' , %20 = ' ', %2F = '/')
Browsed tree uri:
content://com.android.externalstorage.documents/tree/primary%3APSP%20ISO
document uri representation of said folder:
content://com.android.externalstorage.documents/tree/primary%3APSP%20ISO/document/primary%3APSP%20ISO
A document URI of a file inside it that folder:
content://com.android.externalstorage.documents/tree/primary%3APSP%20ISO/document/primary%3APSP%20ISO%2FTekken%206.iso
One level deeper:
content://com.android.externalstorage.documents/tree/primary%3APSP%20ISO%2FWipeout%20Games/document/primary%3APSP%20ISO%2FWipeout%20Games
content://com.android.externalstorage.documents/tree/primary%3APSP%20ISO%2FWipeout%20Games/document/primary%3APSP%20ISO%2FWipeout%20Games%2FDeeper%20folder Deeper folder
So effectively these are files in the "tree" "primary:PSP ISO". The divider between the "tree" specifier and the internal path specifier appears to be simply "/document/". Makes me wonder what happens if you have a directory called "document", as previously mentioned, which is not really hard to imagine - that has to be a bit of a design flaw. I suppose you know that it'll be followed by "primary" if there was a "primary" after tree though, so maybe it's not as ambiguous as I first though. Still, yuck.
I don't believe it actually is ambiguous. A content URI has two or four components, with the first component always being either "tree" or "document" and the third one always being "document" (if there are more than two components). To determine whether something is a tree and/or a document, Android specifically checks the first and third component. It doesn't scan through all components looking for one that says "document".
Ooh I see, the slashes are URL-encoded away inside the component, and the slashes separate the components. Right..
Alright, managed to add some code in #14232 that improves file system navigation below tree URIs (can now go upwards).
So that's a tiny bit of progress...
@nic0lette A somewhat unrelated issue but I know you're watching :)
https://issuetracker.google.com/issues/163120692?pli=1 is a critical bug where accessibility settings and apps breaks game controllers on Android. There seems to be no movement on this, but it even breaks Stadia on some devices. Can you confirm whether the team is aware of it and has a fix for Android 12?
In other news I'm making progress on this, we're probably gonna end up with a workable experience it seems, even though things won't be as smooth as it used to.
As discussed on Discord, some ideas about the root directory issue:
You can't select the root storage, you must select a subfolder.
For new users, this isn't much of an issue. They can create a new folder with any name, or potentially we could encourage them to name it PSP (we don't really care, but existing tutorials around the Internet might be easier to follow if they used "PSP".)
For existing users, this means they would be selecting PSP. However, PPSSPP would normally set this as the root of the memory card, which includes PSP itself, and can potentially include other files.
Some specific examples of things outside PSP:
- seplugins (not supported by PPSSPP anyway)
- N64 (for N64 emulators on the PSP, not a major use case)
- *PICTURE, VIDEO, and MUSIC - default paths from the XMB, so it's possible PSP games may check for files in them or generate files there.
- *PS3 - some PSP games give you bonuses for having saves from PS3 on your memory card.
- ISO - some CFW users may use this convention. Ultimately doesn't affect us really.
I've marked the likely most concerning cases, but even those are definitely edge cases.
So what we might do is the following:
- Sometime in startup, check for SYSTEM/ppsspp.ini and PSP/SYSTEM/ppsspp.ini.
- If both exist, not sure... maybe we keep PSP/. But if only SYSTEM/ppsspp.ini exists, enable special PSP subfolder handling mode.
- When this is active, internal path building should not include PSP (i.e. SYSTEM, TEXTURES, SAVEDATA, PPSSPP_STATE, etc.)
- The filesystem class (perhaps only Android? Directory too?) would strip a leading /PSP/ off paths (noting that
\is a valid directory separator too), if present. For paths like /PS3/FOO it would leave it alone (becoming actually PSP/PS3/FOO on the Android storage.) - This would affect texture replacement, plugins, save data, save states, flash0 override data, ini settings, GE frame dumps, cheats, PBP homebrew loading and downloading, and screenshots most chiefly (just listing things to double check aren't broken.)
I suppose we could handle this by having a separate g_Config.pspDirectory that /PSP/ is always mapped to, and for other platforms it'd be memstickDirectory + PSP.
To avoid confusion it might be best if this behavior was cross platform (i.e. even on Windows, if you had My Documents/PPSSPP /SYSTEM/ppsspp.ini.) That way, if someone does something on Android and it works, they won't get confused when it works differently on another device.
-[Unknown]
Great summary!
Can also, on Android, check the name of the root folder so we don't end up with PSP/PSP , if neither file exists - good for first time setup.
@nic0lette I have another bug to report in the folder browser. After you create a folder in it using the New Folder button, the button "Use this folder" at the bottom of the screen doesn't appear immediately. You have to navigate out of it and back in again.
In the latest update for my Pixel, the previous issue seems to be fixed at least (where you can't use the folder it's opened at).
@nic0lette I have another bug to report in the folder browser. After you create a folder in it using the New Folder button, the button "Use this folder" at the bottom of the screen doesn't appear immediately. You have to navigate out of it and back in again.
Is there an issue for this? (Could you open in this component if not?)
https://issuetracker.google.com/issues/163120692?pli=1 is a critical bug where accessibility settings and apps breaks game controllers on Android. [...snip...]
The team is aware of it, but this is outside the teams I communicate with (Android is big) and so other than I don't know what's going on. Sorry. (My suggestions would be to add a comment to the bug, or +1 it at least, and then add yourself in the 'cc' to get updates?)
I'm actually having trouble reproducing the new folder bug. Maybe it was just some wacky transient thing .. So I guess let's forget about that one, unless I have it happen again - in which case I'll investigate properly.
Right, I know Android is big :) I added a comment.
This is now pretty much resolved with #14619 merged. There may still be a few issues remaining, of course, but we'll fix them one by one.
Closing, finally.