Support for ExFAT filesystem?

Question

Support for ExFAT filesystem?

michaelbaudino opened this issue 4 years ago · comments

Hello,

Recalbox core team member here 👋

We are switching one of our partition filesystem to ExFAT in our next release (due in a few weeks), but from what I can see in this piece of code there's no way to ask NOOBS to create an ExFAT filesystem from our partitions.json file 😭

Are there any plans to support this? Or at least any custom filesystem creation command? If not, which approach do you think we should consider?

Thanks a lot in advance for your help.

PS: I've tried a few things already to reformat and create the filesystem on first boot, but all my attempts failed:

parted refuses to resize the extended partition that contains the logical partition mounted as /
fdisk is made for humans and thus its input is highly inconsistent and not reliably usable by a script
sfdisk doesn't seem to support GPT

Michael Baudino · Answer 1 · Mon Jul 20 2020 17:15:22 GMT+0800 (China Standard Time)

Just to be clear, I'd be happy to provide a PR to support this (let me know if you want it!), but it doesn't sound like a good solution to me because it's not backward compatible: people who own (or receive from a manufacturer) a NOOBS SD-card created with an older version would not be able to install our OS.

maxnet · Answer 2 · Mon Jul 20 2020 17:18:43 GMT+0800 (China Standard Time)

Or at least any custom filesystem creation command?

You can run your own arbitrary commands as post-installation (partition_setup) script.
Windows does that as well.

Regarding partitions.json:
Set "empty_fs" to true to tell NOOBS you don't want the partition formatted.
"partition_type" to whatever MBR partition id exFAT uses (2 hex digits).

sfdisk doesn't seem to support GPT

Neither does NOOBS.

Michael Baudino · Answer 3 · Mon Jul 20 2020 18:39:54 GMT+0800 (China Standard Time)

Awesome idea, I'll try that 🙌
Thanks a lot for the hint @maxnet 🙏

Andrew Scheller · Answer 4 · Tue Jul 21 2020 07:16:43 GMT+0800 (China Standard Time)

I hope you don't mind me asking, purely out of curiosity...

We are switching one of our partition filesystem to ExFAT in our next release

Why? :)

I've just googled and found https://en.wikipedia.org/wiki/ExFAT#Adoption and to answer your original query NOOBS doesn't (yet) use a 5.4 kernel 😉 (Although apparently it does support FUSE, which I didn't realise!)

Andrew Scheller · Answer 5 · Tue Jul 21 2020 07:21:42 GMT+0800 (China Standard Time)

it doesn't sound like a good solution to me because it's not backward compatible

We've had similar backwards-compatibility problems in the past, and that's why current versions of NOOBS download os_list_v3.json with older versions still being supported.

Michael Baudino · Answer 6 · Tue Jul 21 2020 15:41:57 GMT+0800 (China Standard Time)

@lurch:

I hope you don't mind me asking, purely out of curiosity...

We are switching one of our partition filesystem to ExFAT in our next release

Why? :)

I've just googled and found https://en.wikipedia.org/wiki/ExFAT#Adoption and to answer your original query NOOBS doesn't (yet) use a 5.4 kernel 😉 (Although apparently it does support FUSE, which I didn't realise!)

Ahah sure, feel free to ask 🤗

This partition is the one where users can store their media content (roms (games), music, movies, …). Some users have a lot of content.

Since we are targeting non-technical users, we need this partition to be readable/writable on Windows, thus we excluded EXT4. We also excluded FAT32 due to technical limitations (partition size, file size, access rights, no journaling, …). We've been supporting NTFS for some time too, but have a few issues with it (mounting is slow, partitions are randomly not detected, …). It's probably an issue on our side, since NTFS seems quite mature, but we never had time to properly track that kind of not reproductible bugs. Finally, we recently tried ExFAT, which is well supported on both Windows (natively) and Linux, and is quite solid/stable, from what we can see.

Anyway, we may move back to NTFS at some point (since NTFS optimizes space better than ExFAT, iirc), but for now, we're quite happy with ExFAT broad support and performance.

Regarding the driver, we are using this FUSE-based driver which is actually provided by Buildroot. We may switch to native Linux 5.4 support, but we didn't bump to 5.4 on all architectures yet (only on Pi4 at the moment, actually).

Michael Baudino · Answer 7 · Tue Jul 21 2020 18:30:53 GMT+0800 (China Standard Time)

@maxnet:

Regarding partitions.json:
Set "empty_fs" to true to tell NOOBS you don't want the partition formatted.
"partition_type" to whatever MBR partition id exFAT uses (2 hex digits).

It worked 🙌

Well, actually, I also had to add "filesystem_type": "unformatted" too, otherwise NOOBS tries to run mkfs, which is not what we want.

Actually, I was expecting "filesystem_type": "unformatted" to be enough (i.e. that it would imply "empty_fs": true), but no 🤷‍♂️ With only "filesystem_type": "unformatted", NOOBS does not create the filesystem (i.e. it does not run mkfs, as we would expect), but it still tries to download a tarball and populate the (not-existent) filesystem.

Let me try to make this more ~~clear~~ nerdy.

Current behaviour pseudo-code:

if partition is defined in partitions.json then
- create partition
- if filesystem_type is not unformatted then
  - create filesystem (mkfs)
- if empty_fs is not true then
  - populate filesystem with tarball content

Shouldn't it rather be:

if partition is defined in partitions.json then
- create partition
- if filesystem_type is not unformatted then
  - create filesystem (mkfs)
  - if empty_fs is not true then
    - populate filesystem with tarball

Anyway, thanks a lot for your help 🙏