Create new partitions layout on aries and pass it to kernel
PabloPL opened this issue · comments
Since we have uboot working, maybe it would be good idea to change partition table a bit.
We could:
- create new boot and recovery partitions
- create one partition for uboot env
- "hide" bootloader/uboot from kernel (or mark them as read only)
- leave all other (remember about modem/efs/etc)
So uboot would pass those mtd partitions to kernel, which could use them for loading (for example android). We could then also enable fastboot for flashing boot/recovery.
@xc-racer99 Do You know if partition table layout is much different between aries variants?
I'm using now stock PIT file, do you have any for galaxys4, so i could compare them and hardcode in uboot?
This would also allow us to:
- use a standard boot.img and recovery image with android.
- use PostmarketOS with the mainline kernel.
- dual-boot GNU/Linux and Android
The GSM variants have the same partition layout while each one of the fascinate and the galaxy S 4G has it's own.
Fasciante PIT: https://forum.xda-developers.com/attachment.php?attachmentid=405371&d=1285089689
Galaxy S 4G partition layout (commented at the end of the file): https://github.com/xc-racer99/android_kernel_samsung_aries/blob/aosp-7.1/drivers/mtd/onenand/samsung_galaxys4g.h
@PabloPL The big difference in partition layout is that the SGS4G has double the storage capacity. The first few partitions (bootloaders) are the same as on the other GSM versions.
By default, stock has the data partition on the oneNand. Here's a copy of the .pit (VibrantPlus is the SGS4G, Hawk and Sidekick are two more variants that were sold on smaller American carriers IIRC and never had any after-market support to speak of)
Hawk_Sidekick_VibrantPlus_831.pit.zip
@MYEUHD links for the SGS4G partition layout is correct and is the best way to compare (IMO) between the different variants. Note that partition sizes have been adjusted over the different aftermarket versions, notably that the /system size kept increasing :) i believe the other GSM variants also had a few minor changes around ICS but I don't remember exactly what.
The fascinate is essentially the same as the GSM variants, it just doesn't have an EFS partition, I'm not sure how the radio is configured on it.
It should also be noted that currently the Android kernel doesn't support cmdline partitions due to some modifications made to the driver by Google/Samsung. It is possible to do so, however I have had some issues mapping the BML reservoir which is used in Android when updating the boot.img and there are bad blocks.
@xc-racer99 So maybe we should use ubifs? It looks like it's supported on android kernel (3.0).
We wouldn't have also problems with resizing partitions (if they are too small like it was done for /system).
So we would need:
- uboot-env (read only)
- boot (new)
- recovery (new)
- efs (read only?)
- modem (or we could just extract modem from this partition and provide with image as file if it's possible)
- reservoir (read only)
Partitions with bootloaders and uboot won't be mapped.
I have played around with UBIFS, the unfortunate thing is that with 3.0's version of UBIFS SELinux isn't supported so Android won't run nicely if at all (dependent on Android version). Trying to backport would be quite difficult as selinux support wasn't introduced until ~3.18 IIRC. I did backport the ubiblock code (so you could have a read-only block device), but this is only useful for system and modem.
The resizing of /system on the OneNAND was only the SGS4G, the GSM variants had the system partition on LVM so that was easy enough to change. Prior to the introduction of LVM it was a bit of a pain.
efs needs to be read-write as on some variants (mine for sure) the modem will make changes to it.
modem definitely doesn't need its own partition as it's actually just a symlink (https://github.com/CyanogenMod/android_device_samsung_aries-common/blob/cm-10.2/init.aries.rc#L148) to support the closed-source blob. Replicant's Samsung-RIL also uses the original path, so adjusting the symlink is the best way to go.
reservoir (if we map it) should be read-write. However, the only time it is used is when updating u-boot so I would favor not even mapping it for userspace.
uboot-env - does it need to be available from userspace?
I would personally favor having the "boot" or the "recovery" partition in the location of the RECOVERY partition in the stock .pit (same for GSM and SGS4G) so that it can easily be updated by Heimdall. This means the offset is 102*SZ_256K
So, here would be my idea:
Mapped:
boot
recovery
uboot-env
efs (need to determine what filesystem this should use as we probably don't want to use yaffs2 since it isn't mainline)
cache (android-specific and useful for Android's recovery)
misc (remaining space (stopping at beginning of reservoir) going to a partition which could be used as needed for Android support (eg system or datadata))
Unmapped:
BL1
BL2
U-boot
Reservoir
@xc-racer99 I like partition Your partition layout. Just few comments.
If we make uboot-env available from userspace, we can use linux tools to change it params (https://elinux.org/U-boot_environment_variables_in_linux). Could be dangerous, but usefull for people without serial cable. Let's hide (if not possible, mark as read only).
About making boot/recovery partitions updatable by heimdall, i was rather thinking about enabling fastboot support (so users could use fastboot for flashing boot/recovery/other partitions).
Sure, I'm good with making uboot-env available from userspace. However, if it's marked as read-only whats the point? For hidden partitions, are you talking about locking them? I can't find any references to hidden mtd partitions. I'm unsure if the OneNAND driver supports locking or not.
With fastboot, I have a couple of concerns:
- Does the u-boot fastboot support include onenand (ie does common/fb_nand.c support OneNAND as well as regular NAND)?
- I'm still not very confident that I properly fixed writing on the OneNAND - it might be temporarily causing the bad blocks (ie why self-updating and updating the OneNAND boot partition weren't working properly)
For uboot-env - You're right, it doesn't make any sense to make it ro.
For hidden partitions i was thinking about not passing them to kernel.
I don't know yet if fastboot supports onenand and looking at fb_nand.c it looks like it doesn't, so for now just skip this idea (sadly).
About writing on OneNAND, we can check if it's working fine under linux
Ok, for galaxy s it could look like this
new partition: efs
new partition offset: 2
new partition offset: 0x00080000 (2 * 256 * 1024), because one block = 256k. The same for size
new partition size: 40
new partition size: 10M
new partition: uboot-env
new partition offset: 102
new partition offset: 0x01980000
new partition size: 1
new partition size: 256k
new partition: boot
new partition offset: 103
new partition offset: 0x019C0000
new partition size: 30
new partition size: 7680k
new partition: recovery
new partition offset: 133
new partition offset: 0x02140000
new partition size: 30
new partition size: 7680k
new partition: misc
new partition offset: 163
new partition offest: 0x028C0000
new partition size: 1651
new partition size: 422656k
new partition: cache
new partition offset: 1814
new partition offset: 0x1C580000
new partition size: 35m
new partition: modem
new partition offset: 1954
new partition offset: 0x1E880000
new partition size: 50
new partition size: 12800k
It would end as
mtdparts=0:10M@0x00080000(efs),256k@0x01980000(uboot-env),7680k@0x019C0000(boot),7680k@0x02140000(recovery),422656k@0x028C0000(misc),35m@0x1C580000(cache),12800k@0x1E880000(modem)
Well, your efs partition would need to change - you're currently mapping the BML position of efs which since we don't have support for BML we can't access (plus, it's in a proprietary samsung filesystem - RFS). CyanogenMod got around this by create a new MTD efs partition elsewhere and backing up (from the stock recovery) the efs partition and copying it to the new MTD partition once it was in place. The modem is the same way, so CyanogenMod had a copy of the modem.bin (which had to be padded to exactly 14MB IIRC) and placed in its own MTD partition. The original modem partition on BML was overwritten while the EFS one was left untouched in case the user ever wanted to go back to stock. Both the (new) efs and modem partitions were formatted as yaffs2.
Well, your efs partition would need to change - you're currently mapping the BML position of efs which since we don't have support for BML we can't access
Even if i make a backup of efs and modem and then just mount those mtd partitions and put files there (for example on stock recovery/cm/etc)?
It was long time since i last time worked with efs...
Overwriting the modem isn't an issue at all as it isn't device specific and can easily be flashed again. If you overwrite the efs and then somehow lose it again, you no longer have a backup since it's device specific. Since the efs partition doesn't have to be at that offset, it makes more sense to just leave the original blocks there and waste the 12MB of space by creating a new efs partition elsewhere.
IMO, the original efs should be treated as if it were a bootloader and not touched unless absolutely necessary.
efs (need to determine what filesystem this should use as we probably don't want to use yaffs2 since it isn't mainline)
we could use ext4, like it's done on midas
https://github.com/fourkbomb/android_device_samsung_midas/blob/lineage-15.1/rootdir/fstab.midas#L5
@MYEUHD Only if you know how to use ext4 on an mtd device... Read the FAQ at http://www.linux-mtd.infradead.org/faq/general.html for more info.
We're basically limited to UBIFS, JFFS2, or the out of tree YAFFS2. The issue with YAFFS2 is that it isn't in the kernel tree (has to be manually added), the issue with UBIFS is that it doesn't support selinux until around kernel 3.18 so Android doesn't work properly on it until them, I haven't investigated JFFS2.
I will probably investigate backporting a ton of UBI/UBIFS changes for selinux and proper xattr support, but last time I looked it was a huge job.
Alright, here's my proposal:
10240k@26112k(boot),10240k(recovery),4k(uboot-env),12800k(efs),19712k(radio),40960k(cache),{variable}(misc)
EFS and Radio would be able to be mounted on both Android and Linux and would have a JFFS2 filesystem on them (Android kernel needs it enabled, but it is functional wrt to SELinux even on 3.0).
The variable misc partition would be the rest of the partition (starting at 120068k) up to the start of the reservoir (513024k on GSM, 1027072k on SGS4G) so misc sizes would be 392956k on GSM (currently datadata is 430080k) and 907004k on SGS4G.
This seems good to me.
Branch created at https://github.com/xc-racer99/u-boot-galaxys4g/tree/mtdparts but has yet to be more than compile tested as I need to modify the Android kernel/updater.sh to work.
Edit: That branch doesn't boot, needs more work
@xc-racer99 Did You enabled in kernnel CONFIG_MTD_CMDLINE_PARTS ?
With this i was able to get mtd partitions in kernel.
Edit 1:
mtdparts=0:
It can't be 0, but name of mtd device (check in /proc/mtd or something like this).
WIthout this, it won't also create mtd partitions.
Edit 2:
It should be
mtdparts=b0600000.onenand:
Yes, I did (plus needing a change so the driver will actually use the cmdline parts...). I haven't had an issue getting the cmdline parts working.
I might have just been flashing it improperly as I've been working on the ability to have a flashable zip that backups EFS, flashes u-boot, and then restore the EFS partition.
I also need to finish adjusting the Android build system to use the new partition layout. Will rebase this test branch and create PR when I have a booting Android.
@PabloPL Wait, can you boot this branch as is (accounting for the defconfig change?) I can't boot u-boot with CONFIG_ENV_IS_IN_ONENAND set
It's also not booting, but with current mtdparts i got following info in kernel log (after hardcoding partitions, enabling onenand in kernel etc)
[ 2.613855] 7 cmdlinepart partitions found on MTD device b0600000.onenand
[ 2.619340] Creating 7 MTD partitions on "b0600000.onenand":
[ 2.623629] 0x000001980000-0x000002380000 : "boot"
[ 2.627551] 0x000002380000-0x000002d80000 : "recovery"
[ 2.631376] 0x000002d80000-0x000002d81000 : "uboot-env"
[ 2.634794] mtd: partition "uboot-env" doesn't end on an erase/write block -- force read-only
[ 2.652773] 0x000002d81000-0x000003a01000 : "efs"
[ 2.659971] mtd: partition "efs" doesn't start on an erase/write block boundary -- force read-only
[ 2.676906] 0x000003a01000-0x000004d41000 : "radio"
[ 2.684186] mtd: partition "radio" doesn't start on an erase/write block boundary -- force read-only
[ 2.701444] 0x000004d41000-0x000007541000 : "cache"
[ 2.708840] mtd: partition "cache" doesn't start on an erase/write block boundary -- force read-only
[ 2.726242] 0x000007541000-0x00001f500000 : "misc"
[ 2.733546] mtd: partition "misc" doesn't start on an erase/write block boundary -- force read-only
Yeah, I guess they have to be on every 256k as that's the size of the erase block. Can you please test the version I just pushed?
After fixing
0:256k@25856k(uboot-env)
to
b0600000.onenand:256k@25856k(uboot-env)
It looks like it's working fine
[root@alarm ~]# cat /proc/mtd
dev: size erasesize name
mtd0: 00040000 00040000 "uboot-env"
mtd1: 00a00000 00040000 "boot"
mtd2: 00a00000 00040000 "recovery"
mtd3: 00c80000 00040000 "efs"
mtd4: 01340000 00040000 "radio"
mtd5: 02800000 00040000 "cache"
mtd6: 17fc0000 00040000 "misc"
[ 0.902561] Muxed OneNAND 512MB 1.8V 16-bit (0x50)
[ 0.905372] OneNAND version = 0x013e
[ 0.908305] Chip support all block unlock
[ 0.910287] Chip has 4KiB pagesize
[ 0.912342] Chip has cache program feature
[ 0.915152] Scanning device for bad blocks
[ 1.955799] random: fast init done
[ 2.618570] OneNAND eraseblock 2047 is an initial bad block
[ 2.621519] 7 cmdlinepart partitions found on MTD device b0600000.onenand
[ 2.626998] Creating 7 MTD partitions on "b0600000.onenand":
[ 2.631287] 0x000001940000-0x000001980000 : "uboot-env"
[ 2.635602] 0x000001980000-0x000002380000 : "boot"
[ 2.639153] 0x000002380000-0x000002d80000 : "recovery"
[ 2.642925] 0x000002d80000-0x000003a00000 : "efs"
[ 2.651327] 0x000003a00000-0x000004d40000 : "radio"
[ 2.658946] 0x000004d40000-0x000007540000 : "cache"
[ 2.666540] 0x000007540000-0x00001f500000 : "misc"
After making fixing
0:256k@25856k(uboot-env)to
b0600000.onenand:256k@25856k(uboot-env)
Ok, looks like some more work is needed then as it's a different name on 3.0 kernel (well, name isn't present...)
Implemented in c95e295
uboot-env, recovery, and boot are their own separate partitions, everything else is in UBI. 3.0 kernel needs the patches from https://github.com/xc-racer99/android_kernel_samsung_aries/commits/ubifs-selinux plus the ubi volumes need to be created indendently with ubimkvol, ubiformat, and/or ubiattach. Example kernel for SGH-T959P which does it on boot is available at https://github.com/xc-racer99/android_kernel_samsung_aries/tree/ubifs-convert (relies on flashable zip which is more or less implemented at https://github.com/xc-racer99/galaxys4g_ubi_transform/tree/ubifs-mainline-convert)
@xc-racer99 Did You also saw errors when uboot is booting (unaligned memory access or something like this) + it's always restoring default env settings (try to add/modify something, savev env settings and reboot).
Yes, I noticed the unaligned memory access errors. It's related to 2789210#diff-1145f263256c923716d2b8eade2f6689 which uses code I copied elsewhere. There's an upstream fix for where I copied it from at e8f6576
Note that I've moved to https://github.com/xc-racer99/u-boot-galaxys4g/tree/fastboot-v2018.09 where I squished all of the aries-specific patches together and actually dropped the onenand DMA change - the difference in read speed is not significant enough for the bigger code base IMO
Edit: I haven't tried to save/restore the env, but will do so now to see if it works or not...
@xc-racer99 Any plans to upstream some of those fixes (by the way, are You going to submit also some Your linux kernel patches - for example bma/gp2a or i should submit them)?
I probably should go through the process of trying to get them submitted, I never have before. If you want to, please do so - this goes for both the u-boot and linux patches. I don't know when or if I will get the chance to.
Also, with the v2018.09-aries branch setenv/saveenv persists over reboot