adb-sync doesn't copy files properly if names are same on local and remote
alecazam opened this issue · comments
I think this is due to self.both isn't processed for PerformCopies. I have a file with two lines that have the same length, where I comment out one line, and uncomment out another. Then I toggle this, and the file isn't copied. If I change the length of the lines, then src_only picks up the change.
This is with "adb-sync --delete --force ...". I also applied the "--times" fix in another issue, added --times, and fresh synced the content, but that still doesn't fix this issue on subsequent copies.
This self.both list seems to be created if the path on local matches the path on the remote, the stats aren't compared. But a copy is still needed if the modstamp differs. I only see self.both uses in overwrites and deletions, and not in the copy step. PerformCopies only walks the self_only list.
This means that the content on device is not correct, and doesn't reflect any mods that were made to the source directory. The modstamp, filesize, etc can all be different by adb-sync doesn't seem to check those. Please tell me I'm misreading something here, and that this important utility has been broken for all these years.
BuildFileLists
this creates List[Tuple[path, stats]]
def DiffLists(...)
if a_item[0] == b_item[0]: <- seems to only compare the path portion of the tuple from BuildFileLists, should this be comparing a_item[1] == b_item[1], and same on two calls below
both.append((a_item[0], a_item[1], b_item[1]))
a_revlist.pop()
b_revlist.pop()
elif a_item[0] < b_item[0]:
a_only.append(a_item)
a_revlist.pop()
elif a_item[0] > b_item[0]:
b_only.append(b_item)
b_revlist.pop()
else:
def PerformCopies(self) -> None:
"""Perform all copying necessary for the file sync operation."""
for i in [0, 1]:
if self.src_to_dst[i]:
for name, s in self.src_only[i]: <- this is only walking the src_only list build from DiffLists, self.both is ignored
The following code could handle it, but skips if the file sizes match. But the content may be different (modstamps differ) even if the file size is the same. I'm assuming file padding is ignored from st_size. My understanding of overwrites is that they were meant to handle folders <-> files.
def PerformOverwrites(self) -> None:
for name, localstat, remotestat in self.both: <- okay this walks the both list
if stat.S_ISDIR(localstat.st_mode) and stat.S_ISDIR(remotestat.st_mode):
# A dir is a dir is a dir.
continue
elif stat.S_ISDIR(localstat.st_mode) or stat.S_ISDIR(remotestat.st_mode):
# Dir vs file? Nothing to do here yet.
pass
else:
# File vs file? Compare sizes.
if localstat.st_size == remotestat.st_size: <- this earlies out off file size
continue
I'm trying this small change to DiffLists. The issue here is that comparing mod times between OSX files and Android device files doesn't line up. I'm having to munge the times to get the comparison to skip files already copied over previously. I tried snapping to minutes and then adding 16. So how can this sync ever work, if the timestamps don't have even a remote correspondence. I tried with --times and without.
if a_item[0] == b_item[0]:
# if a.mtime is greater, then want in a_only to do the copy
if a_item[1].st_mtime > b_item[1].st_mtime:
a_only.append(a_item)
else:
both.append((a_item[0], a_item[1], b_item[1]))
a_revlist.pop()
b_revlist.pop()
Modification timestamps seems to be an underlying flaw in the Android file system.
https://issuetracker.google.com/issues/36930892#c128
And some circular references back to that unresolved issue, but more of a description of the problem.
https://www.xda-developers.com/diving-into-sdcardfs-how-googles-fuse-replacement-will-reduce-io-overhead/
It's clear from --times still being broken in the release buld that timestamp comparisons in adb-sync were never tested/used.
Also seems like there's an "adb shell stat -c "%y" filename" command that might return more precise timings than "ls -al" which only shows timings to minutes. But this would have to get called for each filename. At least here the seconds are available.
2020-11-25 00:16:05.000000000
There's also "ls -ll" to get more accurate timestamps so they don't need to be rounded, so I'm going to try using that.
adb shell ls -all
drwxrwx--x 14 u0_a184 sdcard_rw 3488 2020-11-25 08:57:14.000000000 -0800 .
I rewrote the timestamp parsing to use microseconds by cropping the nanos returned by ls -ll, and also using the UTC timestamp returned. I stoped using --times since it only sets low resolution timings via "adb touch" and it's slow, and the "adb push" uses to emulate copy already preserves modstamps.
Then I added the modstamp tests on non-directories mentioned above, and fixed the "both" logic to skip copies off a newer modstamp. This all finally got adb-sync to sync files properly to the device.
Since it's an underpinning of adb-sync, I looked into the behavior of adb push by itself on and Android 9 device and macOS. My understanding is that macOS emulates the Android file system where Linux/Win have native implementations.
Looks like "adb push" if you pull the cable to the device in the middle of any push leaves a partial file with the current modstamp set as well. This means you have a partial copy of the file on the device, and since adb sync only checks modstamps, then it breaks our builds.
Also if you push, touch a file, push again and cancel the push then the file is deleted. Wasn't expecting that either. Thought that the file system would still have that original file. push and cancel seems to not leave a file on the device either.
Considering also adding a size check since the "ls -ll" command includes those too. At least it may be a checksum if there is correspondence between file sizes, and they're not thrown off by pagesize on the two file systems.
We ended up dumping adb-sync except for excruciatingly slow deletes that are done one file at a time even if all the files under a folder are deleted. The general idea is as follows, and this is crazy fast and more safe than adb-sync.
# First use "adb sync" to copy newer files to the device.
"adb push --sync folder/* /sdcard/Android/data/com.appdomain/files/"
# Now use python script "adb-sync" to delete files not present in current build. no-clobber so that it doesn't copy any files. Strip this python script down to just handling deletes.
"adb-sync --delete --no-clobber folder/ "/sdcard/Android/data/com.appdomain/files/"