UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 90: invalid start byte
notetiene opened this issue · comments
I have no idea where the error comes from, but it sometimes happens and other times it doesn't.
This is the full output
syncrclone /home/etienne/.config/syncrclone/syncrclone.config.py --debug
2022-02-02 19:20:58: DEBUG: argv: ['/home/etienne/.config/syncrclone/syncrclone.config.py', '--debug']
2022-02-02 19:20:58: DEBUG: CLI config: Namespace(break_lock=None, configpath='/home/etienne/.config/syncrclone/syncrclone.config.py', debug=True, dry_run=False, interactive=False, new=False, no_backup=False, override=[], reset_state=False)
2022-02-02 19:20:58: DEBUG: Localmode: False
2022-02-02 19:20:58: syncrclone (20220103.1.BETA)
2022-02-02 19:20:58: config path: '/home/etienne/.config/syncrclone/syncrclone.config.py'
2022-02-02 19:20:58: A: '/home/etienne/.cache/emacs/org/'
2022-02-02 19:20:58: B: 'webdav:/org/'
2022-02-02 19:20:58: DEBUG: config: Config(remoteA='/home/etienne/.cache/emacs/org/', remoteB='webdav:/org/', workdirA='/home/etienne/.cache/emacs/org/.syncrclone', workdirB='webdav:/org/.syncrclone', name='NcS9y', rclone_exe='rclone', filter_flags=['--exclude', '.#*'], rclone_flags=[], rclone_env={'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf'}, rclone_flagsA=[], rclone_flagsB=[], compare='mtime', dt=1.1, conflict_mode='newer', tag_conflict=False, reuse_hashesA=False, reuse_hashesB=False, always_get_mtime=True, backup=True, sync_backups=False, hash_fail_fallback=None, set_lock=False, action_threads=1, cleanup_empty_dirsA=None, cleanup_empty_dirsB=None, avoid_relist=False, renamesA=None, renamesB=None, save_logs=True, local_log_dest='', pre_sync_shell='', post_sync_shell='', stop_on_shell_error=False, os=<module 'os' from '/usr/lib/python3.8/os.py'>, configpath='/home/etienne/.config/syncrclone/syncrclone.config.py', break_lock=None, debug=True, dry_run=False, interactive=False, new=False, override=[], reset_state=False, workdir0A=None, workdir0B=None)
2022-02-02 19:20:58: rclone version:
2022-02-02 19:20:58: DEBUG: rclone:call ['rclone', '--version']
2022-02-02 19:20:58: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:58: rclone: rclone v1.57.0
2022-02-02 19:20:58: rclone: - os/version: ubuntu 20.04 (64 bit)
2022-02-02 19:20:58: rclone: - os/kernel: 5.13.0-27-generic (x86_64)
2022-02-02 19:20:58: rclone: - os/type: linux
2022-02-02 19:20:58: rclone: - os/arch: amd64
2022-02-02 19:20:58: rclone: - go/version: go1.17.2
2022-02-02 19:20:58: rclone: - go/linking: static
2022-02-02 19:20:58: rclone: - go/tags: none
2022-02-02 19:20:58:
2022-02-02 19:20:58: Refreshing file list on A '/home/etienne/.cache/emacs/org/'
2022-02-02 19:20:58: DEBUG: rclone:call ['rclone', 'lsjson', '--filter', '+ /.syncrclone/LOCK/*', '--filter', '- /.syncrclone/**', '--exclude', '.#*', '-R', '--no-mimetype', '--files-only', '/home/etienne/.cache/emacs/org/']
2022-02-02 19:20:58: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:58: DEBUG: A: Read 5
2022-02-02 19:20:58: DEBUG: A: Read 5
2022-02-02 19:20:58: DEBUG: rclone:call ['rclone', '--retries', '1', 'copyto', '/home/etienne/.cache/emacs/org/.syncrclone/A-NcS9y_fl.json.xz', '/tmp/tmp1xd23xdq/A_prev']
2022-02-02 19:20:58: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:58: 5 files, 14.69 kb
2022-02-02 19:20:58: Refreshing file list on B 'webdav:/org/'
2022-02-02 19:20:58: DEBUG: rclone:call ['rclone', 'lsjson', '--filter', '+ /.syncrclone/LOCK/*', '--filter', '- /.syncrclone/**', '--exclude', '.#*', '-R', '--no-mimetype', '--files-only', 'webdav:/org/']
2022-02-02 19:20:58: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:59: DEBUG: B: Read 5
2022-02-02 19:20:59: DEBUG: B: Read 5
2022-02-02 19:20:59: DEBUG: rclone:call ['rclone', '--retries', '1', 'copyto', 'webdav:/org/.syncrclone/B-NcS9y_fl.json.xz', '/tmp/tmp1xd23xdq/B_prev']
2022-02-02 19:20:59: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:59: 5 files, 14.67 kb
2022-02-02 19:20:59: DEBUG: Found 5 common paths with 0 matching files
2022-02-02 19:20:59:
2022-02-02 19:20:59: DEBUG: Resolving:
{
"A": {
"Path": "notes.archive.org",
"Size": 4878,
"mtime": 1643840884.028013
},
"Ap": {
"Path": "notes.archive.org",
"Size": 4878,
"mtime": 1643840884.028013
},
"B": {
"Path": "notes.archive.org",
"Size": 4878,
"mtime": 1643840888.0
},
"Bp": {
"Path": "notes.archive.org",
"Size": 4878,
"mtime": 1643840888.0
}
}
2022-02-02 19:20:59: DEBUG: 'notes.archive.org': Both A and B compare to prev but do not agree. This is unexpected.
2022-02-02 19:20:59: CONFLICT 'notes.archive.org'
2022-02-02 19:20:59: A: 2022-02-02 17:28:04 (4.8 kb)
2022-02-02 19:20:59: B: 2022-02-02 17:28:08 (4.8 kb)
2022-02-02 19:20:59: Resolving with mode 'newer'(keep B)
2022-02-02 19:20:59: DEBUG: Resolving:
{
"A": {
"Path": "notes.org",
"Size": 10108,
"mtime": 1643847653.652436
},
"Ap": {
"Path": "notes.org",
"Size": 10108,
"mtime": 1643847428.105879
},
"B": {
"Path": "notes.org",
"Size": 10094,
"mtime": 1643847629.0
},
"Bp": {
"Path": "notes.org",
"Size": 10094,
"mtime": 1643847629.0
}
}
2022-02-02 19:20:59: DEBUG: 'notes.org': Neither compare. Both modified or both new
2022-02-02 19:20:59: CONFLICT 'notes.org'
2022-02-02 19:20:59: A: 2022-02-02 19:20:53 (9.9 kb)
2022-02-02 19:20:59: B: 2022-02-02 19:20:29 (9.9 kb)
2022-02-02 19:20:59: Resolving with mode 'newer'(keep A)
2022-02-02 19:20:59: DEBUG: Resolving:
{
"A": {
"Path": "org-annotate-file.org",
"Size": 0,
"mtime": 1643825750.472925
},
"Ap": {
"Path": "org-annotate-file.org",
"Size": 0,
"mtime": 1643825750.472925
},
"B": {
"Path": "org-annotate-file.org",
"Size": 0,
"mtime": 1643826021.0
},
"Bp": {
"Path": "org-annotate-file.org",
"Size": 0,
"mtime": 1643826021.0
}
}
2022-02-02 19:20:59: DEBUG: 'org-annotate-file.org': Both A and B compare to prev but do not agree. This is unexpected.
2022-02-02 19:20:59: CONFLICT 'org-annotate-file.org'
2022-02-02 19:20:59: A: 2022-02-02 13:15:50 (0 b)
2022-02-02 19:20:59: B: 2022-02-02 13:20:21 (0 b)
2022-02-02 19:20:59: Resolving with mode 'newer'(keep B)
2022-02-02 19:20:59: DEBUG: Resolving:
{
"A": {
"Path": "notes.local.org",
"Size": 0,
"mtime": 1643821578.069292
},
"Ap": {
"Path": "notes.local.org",
"Size": 0,
"mtime": 1643821578.069292
},
"B": {
"Path": "notes.local.org",
"Size": 0,
"mtime": 1643821651.0
},
"Bp": {
"Path": "notes.local.org",
"Size": 0,
"mtime": 1643821651.0
}
}
2022-02-02 19:20:59: DEBUG: 'notes.local.org': Both A and B compare to prev but do not agree. This is unexpected.
2022-02-02 19:20:59: CONFLICT 'notes.local.org'
2022-02-02 19:20:59: A: 2022-02-02 12:06:18 (0 b)
2022-02-02 19:20:59: B: 2022-02-02 12:07:31 (0 b)
2022-02-02 19:20:59: Resolving with mode 'newer'(keep B)
2022-02-02 19:20:59: DEBUG: Resolving:
{
"A": {
"Path": "web-capture.org",
"Size": 52,
"mtime": 1643777739.184635
},
"Ap": {
"Path": "web-capture.org",
"Size": 52,
"mtime": 1643777739.184635
},
"B": {
"Path": "web-capture.org",
"Size": 52,
"mtime": 1643777745.0
},
"Bp": {
"Path": "web-capture.org",
"Size": 52,
"mtime": 1643777745.0
}
}
2022-02-02 19:20:59: DEBUG: 'web-capture.org': Both A and B compare to prev but do not agree. This is unexpected.
2022-02-02 19:20:59: CONFLICT 'web-capture.org'
2022-02-02 19:20:59: A: 2022-02-01 23:55:39 (52 b)
2022-02-02 19:20:59: B: 2022-02-01 23:55:45 (52 b)
2022-02-02 19:20:59: Resolving with mode 'newer'(keep B)
2022-02-02 19:20:59: DEBUG: Printing Queueus Initial
2022-02-02 19:20:59: DEBUG: newA []
2022-02-02 19:20:59: DEBUG: newB []
2022-02-02 19:20:59: DEBUG: delA []
2022-02-02 19:20:59: DEBUG: delB []
2022-02-02 19:20:59: DEBUG: tagA []
2022-02-02 19:20:59: DEBUG: tagB []
2022-02-02 19:20:59: DEBUG: backupA ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: backupB ['notes.org']
2022-02-02 19:20:59: DEBUG: transA2B ['notes.org']
2022-02-02 19:20:59: DEBUG: transB2A ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: movesA []
2022-02-02 19:20:59: DEBUG: movesB []
2022-02-02 19:20:59: DEBUG: Printing Queueus After tracking moves
2022-02-02 19:20:59: DEBUG: newA []
2022-02-02 19:20:59: DEBUG: newB []
2022-02-02 19:20:59: DEBUG: delA []
2022-02-02 19:20:59: DEBUG: delB []
2022-02-02 19:20:59: DEBUG: tagA []
2022-02-02 19:20:59: DEBUG: tagB []
2022-02-02 19:20:59: DEBUG: backupA ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: backupB ['notes.org']
2022-02-02 19:20:59: DEBUG: transA2B ['notes.org']
2022-02-02 19:20:59: DEBUG: transB2A ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: movesA []
2022-02-02 19:20:59: DEBUG: movesB []
2022-02-02 19:20:59: DEBUG: Printing Queueus After processing new and tags
2022-02-02 19:20:59: DEBUG: newA []
2022-02-02 19:20:59: DEBUG: newB []
2022-02-02 19:20:59: DEBUG: delA []
2022-02-02 19:20:59: DEBUG: delB []
2022-02-02 19:20:59: DEBUG: tagA []
2022-02-02 19:20:59: DEBUG: tagB []
2022-02-02 19:20:59: DEBUG: backupA ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: backupB ['notes.org']
2022-02-02 19:20:59: DEBUG: transA2B ['notes.org']
2022-02-02 19:20:59: DEBUG: transB2A ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: movesA []
2022-02-02 19:20:59: DEBUG: movesB []
2022-02-02 19:20:59:
2022-02-02 19:20:59: Actions queued on A:
2022-02-02 19:20:59: Backup on A: 'notes.archive.org'
2022-02-02 19:20:59: Backup on A: 'org-annotate-file.org'
2022-02-02 19:20:59: Backup on A: 'notes.local.org'
2022-02-02 19:20:59: Backup on A: 'web-capture.org'
2022-02-02 19:20:59:
2022-02-02 19:20:59: Actions queued on B:
2022-02-02 19:20:59: Backup on B: 'notes.org'
2022-02-02 19:20:59:
2022-02-02 19:20:59: Performing Actions on A
2022-02-02 19:20:59: DEBUG: A dels_back []
2022-02-02 19:20:59: DEBUG: A dels_noback []
2022-02-02 19:20:59: DEBUG: A moves []
2022-02-02 19:20:59: DEBUG: rclone:call ['rclone', 'backend', 'features', '/home/etienne/.cache/emacs/org/']
2022-02-02 19:20:59: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:59: DEBUG: backing up ['notes.archive.org', 'org-annotate-file.org', 'notes.local.org', 'web-capture.org']
2022-02-02 19:20:59: DEBUG: rclone:call ['rclone', 'copy', '-v', '--stats-one-line', '--log-format', '', '--no-check-dest', '--ignore-times', '--no-traverse', '--no-traverse', '--no-check-dest', '--ignore-times', '--retries', '4', '--files-from', '/tmp/tmp1xd23xdq/A_movedel_CJSbz7xwKf', '/home/etienne/.cache/emacs/org/', '/home/etienne/.cache/emacs/org/.syncrclone/backups/2022-02-02T192058_NcS9y_A']
2022-02-02 19:20:59: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:59: rclone: <6>INFO : org-annotate-file.org: Copied (new)
2022-02-02 19:20:59: rclone: <6>INFO : notes.local.org: Copied (new)
2022-02-02 19:20:59: rclone: <6>INFO : web-capture.org: Copied (new)
2022-02-02 19:20:59: rclone: <6>INFO : notes.archive.org: Copied (new)
2022-02-02 19:20:59: rclone: <6>INFO : 4.814 KiB / 4.814 KiB, 100%, 0 B/s, ETA -
2022-02-02 19:20:59: Backups for A stored in '/home/etienne/.cache/emacs/org/.syncrclone/backups/2022-02-02T192058_NcS9y_A'
2022-02-02 19:20:59:
2022-02-02 19:20:59: Performing Actions on B
2022-02-02 19:20:59: DEBUG: B dels_back []
2022-02-02 19:20:59: DEBUG: B dels_noback []
2022-02-02 19:20:59: DEBUG: B moves []
2022-02-02 19:20:59: DEBUG: rclone:call ['rclone', 'backend', 'features', 'webdav:/org/']
2022-02-02 19:20:59: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:20:59: DEBUG: backing up ['notes.org']
2022-02-02 19:20:59: DEBUG: rclone:call ['rclone', 'copy', '-v', '--stats-one-line', '--log-format', '', '--no-check-dest', '--ignore-times', '--no-traverse', '--no-traverse', '--no-check-dest', '--ignore-times', '--retries', '4', '--files-from', '/tmp/tmp1xd23xdq/B_movedel_B4XIRpAKak', 'webdav:/org/', 'webdav:/org/.syncrclone/backups/2022-02-02T192058_NcS9y_B']
2022-02-02 19:20:59: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:21:00: rclone: <6>INFO : notes.org: Copied (server-side copy)
2022-02-02 19:21:00: rclone: <6>INFO : 9.857 KiB / 9.857 KiB, 100%, 0 B/s, ETA -
2022-02-02 19:21:00: Backups for B stored in 'webdav:/org/.syncrclone/backups/2022-02-02T192058_NcS9y_B'
2022-02-02 19:21:00:
2022-02-02 19:21:00: A >>> B 1 files, 9.87 kb
2022-02-02 19:21:00: DEBUG: rclone:call ['rclone', 'copy', '-v', '--stats-one-line', '--log-format', '', '--no-traverse', '--files-from', '/tmp/tmp1xd23xdqA2B_transfer', '/home/etienne/.cache/emacs/org/', 'webdav:/org/']
2022-02-02 19:21:00: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
2022-02-02 19:21:01: rclone: <6>INFO : notes.org: Copied (replaced existing)
2022-02-02 19:21:01: rclone: <6>INFO : 9.871 KiB / 9.871 KiB, 100%, 0 B/s, ETA -
2022-02-02 19:21:01:
2022-02-02 19:21:01: A <<< B 4 files, 4.81 kb
2022-02-02 19:21:01: DEBUG: rclone:call ['rclone', 'copy', '-v', '--stats-one-line', '--log-format', '', '--no-traverse', '--files-from', '/tmp/tmp1xd23xdqB2A_transfer', 'webdav:/org/', '/home/etienne/.cache/emacs/org/']
2022-02-02 19:21:01: DEBUG: rclone: env {'RCLONE_CONFIG': '/home/etienne/.config/rclone/rclone.conf', 'RCLONE_ASK_PASSWORD': 'false'}
ERROR. Dumping logs (with debug) to '/tmp/tmp77grkpus/log.txt'
Traceback (most recent call last):
File "/usr/local/bin/syncrclone", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/syncrclone/cli.py", line 290, in cli
r = SyncRClone(config,break_lock=config.break_lock)
File "/usr/local/lib/python3.8/dist-packages/syncrclone/main.py", line 133, in __init__
self.rclone.transfer('B2A',self.transB2A)
File "/usr/local/lib/python3.8/dist-packages/syncrclone/rclone.py", line 539, in transfer
self.call(cmd,stream=True)
File "/usr/local/lib/python3.8/dist-packages/syncrclone/rclone.py", line 104, in call
for line in iter(proc.stdout.readline, ''):
File "/usr/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 90: invalid start byte
Thanks for this project.
Clearly the problem is that you use emacs :-)
This is very odd to me as Unicode file names are tested. And this part of the code should be just printing what rclone is saying. So it's getting an errant byte from somewhere.
Notice the failed line with the call. Can you run that joined (shlex.join) if it fails again. For example, in the above it would be
$ rclone copy -v --stats-one-line --log-format '' --no-traverse --files-from /tmp/tmp1xd23xdqB2A_transfer webdav:/org/ /home/etienne/.cache/emacs/org/
And can you also post
/tmp/tmp1xd23xdqB2A_transfer
I want to see what output rclone is throwing.
I can (and will later or tomorrow) make it more graceful in these errors since it shouldn't be fatal but I'd also like to solve the underlying issue.
The content of /tmp/tmpdyyfj1vrB2A_transfer is:
notes.archive.org
notes.org
web-capture.org
notes.local.org
org-annotate-file.org
Now executing the command joined (by query replace with Emacs 😉) :
$ rclone copy -v --stats-one-line --log-format --no-traverse --files-from /tmp/tmpdyyfj1vrB2A_transfer webdav:/org/ /home/etienne/.cache/emacs/org/
<6>INFO : There was nothing to transfer
<6>INFO : 0 B / 0 B, -, 0 B/s, ETA
I see nothing offending.
What is <6>?
Either way, I'll add a catch for this. It won't fix the issue but will both let it pass and give us more info if it happens again.
Version 20220204.0.BETA better handles if it can't decode from rclone while transfering.
I still do not understand what is causing this error. rclone should only be returning properly formed UTF8 on both stdout and stderr which is being read directly.
I am going to close this issue for now. I don't like it since it has been band-aided and not really fixed but absent being able to reproduce it reliably, I am not sure what else to do.
Feel free to reopen and/or add more information as it happens.
note: You will should not get an error anymore but if you look at your logs, it'll have things like \x8b in the output.