`git cinnabar fsck` reports bad files
saschanaz opened this issue · comments
> git cinnabar --version
0.5.9a
module-hash: 5e3702f55a47d0bbe5d819cb93d960888138b9be
helper-hash: 35b90b2bf83cfb025e256ca1d65dc8763437b517
> git cinnabar fsck
Checking 8 changeset heads
Loading 17316 manifests
Checking 23 manifest heads
Sha1 mismatch for file media/mozva/mozva.c
revision 8e7f694721797cb82f6573f32c6d3ffb6ef15fac
with parent 49aac12cc4dc6cc360de7c71fa93903341522857
Sha1 mismatch for file media/mozva/va/README
revision 0250c7ce5720e7c10510d41ce30829da3a09fe1b
Sha1 mismatch for file media/mozva/va/va.h
revision d2566169de6a0f824b56dedec361e05fe8f3c916
Sha1 mismatch for file media/mozva/va/va.patch
revision 05a5c75ab6c1cab7c28571c2b2b2b8b5b6831463
Sha1 mismatch for file media/mozva/va/va_dec_av1.h
revision b8bf4cc914f475fabfd505d223f7f8dca2863255
Sha1 mismatch for file media/mozva/va/va_dec_vp8.h
revision bb3beb2542ff23bb0b1525d691fb3a87c5e76831
Sha1 mismatch for file media/mozva/va/va_dec_vp9.h
revision 4feb8c50a635bb97369008c4a1f1ba44bfd7388c
Sha1 mismatch for file media/mozva/va/va_version.h
revision 8017bcff5b3b54798c5408f2c983abb8e86f755b
Sha1 mismatch for file third_party/rust/clap/src/build/app/settings.rs
revision 7d202d211c65ebc03593294d47c9c8589178592b
Sha1 mismatch for file third_party/rust/clap/src/build/arg/mod.rs
revision 682fe035050186d92e7495025d49153174333946
Sha1 mismatch for file third_party/rust/clap/src/build/arg/settings.rs
revision 51adf7ed600b68b0294f1df76af483227bf67675
Sha1 mismatch for file third_party/rust/clap/src/build/arg_group.rs
revision 1eec6e8289428b9eb98d5a4dc9b2612325fd7437
Sha1 mismatch for file third_party/rust/clap/src/build/macros.rs
revision a88060e7a1e2e4b949befa0178dcbe9426bdc9a5
Sha1 mismatch for file third_party/rust/terminal_size/README.md
revision cf8539acb96fde89b45aa27555b29c3d0f1afd75
Checking 47265 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues
Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.
Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.
Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.
But those revision don't seem to actually exist?
Is it the same repo you fixed in #280? Can you create a git bundle the same way you did back then?
Can you do another one with git bundle create $file refs/cinnabar/metadata refs/cinnabar/checked refs/cinnabar/broken? (without --not ...)
Sent it 👍
This time, your repo is more busted than last time, but it's still the same root cause (#207). I'm bummed that the situation was not detected.
You seem to have a setup that amplified the problem, creating a diverging history from gecko-dev. I suspect you have a pref like cinnabar.data or remote.something.cinnabar-data set to always. I wouldn't recommend you use that.
You can fix your repo with
git cinnabar rollback b1fe9b8493dbb38038c230f2fb346bfa803a60fe
and re-pulling, but any local branch you have will need to be rebased.
I think you're the first person to whom this happened twice, so if you don't mind, I'd like to ask you to set something up after I work on some debugging code.
You seem to have a setup that amplified the problem, creating a diverging history from gecko-dev. I suspect you have a pref like
cinnabar.dataorremote.something.cinnabar-dataset toalways. I wouldn't recommend you use that.
No, this is all I have:
> git config --get-regexp cinnabar
cinnabar.version-check 1643634802
cinnabar.helper C:\Users\sasch\Documents\GitHub\git-cinnabar\git-cinnabar-helper.exe
cinnabar.fsck 1637682708
I think you're the first person to whom this happened twice, so if you don't mind, I'd like to ask you to set something up after I work on some debugging code.
👍 Feel free to ping me whenever it's ready.
During rebase...
> git pull
WARNING Mercurial libraries not found. Falling back to experimental native access.
Reading 1836 changesets
Reading and importing 1830 manifests
Reading and importing 3114 revisions of 1130 files
Importing 1836 changesets
Unpacking objects: 100% (40/40), 58.72 KiB | 7.34 MiB/s, done.
ERROR Error in file b'f917b11e765e626d9459c18cc3771e4decd75880'
ERROR Error in file b'8fc2e6a72551b0e749377adf2af35cdbd14c94ff'
Checking 1417 imported file root and head revisions
ERROR Object of type bytes is not JSON serializable
Run the command again with `git -c cinnabar.check=traceback <command>` to see the full traceback.
fatal: Unsupported command: error
fast-import: dumping crash report to .git/fast_import_crash_21684
fatal: error while running fast-import
> git cinnabar fsck
Checking 4 changeset heads
Loading 17606 manifests
Checking 21 manifest heads
Sha1 mismatch for file testing/web-platform/meta/cookies/attributes/secure.https.html.ini
revision 8fc2e6a72551b0e749377adf2af35cdbd14c94ff
Sha1 mismatch for file gfx/angle/checkout/third_party/vulkan-deps/vulkan-headers/src/include/vulkan/vulkan_core.h
revision f917b11e765e626d9459c18cc3771e4decd75880
Checking 48228 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues
Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.
Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.
Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.
👀 Should I send one again?
The crash report has nothing interesting:
fast-import crash report:
fast-import process: 21684
parent process : 1
at 2022-02-01 12:06:15 +0000
fatal: Unsupported command: error
Most Recent Commands Before Crash
---------------------------------
* error
Active Branch LRU
-----------------
active_branches = 0 cur, 5 max
pos clock name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inactive Branches
-----------------
Marks
-----
-------------------
END OF CRASH REPORT
ERROR Error in file b'f917b11e765e626d9459c18cc3771e4decd75880'
ERROR Error in file b'8fc2e6a72551b0e749377adf2af35cdbd14c94ff'
Checking 1417 imported file root and head revisions
ERROR Object of type bytes is not JSON serializable
Is it the first time you see something like this?
> git config --get-regexp cinnabar
cinnabar.version-check 1643634802
cinnabar.helper C:\Users\sasch\Documents\GitHub\git-cinnabar\git-cinnabar-helper.exe
cinnabar.fsck 1637682708
How do you push to try?
Is it the first time you see something like this?
I believe I saw it at least once before this 🤔
How do you push to try?
I use ./mach try.
I believe I saw it at least once before this thinking
I wish you had reported it back then because it showed that the diagnostic code for #207 worked but failed to report the problem correctly with python3 :-/
I use
./mach try
Do you maybe git cinnabar fetch from try?
+1 Feel free to ping me whenever it's ready.
Can you update git-cinnabar to latest master, run git config cinnabar.check unbundler, then start again from #295 (comment) ?
If it doesn't break again on that update, like it did in #295 (comment), can you keep that cinnabar.check value set until the next time it happens, at which time it will hopefully give you some instructions.
I wish you had reported it back then because it showed that the diagnostic code for #207 worked but failed to report the problem correctly with python3 :-/
Oops, sorry 😬
Do you maybe
git cinnabar fetchfrom try?
TIL it exists, but I think I never fetch from try.
Can you update git-cinnabar to latest master, run
git config cinnabar.check unbundler, then start again from #295 (comment) ?
Will do that 👍 (Edit: It didn't break)
TIL it exists, but I think I never fetch from try.
Something doesn't line up. When you push to try, can you check whether git rev-parse refs/cinnabar/metadata returns the same thing before and after you do the push?
> git pull --rebase origin bookmarks/central:main
WARNING Mercurial libraries not found. Falling back to experimental native access.
Reading 299 changesets
Reading and importing 299 manifests
Reading and importing 16 revisions of 12 filesfatal: Malformed file chunk for a59a87e470eca653bd2a0c76662372c9e5e1fe7b
fast-import: dumping crash report to .git/fast_import_crash_9400
Reading and importing 264 revisions of 249 files
Traceback (most recent call last):
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1003, in run
fatal: Unsupported command: error
f retcode = func(args)
ast-import: dumping crash report to .git/fast_import_crash_17980
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\git-remote-hg", line 53, in main
helper.run()
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\remote_helper.py", line 87, in run
self.import_(*args)
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\remote_helper.py", line 422, in import_
getbundle(self._repo, self._store, heads,
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 1071, in getbundle
apply_bundle(store)
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 893, in __call__
for rev_chunk in progress_enum(
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 224, in progress_enum
for count, item in enum_iter:
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 869, in enumerate_files
for count_chunks, (name, chunk) in enumerate(iterator, start=1):
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 837, in iter_files
fh.write(chunk)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\git-remote-hg", line 59, in <module>
run(main, sys.argv[1:])
fatal: error while running fast-import
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1012, in run
message = message or ', '.join(
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1013, in <genexpr>
fsdecode(a) for a in getattr(e, 'args', []))
File "C:\Users\sasch\AppData\Local\Programs\Python\Python310\lib\os.py", line 822, in fsdecode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not int
Exception ignored in atexit callback: <bound method GitHgHelper.close of <class 'cinnabar.helper.GitHgHelper'>>
Traceback (most recent call last):
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\helper.py", line 513, in close
super(GitHgHelper, self).close()
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\helper.py", line 108, in close
self._helper.wait()
File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 835, in wait
fh.close()
OSError: [Errno 22] Invalid argument
> git cinnabar fsck
Checking 3 changeset heads
Loading 331 manifests
Checking 3 manifest heads
Sha1 mismatch for file toolkit/components/glean/tests/browser/browser_fog_gmp.js
revision 61b8f076564ff8c1f2b16b5438b021fe152827e8
with parent 56b7e3dc829cecaac0f860c2479d90babd9a5f71
Sha1 mismatch for file toolkit/components/glean/tests/browser/browser_fog_socket.js
revision d956b2c06e402732a8548180afde2e1b9176fa53
with parent de7589f6616d03e269aeeb5e0509354e136f41a1
Sha1 mismatch for file devtools/client/netmonitor/docs/architecture.md
revision 2e0a5c33feaa57639a305f1c10eaa2789d17592f
with parent 5d7e2850f40bc245751a4388f7435269f675c888
Checking 1232 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues
Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.
Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.
Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.
👀
Hmm, I ran git cinnabar fsck --force and got tons of mismatches 🤔, I wonder how it happened...
did you git cinnabar rollback b1fe9b8493dbb38038c230f2fb346bfa803a60fe as per #295 (comment) ?
I did git cinnabar rollback --fsck instead and thought it should be equivalent 👀 (as it did require rebasing.)
Kind of. I'm baffled you didn't see the other error message before that...
Which error message?
git-cinnabar/cinnabar/githg.py
Lines 1324 to 1336 in 6ee0223
No, haven't seen that one so far, sorry 🥲
Do you still have the repo from #295 (comment) ?
I don't but I made a gitbundle from it, so maybe I can say I do?
Is it a different clone from the previous one?
Same clone but after git cinnabar rollback (and right after the errors)
That doesn't sound true. The bundle you sent me has a bustage that started in august (which explains why you didn't get the bustage detection error), and doesn't contain b1fe9b8493dbb38038c230f2fb346bfa803a60fe
I mean right after #295 (comment), I made the bundle after this as I expected your request to share it, 🤔
Weird that it still had the bustage, I guess git cinnabar rollback --fsck didn't really work then? (I wonder that's why I saw #295 (comment))
But it's a different clone than the one you opened this issue for, right?
It was still the same clone. I didn't do git cinnabar reclone, just the rollback.
So, this exposes a flaw in git cinnabar fsck, which is that it doesn't look at all the files that ever existed, but only those referenced by the current heads... So some files could still be broken, and, in your case, there were broken files in metadata older than b1fe9b8493dbb38038c230f2fb346bfa803a60fe.
You'd probably be better off with a reclone at this point.
You do still seem to have hit #207 several times, so please keep cinnabar.check set to unbundler.
Yeah, I had to reclone to fix the broken pipe error. I'll report again if it happens again.
Just sent you the bundle files again. This time there was some interesting things in cinnabar.log:
ERROR authorization failed
ERROR no suitable response from remote hg
ERROR no suitable response from remote hg
ERROR Error in file db948f474be11dfe25e63d795061343452c4162c
ERROR Error in file 332c47c2565d9cb696af07345e6ccfaa5d850ac9
ERROR Error in file 1dc94db8b234de69246beb6f3e2b2b85a94d7c16
ERROR Error in file 5b8d62bc6ac27f86208ecb6a3d026d3470f2e0fa
ERROR It seems you have hit a known, rare, and difficult to reproduce issue.
Your help would be appreciated.
Please keep a copy of the .git/cinnabar-last-bundle file before doing the following.
Please try either `git cinnabar rollback` followed by the same command that just
failed, or `git cinnabar reclone`.
Please open a new issue (https://github.com/glandium/git-cinnabar/issues/new)
mentioning issue #207 and reporting whether the second attempt succeeded.
Please read all the above and keep a copy of this repository.
But not sure it's actually related as there was no timestamps.