Git 2.24 breaks existing repositories: filename in tree entry contains backslash

Question

Git 2.24 breaks existing repositories: filename in tree entry contains backslash

carlescufi opened this issue 5 years ago · comments

I was not able to find an open or closed issue matching what I'm seeing

What are users supposed to do here, rewrite the history of the repos so that the offending commit is removed?
The workaround is to disable core.protectNTFS:
git config --global core.protectNTFS false

Setup

Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
Git for Windows 2.24.1.windows.2 64-bit

$ git --version --build-options

git version 2.24.1.windows.2
cpu: x86_64
built from commit: 992f0773022527b1b0cb1e0c13aec97dd5248053
sizeof-long: 4
sizeof-size_t: 8

Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.18363.535]

What options did you set as part of the installation? Or did you choose the
defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Interactive Add: Disabled

Any other interesting things about your environment that might be related
to the issue you're seeing?

Installed with Chocolatey

Details

Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

cmd.exe

What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.

git clone https://github.com/zephyrproject-rtos/civetweb.git

What did you expect to occur after running these commands?

Cloned repo successfully.

What actually happened instead?

Cloning into 'civetweb'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 21161 (delta 17), reused 19 (delta 7), pack-reused 21123
Receiving objects: 100% (21161/21161), 24.53 MiB | 7.55 MiB/s, done.
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
Resolving deltas: 100% (13566/13566), done.
Updating files: 100% (1251/1251), done.

If the problem was occurring with a specific repository, can you provide the
URL to that repository to help us with testing?

https://github.com/zephyrproject-rtos/civetweb.git

Johannes Schindelin · Answer 1 · Tue Dec 17 2019 04:59:53 GMT+0800 (China Standard Time)

@carlescufi did you find out which tree entries those might be? They really cannot be checked out on Windows as expected: if there is a backslash encoded in any file name in that repository, it will be mistaken for a directory separator on Windows, which is obviously not what we want here.

Marti Bolivar · Answer 2 · Tue Dec 17 2019 05:42:24 GMT+0800 (China Standard Time)

@carlescufi did you find out which tree entries those might be?

One commit with a bad file is zephyrproject-rtos/civetweb@9d3af98 which adds a test/\/a.txt. However, this file was later removed in zephyrproject-rtos/civetweb@83dc425, and no file in the latest master branch contains a backslash in its name.

They really cannot be checked out on Windows as expected:

No attempt is made to check out a file with a backslash when cloning the latest master.

And indeed, though git clone https://github.com/zephyrproject-rtos/civetweb prints errors, it exits with a 0 return code.

However, this git fetch fails with return code 1:

mkdir civetweb
cd civetweb
git init
git fetch https://github.com/zephyrproject-rtos/civetweb.git

That seems inconsistent and arguably wrong: since we can clone the repository without an error return code (just error messages), we should be able to fetch it, too, just not check out the bad revisions.

Carles Cufí · Answer 3 · Tue Dec 17 2019 19:34:33 GMT+0800 (China Standard Time)

@dscho as @mbolivar pointed out there is no longer any file in the repository with a backslash in its name. This means that, short of changing the behavior of Git for Windows in an upcoming release, the only other alternative is to rewrite the history of the tree to completely remove any traces of commits that introduced filenames with backslashes. This might be very cumbersome on some repositories.

Johannes Schindelin · Answer 4 · Tue Dec 17 2019 20:37:15 GMT+0800 (China Standard Time)

@carlescufi & @mbolivar I am sympathetic to your use case. However, I am still worried about lifting the protections that we carefully put in place. So I'd like to tread very, very carefully here.

To start us in a direction that hopefully will lead to an acceptable compromise that still keeps the Git for Windows users safe, but also allows users to clone civetweb as long as they do not try to check out commits with the inadmissible directory name, I opened #2437.

Carles Cufí · Answer 5 · Tue Dec 17 2019 20:49:08 GMT+0800 (China Standard Time)

@dscho thank you for the understanding and the PR. Please let us know when we can fetch a binary from CI so we can test it ourselves and give a +1.

Johannes Schindelin · Answer 6 · Tue Dec 17 2019 22:03:46 GMT+0800 (China Standard Time)

Please let us know when we can fetch a binary from CI so we can test it ourselves and give a +1.

Whoa. I would like to ask even more explicitly for help with thinking about the problem. Maybe you can give a hand with reviewing the code and trying to get it into a shape where we can be reasonably sure that it does not regress on the vulnerabilities? Before that, I won't even bother with thinking about binaries.

Carles Cufí · Answer 7 · Tue Dec 17 2019 22:19:36 GMT+0800 (China Standard Time)

Please let us know when we can fetch a binary from CI so we can test it ourselves and give a +1.

Whoa. I would like to ask even more explicitly for help with thinking about the problem. Maybe you can give a hand with reviewing the code and trying to get it into a shape where we can be reasonably sure that it does not regress on the vulnerabilities? Before that, I won't even bother with thinking about binaries.

Sorry, since you had posted a Pull Request I assumed that you had considered those already. I will take a look at the current PR with @mbolivar and give feedback here.

Michael T. DeGuzis · Answer 8 · Fri Jan 03 2020 23:08:02 GMT+0800 (China Standard Time)

I have this issue with Jenkins as well, but I have no idea how to implement the workaround, as I don't see a setting for the plugin or a way to use the GitSCM class to work around this. 'bat' script block with the workaround also do not work.

https://issues.jenkins-ci.org/browse/JENKINS-60632

Edit: Tracking the relevant PRs for resolution in addition to the above.

If we are ok with wiping the commit with the offending blackslash-named files from history, how can this be done if we want to fix this now? I've tried git filter-branch to no avail and bfg

$  git log --all --name-only --pretty=format:%H | grep -B 2 '\\'                                                           

hash1
"mypkg/\\"
--

hash2
"mypkg/\\"

# Various escape method to match
git filter-branch --index-filter 'git rm --cached --ignore-unmatch "\"mypkg/\\.*"'

Johannes Schindelin · Answer 9 · Sat Jan 04 2020 06:28:36 GMT+0800 (China Standard Time)

Could y'all please test https://github.com/git-for-windows/git/releases/tag/v2.25.0-rc1.windows.1?

@mtdeguzis If you could test, too? And if successful, recommend to upgrade to v2.25.0 for the affected parties (Git for Windows v2.25.0 is expected to land on January 13th, i.e. 10 days from now)?

Marti Bolivar · Answer 10 · Sat Jan 04 2020 07:17:13 GMT+0800 (China Standard Time)

Could y'all please test https://github.com/git-for-windows/git/releases/tag/v2.25.0-rc1.windows.1?

This solves the problem on my end. The fetch completes without warning or error. Thanks!

Johannes Schindelin · Answer 11 · Sat Jan 04 2020 14:51:22 GMT+0800 (China Standard Time)

@mbolivar could you also, just for shiggles, try to check out the faulty revision? (It should fail)

Michael T. DeGuzis · Answer 12 · Sun Jan 05 2020 09:44:08 GMT+0800 (China Standard Time)

I'm not sure how I can test since mine is affected by Jenkins git plugin. I will test as much as possible though.

Johannes Schindelin · Answer 13 · Sun Jan 05 2020 17:51:14 GMT+0800 (China Standard Time)

@mtdeguzis I thought you would have the option of bundling Git for Windows with the plugin (or if not, be able to install the -rc version on the machine running Jenkins)?

Hans Beck · Answer 14 · Mon Jan 06 2020 17:50:40 GMT+0800 (China Standard Time)

Could y'all please test https://github.com/git-for-windows/git/releases/tag/v2.25.0-rc1.windows.1?

2.25.0.rc1 works for me. Had error fatal: filename in tree entry contains backslash: with git 2.24.1.windows.2 and no error with git 2.25.0.rc1.windows.1
Thanks a lot! :)

Johannes Schindelin · Answer 15 · Mon Jan 06 2020 21:09:09 GMT+0800 (China Standard Time)

@aaniin thank you for confirming!

Michael T. DeGuzis · Answer 16 · Mon Jan 06 2020 23:01:30 GMT+0800 (China Standard Time)

@dscho I can confirm this fixes the issue using Git Bash under v2.25.0-rc1.windows.1 and the Jenkins git plugin. Those using Jenkins can also refer to https://issues.jenkins-ci.org/browse/JENKINS-60632.

Johannes Schindelin · Answer 17 · Mon Jan 06 2020 23:04:57 GMT+0800 (China Standard Time)

Excellent. Thanks, @mtdeguzis!

Marti Bolivar · Answer 18 · Tue Jan 07 2020 02:38:33 GMT+0800 (China Standard Time)

@mbolivar could you also, just for shiggles, try to check out the faulty revision? (It should fail)

Confirmed; it fails.

>git --version
git version 2.25.0.rc1.windows.1
>git checkout 9d3af98
error: filename in tree entry contains backslash: 'test/\/a.txt'

HEAD remained the same after the checkout.

Johannes Schindelin · Answer 19 · Tue Jan 07 2020 05:46:23 GMT+0800 (China Standard Time)

Thank you @mbolivar!

Johannes Schindelin · Answer 20 · Tue Jan 07 2020 05:47:08 GMT+0800 (China Standard Time)

Fixed in v2.25.0-rc1 via git@224c7d7.

Carles Cufí · Answer 21 · Tue Jan 07 2020 18:23:33 GMT+0800 (China Standard Time)

@dscho thank you for all the work on this issue.

inky · Answer 22 · Fri Mar 27 2020 22:37:27 GMT+0800 (China Standard Time)

When I am trying to clone this repo: https://github.com/vinhphunguyen/MPM-Julia
I can clone it, but cannot checkout.

$ git clone https://github.com/inkydragon/MPM-Julia.git
Cloning into 'MPM-Julia'...
remote: Enumerating objects: 237, done.
remote: Counting objects: 100% (237/237), done.
remote: Compressing objects: 100% (211/211), done.
remote: Total 237 (delta 26), reused 233 (delta 25), pack-reused 0
Receiving objects: 100% (237/237), 26.44 MiB | 35.00 KiB/s, done.
Resolving deltas: 100% (26/26), done.
error: invalid path 'CPDI_3D_T4_Cantilever/.\Figs\Cantelever_0.png'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

$ git --version
git version 2.26.0.windows.1

$ git checkout master
error: invalid path 'CPDI_3D_T4_Cantilever/.\Figs\Cantelever_0.png'
error: invalid path 'Classic_2D_2Disk/..\..\Figs\plot_2Disk_Julia.pdf'
error: invalid path 'Classic_2D_SteelAluminum/.\Figs\DiskImpact_1.png'
error: invalid path 'Classic_2D_SteelAluminum/.\Figs\DiskImpact_1001.png'
...

Also, $ git config core.protectNTFS false doesnot work.

This repo seems created in macOS, and has strange filename such as: .\Figs\DiskImpact_1.png.
Yes it's a filename not a path.

Philip Oakley · Answer 23 · Sat Mar 28 2020 00:25:25 GMT+0800 (China Standard Time)