bundle install fails on windows-latest after using bundle config

Question

bundle install fails on windows-latest after using bundle config

mojavelinux opened this issue 4 years ago · comments

If I use bundle config to set bundler options (e.g., bundle config --local without docs), the subsequent call to bundle install fails. The action gives no reason why it failed.

##[error]Process completed with exit code 1.

This does not happen on ubuntu-latest.

Dan Allen · Answer 1 · Thu Jan 30 2020 19:21:27 GMT+0800 (China Standard Time)

In contrast, passing options directly to bundle install works fine:

bundle --path .bundle/gems

However, that results in warnings about using persistent flags with bundle install.

Benoit Daloze · Answer 2 · Thu Jan 30 2020 19:25:35 GMT+0800 (China Standard Time)

Could you link to a log failing because of this? And/or share a workflow triggering it?

Dan Allen · Answer 3 · Thu Jan 30 2020 19:28:18 GMT+0800 (China Standard Time)

There's literally nothing in the log. You can find a workflow example here: https://github.com/asciidoctor/asciidoctor-pdf/blob/master/.github/workflows/ci.yml#L21 (I'm currently messing with it, so I could be changing).

Dan Allen · Answer 4 · Thu Jan 30 2020 19:30:45 GMT+0800 (China Standard Time)

2020-01-30T11:23:35.9402420Z Running `bundle install --jobs 3 --retry 3 --verbose` with bundler 2.1.4
2020-01-30T11:23:36.4473629Z Fetching https://github.com/prawnpdf/prawn-table.git
2020-01-30T11:23:39.8952804Z NOTE: Gem::Specification#rubyforge_project= is deprecated with no replacement. It will be removed on or after 2019-12-01.
2020-01-30T11:23:39.9130110Z ##[error]Process completed with exit code 1.
2020-01-30T11:23:39.9155841Z Cleaning up orphan processes

So the process is just dying.

Dan Allen · Answer 5 · Thu Jan 30 2020 19:33:20 GMT+0800 (China Standard Time)

This may or may not having anything to do with this action. I really have no idea because it's giving absolutely no information when it dies. Maybe that's GitHub Actions suppressing the output. Maybe that's Windows. If this issue only serves as a warning to others not to use bundle config on Windows runners, I would be okay with that. We can only fix what we can fix.

Dan Allen · Answer 6 · Thu Jan 30 2020 19:37:22 GMT+0800 (China Standard Time)

For completeness, here are the warnings we get when using flags instead of bundle config.

[DEPRECATED] The `--path` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set path '.bundle/gems'`, and stop using this flag
[DEPRECATED] The `--without` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set without 'docs lint coverage'`, and stop using this flag

Benoit Daloze · Answer 7 · Thu Jan 30 2020 21:51:15 GMT+0800 (China Standard Time)

As long as the error doesn't happen during the uses: eregon/use-ruby-action@v1 step, I think it is unlikely it has anything to do with this action.
An easy way to make sure would be to use https://github.com/actions/setup-ruby and see if it fails too.

BTW, could you use eregon/use-ruby-action@v1 and not eregon/use-ruby-action@v1.10.0 as recommended in the README? That way you'll get bug fixes, and old releases are not guaranteed to work.

I'm currently messing with it, so I could be changing

y can be used to point to a specific version of a file in GitHub

A link to a log would help me understand better.
I found https://github.com/asciidoctor/asciidoctor-pdf/runs/416999064?check_suite_focus=true
That seems clearly a bug of either GHA or Bundler 2.

I would recommend to not install bundler explicitly, unless you really need a specific version. This action already provides a version of bundle for all Rubies (see https://github.com/eregon/use-ruby-action#bundler).

Marat Radchenko · Answer 8 · Thu Jan 30 2020 22:11:05 GMT+0800 (China Standard Time)

That way you'll get bug fixes, and old releases are not guaranteed to work.

Oh, I didn't know that it works that way.

Benoit Daloze · Answer 9 · Thu Jan 30 2020 22:13:39 GMT+0800 (China Standard Time)

@slonopotamus That's documented in https://github.com/eregon/use-ruby-action#versioning

MSP-Greg · Answer 10 · Thu Jan 30 2020 22:55:09 GMT+0800 (China Standard Time)

@mojavelinux

I'll have a look at this later today. I tried the following locally with ruby master on Windows:

PS C:\Greg\GitHub\asciidoctor-pdf> bundle config --local path .bundle/gems
PS C:\Greg\GitHub\asciidoctor-pdf> bundle config --local without docs lint coverage
PS C:\Greg\GitHub\asciidoctor-pdf> bundle --jobs 3 --retry 3

See fix at asciidoctor/asciidoctor-pdf#1531

There's always been some quirky things with PowerShell in CI. How STDERR output is handled is a configurable item. Same with EXIT codes.

Benoit Daloze · Answer 11 · Fri Jan 31 2020 05:37:30 GMT+0800 (China Standard Time)

Thank you for looking at this @MSP-Greg, it's much appreciated and I had no idea what's going on here. I'll close this issue because I don't think this action can do anything about it, it seems a GitHub Actions/PowerShell issue.

I think it's best to discuss in the PR: asciidoctor/asciidoctor-pdf#1531
Maybe a workaround would be to use shell: bash for multi-line scripts.
That might be useful as I expect many Ruby developers are more familiar with bash than PowerShell, and it allows to use a common syntax on all 3 OS.

Marat Radchenko · Answer 12 · Fri Jan 31 2020 05:38:28 GMT+0800 (China Standard Time)

use shell: bash for multi-line scripts

That explodes with JRuby on Windows.

Benoit Daloze · Answer 13 · Fri Jan 31 2020 05:43:35 GMT+0800 (China Standard Time)

That explodes with JRuby on Windows.

@slonopotamus Can you show an example log where that happens?

Benoit Daloze · Answer 14 · Fri Jan 31 2020 05:45:02 GMT+0800 (China Standard Time)

BTW, I'm happy to improve the README based on the discussion in the PR.

Marat Radchenko · Answer 15 · Fri Jan 31 2020 05:47:12 GMT+0800 (China Standard Time)

That explodes with JRuby on Windows.

@slonopotamus Can you show an example log where that happens?

https://github.com/slonopotamus/epubcheck-ruby/runs/407569369?check_suite_focus=true#step:4:23

Benoit Daloze · Answer 16 · Fri Jan 31 2020 05:51:57 GMT+0800 (China Standard Time)

Interesting.
I noticed the bin/jruby: line 155: [: C:\Program: binary operator expected as well, but it didn't seem fatal. It was fixed in JRuby master, but not yet released: jruby/jruby#6007 (comment)

I'm not sure what

/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file

comes from.
It looks like it happens from bundle exec rake.

Marat Radchenko · Answer 17 · Fri Jan 31 2020 05:56:03 GMT+0800 (China Standard Time)

I didn't investigate it very much. Just added a mark my brain: jruby+windows+bash = no fun, use native shell instead. There are so many more issues with jruby on windows, this is a very small of them.

Dan Allen · Answer 18 · Fri Jan 31 2020 05:56:35 GMT+0800 (China Standard Time)

As long as the error doesn't happen during the uses: eregon/use-ruby-action@v1 step, I think it is unlikely it has anything to do with this action.

True, though I'm kind of exploring if it has to do with how Ruby is installed.

could you use eregon/use-ruby-action@v1 and not eregon/use-ruby-action@v1.10.0 as recommended in the README

Yes, ignore that. We're actively testing specific versions. Once our build is stable, we'll switch to the major.

I would recommend to not install bundler explicitly, unless you really need a specific version. This action already provides a version of bundle for all Rubies

Didn't realize that. Yes, that's what we want.

Maybe a workaround would be to use shell: bash for multi-line scripts.

Is the multi-line script what is causing the problem? That's what I'm trying to determine.

Dan Allen · Answer 19 · Fri Jan 31 2020 05:57:52 GMT+0800 (China Standard Time)

BTW, I'm happy to improve the README based on the discussion in the PR.

If we arrive at a conclusion as to what causes the exit code 1, I think we should do that. I am still not seeing the reason why we get an exit code 1 when using bundle config vs when not using bundle config. Unless it's the multiple run lines.

Benoit Daloze · Answer 20 · Fri Jan 31 2020 06:01:55 GMT+0800 (China Standard Time)

Is the multi-line script what is causing the problem? That's what I'm trying to determine.

Could you try with individual commands to see if that would help?
@MSP-Greg was saying in the PR it might be related to multiple lines + PowerShell settings about how warnings (STDERR output) are treated.

Benoit Daloze · Answer 21 · Fri Jan 31 2020 06:02:37 GMT+0800 (China Standard Time)

Let's reopen until we figure it out better, sorry for the early close.

MSP-Greg · Answer 22 · Fri Jan 31 2020 06:10:37 GMT+0800 (China Standard Time)

Sorry, not sure where to discuss. I'm not sure what is causing this. The warning are coming from RubyGems. I don't know if the command is exiting with a code other than 0, and I know that I've seen Windows CI get confused over STDERR output, especially using PowerShell.

With some AppVeyor build scripts for Ruby, I had to pipe STDERR to STDOUT, otherwise the script stopped.

IOW, this is not an issue with this repo or asciidoctor-pdf. It's somewhere within the space of Actions, PowerShell, Bundler, & RubyGems. The fact that I can't repo it locally doesn't help...

Benoit Daloze · Answer 23 · Fri Jan 31 2020 06:12:29 GMT+0800 (China Standard Time)

Re JRuby on Windows, it seems to work better if we remove the bin/jruby Bash script (and therefore always use the .exe).
Compare https://github.com/eregon/use-ruby-action/runs/418152773?check_suite_focus=true which warns for gem/bundle steps and https://github.com/eregon/use-ruby-action/runs/418163739?check_suite_focus=true which doesn't warn.
I'll do that change.

MSP-Greg · Answer 24 · Fri Jan 31 2020 06:17:40 GMT+0800 (China Standard Time)

Re JRuby on Windows, it seems to work better if we remove the bin/jruby Bash script (and therefore always use the .exe).

Sorry, I haven't really looked at JRuby on Windows. But, unless the bash script does something, I'd use the exe. There is no script with mswin or mingw builds for Ruby itself. All the gem bin files are .bat, .cmd, or .ps1. Windows normally needs the extension to know how to launch the file...

Benoit Daloze · Answer 25 · Fri Jan 31 2020 06:29:24 GMT+0800 (China Standard Time)

I removed the Bash launcher on JRuby, it seems better now.

Dan Allen · Answer 26 · Fri Jan 31 2020 06:43:58 GMT+0800 (China Standard Time)

Using separate steps fixes the problem.

https://github.com/mojavelinux/asciidoctor-pdf/actions/runs/33167078

I see no warnings though, which is very strange. While we work to pinpoint the cause of the exit 1, perhaps recommending using separate steps is a reasonable tradeoff until then.

Benoit Daloze · Answer 27 · Fri Jan 31 2020 07:00:44 GMT+0800 (China Standard Time)

In that run, the existing Bundler is used, which might change something, but probably not.
It seems far more likely the problem is indeed PowerShell with multiple lines.

The full warning is:

Fetching https://github.com/prawnpdf/prawn-table.git
NOTE: Gem::Specification#rubyforge_project= is deprecated with no replacement. It will be removed on or after 2019-12-01.
Gem::Specification#rubyforge_project= called from D:/a/asciidoctor-pdf/asciidoctor-pdf/.bundle/gems/ruby/2.7.0/bundler/gems/prawn-table-515f2db29486/prawn-table.gemspec:19.

And in https://github.com/asciidoctor/asciidoctor-pdf/runs/416845301 it's cut after the first warning line:

Fetching https://github.com/prawnpdf/prawn-table.git
NOTE: Gem::Specification#rubyforge_project= is deprecated with no replacement. It will be removed on or after 2019-12-01.

So it really seems PowerShell just aborts on STDERR output, for multi-line scripts, as @MSP-Greg said.

I think this would be highly valuable to report to GitHub Actions as a bug when using PowerShell with multiple lines. Could you report it?

Dan Allen · Answer 28 · Fri Jan 31 2020 07:05:43 GMT+0800 (China Standard Time)

Thanks for that assessment, @eregon. That seems to be it!

I agree, this should go upstream. I'm not very familiar with the GitHub Actions project. Do you know where it should be filed?

MSP-Greg · Answer 29 · Fri Jan 31 2020 07:08:17 GMT+0800 (China Standard Time)

I just tried the following in a step, and it passed:

- name: ruby test
  run: |
    ruby -e "STDERR.puts 'Causes PS exit?'"
    ruby -v

Benoit Daloze · Answer 30 · Fri Jan 31 2020 07:11:22 GMT+0800 (China Standard Time)

I think https://github.community/t5/GitHub-Actions/bd-p/actions is a good place to report it.

A bit of searching led me to
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#exit-codes-and-error-action-preference
And $ErrorActionPreference = 'stop' is according to https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_preference_variables?view=powershell-7#erroractionpreference:

Determines how PowerShell responds to a non-terminating error, an error that doesn't stop the cmdlet processing. For example, at the command line or in a script, cmdlet, or provider, such as the errors generated by the Write-Error cmdlet.
You can use a cmdlet's ErrorAction common parameter to override the preference for a specific command.
The valid values are as follows:
...
Stop: Displays the error message and stops executing.

And "the Write-Error cmdlet" seems similar as writing to STDERR.
Sounds related at least.

Benoit Daloze · Answer 31 · Fri Jan 31 2020 07:11:36 GMT+0800 (China Standard Time)

@MSP-Greg Could you post the log of the workflow run?

Dan Allen · Answer 32 · Fri Jan 31 2020 07:11:41 GMT+0800 (China Standard Time)

In that run, the existing Bundler is used, which might change something, but probably not.

Nope, it didn't change anything. It works either way when I use separate steps.

Benoit Daloze · Answer 33 · Fri Jan 31 2020 07:33:47 GMT+0800 (China Standard Time)

I can reproduce the failure in GitHub Actions with:
https://github.com/eregon/use-ruby-action/runs/418285035?check_suite_focus=true

    - name: Test
      run: |
        gem install bundler
        bundle config --local path .bundle/gems
        ruby -e 'STDERR.puts %{NOTE: warning}; STDERR.puts 42'
        echo after

Removing the bundle config makes it pass:
https://github.com/eregon/use-ruby-action/runs/418285586?check_suite_focus=true

    - name: Test
      run: |
        gem install bundler
        ruby -e 'STDERR.puts %{NOTE: warning}; STDERR.puts 42'
        echo after

Not installing latest bundler also passes:
https://github.com/eregon/use-ruby-action/runs/418287337?check_suite_focus=true

    - name: Test
      run: |
        bundle config --local path .bundle/gems
        ruby -e 'STDERR.puts %{NOTE: warning}; STDERR.puts 42'
        echo after

Installing latest bundler and bundle config but no STDERR print passes too:
https://github.com/eregon/use-ruby-action/runs/418294467?check_suite_focus=true

    - name: Test
      run: |
        gem install bundler
        bundle config --local path .bundle/gems
        echo after

So somehow gem install bundler + bundle config --local path .bundle/gems + STDERR output cause the failure? Maybe recent Bundler does something strange to the shell?

MSP-Greg · Answer 34 · Fri Jan 31 2020 07:49:02 GMT+0800 (China Standard Time)

What version (Bundler or Ruby) passed? I'll check this locally...

Benoit Daloze · Answer 35 · Sun Feb 09 2020 05:29:22 GMT+0800 (China Standard Time)

I'm not sure what's wrong here, but I already spent too much time debugging weird Windows shell bugs.
Please report the issue to GitHub Actions, for example at https://github.community/t5/GitHub-Actions/bd-p/actions

This is clearly not a bug of this action.

Dan Allen · Answer 36 · Sun Feb 09 2020 06:48:19 GMT+0800 (China Standard Time)

As far as I'm concerned, this is a solved issue either by:

Using a separate step or
Setting $ErrorActionPreference = 'Continue'

It's just not a hill I want to die on trying to solve otherwise.