Giters

chef / omnibus

Easily create full-stack installers for your project across a variety of platforms.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Concurrency issues if two software definitions have the same source directory

albertvaka opened this issue · comments

commented

Description

We have two software definitions that are built from the code in our repo, for which we use source path: '..'. When workers > 1, we see ENOENT failures due to having two FileSyncer trying to sync the same file.

Omnibus Version

master

Platform Version

Any Linux, on both Ruby 2.7 and 2.5.

Replication Case

See description.

Build Output

[PathFetcher: <FIRST RECIPE NAME>] I | 2021-01-04T10:11:06+00:00 | Copying from `..'
[PathFetcher: <SECOND RECIPE NAME>] I | 2021-01-04T10:11:06+00:00 | Copying from `..'
/usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:142:in `stat': No such file or directory @ rb_file_s_stat - ../<FILE PATH HERE> (Errno::ENOENT)
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:142:in `block in sync'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:125:in `each'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:125:in `sync'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/fetchers/path_fetcher.rb:64:in `fetch'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/software.rb:1075:in `fetch'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/project.rb:1306:in `block (3 levels) in download'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:61:in `block (4 levels) in initialize'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:59:in `loop'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:59:in `block (3 levels) in initialize'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:58:in `catch'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:58:in `block (2 levels) in initialize'
commented

Omnibus doesn't have "recipes", so this confused me quite a bit as to what you actually meant. You mean two different software definitions in the same project? I mentally mapped "recipes" onto "projects" first and not "software" and nearly reflexively closed this as having two different projects running concurrently trying to use the same source directory is massively not supported. From your output though and the comment about "workers > 1" it sounds like you're talking about software definitions though.

commented

Thanks for looking at this @lamont-granquist 🙇 Indeed, I meant software definitions, sorry. I've updated the issue now.