When setting/overriding custom repository in Buildfile, http://repo1.maven.org/maven2 is still used.
bigsur0 opened this issue · comments
I still see lockjar/naether trying to use that public maven repo even after overriding the repository in my Buildfile.
cmd: bundle exec lockjar install -v -l my.lock -s default
Installing Jars from my.lock for ["default"]
[pool-1-thread-1] INFO NaetherTransfer - Downloading: http://repo1.maven.org/maven2/org/mortbay/....
[pool-2-thread-1] INFO NaetherTransfer - Downloading: https://my.artifactory.foobar/artifactory/....
The default repo is repo1.maven.org, the remote_repo dsl appends to the list. What will have to be exposed is a clear command, something like clear_remote_repos that would have to be called before remote_repo appends.
An alternative is to not fail/error when one of the repos can't be contacted. It seems like there are multiple threads at play via pools correct? Could the error handling be improved to not fail the full run if one repository is not accessible? Or even just succeed but log ERROR entries for the jars that aren't resolvable?
Can you share the error you are getting? Aether should not be tossing an error if a repo is unaccessible, that is expected to happen. It will toss an error if the dependency is not found in any repo.
I think this captures it. I disabled access to repo1.maven.com via my /etc/hosts file and the timeout exception aborts the lockjar install command see exit status below.
cmd: lockjar install -v -l my.lock -s default
Installing Jars from my.lock for ["default"]
[pool-1-thread-1] INFO NaetherTransfer - Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.3.0-cdh5.1.0/hadoop-common-2.3.0-cdh5.1.0.jar
[pool-2-thread-1] INFO NaetherTransfer - Downloading: https://artifacts.foobar.com/artifactory/org/apache/hadoop/hadoop-common/2.3.0-cdh5.1.0/hadoop-common-2.3.0-cdh5.1.0.jar
com/tobedevoured/naether/impl/NaetherImpl.java:734:in `downloadArtifacts': com.tobedevoured.naether.ResolveException: org.sonatype.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.hadoop:hadoop-common:jar:2.3.0-cdh5.1.0 from/to central (http://repo1.maven.org/maven2/): Error transferring file: Operation timed out
from java/lang/reflect/Method.java:606:in `invoke'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/naether-0.14.1-java/lib/naether/runtime.rb:338:in `download_artifacts'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/lock_jar-0.10.5/lib/lock_jar/resolver.rb:62:in `download'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/lock_jar-0.10.5/lib/lock_jar/runtime.rb:72:in `install'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/lock_jar-0.10.5/lib/lock_jar.rb:40:in `install'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/lock_jar-0.10.5/lib/lock_jar/cli.rb:54:in `install'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/gems/lock_jar-0.10.5/bin/lockjar:6:in `(root)'
from org/jruby/RubyKernel.java:1081:in `load'
from /Users/r6p/.rvm/gems/jruby-1.7.13@default/bin/lockjar:1:in `(root)'
Caused by:
DefaultArtifactResolver.java:538:in `resolve': org.sonatype.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.hadoop:hadoop-common:jar:2.3.0-cdh5.1.0 from/to central (http://repo1.maven.org/maven2/): Error transferring file: Operation timed out
from DefaultArtifactResolver.java:216:in `resolveArtifacts'
from DefaultArtifactResolver.java:193:in `resolveArtifact'
from DefaultRepositorySystem.java:286:in `resolveArtifact'
from RepositoryClient.java:121:in `resolveArtifact'
from NaetherImpl.java:732:in `downloadArtifacts'
from NativeMethodAccessorImpl.java:-2:in `invoke0'
from NativeMethodAccessorImpl.java:57:in `invoke'
from DelegatingMethodAccessorImpl.java:43:in `invoke'
from Method.java:606:in `invoke'
from JavaMethod.java:455:in `invokeDirectWithExceptionHandling'
from JavaMethod.java:316:in `invokeDirect'
from InstanceMethodInvoker.java:61:in `call'
from CachingCallSite.java:326:in `cacheAndCall'
from CachingCallSite.java:170:in `call'
from CallOneArgNode.java:57:in `interpret'
from LocalAsgnNode.java:123:in `interpret'
from NewlineNode.java:105:in `interpret'
from IfNode.java:116:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:182:in `call'
from CachingCallSite.java:326:in `cacheAndCall'
from CachingCallSite.java:170:in `call'
from CallOneArgNode.java:57:in `interpret'
from NewlineNode.java:105:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:182:in `call'
from CachingCallSite.java:326:in `cacheAndCall'
from CachingCallSite.java:170:in `call'
from CallOneArgNode.java:57:in `interpret'
from LocalAsgnNode.java:123:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:290:in `call'
from CachingCallSite.java:376:in `cacheAndCall'
from CachingCallSite.java:247:in `callBlock'
from CachingCallSite.java:251:in `call'
from CallThreeArgBlockPassNode.java:64:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:225:in `call'
from CachingCallSite.java:346:in `cacheAndCall'
from CachingCallSite.java:204:in `call'
from CallTwoArgNode.java:59:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:161:in `call'
from RubyClass.java:527:in `finvoke'
from RubyBasicObject.java:1501:in `send19'
from RubyBasicObject$INVOKER$i$send19_DBG.gen:-1:in `call'
from JavaMethod.java:350:in `call'
from CachingCallSite.java:326:in `cacheAndCall'
from CachingCallSite.java:170:in `call'
from CallSpecialArgNode.java:67:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from IfNode.java:116:in `interpret'
from IfNode.java:118:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from RescueNode.java:221:in `executeBody'
from RescueNode.java:116:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:225:in `call'
from CachingCallSite.java:346:in `cacheAndCall'
from CachingCallSite.java:204:in `call'
from CallSpecialArgNode.java:69:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from IfNode.java:118:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:225:in `call'
from CachingCallSite.java:346:in `cacheAndCall'
from CachingCallSite.java:204:in `call'
from CallTwoArgNode.java:59:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:112:in `call'
from InterpretedMethod.java:126:in `call'
from CachingCallSite.java:296:in `cacheAndCall'
from CachingCallSite.java:72:in `call'
from FCallManyArgsNode.java:60:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from RescueNode.java:221:in `executeBody'
from RescueNode.java:116:in `interpret'
from ASTInterpreter.java:74:in `INTERPRET_METHOD'
from InterpretedMethod.java:139:in `call'
from CachingCallSite.java:306:in `cacheAndCall'
from CachingCallSite.java:136:in `call'
from CallNoArgNode.java:60:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from RootNode.java:129:in `interpret'
from ASTInterpreter.java:121:in `INTERPRET_ROOT'
from Ruby.java:881:in `runInterpreter'
from Ruby.java:2833:in `loadFile'
from ExternalScript.java:66:in `load'
from LoadService.java:359:in `load'
from RubyKernel.java:1089:in `loadCommon'
from RubyKernel.java:1081:in `load19'
from RubyKernel$INVOKER$s$0$1$load19_DBG.gen:-1:in `call'
from DynamicMethod.java:210:in `call'
from DynamicMethod.java:206:in `call'
from CachingCallSite.java:326:in `cacheAndCall'
from CachingCallSite.java:170:in `call'
from FCallOneArgNode.java:36:in `interpret'
from NewlineNode.java:105:in `interpret'
from BlockNode.java:71:in `interpret'
from RootNode.java:129:in `interpret'
from ASTInterpreter.java:121:in `INTERPRET_ROOT'
from Ruby.java:881:in `runInterpreter'
from Ruby.java:889:in `runInterpreter'
from Ruby.java:720:in `runNormally'
from Ruby.java:565:in `runFromMain'
from Main.java:395:in `doRunFromMain'
from Main.java:290:in `internalRun'
from Main.java:217:in `run'
from Main.java:197:in `main'
Caused by:
WagonRepositoryConnector.java:951:in `wrap': org.sonatype.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.hadoop:hadoop-common:jar:2.3.0-cdh5.1.0 from/to central (http://repo1.maven.org/maven2/): Error transferring file: Operation timed out
from WagonRepositoryConnector.java:941:in `wrap'
from WagonRepositoryConnector.java:669:in `run'
from RunnableErrorForwarder.java:60:in `run'
from ThreadPoolExecutor.java:1145:in `runWorker'
from ThreadPoolExecutor.java:615:in `run'
from Thread.java:745:in `run'
Caused by:
LightweightHttpWagon.java:143:in `fillInputData': org.apache.maven.wagon.TransferFailedException: Error transferring file: Operation timed out
from StreamWagon.java:116:in `getInputStream'
from StreamWagon.java:88:in `getIfNewer'
from StreamWagon.java:61:in `get'
from WagonRepositoryConnector.java:601:in `run'
from RunnableErrorForwarder.java:60:in `run'
from ThreadPoolExecutor.java:1145:in `runWorker'
from ThreadPoolExecutor.java:615:in `run'
from Thread.java:745:in `run'
Caused by:
PlainSocketImpl.java:-2:in `socketConnect': java.net.ConnectException: Operation timed out
from AbstractPlainSocketImpl.java:339:in `doConnect'
from AbstractPlainSocketImpl.java:198:in `connectToAddress'
from AbstractPlainSocketImpl.java:182:in `connect'
from SocksSocketImpl.java:392:in `connect'
from Socket.java:579:in `connect'
from Socket.java:528:in `connect'
from NetworkClient.java:180:in `doConnect'
from HttpClient.java:432:in `openServer'
from HttpClient.java:527:in `openServer'
from HttpClient.java:211:in `<init>'
from HttpClient.java:308:in `New'
from HttpClient.java:326:in `New'
from HttpURLConnection.java:996:in `getNewHttpClient'
from HttpURLConnection.java:932:in `plainConnect'
from HttpURLConnection.java:850:in `connect'
from HttpURLConnection.java:1300:in `getInputStream'
from HttpURLConnection.java:468:in `getResponseCode'
from LightweightHttpWagon.java:115:in `fillInputData'
from StreamWagon.java:116:in `getInputStream'
from StreamWagon.java:88:in `getIfNewer'
from StreamWagon.java:61:in `get'
from WagonRepositoryConnector.java:601:in `run'
from RunnableErrorForwarder.java:60:in `run'
from ThreadPoolExecutor.java:1145:in `runWorker'
from ThreadPoolExecutor.java:615:in `run'
from Thread.java:745:in `run'
cmd: echo $?
1
That is the behavior from Aether, looks like it expects the name to resolve correctly.
Below is another example of the failure on from a build machine that isn't allowed to talk to the outside world. Any thoughts on how to work around this? Or is this really a case of exposing a means of clearing the remote repos when using lockjar from the command-line and/or a buildr Buildfile?
Something like this may be appropriate:
cmd: lockjar install -v -l my.lock -s default --clear-remote-repos
cmd: lockjar help install
Usage:
lockjar install
Options:
-l, [--lockfile=LOCKFILE] # Path to Jarfile.lock
# Default: Jarfile.lock
-s, [--scopes=one two three] # Scopes to install from Jarfile.lock
# Default: ["default"]
-v, [--verbose], [--no-verbose] # Verbose output
-c, [--clear-remote-repos] # Clears any default repositories like repo1.maven.org respecting repos listed in lock files
I don't have a suggestion for what might be appropriate for the lockjar dsl syntax in a buildr Buildfile though.
cmd: lockjar install -v -l my.lock -s default
Installing Jars from my.lock for ["default"]
Errno::ECONNREFUSED: Connection refused - Connection refused (http://repo1.maven.org:80)
initialize at org/jruby/ext/socket/RubyTCPSocket.java:126
new at org/jruby/RubyIO.java:851
create_socket at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient/session.rb:799
connect at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient/session.rb:747
timeout at org/jruby/ext/timeout/Timeout.java:168
connect at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient/session.rb:746
query at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient/session.rb:612
query at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient/session.rb:164
do_get_block at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient.rb:1191
do_request at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient.rb:974
protect_keep_alive_disconnected at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient.rb:1082
do_request at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient.rb:969
request at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient.rb:822
get at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/httpclient-2.6.0.1/lib/httpclient.rb:713
download_dependencies at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/naether-0.14.0-java/lib/naether/bootstrap.rb:133
each at org/jruby/RubyArray.java:1613
download_dependencies at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/naether-0.14.0-java/lib/naether/bootstrap.rb:119
bootstrap_local_repo at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/naether-0.14.0-java/lib/naether/bootstrap.rb:83
initialize at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/lib/lock_jar/resolver.rb:33
resolver at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/lib/lock_jar/runtime.rb:54
install at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/lib/lock_jar/runtime.rb:69
each at org/jruby/RubyArray.java:1613
install at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/lib/lock_jar/runtime.rb:68
install at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/lib/lock_jar.rb:40
install at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/lib/lock_jar/cli.rb:54
run at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/thor-0.19.1/lib/thor/command.rb:27
invoke_command at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/thor-0.19.1/lib/thor/invocation.rb:126
dispatch at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/thor-0.19.1/lib/thor.rb:359
start at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/thor-0.19.1/lib/thor/base.rb:440
(root) at /opt/jruby-1.7.13/lib/ruby/gems/shared/gems/lock_jar-0.10.5/bin/lockjar:6
load at org/jruby/RubyKernel.java:1081
(root) at /opt/jruby-1.7.13/bin/lockjar:23
I am also finding that clear_remote_repositories doesn't work as expected. Here's some code I hacked together to test it.
lockjar_runtime = LockJar::Runtime.instance
puts lockjar_runtime.resolver.remote_repositories.inspect
lockjar_runtime.resolver.naether.clear_remote_repositories
puts lockjar_runtime.resolver.remote_repositories.inspect
Basically, the remote repository list always contains https://repo1.maven.org/maven2
But now I see how that happens, I am basically getting a new resolver each time due to the default opts on the Runtime#resolver method:
36 def resolver( opts = {} )
37 ......
38 # XXX: Caches the resolver by the options. Passing in nil opts will replay
39 # from the cache. This need to change.
40 ......
41 unless opts.nil?
42 if opts[:local_repo]
43 opts[:local_repo] = File.expand_path(opts[:local_repo])
44 end
45 else
46 if @current_resolver
47 opts = @current_resolver.opts
48 else
49 opts = {}
50 end
51 end
52 ......
53 if @current_resolver.nil? || opts != @current_resolver.opts
54 @current_resolver = LockJar::Resolver.new( opts )
55 end
56 ......
57 @current_resolver
58 end
So the caching of the resolver is pretty ugly, but the alternative if loading Naether for every request is worse.
The lockjar_runtime.resolver will automatically add a local_repo to the config if one is not passed in, otherwise Naether falls down. This is why repeat calls always returns a new instance, the opts are {:local_repo=>"/path/to/.m2/repository"} but the default {} is passed in.
So I think the best way to tackle this is to update the DSL so the default url can be exclude, something alongs the lines of without_default_maven_repo. For this to work, when the Jarfile is locked, all remote repos must be written into the Jarfile.lock. Right now, it is assumed that the default maven repo is always to be used, so it is never written to the Jarfile.lock.
This change will be a non-backwards compatible, since all existing Jarfile.lock are based on the assumption to always use the default maven repo. I would like to keep everything backwards compatible for the time being. How does adding a new supported version to the DSL sound, something like Current Jarfile.lock already has the gem version it was locked at, so that could be used.lock_verison(2)
Seems like adding a way to pivot on version makes sense. But my older lock files actually have the default maven repo in them, so I am confused by your comment. But maybe those entries were added because we also have build.yaml file for buildr that contained the default repo explicitly.
BTW, I found a work-around:
require 'lock_jar/logging'
LockJar::Logging.verbose!
# 'offline' here means don't use the public https://repo1.maven.org/maven2 repo but it ends-up using only the repos provided in gonzo.lock
LockJar::Runtime.instance.install('my.lock', LockJar.read('my.lock').groups.keys, offline: true)
I basically created a custom buildr task to execute the install and when passing offline: true, the default maven repository is not used and only the repositories listed in the my.lock file are used.
Yeah, offline clears all repos when the Naether instance is built - https://github.com/mguymon/lock_jar/blob/allow_removal_of_default_maven_repo/lib/lock_jar/resolver.rb#L39
Since the Naether instance is cached, if you add remote repos afterwards they will be applied. The danger is if the cache is busted, then the repos would be cleared again.
released 0.12.0 with support for without_default_maven_repo
After running "buildr lock_jar:lock" w/ 0.12.0 and the follow Buildfile syntax, I end-up with a my.lock file that no longer has my custom remote repo listed at all. Is this what you would expect?
lock_jar do
without_default_maven_repo
repository 'https://artifacts.foobar.com/artifactory'
...
end
cmd: git df my.lock
diff --git a/my.lock b/my.lock
index ad8a00a..0988f31 100644
--- a/my.lock
+++ b/my.lock
@@ -1,5 +1,5 @@
---
-version: 0.10.5
+version: 0.12.0
groups:
default:
dependencies: []
@@ -818,5 +818,3 @@ groups:
org.powermock:powermock-reflect:jar:1.5.1: {}
- jar:org.mockito:mockito-all:jar:1.9.5:
transitive: {}
-remote_repositories:
-- https://artifacts.foobar.com/artifactory
That is a bug, the remote repo from the Jarfile are used when locking the file, but not being written to the lockfile
@r6p Pushed out 0.12.1 with the fix
Just tried it out. I get multiple redundant repository entries when running buildr lock_jar:lock w/ v0.12.1.
diff --git a/my.lock b/my.lock
index ad8a00a..f92f953 100644
--- a/my.lock
+++ b/my.lock
@@ -1,5 +1,5 @@
---
-version: 0.10.5
+version: 0.12.1
groups:
default:
dependencies: []
@@ -820,3 +820,4 @@ groups:
transitive: {}
remote_repositories:
- https://artifacts.foobar.com/artifactory
+- https://artifacts.foobar.com/artifactory
is the repo 'https://artifacts.foobar.com/artifactory only being defined in the dsl? or is NAETHER_MIRROR being used as well?
If nothing else, it makes sense to make remote repositories stored as a Set.
We have it defined both in the DSL and our build.yaml file (used by buildr when we want to address specific dependencies). When I remove it from the latter, it only appears in the my.lock file once.
cmd: cat build.yaml
gems:
repositories:
remote:
- 'https://artifacts.foobar.com/artifactory'
artifacts:
jruby_complete: 'org.jruby:jruby-complete:jar:1.7.13'
Released 0.12.2 which uses a Set for remote repositories, preventing dups from being written to the lockfile.
I get the same problem with 0.12.2.
diff --git a/my.lock b/my.lock
index ad8a00a..d2edf37 100644
--- a/my.lock
+++ b/my.lock
@@ -1,5 +1,5 @@
---
-version: 0.10.5
+version: 0.12.2
groups:
default:
dependencies: []
@@ -820,3 +820,4 @@ groups:
transitive: {}
remote_repositories:
- https://artifacts.foobar.com/artifactory
+- https://artifacts.foobar.com/artifactory
Do you think the presence of the extra repo definition in my build.yaml caused this?
Try lock jar 0.13.3, looks like the wrong var changed to a Set
That did the trick. Many thanks!
Only took ~22 days to resolve 😌