Support for private repositories e.g. Github Enterprise
mikkeloscar opened this issue · comments
I didn't see this addressed in any of the docs, so I'll raise the question
here.
Have you considered support for private repositories?
Many of the existing vendor tools does not handle private repositories very
well or at all. godep and govender can sort of handle it because they get
the dependencies from your $GOPATH. But existing tools that fetches directly
from the remote does not work with private repos in my experience (If someone
knows any tools that does work, please let me know :) ).
It would be great if dep could break this trend and work well with private
repositories from the start.
I'm probably not covering all use cases, but these are the use cases I would
like to see solved.
As a developer (consumer) I want to import dependencies from a private Github
Enterprise repositorymy.ghe.com.
Assuming I can already clone repositories either via https or ssh, I don't
want to also configure the authentication indep. Configuring it one time
should be enough e.g. in git (or ssh).
As a developer I want to avoid committing the vendor folder, thus my CI/CD
system must be able to pull in all dependencies (also those from a private
repo) when running a build.
As a developer I don't want to encode the type of repository in my source
code e.i. I don't want to specify a vcs extension in my import: `import
"my.ghe.com/repo/package**.git**"
From my point of view this could be as simple as introducing an environment
variable which maps your private repo URL to a certain protocol. E.g.:
GO_DEP_REPO=my.ghe.com=git+ssh:my.ghe2.com=git+https
Which means whenever dep encounters an import path which starts with
my.ghe.com it will automatically choose git+ssh when fetching the repo. And
since I already have configured an ssh key on my machine it should know how to
authenticate.
Similarly when it encounters an import path with my.ghe2.com it
will choose git+https and ask me for username+password or just work if I
have configured what token to use in .gitconfig.
I'm sure someone could think of a better name for the environment variable and
a better format, but I hope you get my point.
At my company, we solve this exact problem using go's vanity imports. We have something like https://github.com/dominikh/go-vanity which serves up the appropriate protocol depending on various factors and we use the url of the vanity server instead of our ghe url in our import paths.
This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".
Does that work, at least basically for your needs?
With that said it doesn't exist atm.
First - solving this with vanity imports via an external system, as @zevdg describes, is far easier. (I didn't know about go-vanity - that's awesome.) That said, vanity imports still have some of the same issues as what's described here.
Have you considered support for private repositories?
We didn't talk about it much directly within the committee as it wasn't an immediate necessity, but I've thought about this quite a lot. It's in the top three nasty problems I've grappled with over the last year.
To be clear - these problems are solvable, but they add a ton of new, potentially obnoxious failure modes, so I've been very cautious about adding support for them in gps. I've also been thinking about them for way too long, and have gotten somewhat lost in the forest - help with weighing tradeoffs is very welcome :)
This is kind of a slapdash writeup, but hopefully it covers the basic bases.
The essential problem is portability: we have to be able to deduce the source/repo for all import paths we encounter. If that's only possible by relying on custom rules, then anyone trying to work with your project without those custom rules won't be able to resolve those import paths. The really nasty part, though, is that we're unable to provide users in that situation with useful error messages, because it's impossible to tell the difference between "invalid import path" and "invalid import path because you're missing some custom deduction rules." How could we know such custom rules exist?
If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?
We could mitigate this by having the custom rules live in the manifest, alongside constraints/ignores/requires/overrides. composer does this, generally to good effect. But, if you're in a company with a bunch of different projects, then each would need to define the same rules in their manifests. (Pattern definition would have to be a root-manifest-only thing, for basically the same reasons as composer gives). If you have to make any changes in how those rules work, you have to make them in all your projects, and it could potentially make it impossible to use older versions of those projects, because your new rules don't work with the old imports.
But all this works well enough in composer, right? Sure. Things are strictly more difficult for Go, though, because this problem of deducing projects/repositories from import paths is one that composer simply doesn't have. Composer is just dealing with the name of a dep and a location from which to retrieve it and its metadata. Because dep tries hard to be truly idiomatic, we still treat the import graph, not the manifest, as the canonical expression of what code is required (c.f. writeup; we follow option 2). So, rather than just naming a dep and where to get it from, we have to specify a pattern that can be used to match import paths that we encounter and map them back to their repository root.
Plus, composer constrains names to have two parts, but we have no such guarantee - launchpad.net/someproject has two, but github.com/golang/dep has three. Variable-length root names make it impossible to designate a general rule that, say, "all custom rules must have two path parts," which would make ensuring that we don't have any conflicting rules much easier.
Also, as one final kick in the pants, because we're ultimately talking about a system to decide what names (import paths) mean, it would have to be incorporated into the input hashing system we use to determine when you need to regenerate your lock. (That's the memo field in your lock.json - this is a very experimental idea that we may drop)
Again, I do think there are reasonable solutions here. But I have yet to figure out a good, balanced compromise that ratchets down control to avoid rendering the entire system unsafe; I've been enmeshed in this problem for a long time, and have lost a lot of the perspective necessary to weigh tradeoffs. Feedback is super welcome :)
Oh! I forgot to mention...
All of this ugliness is my own single strongest motivation for creating a registry (#175): we can impose meta-rules on a subset of import paths. For example, imagine this pattern:
import "p/gopherhoard.com/user/project"- The leading
pindicates it's a registry source type - The second part is the base domain where the registry is hosted. This would allow the crucial feature of allowing people to host their own private (or public) registries; we might optionally have a shorter alias for the main, public registry.
- There's a fixed number of trailing elements. (We might also just do project name, without namespacing by user).
This solves so many problems.
- There's no more "unknown rule" problem, because there's a meta-pattern followed by all registries
- Root deduction is dead-simple
- The accuracy of input hashing is no longer at risk
- ...all of the other problems that registries solve, independent of this
Thanks for the background! Glad I'm not the only person thinking about this.
The problem that vanity imports don't solve for us is having a centralized cache of 3rd party libraries. We use artifactory to do this from nearly every other language, but since go uses the import path to uniquely identify both the package and its source url, the existing go tool simply cannot handle changing the source url without also changing the package's identifier. We could re-write import paths to point to artifactory, but we'd have to also do that inside of every 3rd party package we use as well so that we don't end up referring to the same package with by different names.
The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency. Something like
{
"dependencies": {
"github.com/golang/protobuf": {
"branch": "master",
"source": "artifactory.corp-intranet.com/github.com/golang/protobuf"
},
"github.com/pkg/errors": {
"branch": "master",
"source": "artifactory.corp-intranet.com/github.com/pkg/errors"
},
"ghe.corp-intranet.com/myteam/lib": {
"branch": "master"
}
}
}So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.
That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency. So on some level, I do want a "custom rule" of some sort, but more generally, I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile. This hook would have the opportunity to also add more updates to the manifest and if it did, dep would re-read and use the new info in generating the lock file. Conceptually, this would be similar to setting up your editor to run goimports followed by go build on save.
In particular, the hook I'd personally want would be along the lines of (in psedo code)
for each package p:
if p is 3rd party:
p.source = "artifactory.corp-intranet.com/"+p.importpath
So that on a properly configured machine, dep ensure github.com/golang/protobuf would inject my alternate source but dep ensure ghe.corp-intranet.com/myteam/lib would not. On an unconfigured machine, everything would still work fine, but one would have to manually specify alternate sources for any new dependencies they add.
I imagine there would be other legit use cases for having an automated edit to the manifest here. I haven't thought too deeply about the implication of allowing a hook like this, but since the manifest is supposed to be editable by humans, never mind other tools, this doesn't seem too dangerous at first glance.
Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?
+1
The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency.
So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.
Yep yep!
That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency.
Oh for sure - but I'll go two steps stronger.
Step 1: it's not just that it requires discipline for devs. It's that you effectively leave to individual developers what may, to meet the workflow/legal/whatever reqs of your organization, need to be questions of policy.
Step 2: when you explicitly record the source values in manifest.json, you're creating a new vector for conflicts - all of your projects relying on github.com/golang/protobuf, for example, need to agree that they are sourced from artifactory.corp-intranet.com/github.com/golang/protobuf. If any don't agree, it's considered a conflict by the solver - not a version constraint conflict, but a source conflict - because how could it not be? We can't source one thing from two different places. (I described a little more how this class of conflict works in #191 (comment) - the mechanism is the same there)
This creates a potentially nasty situation: let's say your company gets acquired, and the artifactory base URL has to change from artifactory.corp-intranet.com to artifactory.newcorp-intranet.com, because 🐙. All of your projects will have to independently cut over to the new artifactory URL at the same time, roll new releases with that change, and none of them will be able to work with older versions. That's exactly the kind of high-risk change scenario that we should all be seeking to avoid.
(The source property is rife with possible outcomes like this. That's why I don't really like it that much as-is, and would ideally like to see it replaced with a few, more narrowly-defined properties that accomplish the same sort of goals.)
I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile.
Yes, I've been imagining something like this since some folks described a similar need in Masterminds/glide#372. A script would probably work well; my plan was that it would take either a project root or a URL (so, github.com/golang/protobuf or https://github.com/golang/protobuf) and transform them into the URL(s) that may be used to interact with the source.
If we went that route, it would be transparent to the manifest, and the onus would be on the implementor of the script to ensure that the mirror is properly reflecting the data from upstream. Failure to do that would compromise the portability of the lock outside of the alternate sourcing universe imposed by the script...which is maybe fine in some cases, but has its risks :)
Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?
Sure - my thinking on these topics is all pretty jumbled up together, so if you can see a way to tease out more specific issues, please do!
Just bumping this to see if you've had any more thoughts on separating out some issues, @zevdg 😄
I'm glad I'm not the only one who have thought about these things :)
At my company, we solve this exact problem using go's vanity imports.
I had this thought maybe 8 months ago but went away from it for reasons I can't quite remember. I think I imagined it would be inconvenient to use a different import url. But looking at it now I see more benefits than trade-offs.
This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".
One minor thing that this doesn't solve as far as I can tell is when you want to vendor for the first time with dep init, then you would have to run dep ensure <package> for all the dependencies instead, which is a bit annoying if you already defined them as imports in your code.
If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?
I can understand wanting to prevent users from misusing a feature like this, but if this would be the only way to get dependencies from gitlab, then it would be better than dep not working with gitlab at all IMO (assuming that not having a built-in rule == it doesn't currently work).
Ok, so my main concerns around rewrite rules for alternate sources seem to be much less pressing due to the paradigm shift that seems to be happening around #213 . If the manifests will not be edited by dep, then we can just make our own manifest generation tooling that writes to the manifest and not have to worry about conflicting with or hooking into dep. If implemented, #221 could help ensure this tooling is installed correctly. The UX could be improved by making some sort of source rewrite rules a first class citizen in dep, but it's a less pressing issue now and it seems like a contentious feature that could be added in a later version of dep. I'll open a ticket for reference and discussion around this issue.
The other main issue is authentication. Even though github has the right go-import meta tags, they are not accessible for any repository (public or private) in my company's GHE necessitating our vanity import server. If I could configure dep with a github access token or with a git-like credential manager, that wouldn't be necessary. To be clear though, AFAIK, this isn't just an issue for GHE, but also for private repositories on regular old github.com We could continue to use this issue for that, but it's now pretty cluttered with other stuff so I'll open a separate issue for that as well.
Giving dep v0.1.0 a try with an existing work project. Since the project is hosted on GitHub Enterprise, I knew it wasn't going to work perfectly given this issue and #286.
When running dep init, I see the following error:
sm.DeduceProjectRoot: unable to deduce repository and source type for
"github.myorg.com/myorg/myrepo": unable to read metadata: go-import metadata not found
In this case myrepo is a reusable package I wrote.
There's nothing tricky about it. This is the remote for that repository:
git remote show -n origin
* remote origin
Fetch URL: https://github.myorg.com/myorg/myrepo.git
There is a 1:1 mapping here -- it's just https://{import path}.git. However, I recognize it's not one of the "blessed" locations and authentication could be "interesting" as well.
Our previous tool was gvt. It doesn't support private repos either.
However, I was able to work around that issue by manually dumping files in vendor and not letting gvt know about them. I don't know that there is any temporary workaround with dep? In fact, dep init didn't generate the starting manifest.
Just noticed I wasn't completely honest in my last comment. There is something a bit tricky.
import "github.myorg.com/myorg/myrepo/mypacakge"
->
https://github.myorg.com/myorg/myrepo.git/mypackage
That could explain it. Of course there's no way of really knowing where the .git repository ends and the paths within it begin.
If I change my import paths to:
import "github.myorg.com/myorg/myrepo.git/mypacakge"
I saw the following message with dep init (dep v0.1.0 0e8607e)
https://github.myorg.com/myorg/myrepo.git does not exist in the local cache and fetching failed: context canceled
Unable to cache github.myorg.com/myorg/myrepo.git
main.getProjectData.func1
/Users/nathany/src/github.com/golang/dep/cmd/dep/init.go:351
runtime.goexit
/Users/nathany/src/go.googlesource.com/go/src/runtime/asm_amd64.s:2197
But then the next time I ran dep init it worked! 👍
Not a big fan of having to put .git into my import paths though. But to get around that, dep would need to know or be told that github.myorg.com/myorg/myrepo.git is the repo so that import paths below that work. And be told in such a way that dep init works, before there is a manifest file.
Looking at private mirroring of our dependencies currently.
So we have the ability to fetch a dependency from a different location via
dep ensure github.com/pkg/foo:git.internal.com/alt/foo
This seems mostly sufficient, just mirror the repositories you want to host internally and I'm assuming everything will work out fine. I have question with this workflow though regarding transitive dependencies.
Let's say we want to add go.uber.org/zap to our manifest.
dep ensure go.uber.org/zap pulls in the transitive dependency github.com/davecgh/go-spew with it and writes it to the lock file.
Let's say we know we are going to pull this dependency in and we mirror it internally.
So what should happen if we do dep ensure go.uber.org/zap:git.internal.com/alt/zap.
- Should transitive dependencies also be pulled from the alternative location?
- What is a good workflow for discovering said transitive dependencies to make sure they also get mirrored?
- I suppose you could do a dry run of the ensure step to inspect the resulting lock file to see the transitive dependencies. Then go and make sure you mirror the dependency you care about and all of it's dependencies.
Any thoughts on what this workflow ought to look like?
Good questions.
Though I'm starting to wonder if we could split up this issue? Personally I'd just like a way to support GitHub Enterprise for dependencies written within the organization.
Transitively mirroring third-party deps is absolutely an important feature, but not one I need where I'm at.
I just don't want one to slow down the implementation of the other... and it becomes a bit difficult to follow the conversation when the issue covers to many different topics.
@nathany I'm happy to move/split however maintainers see fit. The question seemed to fit under "private repository" usage patterns to me.
As an update though on actually trying this out:
dep ensure golang.org/x/sys:git.internal.com/external/golang.org/x/sys
Did not quite work for me, I hit the following
$ dep ensure golang.org/x/sys:http://git.internal.org/external/golang.org/x/sys
ensure Solve(): unable to deduce repository and source type for "http://git.internal.org/external/golang.org/x/sys": unable to read metadata: go-import metadata not found
$ dep ensure golang.org/x/sys:http://git.internal.org/external/golang.org/x/sys.git
ensure Solve(): No versions of golang.org/x/sys met constraints:
master: Could not introduce golang.org/x/sys (from http://git.internal.org/external/golang.org/x/sys.git)@master, as it is not allowed by constraint * from project demo.
Appears I would need to setup a page to serve the correct metadata?
Modifying Gopkg.toml directly did seem to work though:
[[constraint]]
branch = "master"
name = "golang.org/x/sys"
+ source = "http://git.internal.org/external/golang.org/x/sys.git"
dep ensure -v shows it was pulled from the internal mirrored repository.
As I suspected though transitive dependencies continue to be pulled from their original location which is not what we (or presumably other organizations) would want when trying to vendor all third party dependencies. @sdboyer I am inclined to agree that perhaps this part of the issue ought to be split but I'm not particularly sure how you want it organized (or rather what even the right question ought to be).
@aajtodd Won't an [[override]] allow you to do that?
[[override]]
branch = "master"
name = "golang.org/x/sys"
source = "http://git.internal.org/external/golang.org/x/sys.git"This should ensure that golang.org/x/sys (regardless if it's direct or transitive dep) uses http://git.internal.org/external/golang.org/x/sys.git as its source.
Also, the error you are getting is to be expected since dep cannot tell where the project repository exist since the URL is not serving the required go-import metadata:
<meta name="go-import" content="golang.org/x/sys git http://git.internal.org/external/golang.org/x/sys.git">@sdboyer It seems we have more cases where people might not be able to run dep init in a current project (when it's not possible to solve with default params). Most cases require creating the manifest and filling in the required constraints or overrides. How should we deal with this?
Won't an [[override]] allow you to do that?
I don't know haven't tried it, I will attempt to it sometime this week (admittedly golang.org/x/sys was a bad example as it has no transitive dependencies). The documentation on the manifest format and what constraints vs override behavior was not 100% clear to me.
It seems we have more cases where people might not be able to run dep init in a current project (when it's not possible to solve with default params)
I do find init behavior slightly over presumptuous. Perhaps dep init should just create the empty manifest and allow you to fill it in (this is how npm and cargo work for JS/Rust respectively). dep ensure would create the lock file if it is not present. I'm not too worried about it though, it isn't difficult to modify the manifest after the fact (although documentation on this type of workflow will certainly help).
I'm under the assumption that dep ensure golang.org/x/sys:git.internal.com/external/golang.org/x/sys.git should work (note the .git at the end) even if no go-get metadata is available. Could you try that?
what constraints vs override behavior was not 100% clear to me.
Basically, [[constraint]] is used to constraint how dep determine which version to use for one of your direct dependencies. While [[override]] is used to override the constraints defined downstream for one of your transitive dependencies.
I've just started trying out dep in an internal project that is currently using glide. Similar to glide, I really can't use the "init" command. It causes issues between the internal gitlab server deps and the external deps in combination with auth/proxies/etc. So I accepted that and just wrote an equivalent Gopkg.toml for dep. But now I am finding that the ensure command wants to hit the "?go-get=1" http endpoints for the internal gitlab projects, even though I have specified the "source" for all of these. Why is that? I was hoping to answer the question for the tool, but now I seem to be stuck between getting it to work for the external deps vs the internal deps, but never the both.
Is there a way to specify enough information for the internal dependencies that can get this working?
[[constraint]]
name = "golang.org/x/crypto"
branch = "master"
source = "https://github.com/golang/crypto.git"
#source = "https://github.com/golang/crypto.git"
#source = "https://github.com/myfork/package.git"
#source = "https://github.com/golang/crypto"
#source = "github.com/golang/crypto"
[[override]]
branch = "master"
name = "golang.org/x/crypto"
source = "https://github.com/golang/crypto.git"
#source = "https://github.com/golang/crypto.git"
error:
master: unable to deduce repository and source type for "golang.org/x/crypto/ocsp": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/crypto/ocsp?go-get=1": Get http://golang.org/x/crypto/ocsp?go-get=1: dial tcp 216.239.37.1:80: i/o timeout
who can help me ?
thank you .
You sure it's the correct place for that question?
@freeformz , what about my question? I have asked that question not only here, but on golang-nuts and the Go Package Management forum, with no replies. I currently can't use the dep tool at work.
This may not be an option for everyone, but we just use SSH keys to authenticate, and utilize git url rewriting:
git config --global url.git://github.com/.insteadOf https://github.com/
Then dep and glide both work fine. You have to have your ssh key set up locally and on your git repo, but it's working fine atm.
@wadelee1986 not relevant to this issue - please make a new issue.
@justinfx an explanation for why adding source is not sufficient is here: #860
@morganhein but isn't git:// less secure?
Not sure about git:// but SSH to GitHub is pretty secure.
A comparison here: https://gist.github.com/grawity/4392747
If using github.myenterprise.com, it's not entirely clear to me how Git URL rewriting would satisfy dep, go get, or other tools in the Go ecosystem.
Has anyone got this to work, and have an example to share?
https://blog.devzero.com/2014/08/29/useful-git-commands-url-rewriting/
@nathany The problem is most likely that your github enterprise installation doesn't expose the url https://github.myenterprise.com/org/repo?go-get=1 without authentication, so when go get or dep calls this url it will fail and not know how to fetch the repository. This is at least the problem we have with our Github Enterprise setup.
In this case go get will go to the HTTP url before calling git and thus your rewrite rule will have no effect since git is never executed.
It would be great if Github Enterprise could just expose the go metadata without protecting it behind an auth page since there are no secrets in the metadata.
@nathany As for other tools, Glide works correctly in this situation. You just point your dependency at any git repository and it will do the right thing, no matter what package path you use in your source code.
@mikkeloscar or if dep would just do the ssh dance like Glide does.
It would be nice if dep did the ssh dance 💃 🕺 @dsvensson mentioned, or if GitHub implemented what @mikkeloscar suggests.
I'd rather not move work projects from gvt to glide to dep -- and I don't imagine my co-workers would be super happy about switching tools multiple times either.
Plus I'd like to get more experience using dep in "real world" scenarios.
this is really just about #860 - if we were to trust the source declaration fully, it would obviate the need for the HTTP go-get request.
@nathany @mikkeloscar git URL rewriting should work fine with go get and dep to clone private repos. I've used the below for the past couple years with no issues. It seems like you're actually running into http://grokbase.com/t/gg/golang-nuts/13c5jx3g79/go-nuts-problem-with-go-get-with-github-enterprise-repo perhaps you can further rewrite the URL to append the .git back on.
git config --global url."git@github.com:".insteadOf "https://github.com/"
This works because go get does use git/hg to clone repositories.
Ref: https://golang.org/doc/articles/go_command.html#tmp_3
It works for github.com because it's hardcoded in go get (https://github.com/golang/go/blob/9a13f8e11ce487fccef071303164b3d963e6ede6/src/cmd/go/internal/get/vcs.go#L956), it does not work for custom GH enterprise installations because go get goes to the meta-data HTTP endpoint before it calls git, and this fails when the GHE is protected by some login page.
Ha, I just noticed this myself as you posted. Here's the link in the tools repo fwiw.
https://github.com/golang/tools/blob/master/go/vcs/vcs.go#L623
Further discussion here https://groups.google.com/forum/m/#!msg/golang-nuts/AURCoVLjNyc/2Uw7A_-LRfQJ seems to imply that suffixing your imports with .git should work. The first go nuts forum I posted seemed to indicate this worked with git ls-remote
Pretty sure I use rewriting with GitLab instances which Id expect to have the same problem. Beginning to doubt that now though.
to clarify - while the go get behavior is largely equivalent to dep's in this specific regard, go get code should not be attributed explanatory power, as dep has its own implementation and never calls go get.
Thanks for clarifying, I misunderstood your previous comment and did get that impression. Having now also read 830.
if we were to trust the source declaration fully, it would obviate the need for the HTTP go-get request.
This seems like the way to do it, the "ping" seems to create more problems while just letting the vcs error on a bad url seems manageable.
I've tried many workarounds but non of them worked.
I'm trying to dep ensure a package from a private gitlab repo but it's asking for username and then just gets stuck after I hit enter.
$ dep ensure
Username for 'https://git.mygitlab.com:443':
Git clone and go get work as expected.
$ dep version
dep:
version : v0.4.1
build date : 2018-01-24
git hash : 37d9ea0a
go version : go1.9.1
go compiler : gc
platform : linux/amd64
I believe I have a solution for private repos with ssh access, see PR #1717
#1759 - this is a quick sketch of a local registry implementation that doesn't require the editing of any import paths. Let me know if this is useful.
This is the workaround in our team:
-
Make Dep fetch dependencies via SSH:
$ git config --global url."git@github.mycompany.com:".insteadOf "https://github.mycompany.com/"
-
Add the
.gitsuffix for every related project in code:package xxx import ( "github.mycompany.com/chehsunliu/example.git/pkg/calc" ) func DoSomething(a, b int) int { return calc.Add(a, b) }
If one project is likely to be imported, clone it with the .gitsuffix from the beginning.
$ git clone git@github.mycompany.com:chehsunliu/example.git $GOPATH/src/github.mycompany.com/chehsunliu/example.gitOtherwise, it would be quite painful to rename every import statement in the project.
A little sidenote for clarifcation / addition:
meaning of .git and where it is mandatory
you can either go the way of @chehsunliu which has one good point to it, you do not longer need to define constraints manually in this manner:
[[constraint]]
name=code.mycompany.com/whatever/myproject
source=ssh://git@mycompany.com/whatever/myproject.git
So the important one here is, you do not need .git in the package name, you need it in the source. Since @chehsunliu is already importing using .git, this constrain to "remap" name to source is no longer needed.
This said, you could also import the package without .git and do the constraint above .. but this would need a constrain per internal project to remap you non .git import to a .git based repo source
global git url rewrite
You do not need the global configuration @chehsunliu was proposing unless you want to go with "auto imports without constraints", so if you want to skip the constraint definition and want dep to auto-resolve it, you have to import with .git and also remap the non-scheme based import to a ssh based source, since by default, dep will use https://<yourimport .. so if you want to use ssh whatever, you remap https://mycompany.com to git@mycompany.com
You can fix a lot more with the git url rewrites .. even paths and path remapping if the anchor of your repos e.g. is /scm/whatever/project you could remap that https://mycompany.com/whatever/project . from the import will be sourced from git@mycompany/scm/whatever/project and so on. This helps for go projects where you remap / need to fix it / have legacy or when your SCM server has odd paths and you start to golang :) thats why i mention it
git urls
git@mycompany.com my be not enough when you run a non default ssh port, in this case you do
ssh://gitolite@mycompany.com:3302 - this would be needed in the global rewrite or in the constraint - whatever you stick with
With a combination of some of the above answers, I ended using docker to get dependencies and build. Hopefully this workaround will help someone:
# build image
FROM golang:latest AS build-env
ARG GH_TOKEN
ENV GH_TOKEN ${GH_TOKEN:-""}
WORKDIR /go/src/example.company.com/org/some-app
COPY . .
RUN apt-get update
ADD ./company-root.crt /usr/local/share/ca-certificates/
ADD ./company-intermediate.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates
RUN apt-get clean
RUN git config --global url."https://${GH_TOKEN}:@example.company.com/".insteadOf "https://example.company.com/"
RUN go get -u github.com/golang/dep/cmd/dep
RUN dep init || dep ensure
# Set build targets in a Makefile
# example:
# build:
# GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -a -o ./bin/linux/some-app-x86-x64 *.go; \
# GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w" -a -o ./bin/darwin/some-app-x86-x64 *.go;
#
# First build
# docker build --build-arg GH_TOKEN="${GH_TOKEN}" -f ./Dockerfile -t domain/some-app:latest .
#
# Then run with local mounted volume to get the build binaries, dependency vendoring, etc.
# docker run -v $(shell pwd):/go/src/example.company.com/org/some-app -it --rm domain/some-app:latest
CMD ["make", "ghtoken=${GH_TOKEN}", "-s", "build"]
I entirely gave up on dep in this case, moved to go-modules. You need a similar gitconfig alias as before, but other then that, it works without any issues where dep was entirely failing.
gitconfig:
[url "yourgitsshalias:"]
71 insteadOf = https://yourreposystem.tld/scm
Where yourgitsshalias is a ~/.ssh/config entry like
Host yourgitsshalias
hostname yourreposystem.tld
user git
port 22
Where it then uses your ssh id_rsa key to connect. This way go modules are able to fetch the acual repos of every single git repo implementations, since they all support ssh keys authentication and it is fairly secure.
On top of that, it is just less hassle, no dep ensure - just go build and that it is.