install app via git?
tod-uma opened this issue · comments
Would there be any interest in having the ability to install an app from a git repository? We're currently debating how we want to track changes to the apps we use and one option is to store our apps in separate repos in our gitlab installation. There are ansible modules to checkout and update git repos. I should be able to make a pull request that detects when an app in the app list is a git repo and to use that instead of the url module. Any thoughts?
I can't speak to how the rest of the community approaches versioning/packaging apps. This doesn't seem like a bad idea though, but my only worry is there would be a lot more complexity with installing openssh-client, mounting ssh-keys, and possibly mounting known_hosts in each container?
Wondering if gitlab supports some artifact storage, or you're using some other solution for that? If you're separating each app into a separate repo, you can build a workflow to test + publish the final tar-ball to some central location (ex: https://artifactory.internal.com/apps/app-name/master/app-name.tgz). Then your Splunk deployment files could theoretically reference a static URL when creating your topology.
Alternatively you may even want to publish branch-builds of apps (ex: https://artifactory.internal.com/apps/app-name/bugfix-01/app-name.tgz) that can be verified in a Splunk deployment by changing the app URL. That's more-or-less the approach we take when developing apps :)
Well, in our environment we're not using containers much yet. I'm actively working on switching from a custom built set of ansible scripts that deploy the forwarder to these. Git lab can indeed provide a zip or tarball of a branch/tag. That was going to be the route I go if I don't go the git route.
I don't think you'd need ssh on a machine as you can use http to clone a repo just like on github, assuming the permissions are set to allow it. I also wouldn't need ssh keys as gitlab has deploy tokens you can create to accommodate situations like these.
The biggest challenge I see is the app variable(s). I'd want a way to specify a branch/tag and I'm not sure how I'd do that in the current list of strings (without having to parse the string). I was thinking of just making a new variable like apps_from_scm that could be a list of dicts that works like requirements.yml.
Gotcha, I'm mostly concerned that if we advertise we support apps installed from git, the expectation would be all forms of git - whether it be over http/s or ssh. I'd much rather just support http/s only since it comes with way less baggage :P
But will keep this issue open for anyone else to chime.
We could put in the documentation that http(s) should work "out of the box" but other transport methods may require additional libraries on the target hosts.
So, I attempted to use theses scripts to download a tarball from our gitlab and ran into a problem. Since we don't have "public" repositories in our gitlab, we can't download tarballs without a username and password. The gitlab deploy tokens don't work for the tarballs. This is a failing in gitlab and there are open feature requests to change this.
In the meantime, I will probably just write my own separate ansible role to use the git module. If there's interest in merging it into these roles, I can make a pull request to do it.
Hi @tod-uma ! You mentioned that you are not really using containers in your environment. In that case, you should take a look at https://github.com/splunk/ansible-role-for-splunk instead as this supports app deployments from git repositories and was built for non-containerized Splunk environments.
Check out ksconf at https://github.com/Kintyre/ksconf
ksconf provides commands for dealing with splunk conf files specifically targeted towards using git for splunk app deployment.
I am currently using ansible (based on https://github.com/splunk/splunk-ansible rather than https://github.com/splunk/ansible-role-for-splunk ) to deploy apps from git to splunk. I can change code in gitlab, pull the change to our ansible server, start the deployment script which will push the app to either the deployment server or searchhead deployer. Any secrets needed in conf files are taken from the ansible vault.
I plan to integrate ksconf as pre-commit hook to git so it will prevent any commit that has for example syntax errors and such.