CI role doesn't work out of the box

Question

CI role doesn't work out of the box

antgel opened this issue 7 years ago · comments

I have an EC2 instance connecting to my new FOSS smaster. I'm using the CI role as an example, and I get:

root@puppetagent:/home/ubuntu# FACTER_role=ci puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Value for key 'psick::profiles::linux_classes', returned from lookup_key function 'eyaml_lookup_key', when using location '/etc/puppetlabs/code/environments/production/hieradata/role/ci.yaml', has wrong type, expects Puppet::LookupValue, got Hash[Enum['gitlab', 'jenkins'], Runtime[ruby, 'Symbol'], 2, 2] on node puppetagent.sandbox.MUNGED.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Any clues?

Alessandro Franceschi · Answer 1 · Sun Oct 22 2017 05:09:00 GMT+0800 (China Standard Time)

The referenced commit fixes the problem. Going to be merged to production branch soon.

Antony Gelberg · Answer 2 · Mon Oct 23 2017 18:02:00 GMT+0800 (China Standard Time)

@alvagante Thanks, I get further, but it still doesn't get me to the end. Errors:

Error while evaluating a Function Call, Could not autoload puppet/parser/functions/artifactory_sha1: no such file to load -- puppet_x/bodeco/util at /etc/puppetlabs/code/environments/production/modules/tp/manifests/conf.pp:248:21 at /etc/puppetlabs/code/environments/production/modules/psick/manifests/postfix/tp.pp:88 on node puppetagent.sandbox.MUNGED.com

Looking at tp.pp line 88, which is ::tp::conf { $k:, I can get it to go further by removing the :: (the same as your commit above), but I'm not sure that's desired. Also, lines 71 and 101 in the same file also have ::tp prefixes, which would also need changing. What do you think?

Alessandro Franceschi · Answer 3 · Mon Oct 23 2017 18:12:28 GMT+0800 (China Standard Time)

@antgel going to give it a deeper look. The ::prefix inside manifests to refer to classes and defines should not harm, (it would work also on hiera data, if values were quoted) and this issue might be related to something else.
BUT, What do you mean that you can get further by removing the :: in line 88? Does the error message change? (Eventually with same message but different line number??)

Alessandro Franceschi · Answer 4 · Mon Oct 23 2017 18:15:24 GMT+0800 (China Standard Time)

@antgel FYI the ci role has not been widely tested on different OS. What do you actually need? Jenkins or GitLab (CI)? ci role has both, and so is somehow pointless.
Also do you have the latest version of tp module?

Antony Gelberg · Answer 5 · Mon Oct 23 2017 18:29:03 GMT+0800 (China Standard Time)

I need to get Jenkins working (which fails due to the E: Package 'openjdk-7-jdk' has no installation candidate and E: Package 'jenkins' has no installation candidate). Using latest tp (2.0.4).

Here's the full log:

root@puppetagent:/home/ubuntu/src/git/psick/environments/production# FACTER_role=ci puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for puppetagent.sandbox.MUNGED.com
Info: Applying configuration version '180361a - Alessandro Franceschi, Sat Oct 21 23:02:04 2017 +0200 : Fix for #198'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-7-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package openjdk-7-jdk is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'openjdk-7-jdk' has no installation candidate
Error: /Stage[main]/Psick::Java/Tp::Install[openjdk-jdk]/Package[openjdk-7-jdk]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-7-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package openjdk-7-jdk is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'openjdk-7-jdk' has no installation candidate
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-7-jre-headless' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package openjdk-7-jre-headless is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'openjdk-7-jre-headless' has no installation candidate
Error: /Stage[main]/Psick::Java/Tp::Install[openjdk-jre]/Package[openjdk-7-jre-headless]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-7-jre-headless' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package openjdk-7-jre-headless is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'openjdk-7-jre-headless' has no installation candidate
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install jenkins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package jenkins is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'jenkins' has no installation candidate
Error: /Stage[main]/Psick::Ci::Jenkins/Tp::Install[jenkins]/Package[jenkins]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install jenkins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package jenkins is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'jenkins' has no installation candidate
Notice: /Stage[main]/Psick::Ci::Jenkins/Tp::Install[jenkins]/Service[jenkins]: Dependency Package[jenkins] has failures: true
Warning: /Stage[main]/Psick::Ci::Jenkins/Tp::Install[jenkins]/Service[jenkins]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/Tp::Install[gitlab-ce]/Tp::Repo[gitlab-ce]/Exec[tp_aptkey_add_14219A96E15E78F4]/returns: executed successfully
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install gitlab-ce' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package gitlab-ce
Error: /Stage[main]/Psick::Gitlab/Tp::Install[gitlab-ce]/Package[gitlab-ce]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install gitlab-ce' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package gitlab-ce
Notice: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/ssl]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/ssl]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/trusted-certs]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/trusted-certs]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/ssl/puppetagent.sandbox.MUNGED.com.crt]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/ssl/puppetagent.sandbox.MUNGED.com.crt]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/ssl/puppetagent.sandbox.MUNGED.com.key]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/ssl/puppetagent.sandbox.MUNGED.com.key]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/trusted-certs/ca_bundle.crt]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/File[/etc/gitlab/trusted-certs/ca_bundle.crt]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/Exec[gitlab-ctl reconfigure]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/Exec[gitlab-ctl reconfigure]: Skipping because of failed dependencies
Notice: /Stage[main]/Psick::Gitlab/Tp::Install[gitlab-ce]/Service[gitlab-runsvdir]: Dependency Package[gitlab-ce] has failures: true
Warning: /Stage[main]/Psick::Gitlab/Tp::Install[gitlab-ce]/Service[gitlab-runsvdir]: Skipping because of failed dependencies
Info: Tp::Install[gitlab-ce]: Unscheduling all events on Tp::Install[gitlab-ce]
Notice: Applied catalog in 1.64 seconds
root@puppetagent:/home/ubuntu/src/git/psick/environments/production#

Alessandro Franceschi · Answer 6 · Mon Oct 23 2017 18:50:14 GMT+0800 (China Standard Time)

Ok, there's some outdated/wrong tinydata for these apps on Ubuntu. Are you testing on Ubuntu 1604?

Alessandro Franceschi · Answer 7 · Mon Oct 23 2017 18:58:50 GMT+0800 (China Standard Time)

@antgel this commit on tinydata example42/tinydata@c54f75c should fix installation issues on Ubuntu 16.04

Antony Gelberg · Answer 8 · Mon Oct 23 2017 19:43:24 GMT+0800 (China Standard Time)

Thanks for the quick response. Any chance of a release of tinydata so I don't have to fork it? NB The commit doesn't deal with the :: issue from earlier in the thread. :)

Alessandro Franceschi · Answer 9 · Mon Oct 23 2017 19:54:47 GMT+0800 (China Standard Time)

Hadn’t that error in my test environment, eventually check if you have the latest tp version ( there’s a recent update on the forge ).
Will publish tinydata later, be aware anyway that tinydata is not expected to maintain backwards compatibility: always test the data of the apps you use before upgrading tinydata

Alessandro Franceschi · Answer 10 · Mon Oct 23 2017 22:18:37 GMT+0800 (China Standard Time)

@antgel I'm not able to reproduce the error you mentioned about:
Error while evaluating a Function Call, Could not autoload puppet/parser/functions/artifactory_sha1: no such file to load -- puppet_x/bodeco/util at /etc/puppetlabs/code/environments/production/modules/tp/manifests/conf.pp:248:21 at /etc/puppetlabs/code/environments/production/modules/psick/manifests/postfix/tp.pp:88 on node puppetagent.sandbox.MUNGED.com

It might be related to the puppet-archive module used in psick, but in order to test it I need to have more info on your environment (what puppet server version are you using, what versions of puppet on client, what client os and so on).
A thing that I'd try is to run puppet on the puppetserver while being sure you are using an environment which contains the puppet-archive module and then restart the puppetserver service.

Antony Gelberg · Answer 11 · Mon Oct 23 2017 22:49:29 GMT+0800 (China Standard Time)

I pinned tinydata by commit ID, so no rush to release. :) Jenkins installs successfully with:

The tinydata patch you mentioned above.
Removal of the :: before the tp.

To make your life easier, I'm going to start again with vanilla servers, and report back.

Antony Gelberg · Answer 12 · Wed Oct 25 2017 01:03:53 GMT+0800 (China Standard Time)

I'm seeing the ::tp error in the postfix manifest on a vanilla Ubuntu 16.04, this time on an instance destined to be a FOSS puppet master, not ci. I guess it's a wide issue.

Alessandro Franceschi · Answer 13 · Wed Oct 25 2017 01:18:23 GMT+0800 (China Standard Time)

This commit example42/puppet-psick@e26c9dc on puppet-psick module should fix this odd issue

Antony Gelberg · Answer 14 · Wed Oct 25 2017 03:00:02 GMT+0800 (China Standard Time)

I would have hoped so, but it's still happening. Sorry if I removing the :: was a red herring. I just tried it with your psick repo, not my fork, both production and integration branches, and the same happened. This is on Ubuntu 16.04.

Relevant extract from Puppetfile:

mod 'example42/psick',
  :git => 'https://github.com/example42/puppet-psick',
  :commit => 'e26c9dc'

Relevant extract from my userdata script:

git_root=~ubuntu/src/git
clone_dir="${git_root}"/psick
env_dir=/etc/puppetlabs/code/environments
prod_dir="${env_dir}"/production
control_repo=https://github.com/example42/psick
sudo -u ubuntu mkdir -p "${git_root}"
sudo -E -u ubuntu git clone -b integration "${control_repo}" "${clone_dir}"
mkdir -p "${env_dir}"
cd "${clone_dir}"
bin/puppet_install.sh
rm -rf "${prod_dir}"
ln -s "${clone_dir}" "${prod_dir}"
bin/puppet_setup.sh auto
apt-get -qy install puppetserver
export FACTER_role=puppetserver
bin/papply.sh

I wish the error message wasn't so cryptic, I'd happily debug it with more of a clue. I think you should be able to reproduce the error with the above.

Alessandro Franceschi · Answer 15 · Wed Oct 25 2017 03:06:30 GMT+0800 (China Standard Time)

Ugh, I wish we will be able to sensibly reduce the amount of commands you need now in userdata, please give me some days to polish the [semi] automated setup of the foss server.
Given it's generally a one time operation, i hope this is not blocking you for continuing your puppet activities.

Antony Gelberg · Answer 16 · Tue Oct 31 2017 18:08:01 GMT+0800 (China Standard Time)

@alvagante Any clues about this one? Can you reproduce it with the above userdata on Ubuntu 16.04?

Alessandro Franceschi · Answer 17 · Tue Oct 31 2017 18:34:31 GMT+0800 (China Standard Time)

@antgel sorry, I don't have time for a quick response.
Consider buying some of ours to have prompt response for bugs and implement/enhance profiles that then go open source (for example a well working, multiple OS tested jenkins profile (and a Jenkinsfile in the control-repo to drive its own CI pipeline) is currently is the todo list).
About the logs of Jenkins, it looks like some prerequisite repos are still not installed (or apt updated) when jenkins installation starts.
For the userdata , if it sets up correctly a puppetmaster it's should be ok.
The error on jenkins server involves a that working puppet master returned a catalog.

Alessandro Franceschi · Answer 18 · Sat Nov 04 2017 22:02:16 GMT+0800 (China Standard Time)

@antgel error

"Error while evaluating a Function Call, Could not autoload puppet/parser/functions/artifactory_sha1: no such file to load -- puppet_x/bodeco/util at /etc/puppetlabs/code/environments/production/modules/tp/manifests/conf.pp:248:21 at /etc/puppetlabs/code/environments/production/modules/psick/manifests/postfix/tp.pp:88 on node puppetagent.sandbox.MUNGED.com"

seems definitively related to voxpupuli/puppet-archive#225 which looks related to https://tickets.puppetlabs.com/browse/SERVER-94.
In my setup when I bump into this error once, I see it going away after a second Puppet run.

Going to evaluate if I find a workaround or if I can easily remove this module from psick, but it can be hard, as it's a dependency of several public modules.

Antony Gelberg · Answer 19 · Sat Nov 04 2017 22:10:38 GMT+0800 (China Standard Time)

I read the links you provided, it's over my head at this stage to understand how they are related to the error. What I don't understand is that the issue is a year old, but a month or so ago, I could install
a Puppet master with PSICK. Possibly before the psick module was split out. Of course a workaround would be preferred.

Alessandro Franceschi · Answer 20 · Sat Nov 04 2017 22:15:29 GMT+0800 (China Standard Time)

@antgel it's related to problems with modules' pluginsync when puppet environment changes.
Anyway I made my tests in local vagrant environment, I probably need to reproduce your setup on aws.

Also the installation of jenkins has been done flawlessly, in local lab environment, so I better double check if local code is all aligned to upstream.

Alessandro Franceschi · Answer 21 · Sat Nov 04 2017 22:20:07 GMT+0800 (China Standard Time)

FYI, file puppet/parser/functions/artifactory_sha1 comes from https://github.com/voxpupuli/puppet-archive/blob/master/lib/puppet/parser/functions/artifactory_sha1.rb

Alessandro Franceschi · Answer 22 · Sun Nov 05 2017 01:45:32 GMT+0800 (China Standard Time)

@antgel several things changed here:

Error "Error while evaluating a Function Call, Could not autoload puppet/parser/functions/artifactory_sha1: no such file to load -- puppet_x/bodeco/util at /etc/puppetlabs/code/environments/production/modules/tp/manifests/conf.pp:248:21 at /etc/puppetlabs/code/environments/production/modules/psick/manifests/postfix/tp.pp:88 on node puppetagent.sandbox.MUNGED.com" is due to the mentioned archive module. Should/could go away after 2 puppet runs in the same environment, but have to test if the same applies for you.
A possible, untested, workaround is to remove the archive module from Puppetfile, but I still have to evaluate the impact on the existing profiles.
Error "E: Package 'openjdk-7-jdk' has no installation candidate" should be fixed by the updated tinydata module (fix was already done, but not pushed to the forge, so latest tinydata module is online"
Note that psick::ci::jenkins has been replaced by psick::jenkins. You can find relevant hiera data here . Latest version of psick module will be pushed to the forge later today.

Have yet to test the whole thing from scratch on aws.

Antony Gelberg · Answer 23 · Tue Nov 07 2017 21:01:40 GMT+0800 (China Standard Time)

@alvagante Not sure if you made any changes, but I just tried with latest production on our fork, and it seems to work! Closing.

Alessandro Franceschi · Answer 24 · Tue Nov 07 2017 21:17:38 GMT+0800 (China Standard Time)

@antgel honestly I'm surprised. I expected the puppet/parser/functions/artifactory_sha1 error to remain at the very first puppet run (or when an environment is changed). Did you test on an existing setup or a brand new one?

Antony Gelberg · Answer 25 · Tue Nov 07 2017 21:43:43 GMT+0800 (China Standard Time)

Brand new, and I've brought up a few masters today. ;) I did rebase our fork's production branch to remove a few of my commits, in light of fixes that you've done upstream. Perhaps there was a gremlin in there. We're now on upstream production, apart from the postgres module fix.