Hiera backend seems broken in puppet 4.10
edestecd opened this issue · comments
Were building a new puppet 4 environment and I can not seem to get the Hiera backend to work at all. I'm running the simplest of examples from the README. This used to work OK in puppet 3.x.
Our new setup is using puppet 4.10.1, puppetserver 2.7.2, puppetdb 4.4.0.
I have also downgraded to puppet 4.8.x to see if that was the issue with no luck.
The query always returns empty array :(
The cli and puppet functions are working just fine.
I have tried 2.2.0 and 2.3.0 of this module with no success. Still pounding away!
I now believe the issue is related to the changes in hiera 3.x.
This backend is being placed in a legacy mode and I suspect that to be the issue:
https://github.com/puppetlabs/hiera/blob/3.3.1/lib/hiera/backend.rb#L11
I am now updating the backend for the new method signatures....
If that pans out you will have a PR soon ;)
No luck with that approach.
I'm somewhat stumped at the moment.
This line seems to always return nil:
https://github.com/dalen/puppet-puppetdbquery/blob/master/lib/hiera/backend/puppetdb_backend.rb#L33
So it never finds the ::_nodequery variant in any other backends. (Just using yaml here)
OK I think I finally figured it out after placing some debug statements in hiera 3 gem code.
This is failing due to the yaml_backend not being loaded.
https://github.com/puppetlabs/hiera/blob/3.3.1/lib/hiera/backend.rb#L269
In puppet 4.8 and 4.9 Hiera 4 is being used. In puppet 4.10 Hiera 5 is being used.
Both of which are now merged into puppet and not a separate project:
https://docs.puppet.com/puppet/4.10/hiera_intro.html#whats-the-deal-with-hiera-5
Backward compatibility is provided for custom Hiera 3 backends, but it is failing for us.
This means the code in hiera 3 is only loaded for our custom hiera 3 backend and core puppet is doing the yaml lookups and merging the results.
B/c of this the yam_backend in the Heria 3 gem is not being loaded and it is skipped in this lookup:
https://github.com/dalen/puppet-puppetdbquery/blob/master/lib/hiera/backend/puppetdb_backend.rb#L33
The solution:
Possible quick and dirty: Figure out how to make the hiera 3 puppetdb backend load the yaml_backend for our purposes.
Long Term: rewrite the hiera backed in Hiera 5, which is actually just another puppet function in the modern ruby api
I'll have some more time to hack on this tomorrow. Hopefully I can get a solution in place. We have too many of these lookups in our code base to get rid of them easily....
What is the status of this pull request or puppetdbquery and Hiera 5 in general?
I have the code in the PR running in our production environment with puppet 4.10.4.
It looks good from my end, but I don't know if @dalen has had a chance to review it yet.
@edestecd thx, Our puppetserver is running on version 2.7.2 (sigh, puppet and version numbering) and still working with the old backend, but complaining about the deprecated hiera.yaml format. So we will wait until your PR (or something else) got merged and released
@Mosibi what version of puppet and puppetdb are you on? I'm curious how the old backend is working for you? I could not get it to work in puppet 4.8 or 4.10
puppetdb server:
puppet-agent 1.10.1-1trusty
puppetdb 4.0.0-1puppetlabs1
puppetserver(s):
puppet-agent 1.10.1-1trusty
puppetdb-termini 4.0.0-1puppetlabs1
puppetserver 2.7.2-1puppetlabs1
puppetdbquery version 2.1.1
What backends do your queries come from? yaml? json?
That seems consistent with our environment, although we are on CentOS instead of ubuntu
Our current hiera.yaml (v3 format)
---
:hierarchy:
- environments/%{environment}/nodes/%{::fqdn}
- environments/%{environment}/roles/%{::role}
- environments/%{environment}/common
- roles/%{::role}
- common
:backends:
- puppetdb
- yaml
:merge_behavior: deeper
:yaml:
:datadir: '/etc/puppetlabs/code/hieradata/'
Ours is much more complicated, but I reduced it to this while troubleshooting and still no luck:
:backends:
- yaml
- puppetdb
:yaml:
:datadir: "/etc/puppetlabs/code/environments/%{::environment}/hieradata"
:hierarchy:
- common
:merge_behavior: deeperIt is possible we have something else going on in our environment that you do not...
At any rate, there exists some case that the legacy backend fails to work in puppet 4.10.
We are foss with Foreman, so not sure if the ENC could have anything to do with it.