In hiera backend, add ability to use "deep" _nodequery
vincentbernat opened this issue · comments
Hey!
It would be convenient if the hiera backend was able to substitute a node query when returning a structure from hiera. For example, if we have this:
something:
sources::_nodequery: "Class[...]"It would be convient if the hiera backend was able to return:
{ "something" => { "sources" => [ "host1", "host2" ] } }
+1
I am happy to give it a try but I would need a couple guidelines before I start writing this...
It looks like this can be done by editing the lookup function in puppetdb_backend.rb. I am not sure how to make sure this will retrun the whole hash and run the puppetdb query only for the inner deep key which contain the "_noquery" ...
"profile::elasticsearch::config":
bootstrap.mlockall: true
cluster:
name: usersearch
discovery:
zen.minimum_master_nodes: 2
zen.ping.multicast.enabled: false
zen.ping.unicast.hosts::_nodequery: ['ec2_tag_role=usmaster', 'ipaddress']
index:
search.slowlog.threshold.fetch.debug: 500ms
search.slowlog.threshold.fetch.info: 800ms
search.slowlog.threshold.fetch.trace: 200ms
search.slowlog.threshold.fetch.warn: 1s
search.slowlog.threshold.query.debug: 2s
search.slowlog.threshold.query.info: 5s
search.slowlog.threshold.query.trace: 500ms
search.slowlog.threshold.query.warn: 10s
node:
data: false
master: true
name: "%{::hostname}"
Actually we don't need this feature because we can just interpolate the hiera function and let the magic happen:
"profile::elasticsearch::masternodes::_nodequery": ['ec2_tag_role=usmaster', 'ipaddress']
"profile::elasticsearch::config":
bootstrap.mlockall: true
cluster:
name: usersearch
discovery:
zen.minimum_master_nodes: 2
zen.ping.multicast.enabled: false
zen.ping.unicast.hosts: "%{hiera('profile::elasticsearch::masternodes')}"
index:
search.slowlog.threshold.fetch.debug: 500ms
search.slowlog.threshold.fetch.info: 800ms
search.slowlog.threshold.fetch.trace: 200ms
search.slowlog.threshold.fetch.warn: 1s
search.slowlog.threshold.query.debug: 2s
search.slowlog.threshold.query.info: 5s
search.slowlog.threshold.query.trace: 500ms
search.slowlog.threshold.query.warn: 10s
node:
data: true
master: false
name: "%{::hostname}"
Amazing!
Unfortunately, this only works with Hiera 2+ as otherwise, everything hiera can fetch is a string. Right?
I see this behavior too. With my example above I end up with:
hosts: 10.200.31.45610.200.40.12210.200.60.39
but base on the doc (https://docs.puppet.com/hiera/1/puppet.html#hiera-lookup-functions) the hiera function returns arrays and hashes as well...
So it looks like this is a problem with the puppetdb backend not returning an array.
Using hiera CLI or the query_nodes function I am retrieving the array as expected though...
DEBUG: 2016-06-16 00:23:17 +0000: Hiera PuppetDB backend starting
DEBUG: 2016-06-16 00:23:17 +0000: Looking up profile::elasticsearch::masternodes in PuppetDB backend
DEBUG: 2016-06-16 00:23:17 +0000: Hiera YAML backend starting
DEBUG: 2016-06-16 00:23:17 +0000: Looking up profile::elasticsearch::masternodes::_nodequery in YAML backend
DEBUG: 2016-06-16 00:23:17 +0000: Looking for data source node/i-1fc568c2
DEBUG: 2016-06-16 00:23:17 +0000: Looking for data source aws/usdata
DEBUG: 2016-06-16 00:23:17 +0000: Found profile::elasticsearch::masternodes::_nodequery in aws/usdata
DEBUG: 2016-06-16 00:23:17 +0000: Found nodequery ["ec2_tag_role=usmaster", "ipaddress"]
["10.200.31.456", "10.200.40.122", "10.200.60.39"]
I'm also trying to accomplish this exact same thing for elastic search. Too bad a string is returned instead of an array.
Any update on this?
It's a limitation of the hiera function used within a hiera yaml: https://docs.puppet.com/hiera/3.3/variables.html#the-hiera-lookup-function
The hiera() lookup function performs a Hiera lookup, using its input as the lookup key. The result of the lookup must be a string; any other result will cause an error.