Module Data backend over-rides core Hiera backend...
fatmcgav opened this issue · comments
Firstly, thanks for a great addition, I've found it very useful.
Anyhow, the issue I'm seeing is that I'm using the module data backend to ship data for some internal application modules.
This works great for prod, however we now want to use the same modules in Dev, whilst changing some of the values that are pulled in from hiera...
I'd hoped it would have been a simple case of providing the modified values in the core hiera data, however it appears that the module data backend gets looked up after the core backend, thereby preventing us from over-riding application module data...
An example of this from a debug puppet run is:
Debug: hiera(): Looking up remotepost::resources in YAML backend
Debug: hiera(): Looking for data source applications/ncr
Debug: hiera(): Looking for data source applications/remotepost
Debug: hiera(): Found remotepost::resources in applications/remotepost
Debug: hiera(): Looking for data source nodes/rps-node1
Debug: hiera(): Looking for data source os/CentOS
Debug: hiera(): Looking for data source common/common
Debug: hiera(): Looking up remotepost::resources in Module Data backend
Debug: hiera(): Reading config from /tmp/vagrant-puppet-3/modules-0/remotepost/data/hiera.yaml file
Debug: hiera(): Looking for data in source /tmp/vagrant-puppet-3/modules-0/remotepost/data/version/.yaml
Debug: hiera(): Looking for data in source /tmp/vagrant-puppet-3/modules-0/remotepost/data/common.yaml
Is there a way around this?
Cheers
Gav
if this is a normal hiera() lookup, it should stop on first match as always, if its merge lookups like hiera_hash or hiera_array i guess it might go on
I'm using the hiera_resources function, which in-turn uses the hiera_hash function...
So looks like the module data is at the top of the pile, so to speak...
This is indeed how I expect (and count on) it to work ...
I remember I saw some work on Hiera backend ordering but not sure where or how that went.
hiera_hash by design does not override data, it merges data at all layers
But doesn't it merge the same key?
E.g.
-- puppet/hiera/common.yaml
property:
'key':
value: 'value'
-- puppet/module/remotepost/data/common.yaml
property:
'key':
value: 'new_value'should result in a single hash of {'key' => {value => 'new_value'}}?
Proof is in the pudding, show us what you have and what you get?
Hmm, actually it looks like it's not merged it to full depth... I'll try and work up a proper test scenario tomorrow and post back...
:( I've tried creating a simple test case, but am struggling to get the Module Data backend to load :s
The module I've created is: fatmcgav-dim_test.
This includes a simple file resource that will contain a hash pulled from hiera, and has a data folder for use by module_data.
I've then created the following puppet work-dir to test the application: module_data-test.
However the Module data backend just doesn't appear to be loading...
Do I need to do anything special other than include the module_data module in the Puppet modulepath?
Cheers
Gav
getting it loaded is a bit hit and miss with current puppet versions, will be able to fix it properly in 4 only. You can add it to hiera.yaml like any other backend though - add it to the end.
Ah, ok...
Should it be as simple as:
$ git diff hiera/hiera.yaml
diff --git a/hiera/hiera.yaml b/hiera/hiera.yaml
index 675b5a9..6f7d4df 100644
--- a/hiera/hiera.yaml
+++ b/hiera/hiera.yaml
@@ -1,6 +1,7 @@
---
:backends:
- yaml
+ - module_data_backend
:yaml:
:datadir: hiera/hieradata
:hierarchy:As that results in:
$ puppet apply -v --modulepath modules --hiera_config hiera/hiera.yaml manifests/test.pp -d
Info: Loading facts in /home/card.co.uk/gavinw/Puppet/scratch/module_data-test/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /home/card.co.uk/gavinw/Puppet/scratch/module_data-test/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /home/card.co.uk/gavinw/Puppet/scratch/module_data-test/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /home/card.co.uk/gavinw/Puppet/scratch/module_data-test/modules/stdlib/lib/facter/facter_dot_d.rb
Debug: importing '/home/card.co.uk/gavinw/Puppet/scratch/module_data-test/modules/dim_test/manifests/init.pp' in environment production
Debug: Automatically imported dim_test from dim_test into production
Notice: hiera(): Cannot load backend module_data_backend: cannot load such file -- hiera/backend/module_data_backend_backend
Debug: hiera(): Hiera YAML backend starting
Edit: Ahh, don't need the '_backend' bit... :)
$ git diff hiera/hiera.yaml
diff --git a/hiera/hiera.yaml b/hiera/hiera.yaml
index 675b5a9..53fb09e 100644
--- a/hiera/hiera.yaml
+++ b/hiera/hiera.yaml
@@ -1,6 +1,7 @@
---
:backends:
- yaml
+ - module_data
:yaml:
:datadir: hiera/hieradata
:hierarchy:OK, that change meant that the module_data backend is being loaded consistently.
However I'm still not seeing it merge the hash correctly...
As I think about it more, is merging across backends' feasible?
As the yaml backend won't know about the module_data backend and vice-versa...
Currently, with the above hiera.yaml, I get the value from the hiera yaml backend in '/tmp/hash.txt'.
If I switch the order of the backends, so that 'module_data' comes first, then that value is stored in '/tmp/hash.txt'...
So on the surface, it looks like merging across backends' isn't going to work :(
yeah that sounds about right - been ages since i wrote the hiera code but yes, i think you're right
I wonder if this is a scenario that has been address in Puppet 4.0 with the new plugin functionality...
OK, I managed to find an old Redmine ticket which I think would solve this issue: https://projects.puppetlabs.com/issues/10362
Have just pulled it across to Jira and commented it... HI-308
Cheers
Gav
Got some logs from my sample code here illustrating the lack of merge across backends...
Closing this - module is now unsupported, breaks puppet 4 while puppet 4 brought this feature to mainline.
https://www.devco.net/archives/2016/01/08/native-puppet-4-data-in-modules.php
I made a release that complains loudly about being used in Puppet 4, soon I will make one that refuse to work on Puppet 4. Please move to the native feature supported by Puppet Labs.