Upload
puppet
View
236
Download
4
Embed Size (px)
Citation preview
Finding Waldo…
(“the full data enchilada“ - sorry for the original title, I must have been hungry)
About Me• Swedish
• Live on theislandof Gozo,Malta
• Father of 3
• Author of the Puppet 4 Language, Hiera 5, and Puppet Type system, Task Plans.
• On the Puppet Core team
helindbe @hel
Agenda• What is Hiera?
• What does Hiera do?
• Authoring Data
• Differences hiera 3, (4), and 5
• Writing backends
What is Hiera?
[haɪrɑ] or [haɪərɑ]
“hiera" - the gem“hiera" - the command line tool
“hiera" - the function
What Hiera is• A key-value store abstraction with multiple and
extensible set of backends (backend API).
• A key lookup resolution mechanism searching multiple key-value stores (~ query)
• A hierarchical data organization
• A data composition mechanism (defaults, override, merge, unique - etc.)
What do we use Hiera for?
• Explicit lookup
• Automatic Parameter Lookup (APL)
Why a new Hiera?
hiera 3 does this thing…• Similar code in every backend - COPY PASTA - very hard to fix
general things when the logic is in each and every backend. • Easy to make mistakes and leak memory in a backend • Global & static architecture - must restart after changes • Global config pointing into environments - all environments
must change at the same time - yeah right, when you have 1000nds forever changing environments…
• Search uses a cartesian product of levels and backends based on file suffix - lots of trickery and lots of file stats
• Different backend versions not supported in different environments. • No explanation support - need to trace/debug - must restart server • Use of dynamic variables (because of lack of suitable features) makes it impossible to
have efficient cashing to speed up performance. • Has its own backed loading system • Circular dependency on puppet - made it very hard to fix certain types of issues
Hiera Versions
where a gem in puppet
CLI hiera puppet lookup
hiera.yaml version 3 5 ( supports 3 and 4)
explicit lookup hiera() hiera_array() hiera_hash() lookup()
backend API complicated simple using function API
APL options no lookup_options in data, explicit and APL the
same!explain support no --explain
advanced paths no globs, mapped paths
deprecated
Explicit lookup# get value, and…# …fail if not present$x = lookup(‘key’)
# …verify data type$x = lookup(‘key’, Array)
# …return default if not present$x = lookup(‘key’, Array, first,[blue])
# …options hash$x = lookup('key', { <options> })
Many Lookup Options name the key to lookup
value_type the return type to assert
merge merge options
defalt_value if not found use this
default_values_hash if not found pick lookup here
override look here first
default via code block
$x = lookup(‘mykey’) |$x| { # calculate the value compute_it($x)}
Merge Behaviourfirst the first found (default)
unique for Array ( v3 = “array merge”)
hash merge hash, highest prio key wins (no recursion)
deepmerge hash, recursive, higher prio wins on conflict, arrays made unique
Deep Options
knockout_prefix string to match for removal (undef = no knockout)
sort_merged_arrays sorts arrays (false)
merge_hash_arrays if hashes in arrays should be merged (false)
APLclass mymodule::myclass($myparam) { #... }include 'mymodule::myclass'
# Automatically looks up:# 'mymodule::myclass::myparam'# when value is not given.
explicit lookup vs. APL• APL = “Inversion of control” - “Push don’t Pull”
• Much easier to test
• Can be overridden!
• Parameterized classes are documented - your arbitrary keys are not…
• Use APL in your APIs
APL and options
• All lookup options can be set in the data!
• Control “deep merge” etc per key!
• Any backend can return a Hash for the key “lookup_options” with a map of “key” => <options-hash>
• All “lookup_options” are merged
• You can supply defaults in a module for example
APL and lookup_options !
Authoring Datawhere to stick (possibly pieces of) Waldo…
Three Layers
Global Environment Module
Global Layer
• For operational use
• Across all environments
• Overrides environment and modules
• Ok to use a deprecated version 3 hiera.yaml
• In Hiera 3, the only layer (with nasty tricks of referencing into each environment).
Environment Layer
• The typical place to store data
• Across all modules in the env
• Overrides modules
• Use a hiera version 5 hiera.yaml
Module Layer
• Regular hierarchy for overridable and merge-able values
• a default_hierarchy only consulted when not found in regular hierarchy
• Only keys for the module’s namespace
• Must use a hiera 5 version hiera.yaml
the module’s default_hierarchy
A config
Global Environment Module
a::
b::
Strategy ‘first’
Global Environment Module
lookup(‘waldo’)a::
b::
Strategy ‘first’ - not found
Global Environment Module
lookup(‘waldo’)a::
b::
not founderror
Strategy ‘first’ - not found
Global Environment Module
lookup(‘b::waldo’)a::
b::
not founderror
Strategy ‘first’ - simple find
Global Environment Module
lookup(‘b::waldo’)a::
b::
returned
Strategy ‘first’ - simple find
Global Environment Module
lookup(‘b::waldo’)a::
b::
Strategy ‘first’ - find in module
Global Environment Module
lookup(‘b::waldo’)a::
b::
Strategy ‘first’ - find in defaults
Global Environment Module
lookup(‘b::key’)a::
b::
not found
returned
A merging strategy
Global Environment Module
lookup(‘b::key’)a::
b::
Inside a layer
Highest prio
Lowest prio
levels with named entries
Search order
Highest prio
Lowest prio
each level in turn
---version: 5defaults: # Used for any hierarchy level that omits these keys. datadir: data # This path is relative to hiera.yaml's directory. data_hash: yaml_data # Use the built-in YAML backend.
hierarchy: - name: "Per-node data" # Human-readable name. path: "nodes/%{trusted.certname}.yaml" # File path, relative to datadir. # ^^^ IMPORTANT: include the file extension!
- name: "Per-datacenter business group data" # Uses custom facts. path: "location/%{facts.whereami}/%{facts.group}.yaml"
- name: "Global business group data" path: "groups/%{facts.group}.yaml"
- name: "Per-datacenter secret data (encrypted)" lookup_key: eyaml_lookup_key # Uses non-default backend. path: "secrets/%{facts.whereami}.eyaml" options: pkcs7_private_key: "/etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem" pkcs7_public_key: "/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem"
- name: "Per-OS defaults" path: "os/%{facts.os.family}.yaml"
- name: "Common data" path: "common.yaml"
a level
Inside a layer
Highest prio
Lowest prio
multiple paths/globs etc. per level (backend only called if file exists)
---version: 5defaults: # Used for any hierarchy level that omits these keys. datadir: data # This path is relative to hiera.yaml's directory. data_hash: yaml_data # Use the built-in YAML backend.
hierarchy: - name: "Per-node, datacenter, and business group data" paths: - "nodes/%{trusted.certname}.yaml" - "location/%{facts.whereami}/%{facts.group}.yaml" - "groups/%{facts.group}.yaml"
- name: "Per-datacenter secret data (encrypted)" lookup_key: eyaml_lookup_key # Uses non-default backend. path: "secrets/%{facts.whereami}.eyaml" options: pkcs7_private_key: "/etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem" pkcs7_public_key: "/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem"
- name: "Defaults per os and common" paths: - "os/%{facts.os.family}.yaml" - "common.yaml"
multiple
Different ways to reference data files / sources
Key Data type Expected valuepath paths
String Array
One file path. Any number of file paths. This acts like a sub-hierarchy: if multiple files exist, Hiera searches all of them, in the order in which they’re written.
glob globs
String Array
One (or several) shell-like glob patterns, which might match any number of files. If multiple files are found, Hiera searches all of them in alphanumerical order (ignoring the order in which multiple globs were given).
uri uris
String Array
One, (or several) URIs that are not checked for existence. One call to the backend is performed for every given URI.
mapped_paths Array or Hash
A fact that is a collection (array or hash) of values. Hiera expands these values to produce an array of paths.
mapped_paths: [services, tmp, "service/%{tmp}/common.yaml"]
Tips and Tricks
• Have keys with ‘.’ in them?Quote the key when looking up to prevents the built in “dig” behaviour: lookup("'my.dotted.key'")
Writing Backends
Three kinds of backends
data_hash
Produces all of the key => valuepairs at once as a Hash
Good for small to moderate data volume and where most of the data is always used. Limit; static in nature.
Reading a json file
function mymodule::myjson(Hash $options, Puppet::LookupContext $ctx) { $options[‘path’].file.parsejson()}
lookup_key
Produces values per key - called multiple times.
Slightly more complex because of the added flexibility/power, but still not complicated to implement.
A “prefixer” added to our hiera.yaml
- name: "Using example with prefix" path: “examples/%{trusted.certname}.yaml" lookup_key: mymodule::json_with_prefix # the example function options: prefix: “Yo, Waldo! The value is: "
transforming values by key…
function mymodule::myjson_with_prefix( Variant[String, Numeric] $key, # the key being looked up Hash $options, # the options from hiera.yaml Puppet::LookupContext $ctx # the context/helper){ $hash = $ctx.cache_file($options[‘path’]) |$content| { $content.parsejson() } case $val = $hash[$key] { String : { "${options[‘prefix']}${val}" } NotUndef: { $val } default : { $ctx.not_found() } }}
data_dig
Like lookup_key but is responsible for any digging into the key.
because lookup(“users.jane_doe.pager_nbr”) would be terrible if there are thousands of users…
Puppet::LookupContext object
Key Expected valuenot_found() Immediately returns from the function and tells hiera there is no
value for the keyinterpolate(value) Perform hiera style interpolation on the given string value
environment_name() module_name()
Produces information about the container where this function is part of a hiera.yaml
cache(key, value) cache_all(hash)
Adds values to a cache.
cached_value(key) cache_has_value(key) cached_entries()
Retrieves value(s) from the cache
cached_file_data(path) |$content| {...}
Reads and caches the contents of a file, or the transformed content of a file
explain() || { 'message' } Emits an “explain” message if —explain mode is on
Ideas for backends• DRY up data - use lookup inside backend to compose
values from lookups - earlier not possible for arrays and hashes.
• Computed values - given input from hiera.yaml (maybe even a key), other values can be derived
• Provide different data sets (in a module; for example “standalone” vs. a “client server” configuration) that can be integrated. While a module cannot directly supply global keys, it can provide the data/backend that does so if added to the env’s hiera.yaml!
Cage Fight !
• Faster • Cleaner • More powerful, yet Simpler • Explain Support
3 vs 5 - deprecated bad magic• Nothing good came from using these hiera 3 magic variables:
$calling_module$calling_class$calling_class_path
• Hiera 3 could use these as a hacky predecessor of module data, but anything you were doing with them is better accomplished with the module layer. You can continue using these in a version 3 hiera.yaml file, but you’ll need to remove them once you update your global config to version 5.
• If used to split up data in multiple files (per module etc). Use the ‘glob’ pattern.
3 vs 5• Use lookup() instead of hiera_xxx() • Use lookup() + include() instead of hiera_include() • Use lookup CLI instead of hiera CLI • No global merge/deep-merge setting (was: horrible!) - use lookup options. • Move to using hiera 5 backends! • The ‘data binding terminus’ (advanced hackery) is no longer used - write a
backend instead • Hiera 5 is faster (much thanks to caching) and with greatly reduced risk of
memory leaks due to mistakes in backends • You can call lookup() from within backend functions! Can do what hiera 3
alias never could (hiera 3 - limited to strings). • The lookup_key function opens up for advanced data composition - merge
multiple (different) keys into one etc.
And the winner is…
Questions?