munin | nuknad.com

In one of my previous posts, I wrote about using foreman (a node classifier and dashboard for puppet) to retrieve a list of hosts (and meta information) so that you can use it in a puppet manifest. We’re looking to do something like iterate over a set of nodes in a template (I use it for generating the Munin server config). Stored resources provide a way to centralize information from nodes, but it isn’t very intuitive, and gets a little tricky to plan and maintain.

While foreman is good for a lot of use-cases, not everyone uses it. So, I want to provide an alternative for those that don’t use foreman. The alternative uses MCollective to populate a list of hosts based on information given to us by the MC Registration plugin. Before I dive in, I’d like to quickly cover off a blog from the MCollective architect, R.I.Pienaar, on this very topic. His blog post (PuppetSearch) brings together 3 things. MongoDB, MC Registration, and Puppet in a very powerful way. It is a really good solution, is more robust, and we’ve since moved on to something similar. The following blog is more just to help understand how flexible puppet can be, and how well it integrates with MCollective. The advantage of using PuppetSearch is that you can load a specific node, and you can query using MongoDB syntax.

So, we’re looking to achieve something quite simple really. We want a subset of hosts matching a basic query, with their meta information, in a variable in a puppet manifest.

MCollective and Registration
MCollective is great, and very flexible. One of the core plugins for MCollective is called registration. Essentially, every node/host sends a registration message at a pre-determined interval set in the configuration. The registration message is sent as a broadcast, and so any client can pick up the registration messages of any other client. We only want one registration handler, but its nice to know that there can be more than one handler (1 per node).

The registration message can be anything, and in our case, we want to send the client identifier, it’s facts, and maybe you want to add some more information along the way. This message is picked up by a handler, which then processes the message.

Currently, we’ve taken the lead of R.I.Pienaar (from his blog above) and shoved the messages straight into a MongoDB instance so that we can query it and use it for different parts of our operations infrastructure. Since that’s done, I’m going to cover off the plain old text file version. It’s extremely similar in architecture, but the code does completely different things.

Installing the registration plugin isn’t too difficult. We need to do it in 2 parts which is pretty standard and has been well documented.

Part 1 is getting the clients to register with all the information they have, including facts, and anything else you like. For this, one of the documented registrations will do just fine. This file will sit in your MCollective plugins/mcollective/registration directory. You then need to adjust your config file for the server to say what registration plugin you’re using and how often.

server.cfg

registerinterval = 300
registration = Meta

After this, you should have registration messages flying around. You need to handle them. The RegistrationAgent provides a simple handler which will write the messages to a text file per client. check_mcollective is optional, and we won’t be using it. It provides a link to nagios if you wish to explore. So, for the client we want to handle the registration messages (probably your mcollective/puppet server), we want to put registration.rb in MCollective plugins/mcollective/agent.

Restart MCollective on all nodes that have been affected, and you should start seeing the registrations text files populating in /var/tmp/mcollective/. You can change this directory by specifying it in the client.cfg on the node where the handler is. (plugin.registration.directory = '/etc/mcollective/registered/')

Ok, so that’s MCollective registration done. If you want to add more information, just make some changes to meta.rb.

Hostlist (Puppet function)

Ok, here we come onto the real topic which is to use what we’ve implemented above to retrieve a list of hosts with their facts. The function itself is very easy because all the information we have is already in YAML and we can just load it, and spit it out!

So the function below is a puppet parser function which can be used in manifests. Please excuse my Ruby…


# mc_hostlist.rb
# Duncan Phillips

# Retrieve a list of hosts and their meta information by querying the data stored by the registration agent.
# info on the registration agent can be found at http://marionette-collective.org/reference/plugins/registration.html

# Usage: mc_hostlist([class],[fact])
# If neither is specified, all hosts are returned.
# Class and Fact are filters and can both be specified
# Fact can be specific or non-specific. i.e. machine with fact, or machine with fact=z

# e.g. mc_hostlist(class=hosting, fact=operatingsystem=Ubuntu)
#[hostname => {facts : {fact1 : value1}, classes : {class1 : value 1}}]

require 'yaml'

module Puppet::Parser::Functions
	newfunction(:mc_hostlist, :type => :rvalue) do |args|
		#populate our array/map
		hosts = Dir.entries("/var/tmp/mcollective")
		for h in hosts do
			begin
				if (h == '.') or (h == '..')
					hosts[hosts.index(h)] = nil; 
				else
					hfile = open("/var/tmp/mcollective/"+h)
					raw = hfile.read.gsub("!ruby/sym ","")
					hosts[hosts.index(h)]=YAML.load(raw).merge({"fqdn"=>h})
				end
			rescue Exception => e
				raise Puppet::ParseError, "There was an exception: " + e + "\n"
			end
		end

		args.each do |arg|

			name, value, factvalue = arg.split("=")

			case name
			when "fact"
				hosts=hosts.compact
				for h in hosts do
					if hosts[hosts.index(h)]["facts"][value]
						if (factvalue) and (hosts[hosts.index(h)]["facts"][value] != factvalue)
							hosts[hosts.index(h)]=nil
						end
					else
						hosts[hosts.index(h)]=nil
					end
				end
			when "class"
				hosts=hosts.compact
				for h in hosts do
					if hosts[hosts.index(h)]["classes"].index(value) == nil
						hosts[hosts.index(h)]=nil
					end
				end
			 
			else
				raise Puppet::ParseError, "mc_hostlist: Invalid parameter #{name}"
			end #case
		end #args
	
		return hosts.compact

	end #func
end

This is a puppet parser, and so it needs to be installed into a module as such. You can find out more about this here. I recommend just putting it into the common module, in which case it will go into MCollective modules/common/lib/puppet/parser/functions/. After this you’ll need to resync the plugins (usually a puppet run will suffice if pluginsync is turned on in the configs).

Into the manifest

So, how do we use this? I’m going to give some insight into how one can use this to generate a Munin conf file… I won’t go into other bits, but will look at what’s relevant here.

Below is an example of how one might use a list of all hosts which have the class ‘Web‘ to create an aggregate graph. We can aggregate anything, for now we’ll create a graph of the load for every node in the list.


	$hl_web  = mc_hostlist("class=Web")

	file {	"/etc/munin/munin.conf": content => template("munin/munin.conf"), }

We can then use this in the template file as below (Once again, excuse my Ruby):


<% hl_web.each do |h| -%>

# Register the nodes

[GroupName;<%= h['fqdn'] %>]
        address <%= h['fqdn'] %>
        use_node_name yes
<% end %>

# Create a new group Totals which holds aggregate graphs

[Totals; GroupName]

        # Generate our aggregate graph

        web_load.graph_title Load Average
        web_load.graph_category GroupName
        web_load.graph_scale no
        web_load.graph_vlabel Load
        web_load.graph_order \<% hl_web.each_with_index do |h,i| -%><% if (h != '') %>
                <%= h['fqdn'][/[a-zA-Z0-9]*/] %>=GroupName;<%= h['fqdn'] %>:load.load <% if i != hl_web.size-1 -%>\<% end -%><% end -%><% end %>
        <% hl_web.each_with_index do |h,i| %>
        <% if (h != '') -%>web_load.<%= h['fqdn'][/[a-zA-Z0-9]*/] %>.draw LINE1<% end %>
        <% end %>

End result: A nice aggregate graph that will dynamically add hosts as they register.

I’m on a mission to completely automate the configuration of munin by taking the excellent work of duritong on puppet-munin, and embellishing it a little in order to create an Overview section which can aggregate a cross-section of hosts. One of the main enablers for this will be a little-known wrapper function that comes with puppet-foreman.

The function provided is a ‘Query Interface’ to foreman, and queries foreman via an API, which returns a list of hosts. This wrapper function actually makes a call via a ruby script provided with the foreman installation. As the docs say, you’ll have to modify line 8 of this script to point to your foreman installation.

If you would like to test that the function is pointing correctly and working, create a hostgroup, attach to a host, and use the following curl:

curl -v http://foreman?hostgroup=somehostgroup&format=yml&state=all

If all is good you’ll see something like

---
  - hostname.domain

The next thing we want to do is use the puppet-foreman provided (example) function, foreman, to make a call for us and return a list of hosts. To do this, we just need to import the foreman module into our namespace and make the call. The call comes standard as taking a number of optional parameters as follows:

hostgroup=groupname : list hosts that are in the group

fact=factname : list hosts with a specific fact.

class=classname : list hosts with a class attached

verbose=yes : output the hosts facts as well

A simple class that implemented this might look like:


class test {
    $hostlist = split(gsub(foreman("hostgroup=cluster1"),"[ -]*",""),'\n')
    file {
        ... some template that uses hostlist
    }
}

where some template could be as simple as listing the fqdn of the hosts that match your query. Later on we will have a look at how to use the facts supplied from clients in a central manifest.


<% hostlist.each do |host| %>
    <%= host %>
<% end >%

The code for the manifest is a bit obscure and I came across it in some old mail archive’s. Essentially, the output from the wrapper function is a string and so it needs to get passed through the split and gsub functions provided in the common module. The split function returns an array which we can then use to pass into our template. This is something we’ll look into just now.

Something that caught me out initially, is that if you are trying to list hosts by classname, the classes have to be directly attached, as opposed to implicitly via a hostgroup. e.g. If you have a node belonging to a hostgroup which has class x. A search for hosts with class x will not return the node.

The example wrapper that has been given in the docs is a little basic and has some elements missing which are available in the actual Query Interface. For instance, when specifying a fact, you can only specify whether a host contains has that fact, and not whether it has a fact with a specific value.

A better puppet parser function
Looking at some missing elements from the query interface, there a couple improvements we would like to get from the wrapper function:

Be able to return a list of hosts with fact=value.
Return an array / hash instead of a string and use those facts given in a verbose response
Option to include all hosts irrespective of their state

Here is my code for the updated wrapper function based on the original with some modifications.
/etc/puppet/modules/foreman/lib/puppet/parser/functions/foreman.rb


require 'net/http'

# Query Foreman
module Puppet::Parser::Functions
 newfunction(:foreman, :type => :rvalue) do |args|
	#URL to query
	host = "foreman"
  url = "/hosts/query?"
  query = []
  args.each do |arg|
    name, value, fact_val = arg.split("=")
    case name
    when "fact"
      query << "#{name}=#{value}-seperator-#{fact_val}"
    when "class", "hostgroup"
      query << "#{name}=#{value}"
    when "verbose"
      query << "verbose=yes" if value == "yes"
    when "state"
      query << "state=all" if value == "all"  
    else
      raise Puppet::ParseError, "Foreman: Invalid parameter #{name}"     
    end   
end   

begin
     response = Net::HTTP.get host,url+query.join("&")+"&format=yml"
rescue Exception => e
    raise Puppet::ParseError, "Failed to contact Foreman #{e}"
end

  begin
    hostlist = YAML::load response
  rescue Exception => e
    raise Puppet::ParseError, "Failed to parse response from Foreman #{e}"
  end
  return hostlist
 end
end

With this modified wrapper function in place (after running a pluginsync) we can simplify our class to:


class test {

    $hostlist_cluster1 = foreman("hostgroup=cluster1", "state=all")
    $hostlist_os = foreman("fact=operatingsystem=Ubuntu", "state=all")

    file {
        ... some template that uses hostlist
    }
}

where some template could be


<% hostlist.each do |host| %>
    <%= host %>
<% end >%

Now, if you’re using puppet 2.6, you can also use the verbose option to return hosts and their facts which just adds another level of awesomeness. The verbose output adds in the meta information to the YAML, and if you do a curl as follows:

curl -v http://foreman?hostgroup=somehostgroup&format=yml&state=all&verbose=yes

#You can also try parse through ruby to get a cleaner output:
curl -v 'http://foreman?hostgroup=somehostgroup&format=yml&state=all&verbose=yes' > /tmp/f.list
cat /tmp/f.list | ruby -e 'require "yaml"; y=STDIN.read; y=y.gsub("!ruby/sym ",""); print YAML.dump(YAML.load(y))'

You should see a yours hosts coming back with this kind of structure

---
   - host.domain
        - facts
             - somefact=x
             - somefact=x
        - classes
             - someclass

So, to use this information from the updated foreman function, we use verbose as:


    $hostlist = foreman("fact=operatingsystem=Ubuntu", "state=all", "verbose=yes")

An in our template, we can reference the facts:


<% hostlist.each do |host| %>
    <%= host['facts']['fqdn'] %> runs <%= host['facts']['operatingsystem'] %>
<% end >%

This final piece of the puzzle, bringing facts and meta-data into the picture, will enable fairly complex setups for distributed systems with a centralized configuration. There are other options and I’d encourage anyone reading this to get familiar with ExtLookup from RI Pienaar, as well as the exported resources which are available as part of the core.

nuknad.com

Posts Tagged ‘munin

Pulling a list of hosts from MCollective for Puppet

Pulling a list of hosts from foreman for Puppet

Recent Posts

Archives

Tags

Follow “nuknad.com”