I’m going to start with a quick bit of background on my experience because its important. I am not a Chef evangelist. I arrived on the scene with a company that already used Puppet and had invested some time there. It had been a bit slow moving and so over a few months I helped to pick it up. The devops team brought puppet to a state where it was usable, fitted with our systems and our idea of how data and code should be managed. We developed ideas around provisioning machines, what should be managed, and how. We integrated with products like MCollective to help us manage servers, and integrated with some of our own internal systems. During the last 10-12 months of learning, I have been learning both the intricacies of Puppet and the principles of Configuration Management. A lot of this time was really evolving the puppet architecture and the supporting systems into a state that worked in context.
Over the past week and a bit I have installed a chef server and brought up a fairly complex machine. I’m going to touch on a few areas of Chef that i’ve worked with…
Installation
The Chef installation process is good. For a first time installer, the documentation is there, but you have to jump around the concepts and terminology a bit before you realize what needs to be done. I also hit a bug in rabbitmq where if your chef domain name doesn’t resolve, the whole install will fail. I had setup /etc/hosts, but the resolution doesn’t use it, and so you need to use a resolving domain name.
They’ve put a whole lot of thought into automatically registering nodes. Anything special you want to do like creating self-classifying nodes is easy enough considering you have the flexibility of ruby and all the node information at your disposal, including the ohai data.
Once you understand the authorization mechanisms and how to use the tools, you can install Chef on client and server in under 5 minutes.
Writing Recipes
I’ll go into tools later, but when it comes to a new cookbook, you can just issue a simple knife command to create a template to work from. Remove a few directories you don’t need, and start writing recipes. Recipes are pretty simple to write, and are basically just configuring a resource (packages, files, links, etc).
Chef recipes have two really strong points. They are deterministic, and they can use ruby code.
A deterministic recipe makes the world of difference in some situations. Configuration management systems can use dependencies to specify run order, but I find this can be complex, hard to read, and difficult to debug. I like determinism because it is simple implicit dependency management.
Ruby code is also a big win. You can use data from anywhere to make decisions about what should run. You can make decisions based on whether a file exists on the node, or whether some data has a specific value on another node somewhere. You can construct variables that can be used globally, and really, whatever you want. It opens you up to make the mistake of side-stepping best-practices, but I don’t think Chef lacks features that would make you want to do this.
Managing Data
Data comes from 4 sources: database, ohai, attributes, recipes.
The database, AKA data-bags, is the most removed data source for your more miscellaneous kind of data. It has a very clear place in my mind and I know exactly the sort of data to store here. Information that is relevant to multiple nodes such as usernames and passwords for services. Relevant to that is the added benefit of having encrypted data bags. This adds a level of security and is a great feature. For anyone concerned about keeping information in the database, a best practice is to keep the stored json in subversion, and simply upload to the database. Nice.
Ohai is the node level information reported in by the node such as number of cpu’s, ip-address, ec2-ami-id, load, etc. I find the information to be very good and one of my favourite things about Ohai is that the format is nested (like json). This way its easy to keep the namespace clean and be clear about your data. I found it to be easily extensible as well, but more on that further down.
Attributes are recipe level variables. You would set up something like the default apache port here and use it in your template. There has been a lot of thought on how recipes interact and so there is a nice multi-tier mechanism to override variables. Often you will end up overriding these variables using information from other data sources or just hard-coding them (as an override, not in the module).
Finally, recipes can set variables, change them, etc. These variables are persistent and can be used by other recipes.
These data sources together provide a great deal of flexibility in how you manage your data and ensures separation of data and code.
Extending
I haven’t played a lot with extending chef. I dabbled with an ohai plugin, and wrote a quick Light-weight Resource Provider (LWRP). I found the documentation to be a bit lacking for ohai plugin writing, and the examples given were all very simple. My ruby isn’t amazing by any standards and so I struggled to figure out a few things that might come more naturally to others. Distributing the ohai plugin was easy, and you just need to put it in the run-list high up, then, any cookbooks after that will have access to the data in the plugin.
Writing a LWRP was pretty simple given the documentation and examples. A LWRP lets you abstract away a resource (eg. file, symbolic link, package) and a provider (something that manages the resource on the platform). These LWRP’s let you quickly encapsulate common code into a single resource which helps reduce code size, and improve readability. Its nice to be able to abstract resources and providers away even at a basic level, it makes for much neater and more understandable code.
There are more avenues for extending, but I haven’t explored them yet.
Tools
Chef comes with two main tools that I know of. Knife and Shef.
Knife is, well, the Swiss army knife of chef. It lets you manage chef from anywhere, and includes commands for everything you will need to do. The basic commands cover nodes, clients, cookbooks, data-bags, environments and roles. It also has commands to configure clients through a Q&A interface which is used as part of the install process. Knife connects to the server, and so its strength is that everything you need to control chef can be done remotely from any machine. You do not need to be on the central server to upload new cookbooks, modify data, change parameters, etc…
Shef is a ruby shell running in the Chef client context. You run this from an end node and use it to debug cookbooks and recipes step by step. You can assume roles, run any cookbook, set any variables, etc. It really lets you get involved in the debugging of a cookbook. I haven’t spent much time on Shef yet, but I plan to get to know it very well.
Community
The Chef community is good. I’ve been on IRC asking a lot of questions, and mostly got back really good answers. Opscode and 37-signals have opensourced a whole lot of cookbooks which is really helpful in getting up and running quickly. They are all excellent and I didn’t need to modify anything at all for my purposes. They helped with learning as often you have to read the code to figure out what they do, how they do it, and what data they need. Big thanks to both teams for their hard work in this area.
What next…
I will be looking at bringing up a few more machines and see what challenges come up. I have been impressed on all fronts, and personally, it suits a lot of the ways that I think about problems and how to solve them. I still need to draw my own conclusions on whether Puppet or Chef is right for the systems and architecture I work with, and I advise anyone looking into Configuration Management to do the same and explore their options.
