Update:
I am having problems using the chef dev tools/client from the HEAD of the chef git repository with the Opscode Alpha Server service. I’m not sure if its me or if the latest versions of the chef client from HEAD is compatible with the Alpha Server Service. So the following is still useful for understanding how to build from HEAD, but it will not work with the Opscode Alpha SaaS server. It will work with the server you build from HEAD. See the next article Creating an Amazon EC2 AMI for Opscode Chef 0.8 for info on creating a Chef client and server on EC2.
Introduction
Opscode is introducing a pretty major set of changes in Chef in the 0.8 release. Its a major step forward and has some major changes as to how one interacts with Chef. (as well as some major bug fixes that alone make it worth the move). The Opscode Alpha Program introduces a new service where Opscode runs the actual Chef Server as a service.
This post will describe setting up your User/Dev environment by building your own Chef Client / Dev Gems from the latest HEAD of the Chef repo from Github. It assumes that you did sign up for the Alpha program and have access to the Opscode Alpha Server. Though much of it would be the same if you were running your own chef server also built from the latest source from github. This post does not show how to actually use Chef and the chef-client on a target node. Hope to have a post on that in the next few days.
The documentation on how to move to and use Chef 0.8 is still very sparse, so I figured I would jot down some of the things we are learning as we apply this to our infrastructure at Runa. If any of you OpsChefs out there see something wrong or something I left out, let me know in the comments or via email.
The Opscode Chef Alpha Environment
If you are in the Opscode Alpha program, you would have been given login[s] and some pem keys. I won’t go into the details of this since they do have pretty good docs on setting this up (if you have an alpha login you can get them at http://opscode.zendesk.com/forums/58858/entries/49336). Its probably a good idea to follow these and start with their 0.8.0 gem to make sure you are talking with the Alpha Server before trying to use the Chef Git Repository to build your own gems.
The Alpha instructions use a Chef gem that is frozen at 0.8.0. But the Chef folks have already progressed much further than the Oct 29h release of 0.8.0 in the Chef Git Repository.
The HEAD of the Git Repository has many changes since 0.8.0. Some big ones include:
- The Knife sub commands are completely different
- There is now a Chef Shell (A REPL like irb but for the chef client)
- Lots of Bug Fixes
And if we’re going to be on the bleeding edge, we might as well go all the way! So the rest of this blog will be about using the Chef HEAD branch from the Chef git repository. We’ll still use the Alpha Chef Server at least to start with.
Configuring your Dev Environment
Prerequisites
I’m using Mac OS X 10.6 (snow leopard). Our target environments are Ubuntu Linux on Amazon EC2. But assuming you have *nix, Ruby and Ruby Gems set up on your environment it should generally be the same (don’t know about people stuck in the Legacy Windows environment though).
So you will need to have installed and know how to use:
- Ruby
- RubyGems
- Git
And the following Ruby Gems should be installed (I think this is the minimum you need, these will include their own dependencies:
- rake
- rspec
- cucumber
- uuidtools
- nanite
- gemcutter
- jeweler
You will need http://gems.opscode.com as a gem source for the following. You can use the command:
sudo gem sources -a http://gems.opscode.com- mixlib-authentication
Getting and building the code/GEMs for the Dev Environment
The instructions that are in the README.doc of the Chef Git Repository are out of date as of now (Dec 20, 2009). The instructions on the wiki, Installing Chef from HEAD are more accurate. Even though it seems like one can use the mixlib gems as the repository and the gems have the same version number, I found that I needed to install the mixlib libraries from source.
Getting and building Ohai & Mixlib Gems from Github
We won’t be making any changes in these, so we’ll just git clone and build it:
cd to where you want to keep your local repositories
git clone git://github.com/opscode/ohai.git
cd ohai
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-config.git
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-log.git
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-cli.git
sudo rake install
cd ..
Getting the Chef code from github
You can get the Chef repository from github. The readme there has most of the info you need for
If you plan to submit any patches or other changes back to Opscode, or you would like to have your own repository of this, you can fork the Opscode repository into your own Github account. This is what I did and will demonstrate below. If you don’t want any hardcore forking action, you can just git clone the opscode repository as shown here (assuming your current working directory is where you want the local directory repository placed. It will be named using the default “chef”):
git clone git://github.com/opscode/chef.gitIf you have forked into your own github account (mine is rberger), you would git clone using the “Your Clone URL”:
git clone git@github.com:rberger/chef.git rberger-chefThis assumes you want your local directory name for the repository to be rberger-chef, just so you can distinguish it from the official opscode one. (I will refer to the top of the local repository as rberger-chef from now on).
What’s in the Chef Git Repository
Change directory into the local repository and do an ls. You’ll see that there are several components here.
$ cd rberger-chef
$ ls
CHANGELOG README.rdoc chef-server chef-solr scripts
LICENSE Rakefile chef-server-api cucumber.yml
NOTICE chef chef-server-webui features
There are 2 main trees:
- chef: chef-client and dev gem
- chef-server: Chef Server gem. Used only if you build your own server
- chef-server-api: Implements the REST interface sub-system as part of the full Chef Server
- chef-server-webui: Implements the WebUI as part of the full Chef Server
- chef-solar: Implements the Solar Search sub-system as part of the full Chef Server
- features: Not 100% sure all its used for, definately for the cucumber tests. But is part of the Server as far as I can tell
For now we are only interested in the chef tree. That will be used to set up the local dev environment. We’re not going to follow the outdated instructions that are in the README.doc in the root of the chef repository which assumes you are setting up the whole stack on the Dev machine. We’re going to just install the chef client and tools from the chef sub-tree on the dev machine.
This post will not describe how to build /use the chef-server, though you can pretty much build everything by running
sudo rake installfrom the top of the distro. There are more gem dependencies that need to be installed before you can build the chef-server trees.
Building and Installing the Chef Client / Dev tools
Change directory to the chef subdirectory so you should be in rberger-chef/chef (or if you have a direct clone of the opscode chef repository: chef/chef)
cd chefSome minor tweaks to the Source
(shef is now included in the executables in the latest repository and setting my own sub-version number was lame)
I have done a few mods to the source. Mainly to set the version number to something that will not conflict with the official numbering now or when new releases come out and to have shef be installed by the gem.
- Changed line 30 in the Rakefile to
s.executables = %w( chef-client chef-solo knife shef )so the install puts shef in /usr/bin - Changed line 7 in the Rakefile to
CHEF_VERSION = "0.8.0.1" - Change line 30 in lib/chef.rb to
VERSION = '0.8.0.1'
Build and install
rake installIts going to eventually ask for your sudo password as it needs to use sudo to do the gem install. The run should look something like:
(in /Users/rberger/work/Chef/rberger-chef/chef)
mkdir -p pkg
WARNING: no rubyforge_project specified
WARNING: description and summary are identical
Successfully built RubyGem
Name: chef
Version: 0.8.0.1
File: chef-0.8.0.1.gem
mv chef-0.8.0.1.gem pkg/chef-0.8.0.1.gem
sudo gem install pkg/chef-0.8.0.1 --no-rdoc --no-ri
Password:
Building native extensions. This could take a while...
Successfully installed eventmachine-0.12.10
Successfully installed amqp-0.6.5
Successfully installed thor-0.12.0
Successfully installed deep_merge-0.1.0
Successfully installed moneta-0.6.0
Successfully installed chef-0.8.0.1
6 gems installed
Using Chef with the Opscode Alpha SaaS Server
This just touches on some of the things that are described in The Official Guide to Getting Started With Opscode
Setting up your Dev Environment
Its not clear if you really have to do everything as described in the document if you are building the latest release from the chef repository and using the ~/.chef/knife.rb config described below. For instance I didn’t have to set the environment variables for OPSCODE_USER and OPSCODE_KEY since they are now set in the knife.rb nor did I have to create /etc/chef/client.rb. And even without the global Chef config, I was able to use most of the knife commands. But not some like the ec2 instances data seemed to need the organization validation key to be in /etc/chef/validation.pem
Copy your assigned validation key to /etc/chef
When you got your Opscode Alpha welcome stuff, you should have gotten your user keys and a key for your organization. Copy your organization (in our case runa.pem) to /etc/chef/validation.pem. You will probably have to create /etc/chef directory first.
The User Chef/Knife config
You must configure a knife config file in your home directory under ~/.chef/knife.rb and have your key that you got from Opscode somewhere pointed to by a line in ~/.chef/knife.rb. The configuration parameters are described on the Knife Wiki Page. For instance my config file:
log_level :info
log_location STDOUT
node_name 'rberger'
client_key '/Users/rberger/.chef/rberger.pem'
chef_server_url "https://api.opscode.com/organizations/runa"
cache_type 'BasicFile'
cache_options( :path => '/Users/rberger/.chef/checksums' )
Once you have this set up you can now use knife and the chef rake commands. You can test things out by saying something like:
knife client listWhich should return and empty list assuming you haven’t set up any clients on this server yet.
The first real useful command you want to do is to upload your cookbooks to the Opscode Server:
cd to where your chef cookbook repository is
rake upload_cookbooksYou can also do it with just knife:
knife cookbook upload -aThis may take a while as it will upload all the cookbooks in cookbooks and site-cookbooks in your current repository.
After that you can upload single cookbooks
knife cookbook upload
Just remember the knife documentation on the Alpha site no longer applies to the knife that you get from building from the HEAD of the chef git repository. Strangely enough, the knife documentation on the wiki is accurate.
Conclusion
Once you’ve been thru it, its all quite simple. I hope to post some more on using 0.8.0+ soon. See a more recent blog post for building your own Chef Server Creating an Amazon EC2 AMI for Opscode Chef 0.8
