Creating an Amazon EC2 AMI for Opscode Chef 0.8 Client and Server

Changes Since Original

• 1/13/10: Fix various minor inaccuracies and improved description on how to set up the chef-server. Also removed nanite as a requirement (its no longer used)
• 1/17/10: Add the requirement to build and install mixlib-authentication for the chef-client
• 1/21/10: Added a mkdir for /var/log/chef
• 1/22/10: Added step to insure that /tmp permissions are set

Introduction

Here’s my experience setting up an Amazon EC2 AMI and Instance for a Chef Server and Client. It is based mostly on Bryan Mclellan (btm)‘s post of Nov 24, 2009 Installing Chef 0.8 alpha on Ubuntu Karmic and his more up to date GIST: chef 0.8 alpha installation. It has a slightly different focus and is a bit stale if you are building your own 0.8 gems from the source.

Instantiate an Amazon EC2 Instance

We’ll start with the Canonical Ubuntu 9.10 Karmic AMI. I always go to Eric Hammond’s site  alestic.com to get the pointers to the right AMIs. In this case we’re using a 32bit image for the US-West Region: ami-7d3c6d38 US-East 32bit: ami-1515f67c. You can use the US-West 64bit image ami-7b3c6d3e, US-East 64bit: ami-ab15f6c2

Start the instance from your local dev machine using the command line ec2-api-tools (available as a package or directly from Amazon) or using something like the Firefox Elasticfox and then ssh into the instance so that you can do the following steps on the instance. For the sake of this example, lets say that the Public DNS name for the instance you started is ec2-204-222-170-10.us-west-1.compute.amazonaws.com and the ssh keypair you associated with this new instance is now on your local dec machine in ~/.ssh/gsg-keypair

Prerequisite preparation

The first set of steps need to be done on the instance you just created so login via ssh:

ssh -i ~/.ssh/gsg-keypair ec2-204-222-170-10.us-west-1.compute.amazonaws.com

If on Amazon us-west

There is a bug in the current us-west Canonical AMI where it does not use the us-west apt server. So you have to correct the apt soruces.list:

sed -i.bak '1,$s/us.ec2.archive.ubuntu.com/us-west-1.ec2.archive.ubuntu.com/' \
/etc/apt/sources.list

For all cases

sudo sed -i.bak2 '1,$s/universe/universe multiverse/' /etc/apt/sources.list
sudo apt-get -y update
sudo apt-get -y upgrade
sudo apt-get -y install emacs23 # Of course this is the first package to install!

# Will need these to manipulate ec2 images
sudo apt-get -y install ec2-api-tools ec2-ami-tools 

Set up the ruby environment and install rubygems

Install Ruby and needed packages

sudo apt-get -y install -y ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb \
build-essential wget ssl-cert git-core rake librspec-ruby libxml-ruby \
thin couchdb zlib1g-dev libxml2-dev

Install Rubygems

Rubygems will be installed from source since debian/ubuntu try to control rubygems upgrades. If you don’t care you can install it via apt-get install rubygems

cd /tmp
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar zxf rubygems-1.3.5.tgz
cd rubygems-1.3.5
sudo ruby setup.rb
sudo ln -sfv /usr/bin/gem1.8 /usr/bin/gem
sudo gem sources -a http://gems.opscode.com
sudo gem sources -a http://gemcutter.org

Install Pre-requisit Gems

sudo gem install cucumber merb-core jeweler uuidtools \
json libxml-ruby --no-ri --no-rdoc

Building and Installing Chef Related Gems

Until there are final 0.8.x Chef gems, you will have had to build them on your local machine and upload them to this instance. On your dev machine (this example builds things in ~/src, but it could be anywhere appropriate) follow these instructions to build all the gems and install gems you might need to use your local machine. You will use your local dev machine to develop and manage cookbooks and to manage a remote chef-server:

mkdir ~/src
cd ~/src
git clone git://github.com/opscode/chef.git
git clone git://github.com/opscode/ohai.git
git clone git://github.com/opscode/mixlib-log
git clone git://github.com/opscode/mixlib-authentication.git
# Need to get mixlib-log for client & server and
# mixlib-authentication for the client from git till the 1.1.0 update hits
# See http://tickets.opscode.com/browse/CHEF-823
cd mixlib-log
sudo rake install
cd mixlib-authentication
sudo rake install
cd ../ohai
sudo rake install
cd ../chef
rake gem
# Now cd into ~/src/chef/chef to install the chef client/dev gem on your local machine
cd chef
rake install 

Upload the gems needed for the client to your instance. From ~/src on your local dev machine do:

scp -i ~/.ssh/gsg-keypair chef/chef/pkg/chef-0.8.0.gem  ohai/pkg/ohai-0.3.7.gem \
mixlib-authentication/pkg/mixlib-authentication-1.1.0.gem \
mixlib-log/pkg/mixlib-log-1.1.0.gem  ec2-204-222-170-10.us-west-1.compute.amazonaws.com:

Set up the Chef Client on the new Instance

Now back in your home directory on the instance ec2-204-222-170-10.us-west-1.compute.amazonaws.com install the gems you just copied over:

sudo gem install mixlib-log-1.1.0.gem ohai-0.3.7.gem
sudo gem install chef-0.8.0.gem 

Create the client config file

mkdir /var/log/chef
mkdir /etc/chef
chown root:root /etc/chef
chmod 755 /etc/chef

Put the following in /etc/chef/client.rb:

# Chef Client Config File

require 'ohai'
require 'json'

o = Ohai::System.new
o.all_plugins
chef_config = JSON.parse(o[:ec2][:userdata])
if chef_config.kind_of?(Array)
  chef_config = chef_config[o[:ec2][:ami_launch_index]]
end

log_level        :info
log_location     "/var/log/chef/client.log"
chef_server_url  chef_config["chef_server"]
registration_url chef_config["chef_server"]
openid_url       chef_config["chef_server"]
template_url     chef_config["chef_server"]
remotefile_url   chef_config["chef_server"]
search_url       chef_config["chef_server"]
role_url         chef_config["chef_server"]
client_url       chef_config["chef_server"]

node_name        o[:ec2][:instance_id]

unless File.exists?("/etc/chef/client.pem")
  File.open("/etc/chef/validation.pem", "w") do |f|
    f.print(chef_config["validation_key"])
  end
end

if chef_config.has_key?("attributes")
  File.open("/etc/chef/client-config.json", "w") do |f|
    f.print(JSON.pretty_generate(chef_config["attributes"]))
  end
  json_attribs "/etc/chef/client-config.json"
end

validation_key "/etc/chef/validation.pem"
validation_client_name chef_config["validation_client_name"]

Mixlib::Log::Formatter.show_time = true

Set up the /etc/init.d/chef-client

Copy the example init.d script (You can also use runit instead, but we’re not going to describe that here)

cp /usr/lib/ruby/gems/1.8/gems/chef-0.8.0/distro/debian/etc/init.d/chef-client /etc/init.d
cd /etc/init.d
update-rc.d chef-client defaults

Create an Init script to set /tmp to proper permmissions

It looks like the Canonical Images will not have /tmp with proper permissions if you exclude /tmp from your bundle process. Eric Hammond recommends doing the following.

Create a file /etc/init.d/ec2-mkdir-tmp with the following contents:

#!/bin/sh
#
# ec2-mkdir-tmp Create /tmp if missing (as it's nice to bundle without it).
#
mkdir -p    /tmp
chmod 01777 /tmp

Then set up the /etc/rc dirs to launch this on boot:

chmod a+x /etc/init.d/ec2-mkdir-tmp
ln -s /etc/init.d/ec2-mkdir-tmp /etc/rcS.d/S36ec2-mkdir-tmp
Build the EC2 Image
The always amazingly helpful Eric Hammond has a post, Creating a New Image for EC2 by Rebundling a Running Instance, that describes the basics of how to do this. The following is pretty much a direct synopsis with minimal explanation. See his blog post for more details.
Clean up potential security holes
Remove stuff you don’t want to freeze into your image.
sudo rm -f /root/.*hist* $HOME/.*hist*
sudo rm -f /var/log/*.gz
Copy AWS Certs to Instance
Back on your local development system, copy your Amazon certificates to the instance.

remotehost=<ec2-instance-hostname>
remoteuser=ubuntu
scp -i <private-ssh-key> \
  <path-to-certs>/{cert,pk}-*.pem \
  $remoteuser@$remotehost:/tmp

Create the new Image on the Instance
Back on the ec2 instance, you’ll do the following to create the image.
Define where to store the image on S3
This assumes you have an S3 account setup on AWS. You don’t have to have already created the bucket. Set some bash variables that will be used by the commands that follow. You should set the prefix to something that is meaningful. Below is what I used as an example. You’ll want to make it unique to your environment. The Bucket name must be Globally unique across all of Amazon S3.
bucket=runa-west-amis
prefix=runa-ubuntu-9.10-i386-20100101-base
Define your AWS credentials and target processor
export AWS_USER_ID=<your-value>
export AWS_ACCESS_KEY_ID=<your-value>
export AWS_SECRET_ACCESS_KEY=<your-value>

if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
else
  arch=i386
fi

Bundle the files

This also runs on the current instance and will bundle the everything on the instance file system except for dirs specified with the -e flag into a copy of the image under /mnt:
sudo -E ec2-bundle-vol           \
  -r $arch                       \
  -d /mnt                        \
  -p $prefix                     \
  -u $AWS_USER_ID                \
  -k /tmp/pk-*.pem               \
  -c /tmp/cert-*.pem             \
  -s 10240                       \
  -e /mnt,/tmp,/root/.ssh,/home/ubuntu/.ssh

If you are deploying to US-West-1 AWS Region
Looks like the Amazon ec2 ami tools are not super aware about us-west yet. So you have to do this extra step right now. You’ll have to change the –kernel and –ramdisk to the ones appropriate for your kernel. You can inspect the values used for the AMI you used to boot the original instance. You can do this with ElasticFox or with the command (specify the AMI and region its in thatyou want to check):
ec2-describe-images ami-7d3c6d38   -C /tmp/cert-*.pem -K /tmp/pk-*.pem --region us-west-1
Then execute the following command and specify the right kernel and ramdisk
sudo -E ec2-migrate-manifest        \
  -c /tmp/cert-*.pem             \
  -k /tmp/pk-*.pem               \
  -m /mnt/$prefix.manifest.xml   \
  --access-key $AWS_ACCESS_KEY_ID  \
  --secret-key $AWS_SECRET_ACCESS_KEY \
  --kernel aki-773c6d32          \
  --ramdisk ari-713c6d34         \
  --region us-west-1
 
Upload the bundle to a bucket on S3:
sudo -E ec2-upload-bundle        \
    -b $bucket                   \
    -m /mnt/$prefix.manifest.xml \
    -a $AWS_ACCESS_KEY_ID        \
    -s $AWS_SECRET_ACCESS_KEY    \
    --location us-west-1

You may be prompted with something like:
You are bundling in one region, but uploading to another. If the kernel or ramdisk associated with this AMI are not in the target region, AMI registration will fail.
You can use the ec2-migrate-manifest tool to update your manifest file with a kernel and ramdisk that exist in the target region.
Are you sure you want to continue? [y/N]

You should enter y return to accept.
Register the AMI
Back on your local development machine:
ec2-register $bucket/$prefix.manifest.xml --region us-west-1
The output of this will be the ami-id of your new instance. You can use this to instantiate your new ami.
You now have a private ami image you can start just like any other image. If you want to make it public
ec2-modify-image-attribute -l -a all 
Using the new AMI Image
You can now use this instance as the basis for chef clients and also the basis to create a Chef Server. Use the Amazon EC2 tool, ElasticFox or whatever you favorite tool for managing EC2 instances to make a new instance first to create a Chef Server. Then after that you can create clients and have them load their roles and recipes from the chef server. Once you have a Chef Server, you can use knife ec2 instance command to create user data that includes a run list, credentials and other json that can be passed to the general ec2 tools to build specific instances.
Creating a Chef Server from your new Image
Using an EC2 tool like ec2-tools or elasticfox, create a new instance based on the AMI created earlier. You should use at least a c1.medium as the m1.small is just too painfully wimpy to use. Assume the new instance has the Public DNS name: ec2-204-203-51-20.us-west-1.compute.amazonaws.com

Copy the chef server gems to the new instance from the ~/src directory in your local dev environment to the new instance:
scp -i ~/.ssh/gsg-keypair chef/*/pkg/*.gem \
ec2-204-203-51-20.us-west-1.compute.amazonaws.com:
ssh to the new instance and do the following:
sudo gem install chef-server-0.8.0.gem chef-server-api-0.8.0.gem \
chef-server-webui-0.8.0.gem chef-solr-0.8.0.gem
Set things up to use bootstrap client using chef-solo
We’ll be using the last part of BTM’s GIST, and danielsdeleo (Dan DeLeo)’s bootstrap cookbook and chef-solo to set up this initial server.
mkdir -p /tmp/chef-solo
cd /tmp/chef-solo
git clone git://github.com/danielsdeleo/cookbooks.git
cd cookbooks
git checkout 08boot

Create ~/chef.json:
{
  "bootstrap": {
    "chef": {
      "url_type": "http",
      "init_style": "runit",
      "path": "/srv/chef",
      "serve_path": "/srv/chef",
      "server_fqdn": "localhost"
    }
  },
  "recipes": "bootstrap::server"
}
# End of file

Create ~/solo.rb with the following content:
file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
# End of ~/solo.rb file

Run chef-solo which will execute the chef bootstrap recipes using the bootstrap params in ~/chef.json to actually setup and configure this chef server
If you had installed rubygems with the ubuntu apt package you may have to specify the path:
/var/lib/gems/1.8/bin/
instead of:
/usr/bin
for the knife and various chef commands in the following code.
/usr/bin/chef-solo -j ~/chef.json -c ~/solo.rb -l debug
You will see a lot of Debug statements go by and it will take several minutes to complete. It should complete with something like:
[Thu, 14 Jan 2010 00:19:38 +0000] INFO: Chef Run complete in 38.59808 seconds
[Thu, 14 Jan 2010 00:19:38 +0000] DEBUG: Exiting
Setup basic cookbooks
The following will install the standard cookbooks on the chef server
cd
git clone git://github.com/opscode/chef-repo.git
cd chef-repo
rm cookbooks/README
git clone git://github.com/opscode/cookbooks.git

Now upload the standard cookbooks using the credentials set up by the bootstrap process (user chef-webui)
knife cookbook upload --all -u chef-webui \
-k /etc/chef/webui.pem -o cookbooks

Startup the Chef Server web ui
Do to a bug (http://tickets.opscode.com/browse/CHEF-839) you have to run this twice, the first time will create the admin user:
sudo /usr/bin/chef-server-webui -p 4002
But the first time will abort with an error message like:
Loading init file from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/init-webui.rb
Loading /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/environments/development.rb
~ Loaded slice 'ChefServerWebui' ...
WARN: HTTP Request Returned 404 Not Found: Cannot load user admin
~ Compiling routes...
~ Could not find resource model Node
~ Could not find resource model Client
~ Could not find resource model Role
~ Could not find resource model Search
~ Could not find resource model Cookbook
~ Could not find resource model Client
~ Could not find resource model Databag
~ Could not find resource model DatabagItem
/usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/init-webui.rb:32: uninitialized constant OpenID (NameError)
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `call'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `each'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:99:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:172:in `bootup'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:42:in `start'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core.rb:173:in `start'
from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/bin/chef-server-webui:76
from /usr/bin/chef-server-webui:19:in `load'
from /usr/bin/chef-server-webui:19
Then again to actually start the WebUI and have it run in the background. You might want to start it in screen for now or possibly redirect its output to a log file The following example shows sending the output of the command to a log file. You’ll want to check that log file after starting to make sure there were no errors.
sudo sh -c '/usr/bin/chef-server-webui -p 4002 > /var/log/chef-server-webui.log' &
If you look at the output of a ps, you’ll see the shell command above, but the real work is being done by a merb instance with the port you specified (4002):
#ps ax | grep webui
5533 pts/0    S      0:00 sh -c /usr/bin/chef-server-webui -p 4002 > /var/log/chef-server-webui.log
#ps ax | grep merb
3694 ?        Sl     0:55 merb : worker (port 4000)
5534 pts/0    Sl     0:07 merb : worker (port 4002)
The first merb worker is the chef-server itself, the second is the WebUI server.
Accessing the Chef Web UI
You can access the Chef Web UI web server using a web browser at the IP address / Public DNS name of this server that was just set up. Assuming the Public DNS is
ec2-204-203-51-20.us-west-1.compute.amazonaws.com
Assuming that you set up this instance to allow you to access port 4002 from the IP adddress of your local dev machine, you should be able to access the Web UI at
http://ec2-204-203-51-20.us-west-1.compute.amazonaws.com:4002
You can allow access to port 4002 from specific ip address ranges by updating your security group. You can do that with ElasticFox (easy) or via the command line tools (a pain for a one off). Eventually you (or hopefully Opscode) will  set up an apache or nginx reverse proxy, Passenger or equiv to allow normal port 80 / 443 http/https access.
Conclusion
You should now be able to use  knife your local dev environment to develop cookbooks and upload roles and cookbooks to your new Chef Server and spin up new chef cookbook driven instances. You should use the knife documentation from the Opscode main wiki Knife Page NOT the docs in the Alpha Forums / Getting Started With Opscode / Knife – Commandline API as the later is actually more obsolete in terms of the version that you built from the opscode git repository. There is also a man page and knife –help gives you pretty much the same correct info as the wiki.
I hope to have a follow up post on how to do this in more details.
Feel free to leave comments if you find problems or have questions.


	
Related