How to create custom CentOS/RHEL images for Openstack using Boxgrinder

Introduction

Migrating existing infrastructure to an Openstack or AWS cloud often requires one to create custom virtual machine (VM) images. These images can have any predefined list of packages installed and thus facilitate quick application deployment in the cloud. For example, to deploy a web application one can create a custom virtual machine image with database installed and another with web server. Complex distributed web applications will have many more component such as, message queue, memcached etc. In order to deploy a cluster in a cloud quickly, the best practice will be to have image created for each of these components.

In this blog, I shall describe how to create custom CentOS/RHEL images using Boxgrinder The process for creating images for Ubuntu is similar.

Obtaining Boxgrinder

To build VM images for Openstack, we shall use Boxgrinder meta appliance. This appliance can be downloaded directly from the Boxgrinder's website. I recommend using the RAW x86_64 format. While Boxgrinder VM can be setup on a local laptop, I shall explain how to set it up on Openstack itself.

Steps for setting up Boxgrinder meta appliance

1. Download the boxgrinder meta appliance using this link: <http://dist.stormfolio.boxgrinder.org/meta/boxgrinder-meta-1.7-fedora-15-x86_64-raw.tgz>
2. Copy the downloaded file on the Openstack controller
3. Uncompress using 

     tar xzvf /boxgrinder-meta-1.7-fedora-15-x86_64-raw.tgz 

You should see a file named boxgrinder-meta-sda.qcow2. 

4. Setup OS_USERNAME, OS_PASSWORD and other environment variables required to use Openstack APIs. 

5. Run `glance image-create --name boxgrinder --disk-format=qcow2 --container-format=bare < boxgrinder-meta-sda.qcow2` 

6. Now create a VM using this instance. NOTE: The VM must have at least 10GB of diskspace so that we don't run out of disk space while creating images. Please check which `flavor id` to use by running a nova flavor-list command. Then run the following command to bring up a

Boxgrinder VM: 

   nova boot --image boxgrinder --flavor 2 boxgrinder-vm

7. Wait till the VM becomes active (Run `nova list` to check the VM's status and IP address). 

8. Once the VM becomes active, you should be able to login to it by ssh'ing to it. The default password is 'boxgrinder'. 

Image creation with Boxgrinder

Boxgrinder is tool that allows one to create virtual box images in different formats. One of the reasons why we chose Boxgrinder is it supports creating CentOS based images in qcow2 disk format. Creating an image with boxgrinder involves only 2 steps: 

• Fixing arguments used during the the boot process. 
• Defining the box or VM in YAML format
• Running boxgrinder-build command to generate the image. 

Boot parameter customization

Boxgrinder, unfortunately, doesn't allow customizing certain boot parameters, for example, keyboard layout, timezone etc. These parameters, however, can be modified by changing a kickstarter file template located in 

/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.10.0/lib/boxgrinder-build/plugins/os/rpm-based/src/appliance.ks.erb 

Box definition

Appliance definition files are written in YAML, which is rather easy to read and understand. The YAML file has several key sections and their details are better described in the Boxgrinder appliance definition format. 

Here is a sample CentOS 5.x appliance definition in the YAML format.

name: CentOS-5

summary: Custom CentOS.

# Begin 

os: 

 name: centos

 version: 5

 password: secret

hardware: 

 cpus: 2

 memory: 1024

 partitions: 

 "/": 

 size: 5

 type: ext3

# Packages to install

packages: 

 - coreutils

 - yum

 - lvm2

 - e2fsprogs

 - zlib

 - keyutils

 - openssl

 - dhcp

 - tftp-server

 - OpenIPMI

 - OpenIPMI-tools

 - OpenIPMI-libs

 - cpuspeed

 - rpm

 - e2fsprogs

 - grub

 - sysstat

 - ntp

 - openssh-server

 - openssh-clients

# truncated for the sake of brevity

............................

 - xorg-x11-xauth

 - tcpdump

 - strace

 - pam

 - procmail

 - rubygem-chef

# Disable default mirror repo and enable repo for the data center

default_repos: false

repos:

 - name: "3rdParty"

 baseurl: "http://reposerver-internal/repo_mirror/3rdParty/"

# Any files that need to be copied to the guest VM.

files: 

 "/etc/yum.repos.d": 

  - "cobbler.repo"

 "/tmp"

  - "rc.local" 

# Configuration steps during boot time.  

post: 

 base: 

    - "mv /tmp/rc.local /etc/rc.d/rc.local"

    # Fix console log issue for CentOS images

    - "sed -i -e '/kernel/ s/$/ console=ttyS0 console=ttyS0,115200n8/' /boot/grub/grub.conf"

    - "sed -i -e 's/^title/serial -unit-0 -speed=115200\\nterminal -timeout=10 console serial\\ntitle/' /boot/grub/grub.conf"

In this file, we defined the OS type (CentOS) and version 5. The list of packages were obtained from the kickstarter file templates available on ops-tools git repo. Boxgrinder seemed extremely sensitive about the indentations in each line and it does not like tabs. So while defining YAML file for Boxgrinder, you must use spaces very carefully.


After defining the package section, we specified which repos it should look for obtaining these packages. We set "default_repos" to false. Very often access to public repositories is restricted in private data centers. Specifying the correct repo location is critical in such scenario.

File injection

Boxgrinder allow injecting files into the guest VM. This is a very useful feature through which we can "bake in" configuration or other files into an image which don't vary from one VM instance to the other. 

files: "/etc/yum.repos.d": ---> defines the location where the file will be installed. - "cobbler.repo" --> name of the file.

Note: Boxgrinder expects the file (e.g. cobbler.repo) to be in the same working directory where the appliance definition file is. 

Post processing

Finally, the post section allows us to put any post image creation steps.  Here we can put any commands that will be executed after the packages are installed. Typically, we want to put only the configuration steps that are common for all instances of the same image. For example, defining which services should be running at startup (chkconfig settings) or firewall configuration. 

In the following example,  we define in the post section an action that we add to /etc/rc.local file. The goal of this action is to fetch a script file from Openstack's metadata service, save it as boot-init.sh file. By putting the boot-init.sh file in /etc/rc.local file which runs when a machine boots. 

post:
base:
- "mv /tmp/rc.local /etc/rc.d/rc.local"
# Fix console log issue for CentOS images
- "sed -i -e '/kernel/ s/$/ console=ttyS0 console=ttyS0,115200n8/' /boot/grub/grub.conf"
- "sed -i -e 's/^title/serial -unit-0 -speed=115200\\nterminal -timeout=10 console serial\\ntitle/' /boot/grub/grub.conf"

Running Boxgrinder to generate image

Once the appliance definition is created, we use the following command to build the image in qcow2 format. 

boxgrinder-build chef-server.appl --os-config format:qcow2

Uploading images in glance

Images can be uploaded in glance with the following command: 

 glance image-create --name centos58 --disk-format=qcow2 --container-format=bare --is-public=true --file=centos-5-sda.qcow2 

Where centos58-e2 is the name we want to store the image as and centos-5-sda.qcow2 is the image file. 

Customizing instances at boot time with Openstack

Since certain parameters cannot be baked into an image and vary for each instance of an image, we want to use nova boot options to configure these parameters.  For example, hostnames, DNS server configurations, chef server URL should not be part of an image. Nova boot command offers several options further customizing images at runtime:

1. Injecting files – ideal use case for this option is to pass chef validation.pem file or ssh authorized keys. 
2. Using meta-data service – can be used to retrieve any meta data associated with an instance by running commands against the meta-data service. 
3. Using user-data  – user data is a file that can contain a shell script or file of any format. An instance can fetch this file while booting and do something with it. We used this option to pass a script that performs some initialization tasks. 

Sample nova boot command that we used: 

nova boot --image 3aec87e4-d490-4feb-9f5b-49474ca52518 --flavor 1 --file /etc/chef/validation.pem=./validation.pem --user-data ./boot-init.sh --security_groups default --key_name e2_admin --availability-zone nova:wpc0048 e2-slave 

NOTE: Although we used /etc/rc.local file for running some post install tasks at boot time, Openstack recommends cloud-init package. This package offers many benefits among which one is to be able to pass and run chef recipes as a part of the booting process.