Running continuous integration of Openstack on Docker

Motivation

There are many configuration management tools -- Chef, Puppet and Juju to name a few -- available for automatically deploying an Openstack cloud. My recent project involved evaluating community developed Chef cookbooks for deploying Openstack in a data center. During the evaluation process, I encountered a number of issues which were not address by any of the open source community projects: 

• The development environments used by community do not resemble how Openstack is deployed in production. For example, a typical production environment deploy multiple compute nodes, DNS services,  LDAP, clustered databases and message queue. However, all of the development environment widely used by the community only use either an all-in-one or simple two-node configuration. 
• Simulating production-like environment with Virtual Box or VMWare is too resource consuming and impractical. 
• In a collaborative team environment, we would like to easily build and share the common infrastructure services such as, LDAP, database and message queue across the team. This is rather difficult to achieve using VM images or snapshots. 
• Once Openstack is installed, there is no automated way to validate functionality of Openstack services. 
• Existing tools do not allow developers to only test the components they are working on. This means even to test a single line of code change, one needs to rebuild an entire cluster, a process that takes a significant toll of development productivity. 

This post describes how we addressed the aforementioned issues by building a continuous integration framework using Docker containers.

Why Docker? 

Docker containers offer a very lightweight alternative to virtual machines and allow easy deployment and packaging of application/services. Since containers are lightweight, we can have many containers running on a laptop without experiencing significant resource crunches. Second, using Docker registry service (Docker hub), it is easy to build certain components, for example Jenkins server and Tempest, and share the container images with other developers. Using these images developers can easily bring up common infrastructure services on their development environment and focus more on the components they are working on. Third, once a service is installed in a container, we can take a snapshot of the component using Docker's image service. By taking a snapshot or image of a container, we can avoid building certain services, for example MySQL, if there are no code changes for that component.

Deployment model

For our setup, we used a single Vagrant box running Fedora 20 and ran five containers in it. This include:

• Chef server
• Openstack controller
• Openstack compute
• Tempest (for testing Openstack)
• Jenkins

To simulate our production environment as closely as possible, all containers had CentOS 6.5 image. 

The Vagrant box ran Docker engine and a DNS service. Since Docker randomly assigns IP addresses to its containers, we chose to use a DNS service and registered all Docker containers with this service. We wrote a script to automate hostname registration of all Docker containers with this DNS service. As a result, from one container we could reach any other container by its hostname. For example, one can reach the Chef server from any other containers using its hostname chef.mydomain.com.

The Openstack controller and compute nodes are provisioned using the Chef server. For Openstack controller and compute nodes, we created a custom Docker image which had Chef client pre-installed. This allowed us to easily launch these containers and run a single command to provision them to their desired role.

All Docker images used for this project (Chef server, Chef client, Tempest and Jenkins) are available at the Docker hub (https://hub.docker.com/u/imtiaz/).

Running continuous integration jobs

To allow developers to quickly test their code changes in local environment, we setup a Jenkins server. While running a Jenkins locally seem to be a bit of overkill, it provides the following added benefits over simple automation scripts: 

1. Nice user interface.
2. Central place for storing and viewing logs. One can view any Chef run failures without having to log into the the servers.
3. JUnit plugin for Jenkins offers a great way to compare test results and execution time from one build to another. 

To bring up the entire CI framework, we perform the following steps:

1. Launch Jenkins
2. Launch Chef server
3. Launch a container with Chef client image
4. Provision it to become an Openstack controller
5. Repeat steps 3-4 with different Chef roles to create an Openstack compute node.
6. Repeat steps 3-4 with different Chef roles to create a Tempest node.
7. Run Tempest test cases and publish results.

The follow sequence diagram explains the process:

Demo

Here is a video demo showing how the CI framework works. I apologize in advance for failing to capture the entire screencast. 

Challenges

While Docker allowed us to setup a continuous integration framework with only one virtual machine, achieving this was not without pain. In order to build a stable and repeatable environment, I had to troubleshoot many complex issues and at one point even recompiled entire Linux kernel. In my next post, I shall cover the issues I faced to be able to run a fully functional Openstack on Docker.

How to create base docker images for CentOS

Background

Docker is a lightweight container technology that is gaining a lot of momentum in the development community. Compared with traditional methods of virtualization, Docker allows rapid prototying, building and shipping an application.

I used Docker to simulate the production environment on my laptop. First step in this process is to have an base image that replicates the virtual or real machines used in production. In this blog, I describe how to create base Docker images from Kickstart files which can be used for creating Openstack/AWS virtual machines or Docker container.

Pre-requisite

1. Start a CentOS virtual machine (on VirtualBox, VMWare Fusion or Vagrant). You can download CentOS6.5 virtual machines from [https://github.com/opscode/bento]
2. Login to the machine.
3. Add the EPEL repository from where we shall install the tools required to create images.

   sudo yum install -y http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
   

Creating base image

1. Install image creation tools on Centos VM.

   sudo yum install -y appliance-tools libguestfs-tools
   

2. Create an image in QCOW2 format appliance-creator can be used to automatically install a virtual machine using the kickstart file.

   sudo appliance-creator -c container-65.ks -d -v -t /tmp \
   -o /tmp/myimage --name "centos65" --release "6.5" \
   --format=qcow2;
   

3. Export from QCOW2 to tar

   sudo virt-tar-out -a /tmp/myimage/centos65/centos65-sda.qcow2 / - | gzip --best > centos65.tar.gz
   

4. Import image to docker registry

   cat centos65.tar.gz | sudo docker import - imtiaz/centos65:latest
   

5. Optional step: Publish image to docker hub (public image repository for docker images)

   sudo docker push imtiaz/centos65

How to setup stand-alone Openstack Heat service

Background

Heat is an orchestration service from Openstack. While this service can be installed on an Openstack controller along with other services, Heat can run as an independent service and can be used to provision any remote clouds. This is useful, for example  if you want to deploy a cloud application based HOT or AWS CloudFormation template on an Openstack cloud. 

Setting up Heat service (on Mac laptop)

Pre-requisites

• MySQL (5.5)
• RabbitMQ (or any other message queue)
• Python-mysql module

NOTE According to the documentation, it should be possible to setup Heat without having to install MySQL or RabbitMQ but I ran into a number of issues in trying to do that. 

Install MySQL

Download MySQL 5.5 64 bit and install on your Mac. I used the DMG package that is available. Please note, newer versions of MySQL didn't work with Heat and required a lot of configuration to get it to work. 

Once installed, you can start MySQL service from "Systems Preferences" Window. 

Create Heat DB on MySQL

After installing MySQL, create a heat DB. 

> mysql -u root CREATE DATABASE heat CHARACTER SET utf8 ; GRANT ALL PRIVILEGES ON heat.* TO 'heat'@'localhost' IDENTIFIED BY 'heat_dbpass'; GRANT ALL PRIVILEGES ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heat_dbpass';

Install Message Queue

Download RabbitMQ and install. 

Start the RabbitMQ service. 

Install Python MySQL 

Installing MySQL client on Mac proved quite challenging as it required some compilation of some libraries and Mac's gcc compiler didn't understand the compiler flags. Here are are steps to overcome these issues: 

1. Install XCode 4.1 (or latest) from App store. 
2. Install MySQL python client using the following command. 

export DYLD_LIBRARY_PATH=/usr/local/mysql/lib export PATH=/usr/local/mysql/bin:$PATH sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install mysql-python

Installing Heat Service

1. Check out the source from github.
   

   git clone https://github.com/openstack/heat.git

   
   
2. (Optional) Install Heat by running sudo python setup.py install from the heat directory
   
   

Configuring Heat

In order to run Heat services, it must be registered with Keystone. This Keystone service can run locally (on Mac) or anywhere else (a Vagrant box for example). The following steps show how: 

export SERVICES_TENANT_NAME="service" # or 'services'  #set HEAT_HOSTNAME to IP address of the server running Keystone. export HEAT_HOSTNAME=   $ keystone user-create --name heat --pass ${HEAT_USER_PASSWORD_OF_CHOICE} --tenant ${SERVICES_TENANT_NAME} $ keystone user-role-add --user heat --role admin --tenant ${SERVICES_TENANT_NAME} $ keystone service-create --name heat --type orchestration   # Note the service Id after this command. This is the HEAT_SERVICE_ID used in the commands to follow.  $ keystone service-create --name heat-cfn --type cloudformation $ keystone endpoint-create --region RegionOne --service-id ${HEAT_CFN_SERVICE_ID} --publicurl "http://${HEAT_HOSTNAME}:8000/v1" --adminurl "http://${HEAT_HOSTNAME}:8000/v1" --internalurl "http://${HEAT_CFN_HOSTNAME}:8000/v1" $ keystone endpoint-create --region RegionOne --service-id ${HEAT_SERVICE_ID} --publicurl "http://${HEAT_HOSTNAME}:8004/v1/%(tenant_id)s" --adminurl "http://localhost:8004/v1/%(tenant_id)s" --internalurl "http://localhost:8004/v1/%(tenant_id)s" # where HEAT_SERVICE_ID is the ID of the Heat service

Setting Heat.conf file

In the etc/heat/heat.conf file, set the properties for database, RabbitMQ and Keystone endpoints. The following example shows the minimum required configuration to start Heat services.  

[DEFAULT] # plugin_dirs=/Users/imtiaz.chowdhury/git/heat/contrib   sqlite_db=heat.sqlite sqlite_synchronous=false   rabbit_host=localhost rabbit_port=5672 rabbit_userid=guest rabbit_password=guest rabbit_virtual_host=/     [auth_password]   [database] sql_connection=mysql://heat:heat_dbpass@localhost/heat   [keystone_authtoken] auth_host=localhost auth_port=35357 auth_protocol=http auth_uri=http://localhost:35357/v2.0 admin_user=heat admin_password=heat_password admin_tenant_name=services

Starting Heat

1. Start Heat API service. This will start the REST API services
   

   heat-api --d --config-dir <directory of the Heat configuration, e.g. /Users/imtiaz/git/heat/etc/heat >

2. Start Heat engine.
   

   heat-engine --d --config-dir <directory of the Heat configuration, e.g. /Users/imtiaz/git/heat/etc/heat >

Verifying Heat installation

Now that Heat is installed, you should be able run Heat client and get stack information. First, you'll need to install the Heat client. 

sudo pip install python-heatclient

Next set the username, password and Keystone endpoint the Heat client will use to authenticate before running any Heat commands. For example,

OS_PASSWORD=b6211c965173421b OS_AUTH_URL=http://localhost:5000/v2.0/ OS_USERNAME=admin OS_TENANT_NAME=admin

NOTE The above setting will allow Heat to create resources for admin project as admin user. If you want to use Heat to create resources for other tenants then you should use user credentials associated with that tenant 

Running heat stack-list now return an empty list. 

You should now be able to launch VM instances or create other resources by using a Heat template. Many sample templates are available on https://github.com/openstack/heat-templates.