Containerization technology is gaining traction in production systems but the benefits of it should not be downplayed for development and testing purposes. Here is a quick anecdote on how Docker has shortened the development cycle for the team working on Qstack from days to minutes.
What Docker has allowed us to do is to run Qstack within Qstack!
If we go back 3 years, when Qstack was still in development, a new developer on the Qstack dev team would have had to:
- Request access from Ops to a shared development host (where everybody ended up with root access and could get unlucky and end up on a busy host).
- Run Chef scripts to create a basic environment using Vagrant.
- Manually configure the environment for development purposes (Chef scripts were mainly designed for production deployments).
- Manually upgrade and maintain all components in their environment.
Since with many of these steps the developer needed help from someone outside the dev team and included technologies developers weren’t used to it frequently took a couple of days before a new developer had a version of Qstack up and running. This made developers infrequently upgrade their instance and that caused frequent problems with inconsistencies between developers when tracking down problems. This also had the side effect of testing of different product versions was a very painful process.
Today we can use Qstack to develop Qstack. So when a new developer starts working on Qstack the steps required are:
- Get someone to create a personal account on our internal Qstack development environment.
- Run the DENV script that creates a new Qstack instance.
The scripts takes approximately 15 minutes to run and creates a new Virtual Machine with a fully functioning Qstack environment running the version you specify!
The DENV is in the details
We have a set of scripts that allow you to create, upgrade, restart and manage your Qstack deployment. We call it a “DENV” – short for Development ENVironment.
For example the denv-new scripts goes automatically through these steps:
- Create a virtual machine using the EC2 API support in Qstack.
- Register the instance in our local DNS (so you can simply click the URL when done to log into Qstack).
- Create a management “server” Docker instance – mc0. We have a build system that builds Docker images and stores in our internal Docker Registry.
- Create two KVM instances. We have managed to wrap KVM into Docker instances to support nested Virtualization (VM in a container within a VM!).
- Create a Zone and register the KVM instances in it.
- Install a set of templates for testing. Last two steps are using our recently open sourced https://pypi.python.org/pypi/minicloudstack library.
Qstack running within Qstack
The same scripts are used in our automatic builds that nightly refresh all our test systems with all deployed versions and automatically run our integration tests to make sure there have been no regressions.
The ease of creating an new fully functional instance of Qstack has leapfrogged our testing and change validation process!
What is next?
We have a working version of our scripts that can create multiple zones, and also a version that can emulate bare metal deployment with IPMI and custom switch hardware (yes all running as Docker containers). And since we recently released a major new version of Qstack with Kubernetes support we are of course looking at leveraging Kubernetes to deploy Qstack in Qstack!