Let’s see how Nova — the compute component of OpenStack — uses a disk image from the time it was initially imported into Glance (OpenStack’s repository for virtual machine disk images) till a new virtual machine instance is booted and running.
I started by downloading a Fedora-17 disk image from here
(Keep a note of the size of the image — 241 MB )
[tuser1@interceptor ~(keystone_user1)]$ ls -lash f17-x86_64-openstack-sda.qcow2 241M -rw-rw-r--. 1 tuser1 tuser1 241M Jan 13 2012 f17-x86_64-openstack-sda.qcow2 [tuser1@interceptor ~(keystone_user1)]$
Import the above Fedora 17 disk image into Glance:
[tuser1@interceptor ~(keystone_admin)]$ glance image-create --name="fedora-17" --is-public=true \ --disk-format=qcow2 --container-format bare < f17-x86_64-openstack-sda.qcow2
(Note that I have to source Keystone admin credentials to list images from Glance).
Let’s list the current images in Glance:
tuser1@interceptor ~(keystone_admin)]$ glance image-list +--------------------------------------+-----------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+-----------+-------------+------------------+------------+--------+ | 1e6292f9-82bd-4cdb-969e-c863cb1c6692 | fedora-17 | qcow2 | bare | 251985920 | active | | acc4c853-9153-4e80-b3c8-e253451ae983 | rhel63 | qcow2 | bare | 1074135040 | active | +--------------------------------------+-----------+-------------+------------------+------------+--------+ [tuser1@interceptor ~(keystone_admin)]$
List current running Nova instances :
[tuser1@interceptor ~(keystone_user1)]$ nova list +--------------------------------------+-----------+--------+-------------------+ | ID | Name | Status | Networks | +--------------------------------------+-----------+--------+-------------------+ | 08d616a9-87a1-4c0d-b986-7d6aa5ed6780 | fedora-t1 | ACTIVE | net1=ww.xx.yyy.zz | | 3e487977-37e8-4f26-9443-d65ecbdf83c9 | fedora-t2 | ACTIVE | net1=ww.xx.yyy.zz | | 48d9e518-a91f-48db-9d9b-965b243e7113 | fedora-t4 | ACTIVE | net1=ww.xx.yyy.zz | +--------------------------------------+-----------+--------+-------------------+ [tuser1@interceptor ~(keystone_user1)]$
Let’s also list the guests running using libvirt’s virsh :
[tuser1@interceptor ~(keystone_user1)]$ sudo virsh list Id Name State ---------------------------------------------------- 12 instance-0000000c running 13 instance-0000000d running 22 instance-00000012 running [tuser1@interceptor ~(keystone_user1)]$
Find the block device in use for one of the running instances to examine it further — instance-0000000c, in this case:
[tuser1@interceptor ~(keystone_user1)]$ sudo virsh domblklist instance-0000000c Target Source ------------------------------------------------ vda /var/lib/nova/instances/instance-0000000c/disk [tuser1@interceptor ~(keystone_user1)]$
Let’s get information about the disk in use by the above nova instance; specifically, find its backing file:
[tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/instance-0000000c/disk image: /var/lib/nova/instances/instance-0000000c/disk file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 149M cluster_size: 65536 backing file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
Now, get information about the “backing file” used by the overlay (used by the running Nova instance above):
[tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 image: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 file format: raw virtual size: 20G (21474836480 bytes) disk size: 740M [tuser1@interceptor ~(keystone_user1)]$
It’s worth noting here that the original disk image initially uploaded into glance was a qcow2 image. However, the base image being used by Nova is a raw disk image.
From this, we can see: When booting a virtual machine instance for the first time, Nova does a few things:
- Make a copy of the original qcow2 disk image from Glance, convert it into a ‘raw’ sparse image & make it a base image (its location – /var/lib/nova/instances/_base) — Reason for turning non-raw to raw, refer “Background info” section below.
- Expands the size of the base image to 20GB (because, I used the m1.small flavour from Nova, when initially booting the image.)
- Use this base image & instantiate a copy on write overlay (qcow2) to boot the nova instance.
To demonstrate this post, I used Red Hat OpenStack Folsom, RHEL 6.4, single node, running these services — nova, glance, keystone, cinder; using KVM as the underlying hypervisor.
Background info:
[1] From nova’s git log, the conversion from non ‘raw’ images to ‘raw’ is tracked down to this commit
commit ff9d353b2f4fee469e530fbc8dc231a41f6fed84
Author: Scott Moser
Date: Mon Sep 19 16:57:44 2011 -0400
convert images that are not 'raw' to 'raw' during caching to node
This uses 'qemu-img' to convert images that are not 'raw' to be 'raw'.
By doing so, it
a.) refuses to run uploaded images that have a backing image reference
(LP: #853330, CVE-2011-3147)
b.) ensures that when FLAGS.use_cow_images is False, and the libvirt
xml written specifies 'driver_type="raw"' that the disk referenced
is also raw format. (LP: #837102)
c.) removes compression that might be present to avoid cpu bottlenecks
(LP: #837100)
It does have the negative side affect of using more space in the case where
the user uploaded a qcow2 (or other advanced image format) that could have
been used directly by the hypervisor. That could, later, be remedied by
another 'qemu-img convert' being done to the "preferred" format of the
hypervisor.
[2] Pádraig Brady pointed out (thank you !) this bz — https://bugs.launchpad.net/nova/+bug/932180 to note further reasons for converting images from non raw to raw.
As an aside, more informaion on disk image size allocation/ performance improvements upstream – https://blueprints.launchpad.net/nova/+spec/preallocated-images
UPDATE : Pádraig Brady writes in excellent detail about the life of an openstack libvirt image — describing Openstack — http://www.pixelbeat.org/docs/openstack_libvirt_images/
