Nova uses disk image cache management to handle unused base disk images, validate their integrity, etc. In this entry, let’s try to find out how unused base images are cleaned up. For context on Openstack’s libvirt base images, take a look at Pádraig’s excellent write-up on life of an OpenStack libvirt image. And a prior related post on disk image usage by nova.
Set the relevant config directives
Let’s set periodic_interval=1 — the no. of seconds between running periodic tasks; image_cache_manager_interval=1 — the no. of periodic scheduler ticks to wait between runs of the image cache manager. I’ve set both of them to ’1′ for test purpose, and grepped to ensure the change reflects
#-----# [tbox ~keystone_user1]$ sudo echo -e "periodic_interval=1\nimage_cache_manager_interval=1" >> /etc/nova/nova.conf #-----# [tbox ~keystone_user1]$ sudo egrep '^image_cache_manager_interval|^periodic_interval' /etc/nova/nova.conf periodic_interval=1 image_cache_manager_interval=1 [tbox ~keystone_user1]$ #-----#
Restart all openstack nova services, so the config changes take effect:
[root@tbox ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done
Delete a running nova instance, verify logs to see the effect of config changes
Let’s stop and delete the only nova instance running — so that nova’s image cache manager knows the image is no longer in use, and the previously enabled config options can take effect.
#-----# [tbox ~keystone_user1]$ nova list +--------------------------------------+---------+--------+-------------------+ | ID | Name | Status | Networks | +--------------------------------------+---------+--------+-------------------+ | 11c1997a-b1ff-46ef-b96c-f329576de8d6 | fedora4 | ACTIVE | net1=10.xx.yyy.zz | +--------------------------------------+---------+--------+-------------------+ [tbox ~keystone_user1]$ nova stop 11c1997a-b1ff-46ef-b96c-f329576de8d6 #-----# [tbox ~keystone_user1]$ nova delete 11c1997a-b1ff-46ef-b96c-f329576de8d6 [tbox ~keystone_user1]$ #-----# [tbox ~keystone_user1]$ nova list #-----# [tbox ~keystone_user1]$ sudo virsh list --all Id Name State ----------------------------------------------------
List out the base_images (now they should become invalid) :
[tbox ~keystone_user1]$ sudo ls -lash /var/lib/nova/instances/_base/ total 1.7G 4.0K drwxr-xr-x. 2 nova nova 4.0K Feb 18 00:41 . 4.0K drwxr-xr-x. 3 nova nova 4.0K Feb 18 04:01 .. 739M -rw-r--r--. 1 nova nova 9.8G Feb 18 04:01 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 741M -rw-r--r--. 1 nova nova 20G Feb 18 04:01 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 241M -rw-r--r--. 1 nova nova 241M Feb 18 00:41 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3.part [tbox ~keystone_user1]$
At this point, the — now invalid — base_images should’ve been deleted. Let’s look at nova’s compute logs, location: /var/log/nova/compute.log
. . 2012-02-18 04:13:03 37747 INFO nova.compute.resource_tracker [-] Compute_service record updated for interceptor.foo.bar.com 2012-02-18 04:13:03 37747 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 2012-02-18 04:13:03 37747 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2012-02-18 04:13:03 37747 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 [tbox ~keystone_user1]$
Now, it says, base-files are too young to remove:
. . 012-01-18 04:16:46 37747 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 2012-02-18 04:16:46 37747 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
Let’s drill down a bit further, by searching in upstream nova code for how young is ‘too young’:
kashyap@test$ git clone git://github.com/openstack/nova.git && cd nova
kashyap@nova$ grep FLAGS.remove_unused_original_minimum_age_seconds nova/virt/libvirt/imagecache.py
maxage = FLAGS.remove_unused_original_minimum_age_seconds
kashyap@nova$
kashyap@nova$ git grep remove_unused_original_minimum_age_seconds
etc/nova/nova.conf.sample:# remove_unused_original_minimum_age_seconds=86400
nova/virt/libvirt/imagecache.py: cfg.IntOpt('remove_unused_original_minimum_age_seconds',
nova/virt/libvirt/imagecache.py: maxage = FLAGS.remove_unused_original_minimum_age_seconds
kashyap@nova$
Ok, 86400 seconds (24 hours) appears to be the default time. Let’s, set that value to 60 seconds in nova.conf and restart all nova services again:
#-----# [tbox ~keystone_user1]$ sudo echo -e "remove_unused_original_minimum_age_seconds=60" >> /etc/nova/nova.conf #-----# [tbox ~keystone_user1]$ sudo egrep '^image_cache_manager_interval|^periodic_interval|^remove_unused_original_minimum_age_seconds' /etc/nova/nova.conf periodic_interval=1 image_cache_manager_interval=1 remove_unused_original_minimum_age_seconds=60 [tbox ~keystone_user1]$ #-----#
Restart all nova services again:
[tbox ~keystone_user1]$ sudo -i
[root@tbox ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done
Now, the /var/log/compute/compute.log indicates one of the base-files being removed, that’s good:
. . 2012-02-18 04:24:17 41389 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810 a0d1d5d3_20 2012-02-18 04:24:17 41389 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810 a0d1d5d3 /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2012-02-18 04:24:17 41389 INFO nova.virt.libvirt.imagecache [-] Removing base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3
Let’s check in the nova base images directory. Now, it seems like, two more base_images are still remaining:
[tbox ~keystone_user1]$ sudo ls -1 /var/lib/nova/instances/_base/ 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3.part [tbox ~keystone_user1]$
Let’s get back to nova compute.log, it still says ‘Unknown base file’ and ‘Base file too young to remove’:
. . 2012-02-18 04:50:45 41389 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2012-02-18 04:50:45 41389 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2012-02-18 04:50:45 41389 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
Let’s check the nova compute.log again — now it’s indicating one of the other two unused base images being deleted — from time-stamps, in 10 minutes.
. . 2012-02-18 05:01:24 41389 INFO nova.compute.resource_tracker [-] Compute_service record updated for interceptor.foo.bar.com 2012-02-18 05:01:24 41389 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2012-02-18 05:01:24 41389 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2012-02-18 05:01:24 41389 INFO nova.virt.libvirt.imagecache [-] Removing base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
Now, the only remaining one appears to be a .part image.
[tbox ~keystone_user1]$ sudo ls -1 /var/lib/nova/instances/_base/ 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3.part [tbox ~keystone_user1]$
From further investigation by Padraig , the stale .part files turned out to be a different bug (for which he posted a fix upstream) — When converting non-raw (in this case, qcow2) to raw, libvirt leaves behind the original non-raw image on disk.
Finally
Let’s turn the config directives back to sane defaults, and comment out the config directives periodic_interval, image_cache_manager_interval. And, restart all nova services to take the config change effect.
#-----#
[tbox ~keystone_user1]$ sudo sed -i 's/remove_unused_original_minimum_age_seconds=60/remove_unused_originalminimum_age_seconds=86400/' foobar1.txt
#-----#
[tbox ~keystone_user1]$ sudo sed -i 's/periodic_interval=1/\#periodic_interval=1/;s/image_cache_manager_interval=1/\#image_cache_manager_interval=1/' >> /etc/nova/nova.conf
#-----#
[root@tbox ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done
#-----#
UPDATE: I also had to set the config directive remove_unused_resized_minimum_age_seconds=60 in nova.conf, to remove the re-sized — according to the flavor chosen, in this case, it’s m1.small — raw images from the _base directory.
