I want to show you how to configure a host server so, when it is shut down, it executes a script that runs commands on any running virtual machines before the host tries to stop them. I will configure the host server to wait until the script completes configuring the virtual machines before continuing with the shutdown process, shutting down the virtual machines, and eventually powering off.
I had to learn how Systemd service unit configuration files work and some more details about how Libvirt is configured in different Linux distributions. Read on to see the solution, plus some details about how to test the solution in Ubuntu and CentOS.
Solution Summary
Install Libvirt and libguestfs-tools. Ensure that the libvirt-guests service is already started and enabled, and is configured appropriately. Create a new Systemd service named graceful-shutdown that runs a script when the host system shuts down, but before Libvirt shuts down any virtual machines.
The graceful-shutdown.service unit configuration file
Create a new Systemd unit configuration file named graceful-shutdown.service and save it in the directory, /etc/systemd/system, where it is advised you put custom configuration files.
For example:
# vi /etc/systemd/system/graceful-shutdown.service
Enter the following text into the file, then save it:
[Unit]
Description=Graceful Shutdown Service
DefaultDependencies=no
Requires=libvirt-guests.service
After=libvirt-guests.service
[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/sbin/stoplab.sh
[Install]
WantedBy=multi-user.target
Then, enable and start the service with the commands:
# systemctl enable graceful-shutdown
# systemctl start graceful-shutdown
The graceful-shutdown service will cause Systemd to run a Linux shell script, /sbin/stoplab.sh before Systemd stops the libvirt-guests service. Because the service type is oneshot, Systemd will wait until the script exits before it continues with its standard shutdown process.
If you are not familiar with the way Systemd works when starting up a Linux system and when shutting it down, you may think that the order of execution in the unit file is backwards. However, it is correct. I will explain how to read service unit files when I test the solution later in this post.
The stoplab.sh script
The file you created will cause Systemd to run a Linux shell script, /sbin/stoplab.sh. You need to create that script yourself so it meets your own needs. In my case, the stoplab script runs commands that prepare the lab virtual machines before they are shut down. I will show an example of a simple stoplab script when I test the solution.
Naturally, the script must be executable:
# chmod +x /sbin/stoplab.sh
The libvirt-guests service
The graceful-shutdown service relies on the libvirt-guests service.
If the libvirt-guests service is not running, the graceful-shutdown service will not trigger the stoplab script so the virtual machines will not be gracefully shut down. Without libvirt-guests service, Libvirt will power off the virtual machines, which may cause corruption on processes or databases running on those virtual machines.
Ubuntu
Ubuntu will enable and start the libvirt-guests service when you install Libvirt. The libvirt-guests default configuration on Ubuntu works well for my needs. It runs the shutdown command on any running virtual machines when the host is shut down and it does not restart them when the host is started again.
CentOS and Fedora
CentOS does not enable or start the libvirt-guests service when you install the Virtualization Host group package. You must enable and start the service. In addition, the libvirt-guests default configuration on CentOS causes running VMs to be suspended instead of shut down, and automatically resumes when the host start again. This is not the behaviour I want, so I recommend editing the libvirt-guests configuration file.
Verify the solution is working
To verify that the solution is working, review the Systemd log files after you shut down the host server and restart it. Use the journalctl command to read system logs. You can list all logs related to the graceful-shutdown service with the following command:
# journalctl -u graceful-shutdown
If you want to filter the logs so you see only the logs from the previous boot session, the end of which is when you shut down the system and generated the logs you want to view, use the following command:
# journalctl -b -1 -u graceful-shutdown
Persistent logs in CentOS
The above commands work in Ubuntu. However, CentOS does not enable persistent logs by default so you will lose the shutdown logs unless you set up a persistent log file.
In CentOS, run the following commands to enable a persistent log file.
# mkdir /var/log/journal
# systemd-tmpfiles --create --prefix /var/log/journal
# systemctl restart systemd-journald
Then shut down the host, restart it, and run the journalctl command to view the logs related to the graceful-shutdown service.
Summary conclusion
I covered the basics of the solution for those who may already be familiar with Systemd and with Libvirt. If you would like to know more about why I created this solution and how it works, please read on.
Graceful shutdown use case
I build open-source network emulation labs using cloud computing service providers, like Google Cloud or Microsoft Azure, that support nested virtualization on cloud instances. Each lab consists of interconnected nested virtual machines running on a single instance. Network emulation labs require a large amount of resources and can be costly to run.
To keep costs low, I ask lab users to stop their cloud instances when they are not actively using them. I want users, some of whom are beginners, to be able to stop the instances simply, by pushing a button on a web portal, or I configure the cloud instances to shut down at a scheduled time. To avoid corrupting virtual machines or losing virtual network node configurations when the instance stops, I created scripts that start when the host VM begins shutting down and that run commands on the nested virtual machines before they are shut down.
The graceful shutdown service I describe and test in this post enables me to save virtual network node configurations, avoid database corruption on virtual network appliances, and to more quickly shut down virtualized commercial network nodes that do not respond to Libvirt’s normal shutdown commands.
Save configurations
Some open-source and commercial networking operating systems allow users to make changes to a running configuration in memory without saving the changes to a persistent storage, like the system’s disk. Users must take an extra step to save the configuration. The stoplab script can run commands that log into each running virtual network node and run the appropriate commands to save running configurations to disk. This will ensure the configurations are recovered when the system starts again.
Stop databases
Before shutting down a virtual network appliance that is running a database, you should first stop the database to ensure it does not become corrupted during the shut down process. Most of the network appliances I work with come with a set of procedures required to shut down the system safely. If you have similar systems running in your virtual lab, read their documentation and integrate those procedures into your stoplab script.
Shut down virtualized commercial network appliances
The libvirt-guests service uses the virsh shutdown command, which requires that the guest operating system is configured to handle ACPI shut down requests. Commercial networking equipment operating systems do not always support ACPI shutdown requests, or may require additional configuration to accept ACPI shut down requests. Instead of allowing the libvirt-guests service to shut down the virtual network node running a commercial networking operating system, you may need to run the virsh destroy command in the stoplab script to stop the virtual network node. This will help your server to shut down more quickly.
Testing the solution
To test the graceful-shutdown service, you must first set up a Linux system with Libvirt and KVM, plus some additional virtualization tools. I describe how to set up a network emulation lab using Libvirt in a previous post. Below, I show how to set up a simple test using one nested virtual machine on both Ubuntu and CentOS.
Create a guest VM in Ubuntu
If you are using an Ubuntu system, run the following commands to install the virtualization software and set up a virtual machine.
$ sudo apt install -y qemu-kvm \
libvirt-daemon libvirt-daemon-system \
bridge-utils libguestfs-tools \
virt-manager libnss-libvirt virt-top \
libosinfo-bin sshpass expect
Set up groups so you can use Libvirt without root permissions:
$ newgrp libvirt
Then, to allow libguestfs-tools to work, change the permissions on the Linux kernel so other users can read the kernel file ((The libguestfs-tools used in this example require read access to the host’s Linux kernel file but the Ubuntu developers decided to make the Linux kernel readable only by the root user. Canonical says they did this to improve security but others strongly disagree with them. The libguestfs tools install guide refutes Canonical on this point so I set the Linux kernel file to be readable by all users on my host system, instead of running virtualization tools with root privileges. I suggest you do the same.)).
$ sudo chmod 0644 /boot/vmlinuz*
On your Ubuntu host server, run the following commands to quickly set up a simple virtual machine that we will use to test the graceful-shutdown service.
$ mkdir ~/images
$ virt-builder centos-7.6 \
--output ~/images/testvm01.qcow2 \
--arch x86_64 \
--format qcow2 \
--size 6G \
--hostname testvm01 \
--root-password password:root
$ virt-install --import --name testvm01 \
--virt-type=kvm \
--ram 2048 \
--vcpus 2 \
--disk path=~/images/testvm01.qcow2,bus=virtio,format=qcow2 \
--os-type linux \
--os-variant rhel7.6\
--network bridge=virbr0 \
--graphics none
This creates and starts a new VM named testvm01 and opens a console to the VM. Get out of the console by pressing the CTRL–] key combination.
testvm01 login:
CTRL - ]
Domain creation completed.
$
Set the new VM to autostart when the host system starts:
$ virsh autostart testvm01
Create a guest VM in CentOS
When using CentOS, I’ve found that VMs must be created by the root user in order for this solution to work.
Only Libvirt-managed VMs visible to the root user can be shut down by the libvirt-guests service during system shutdown. This may be an issue with the way CentOS handles Libvirt URI’s. ((Also see “First few steps with libvirt” in How to get started with libvirt on Linux by @rabexc.)) VMs created by a normal user could not be seen by root and, for some reason, were shut down immediately when the host shut down, instead of waiting for the graceful-shutdown and libvirt-guests services to stop, in that order. The root user is unable to run libguestfs-tools on VMs unless they are in the default Libvirt images folder, /var/lib/libvirt/images, due to a bug in CentOS that looks like it will never be fixed.
Install the virtualization software in CentOS.
$ sudo su
# yum -y install epel-release
# yum -y update
# yum groups mark convert
# yum -y group install "Virtualization Host"
# yum -y group install "Virtualization client"
# yum -y install libguestfs-tools
# systemctl enable libvirtd
# systemctl start libvirtd
# yum -y install expect openssl sshpass
Edit the /etc/sysconfig/libvirt-guests file to configure it to shut down guest VMs when the host shuts down.
# sed -i 's/#ON_SHUTDOWN=suspend/ON_SHUTDOWN=shutdown/g' \
/etc/sysconfig/libvirt-guests
# sed -i 's/#SHUTDOWN_TIMEOUT=300/SHUTDOWN_TIMEOUT=240/g' \
/etc/sysconfig/libvirt-guests
# sed -i 's/#ON_BOOT=start/ON_BOOT=ignore/g' \
/etc/sysconfig/libvirt-guests
We set the timeout to less than 5 minutes, because most cloud services shutdown timeout is five minutes so we need to complete all scripts before five minutes or the cloud service will force the cloud instance to shut down, anyway.
Then, start the libvirt-guests service:
# systemctl enable libvirt-guests
# systemctl start libvirt-guests
Since we will use Libvirt as root, we do not need to modify the groups for any users.
On your CentOS host server, run the following commands to quickly set up a simple virtual machine that we will use to test the graceful-shutdown service.
# virt-builder centos-7.6 \
--output /var/lib/libvirt/images/testvm01.qcow2 \
--arch x86_64 \
--format qcow2 \
--size 6G \
--hostname testvm01 \
--root-password password:root
# virt-install --import --name testvm01 \
--virt-type=kvm \
--ram 2048 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/testvm01.qcow2,bus=virtio,format=qcow2 \
--os-type linux \
--os-variant rhel7.6\
--network bridge=virbr0 \
--graphics none
This creates and starts a new VM named testvm01 and opens a console to the VM. Get out of the console by pressing the CTRL–] key combination.
testvm01 login:
CTRL - ]
Domain creation completed.
#
Set the new VM to autostart when the host system starts:
# virsh autostart testvm01
The stoplab.sh script
To test the graceful-shutdown service, create a stoplab.sh script that connects to the VM and runs some commands on it when the host system shuts down. I chose to name the script stoplab.sh and place it in the /sbin directory.
$ sudo vi /sbin/stoplab.sh
An example of the script is shown below. It shows two ways to send commands to a nested VM from a script running on the host. In each case, it writes the time to a file, waits 20 seconds, and writes the time again. This will show that Systemd waits until all scripts complete on the VM before continuing to shut down the host.
#!/usr/bin/env bash
# Gracefully shut down lab VM
rm -f /root/.ssh/known_hosts
echo "Starting lab server shutdown"
date
# One way to send commands to the VM
expect -c "
set timeout 120
spawn ssh -o StrictHostKeyChecking=no [email protected]
expect \"\*\?assword\" { send \"root\r\" }
expect \"# \" { send \"echo 'some commands' \>\> /root/example.txt\r\" }
expect \"# \" { send \"date \>\> /root/example.txt\r\" }
expect \"# \" { send \"sleep 40\r\" }
expect \"# \" { send \"date \>\> /root/example.txt\r\" }
expect \"# \" { send \"exit\r\" }
expect eof
"
# Another way to send commands to the VM
sshpass -proot \
ssh -o StrictHostKeyChecking=no \
[email protected] \
"echo 'some more commands' >> /root/example.txt"
sshpass -proot \
ssh -o StrictHostKeyChecking=no \
[email protected] \
"date >> /root/example.txt"
sleep 40
sshpass -proot \
ssh -o StrictHostKeyChecking=no \
[email protected] \
"date >> /root/example.txt"
sshpass -proot \
ssh -o StrictHostKeyChecking=no \
[email protected] \
"shutdown -h now"
# Shut down the VM
virsh shutdown testvm01
# Finish and clean up
echo "Finished lab server shutdown"
date
rm -f /root/.ssh/known_hosts
exit 0
Remember to make it executable:
$ chmod +x /sbin/stoplab.sh
The Systemd unit configuration file
Systemd is the initialization system used by most Linux distributions. Immediately after a Linux system boots, Systemd begins the process of starting all the programs and servers that create a running Linux system. Systemd follows instructions in the Systemd unit configuration files. Each Linux distribution may arrange these files a bit differently and administrators may modify them or create their own unit configuration files to define custom services.
We previously created a custom service by creating the file /etc/systemd/system/graceful-shutdown.service. Here, I will walk through the information in the file to show how Systemd interprets it.
The Unit section
The first section of the file is the Unit section. It tells Systemd which other services the graceful-shutdown service depends on and the order in which this service will start (or stop) in relation to other Systemd services or targets.
[Unit]
Description=Graceful Shutdown Service
DefaultDependencies=no
Requires=libvirt-guests.service
After=libvirt-guests.service
In the Unit section shown above, we see the graceful-shutdown service requires that the libvirt-guest service be running before it will start and that it will start after the libvirt-guest service starts.
In our case, we are more interested in what will happen during the system shutdown process, when Systemd will be stopping services. When services are being stopped, Systemd goes through all the services in reverse order. This means that, when we tell Systemd to the start the graceful-shutdown service after the libvirt-guests service, Systemd will do the reverse during shutdown: it will ensure the graceful-shutdown service executes any stop commands and exits before the libvirt-guest service executes its stop commands.
Also, we need to have both an After line and a Requires line in the unit file.
The Service section
The second section of the file is the Service section. This contains options that define the type of service and the commands that will run to start the programs or perform the actions related to the service. The file I created has the following options:
[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/sbin/stoplab.sh
In this example, the service type is oneshot, which will run a command when it starts and another command when it stops. In our example, we do not run any command when the service starts and we want the service to remain active so it will run a command when it is stopped during the shutdown process. So, we add in the option, RemainAfterExit=true. Finally, we define the command that will run when the service stops in the ExecStop option.
The last section of the unit file is the Install section, which tells Systemd what to do with the service when you enable it.
[Install]
WantedBy=multi-user.target
In this case, when you enable the service, Systemd will install a symbolic link in the /etc/systemd/system/multi-user.target.wants directory, which will point to the unit file associated with the service. This adds the graceful-shutdown service into the chain of files that will be parsed by Systemd when the system starts up or shuts down.
Test the graceful-shutdown service
We created the graceful-shutdown service, enabled it, and started it. We created the /sbin/shutdown.sh script and made it executable. We installed the virtualization software and ensured that the libvirt-guests service was configured and enabled.
If you are using CentOS, remember to enable persistent system logs:
$ sudo mkdir /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal
$ sudo systemctl restart systemd-journald
Now, we can shut down the host system and see how the graceful-shutdown service works. If you created your Linux system on a cloud service, go to the user portal and stop the cloud instance. After it finishes stopping, start it again.
If you are testing on a local PC, use the operating system’s restart command.
$ sudo reboot
After the system finishes restarting, login and check the system logs:
$ sudo journalctl -b -1 -u graceful-shutdown
-- Logs begin at Wed 2019-09-25 22:43:51 UTC, end at Fri 2019-09-27 16:43:24 UTC. --
Sep 27 15:10:17 ubuntu systemd[1]: Started Gracefully stop running labs.
Sep 27 15:19:55 ubuntu systemd[1]: Stopping Gracefully stop running labs...
Sep 27 15:19:55 ubuntu stoplab.sh[2738]: Starting lab server shutdown
Sep 27 15:19:55 ubuntu stoplab.sh[2738]: Fri Sep 27 15:19:55 UTC 2019
Sep 27 15:19:55 ubuntu stoplab.sh[2738]: spawn ssh -o StrictHostKeyChecking=no [email protected]
Sep 27 15:19:56 ubuntu stoplab.sh[2738]: Warning: Permanently added '192.168.122.81' (ECDSA) to the list of known
Sep 27 15:19:56 ubuntu stoplab.sh[2738]: [email protected]'s password:
Sep 27 15:19:57 ubuntu stoplab.sh[2738]: [root@testvm01 ~]# echo 'some commands' >> /root/example.txt
Sep 27 15:19:57 ubuntu stoplab.sh[2738]: [root@testvm01 ~]# date >> /root/example.txt
Sep 27 15:19:57 ubuntu stoplab.sh[2738]: [root@testvm01 ~]# sleep 40
Sep 27 15:20:37 ubuntu stoplab.sh[2738]: [root@testvm01 ~]# date >> /root/example.txt
Sep 27 15:20:37 ubuntu stoplab.sh[2738]: [root@testvm01 ~]# exit
Sep 27 15:20:37 ubuntu stoplab.sh[2738]: logout
Sep 27 15:20:37 ubuntu stoplab.sh[2738]: Connection to 192.168.122.81 closed.
Sep 27 15:21:19 ubuntu stoplab.sh[2738]: Connection to 192.168.122.81 closed by remote host.
Sep 27 15:21:19 ubuntu stoplab.sh[2738]: Domain testvm01 is being shutdown
Sep 27 15:21:19 ubuntu stoplab.sh[2738]: Finished lab server shutdown
Sep 27 15:21:19 ubuntu stoplab.sh[2738]: Fri Sep 27 15:21:19 UTC 2019
Sep 27 15:21:19 ubuntu systemd[1]: graceful-shutdown.service: Succeeded.
Sep 27 15:21:19 ubuntu systemd[1]: Stopped Gracefully stop running labs.
You can see that the graceful-shutdown service ran the stoplab.sh and sent the output from the script to the system logs.
The testvm01 VM should be running. Log into the testvm01 VM and verify that the script successfully wrote to a file on the VM before it was shut down.
$ ssh [email protected]
[email protected]'s password:
[root@testvm01 ~]# cat example.txt
some commands
some more commands
some commands
Fri Sep 27 10:32:53 EDT 2019
Fri Sep 27 10:33:33 EDT 2019
some more commands
Fri Sep 27 10:33:34 EDT 2019
Fri Sep 27 10:34:14 EDT 2019
[root@testvm01 ~]#
You can see that the script waited 20 seconds each time it wrote the dates to the file. This proves the graceful-shutdown service made Systemd wait until the script completed before continuing with the shutdown process.
Conclusion
I showed how to create a custom Systemd service that, when the host server shuts down, will ensure any running VMs are first gracefully shut down by a custom shutdown script.