Microsoft Azure unofficially supports nested virtualization using KVM on Linux virtual machines, which makes it possible to build network emulation scenarios in the cloud using the same technologies you would use if you were using your own PC or a local server.
In this post, I will show you how to set up a Linux virtual machine in Microsoft Azure and then create a nested virtual machine inside the Azure virtual machine. This is a simple example, but you may use the same procedure as a starting point to create more complex network emulation scenarios using nested virtualization.
Prerequisites
To follow this tutorial, you need an Azure account. Microsoft offers a free-trial period that provides up to $300 in credits for up to 30 days. Creating a free trial account is easy: follow the instructions at: https://azure.microsoft.com/free.
If you have not used MS Azure before, I recommend the free training offered on their web site. The first course you should take is the beginner-level Azure Administrator course, which demonstrates all the basic topics you will need to understands when managing virtual machines in Azure.
In this tutorial, I will use the Azure CLI to create and manage infrastructure in Azure, instead of using PowerShell or the Azure Portal. I find that the Azure CLI is easier to read than PowerShell. If you are using the GUI provided by Azure Portal, you can still follow along, using the CLI commands shown below as a guide. I use the Azure Cloud Shell to run my CLI commands.
Azure Shell and Azure CLI
The simplest way to start using Azure CLI is to use the web-based Azure Cloud Shell so you don’t need to install Azure CLI on your computer. If you prefer to install Azure CLI on your own PC, follow Microsoft’s documentation.
Log into Azure Cloud Shell by typing https://shell.azure.com into your browser navigation bar. Log in using your Microsoft account ID. In the upper left corner of the shell window, choose the Bash shell. This enables Azure CLI in the shell. It also supports other Linux-based applications like Ansible, Terraform, and Python so the Azure Cloud Shell is a powerful tool.
The Cloud Shell will disable itself after twenty minutes of inactivity but it saves your files in an attached storage account. So when you restart the Cloud Shell, your files and scripts will still be available.
Gather information
To create a virtual machine, you first need to understand a bit about the resources available. You need to tell Azure in which datacenter you want the VM created, so you need to get a list of all available datacenters. Then, you need to know which VM sizes are supported in the datacenter you chose, and which base images are available in the Azure Marketplace. Then you can use this information to build an Azure resource group and an Azure virtual machine.
To list all available datacenters, run the Azure CLI command:
$ az account list-locations
This produces a long list with many columns. To clean up the output, use some extra options with the command, as shown below:
$ az account list-locations \
--query '[].{Location:displayName,Name:name}' \
--output table
I use the –query option to choose specific columns of information displayed and use the –output option to format the information as a table. This will output a long list of locations. A sample of the output is shown below:
Location Name
------------------- ------------------
East Asia eastasia
Southeast Asia southeastasia
Central US centralus
East US eastus
East US 2 eastus2
West US westus
North Central US northcentralus
Scan through the list until you find a datacenter you want to use. You may choose a datacenter that is closest to you, for example.
Then, list all the available VM sizes in the datacenter. Azure supports nested virtualization only on Dv3 and Ev3 virtual machine types. So we must check that they are supported in the datacenter we chose. If they are not, choose another datacenter.
To get the list of VM sizes supported in the datacenter, run the following command. In this example, I am using the eastus datacenter.
$ az vm list-sizes \
--location eastus \
--query '[].{Name:name,CPU:numberOfCores,Memory:memoryInMb}' \
--output table | grep _v3
Above, I use the –query option to reduce the amount of information displayed and pipe the output through grep to filter only “_v3” sizes. A sample of the output is shown below.
Standard_D2s_v3 2 8192
Standard_D4s_v3 4 16384
Standard_D8s_v3 8 32768
Standard_D16s_v3 16 65536
Standard_D32s_v3 32 131072
Standard_D2_v3 2 8192
Standard_D4_v3 4 16384
Use either Ev3 or Dv3 machine types. In this example, I will use the Standard_D4s_v3 size, which offers 4 vCPUs and 16GB of memory.
Finally, list the images available from the Azure Marketplace at the datacenter you chose — in this case, eastus. As an example, look for a CentOS 7.5 image:
az vm image list \
--location eastus \
--offer centos \
--sku 7.5 \
--all \
--query '[].urn' \
--output table
As before, I use the –query option to reduce the information displayed. We need the URN of the image. The output is:
Result
---------------------------------
OpenLogic:CentOS:7.5:7.5.20180522
OpenLogic:CentOS:7.5:7.5.20180529
The latest available CentOS 7.5 image in the East US datacenter is OpenLogic:CentOS:7.5:7.5.20180529. You may substitute the version number with the text “latest” when requesting an image so use the Image URN OpenLogic:CentOS:7.5:latest when creating the VM.
Create the virtual machine
To create resources in Azure you must first create a resource group, which is like a folder that will contain your resources. It organizes your infrastructure in Azure and sets the default data center location. In the Cloud Shell window, enter the Azure CLI command:
$ az group create \
--name testRG \
--location eastus
When creating the VM, enter the information you previously gathered and also choose the VM name and user name. Use the –storage-sku option to choose the lower-priced Standard_LRS disk type.
$ az vm create \
--name Test1 \
--resource-group testRG \
--size Standard_D4S_v3 \
--image OpenLogic:CentOS:7.5:latest \
--generate-ssh-keys \
--admin-username brian \
--storage-sku Standard_LRS
Azure will use default values to create the virtual network and subnet, will create a security group, and will generate the SSH keys needed to log in to the VM.
The virtual machine takes a few minutes to start. When is starts, Azure CLI will output some information about it, as shown below:
{
"fqdns": "",
"id": "/subscriptions/abcfake3-8a56-43e0-ae4c-a38faked8dd7c/resourceGroups/testRG/providers/Microsoft.Compute/virtualMachines/Test1",
"location": "eastus",
"macAddress": "00-0D-3A-1C-79-CC",
"powerState": "VM running",
"privateIpAddress": "10.0.0.4",
"publicIpAddress": "23.96.23.54",
"resourceGroup": "testRG",
"zones": ""
}
Look through the output. You are most interested in the virtual machine’s public IP address. In this case, it is 23.96.23.54. Make a note of the public IP address.
Log in to the Azure VM
You need the private SSH key to log in to the Azure VM. When we created the VM, Azure automatically created an SSH key pair for us and stored it on the Azure storage account connected to the Cloud Shell.
In the Cloud Shell, go to the .ssh directory and list the contents.
$ cd ~/.ssh
$ ls
id_rsa id_rsa.pub
You see the private and public keys. You may copy the private key to your local PC if you want to run SSH locally — just list the contents of id_rsa and copy-and-paste it to a text file on your local PC. Or, you may connect to the Azure VM using the Cloud Shell. In this example, I am using the Cloud Shell.
Connect to the Azure VM using the SSH command. Use the SSH provate key and the username you specified when you created the VM.
$ ssh -i ~/.ssh/id_rsa [email protected]
Now we have a terminal window connected to the remote virtual machine and we are ready to configure it.
Azure Network Security Groups
When you created your virtual machine, Azure also automatically created the resources that support that VM, such as storage disk, a network, and a network security group. The network security group is a set of software firewall rules applied to each VM associated with it. For Linux VMs, the default security group’s ingress rules block all incoming traffic except SSH traffic, and its outbound rules allow all outbound connections. This is reasonably good security, especially of we use SSH keys to authenticate user access to the VM.
If you need to run other applications on the virtual machine, you may need to create some more rules that allow other applications to connect to the virtual machine. For example: web browsers or remote desktop protocol. So if you have trouble configuring an application on your virtual machine or nested virtual machines, check the network security group rules.
First, get the name of the network security group that Azure created by default:
$ az network nsg list \
--resource-group testRG \
--output table
See there is only one network security group in the resource group, and it is named, Test1NSG.
Location Name ProvisioningState ResourceGroup ResourceGuid
---------- -------- ------------------- --------------- ------------------------------------
eastus Test1NSG Succeeded testRG 1571dbf1-b3ac-48fc-a386-5ed621700da3
Then list the rules for the network security group, Test1NSG.
$ az network nsg rule list \
--nsg-name Test1NSG \
--resource-group testRG \
--output table
The output shows the firewall rules, as seen in the listing below:
Name ResourceGroup Priority SourcePortRanges SourceAddressPrefixes SourceASG Access Protocol Direction DestinationPortRanges DestinationAddressPrefixes DestinationASG
----------------- --------------- ---------- ------------------ ----------------------- ----------- -------- ---------- ----------- ----------------------- ---------------------------- ----------------
default-allow-ssh testRG 1000 * * None Allow Tcp Inbound 22 * None
We see that traffic is allowed between resources in the same virtual network in Azure and that inbound connections on port 22 are allowed. All other inbound ports are blocked. However, applications running in the Azure VM, or on a nested VM in the Azure VM, may initiate outbound connections.
Configure the Azure VM
Verify that the hardware support for virtualization is available in this virtual machine:
$ grep -cw vmx /proc/cpuinfo
4
We expect to see a value of “4” because we assigned four vCPUs to this VM.
Install software
Check for and install any updates:
$ sudo yum update
Install tmux on the Azure VM. If you are using Azure Cloud Shell to manage the virtual machine, it will disconnect after 20 minutes of inactivity and, sometimes, it hangs up for no reason. The tmux utility prevents interuption of processes started from your terminal session, if you are disconnected. Install and start tmux:
$ sudo yum install tmux
$ tmux
Install the kvm and libvirt packages needed to create the nested virtual machines.
$ sudo yum install qemu-kvm qemu-img virt-manager \
libvirt libvirt-python libvirt-client virt-install \
virt-viewer
Add your userid to the kvm and libvirt groups. In this case, the userid is brian.
$ sudo usermod -aG kvm,libvirt brian
Restart the libvirtd service
$ sudo service libvirtd restart
Create nested virtual machine
Now, create a virtual machine that will run on the Azure virtual machine. In this example, create an Ubuntu 18.04 nested virtual machine.
Create the disk image that the nested virtual machine will use.
$ mkdir Images
$ qemu-img create -f qcow2 \
/home/brian/Images/ubuntu1804.qcow2 4G
Load the VM install image from an ubuntu mirror server. Get the list of mirrors from https://launchpad.net/ubuntu/+cdmirrors. I chose the Princeton mirror because it offers high bandwidth and is close to the US East Data Center.
$ virt-install \
--name ubuntu1804 \
--ram 1024 \
--disk path=/home/brian/Images/ubuntu1804.qcow2,size=4 \
--vcpus 1 \
--os-type linux \
--network bridge=virbr0 \
--graphics none \
--location 'http://mirror.math.princeton.edu/pub/ubuntu/dists/bionic/main/installer-amd64/' \
--extra-args='console=ttyS0'
After the nested VM starts, you will see the text-based installer for Ubuntu 18.04. Follow the prompts, enter the requested information and choose the options you want.
In this example, I configured the nested VM with the server name nested, userid brian, and a password. When asked to select software, choose “OpenSSH Server” and “Basic Ubuntu Server” options.
NOTE: The nested VM Linux installation may take up to 10 minutes. If at some point during the install you stepped away from your computer for more than 20 minutes, the Cloud Shell will terminate due to lack of user activity. If this happens, restart the cloud shell, use SSH to log in to the VM and then run the tmux a command to reattach to the running tmux session. If you forgot the Azure VM’s public IP address, run the command az network public-ip list --resource-group testRG and look for the “ipAddress” field in the output.
Fix the serial interface
When the installation process ends, the nested virtual machine will reset itself. The Cloud Shell terminal will show a blank screen because it cannot access the nested VM’s serial interface.
To fix this, return to the Azure VM’s prompt by pressing the CTRL-[ key combination. Then, find the VM’s IP address with the command.
$ arp -an
Which output the VM’s ARP teable. The table should be very small, and we are only looking for IP addresses associated with the bridge named virbr0. For example:
$ arp -an
? (10.0.0.1) at 12:34:56:78:9a:bc [ether] on eth0
? (192.168.122.37) at 52:54:00:83:0f:81 [ether] on virbr0
We see the nested VM’s IP address on the bridge virbr0.
Login to the nested VM using the SSH command (In this example, I configured the userid brian on the nested VM):
$ ssh [email protected]
Now, in the nested VM, Edit the file /etc/default/grub.
nested:$ sudo nano /etc/default/grub
Add the text console=ttyS0 in the GRUB_CMDLINE_LINUX_DEFAULT parameter as shown below. After updating it, the file should look like below:
GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash console=ttyS0"
GRUB_CMDLINE_LINUX=""
Then update GRUB
nested:$ sudo update-grub
Now, exit the nested VM:
nested:$ exit
Restart the nested virtual machine:
$ virsh destroy ubunut1804
$ virsh start ubuntu1804
Connect to nested VM
Now, when you connect to the virtual machine’s console, the serial interface will function correctly.
$ virsh console ubuntu1804
You are connected to a virtual machine running inside another virtual machine on Azure. You have successfully implemented nested virtualization on Microsoft Azure.
Shut down
When your testing is complete, shut down your Azure VM to avoid additional costs. First, shut down the nested VM. If you are already connected to the nested VM’s console, disconnect from it by pressing CTRL-]. Then shut down the VM with the command:
$ virsh destroy ubuntu1804
Next, exit the Azure VM
$ exit
You are now back at the Azure CLI prompt on the Cloud Shell.
Shut down the Azure VM with the following command:
$ az vm deallocate \
--resource-group testRG \
--name Test1
Delete resource (optional)
If you are completely finished, you may delete all the resources you created simply by deleting the resource group. Deleting all resources deletes all the resources created by Azure to support the VM, which includes storage, the public IP address, and more. These resources cost very little so you do not need to worry about them using up your free trial account credits but, if you are completely done, you may delete them.
$ az group delete \
--name testRG
Conclusion
In this post, I showed you how to set up your first virtual machine in Azure and discussed a little bit about the resources created by Azure to support that virtual machine. I encouraged you to view the free training available on the Mictrosoft Azure web site, which will give you a good overview of how to work in the Azure cloud environment.
I also showed you how you can configure your virtual machine after you get it started. I showed how you can run more virtual machines nested in this virtual machine because Azure supports nested virtualization.
One nested virtual machine by itself is not very interesting, except for performance analysis. Nested virtualization allows you to run multiple nested VMs on the same cloud instance and network them together using any Linux networking technology you wish to use, such as Linux bridging or Open vSwitch. I will discuss building complex network emulation scenarios in the cloud with nested virtualization in a future post.