Google Cloud Platform introduced nested virtualization support in September 2017. Nested virtualization is especially interesting to network emulation research since it allow users to run unmodified versions of popular network emulation tools like GNS3, EVE-NG, and Cloonix on a cloud instance.
Google Cloud supports nested virtualization using the KVM hypervisor on Linux instances. It does not support other hypervisors like VMware ESX or Xen, and it does not support nested virtualization for Windows instances.
In this post, I show how I set up nested virtualization in Google Cloud and I test the performance of nested virtual machines running on a Google Cloud VM instance.
Summary
I assume that you are a new users of Google Cloud. If you are already experienced with Google Cloud, you may skip to the nested-virtualization section and then to the test results.
In this post, I show how to create a Google Cloud account. I suggest you take advantage of the generous free trial offered by Google. Then, install the gcloud command-line tool on your PC. Initialize the gcloud tool configuration to set up networking between your PC and Google Cloud and to set up your first project. You may need to define more than one gcloud configuration if you use your PC in multiple networks.
Next, create a new VM image that has a license to use the new nested-virtualization feature. Use this image to launch new VMs. Be sure to set up your project SSH keys correctly so you can access new VMs using standard SSH clients.
Finally, I installed and ran a series of benchmark tests on baseline hardware, on Google Cloud VM instances and on nested VMs running on Google Cloud instances.
Create Google Cloud account
Sign up for a free trial on Google Cloud. Google offers a generous three hundred dollar credit that is valid for a period of one year. You pay nothing until either you have consumed $300 worth of services or one year has passed. I have been hacking on Google cloud for one month, using relatively large VMs, and I have consumed only 25% of my credits.
If you already use Google services like G-mail, then you already have a Google account and adding Google Cloud to your account is easy. Just sign in with your existing Google credentials. Then set up your billing information.
Install gcloud command-line tool
At the time I am writing this post, nested virtualization is still a beta feature in Google Cloud so it cannot be enabled from the Google Cloud Console web app. We must use the gcloud Command Line Interface or the API to create a custom image that supports nested virtualization. That custom image can then be used to start instances that support nested virtualization. Also, gcloud commands provide a faster way to perform most Google Cloud operations, once you learn the CLI.
Install the gcloud command-line tool on your PC. Google has gcloud clients for Windows, Mac, and PC. In this example, I am installing the gcloud command line tool in Ubuntu Linux.
To install gcloud on my Linux PC, I executed the following commands in a Terminal window:
$ sudo apt install curl
$ export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
$ echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
$ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo apt-get update && sudo apt-get install google-cloud-sdk
Set up your Google Cloud project
Now, initialize the gcloud command to set up networking to reach Google from your local network — gcloud will help you configure proxy servers, for example — and to set up your Google Cloud project. To initialize the gcloud configuration, run the command:
$ gcloud init
Follow the prompts to set up your network connection (if needed), your Google account credentials, create a new project (in my case, I named my project, net-sims), set the default project, and set the default data center region and zone. Be sure to choose a zone that supports Haswell processors or later by default.
Run gcloud --help to see the Cloud Platform services you can interact with using gcloud. And run gcloud help <<COMMAND>> to get help on any gcloud command.
Multiple configurations
The gcloud command supports multiple configurations. You may need multiple configurations to support different networking setups. For example, you may need different network configurations if you use your PC on multiple networks, or you may want to set different defaults for various situations. Create more than one configuration by running gcloud init again and choosing Create a new configuration at the first prompt.
You can see all your gcloud configurations by running the command:
$ gcloud config configurations list
You can switch to a different configuration with the command:
$ gcloud config configurations activate <<CONFIG-NAME>>
You can view the settings of any configuration by running the command:
$ gcloud config list --configuration <<CONFIG-NAME>>
Create nested-virtualization-enabled image
The Google Cloud documentation shows how to create an image from a VM instance’s disk image. This allows you to add nested virtualization support to an existing VM but, when you want to create a new VM that supports nested virtualization, that procedure seems like one too many steps to me.
I prefer to build nested-virtualization images directly from the base images available in the Google Cloud image library.
To build a custom image based on an available base image, you must first find the image project and image family for the base image you wish to use. First list all available images using the command:
$ gcloud compute images list
This command prints a table of images to choose from. I show a few examples from the list below:
NAME PROJECT FAMILY
centos-7-v20171213 centos-cloud centos-7
ubuntu-1604-xenial-v20171212 ubuntu-os-cloud ubuntu-1604-lts
ubuntu-1710-artful-v20171213 ubuntu-os-cloud ubuntu-1610
Users enable nested virtualization by building a custom image with an enable-vmx license key added. The following command creates a new image named nested-virt using the latest Ubuntu 16.04 image in the library and the enable-vmx license key:
$ gcloud compute images create nested-virt \
--source-image-project=ubuntu-os-cloud \
--source-image-family=ubuntu-1604-lts \
--licenses="https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"
As you can see above, I specify the image family in the command to choose the latest available Ubuntu 16.04 LTS image. Since the default project is already set, I need to also specify the image’s project so the gcloud command can find it.
Now we have a new custom image saved in our default project. We will use this image to create VM instances that support nested virtualization.
Create virtual machine
Now create a new VM instance using the nested-virt image you created in the previous step.
Add SSH keys
First, create a project-level SSH key pair. Google Cloud will automatically add the public key to every VM you create in the project.
It is easier to add SSH keys using the Google Cloud Console web app. Access the web app at the following URL: https://console.cloud.google.com/home/dashboard. Then, generate an SSH key pair on your local PC. List the public key and copy it to your clipboard.
Click on the menu icon on the top right-hand corner of the web page. Click on Compute Engine, then click on “Metadata*. In the window that appears, click on SSH Keys.
You may see some SSH keys that were generated by Google. These are used when you use the built-in SSH function in the web app to access a VM instance. They are not useful when using standard SSH clients. You need to add your own public key.
Click on the Edit button. Then click on Add Item at the bottom of the screen. Paste you public key into the text box that appears. Then, edit the key so that the text at the end is exactly the same as your Google Cloud userid (in this example: I replaced text like brian@t420 with my Google Cloud userid, which is brian_not_realname). The SSH public key text should look like:
ssh-rsa AAAAB3NzaC1yc2AQEAhNRXc1RClJnotrealkeytpYuEk/FuGBLRaP29AI4BKx+notrealkeyM30AayFM9G0iN5HhfRwxUcs7hqxQKnotrealkeyTVJo0Q/8fpPy3PC3x3B+JznotrealkeyT9vTeJdedGcs7Zc673aUARCDkhijncJjufz1rFtyPlwNQd/h7NUFKjPnotrealkeyLpCrgbWZAIzrQDB8S7zZ2KAaHVM0swQJZYlGYbnotrealkyAZmeEx+dmBeJsw/CwlUWwM== brian_not_realnameThen click Save.
An important note about the format of the public SSH key: When using the Google Cloud web app, you need to modify the SSH public key you paste into the web app to that it ends with the your Google Cloud username. As you can see above, I changed the extra data at the end of the key to my Google Cloud userid, which is “brian_not_realname”. The SSH public key suffix must be your Google Cloud username.
Create VM instance using Google Console
Since we are already using the Console at this point, we will create a new VM using the Console.
Click on VM instances in the Compute Engine sidebar menu. Click on Create Instance near the top of the window. Fill in the instance name, use the default zone (unless you have a reason to change it), pick the machine size, then select the boot disk image. Click the Change button next to the Boot disk prompt.
In the window that pops up, click on the Custom Images tab at the top. Find the nested-virt image you created in the previous step and pick it using the radio button next to it. Then select the type (SSD or HD) and the size that best supports your application. I usually choose SSD and 40 GB in size. Click the Select button.
Leave all other settings at default values, unless you have a reason to change them.
Note that, before starting the VM, you may click on the Equivalent REST or command line link to see the equivalent gcloud command that would perform the same action. It’s a good idea to copy and paste this command into a text file and use it as a basis for crafting gcloud commands to start VMs in the future.
Then, click on the Create button. Within one minute, you will have a new VM running that supports nested virtualization.
You will see the VM on the VM instances page. From this page you can copy the machine’s external IP address for use in accessing it via an SSH client.
Create VM instance using gcloud commands
I find using the gcloud to be a faster way to build multiple virtual machines. It helps to have complex commands saved in a text file so you can edit them and re-use them to build more virtual machines.
When we clicked on the Equivalent REST or command line link above, we saw a long, complex gcloud command. We do not actually need most of these parameters since they merely describe the project’s default settings. Assume we will use the default project settings, we may modify and reduce the command syntax to something like:
$ gcloud compute instances create "nest-1" --zone "us-east1-b" --machine-type "n1-standard-8" --min-cpu-platform "Intel Haswell" --image "nested-virt" --boot-disk-size "40" --boot-disk-type "pd-ssd" --boot-disk-device-name "nest-1"Note that we specified the minimum CPU platform in the new version of the command. Google Cloud requires a Haswell or later processor to run instances that support nested virtualization. Users may choose a region that supports these more modern processors by default or they may select a minimum processor type when starting an instance using the gcloud CLI or the API. If you are using a zone in which the default instance type uses a platform older than Haswell, you will need to start VMs using gcloud commands so you can specify the minimum platform. That is one case where you will need to use gcloud commands to start a VM instance instead of the Console.
When the VM starts, it will output its IP address on the screen. You may use this to address connect to the VM.
NAME ZONE MACHINE_TYPE INTERNAL_IP EXTERNAL_IP STATUS
nest-1 us-east1-b n1-standard-8 10.142.0.2 203.0.113.51 RUNNINGYou may get information about the instances in your project at any time by running the gcloud command:
$ gcloud compute instances list
Connect to virtual machine
Connect to the virtual machine using an SSH client. I’ve provided examples of how to configure SSH clients for all the major operating systems in my previous post about building a virtualization server.
Alternatively, you may access the VM instance using the SSH function built into the Google Cloud Console. This is a convenient way to access VMs to troubleshoot access problems. To access the VM instance, just click on the SSH button next to it in the Console.
But, you will get more function, including X windows forwarding, by using a standard SSH client on your computer.
In my case, I will access the instance using OpenSSH from my Linux PC (note that I named my private key google:
$ ssh -X -i /.ssh/google [email protected]
Test nested virtualization
Google stated that L2 virtual machines running nested on a Google Cloud VM instance should incur less than a ten percent performance penalty. My testing shows this to be true. In fact, the performance penalty for CPU tasks is very low. I did not test the I/O performance penalty. Network emulation performance is mostly a CPU-bound problem.
After connecting to the Google Cloud VM instance, check that nested virtualization is enabled:
$ grep -cw vmx /proc/cpuinfo
You should see a non-zero result that corresponds to the number of vCPUs provided by the instance. In this example: 8.
Next I used the Cloonix network emulator to create one VM running Debian Linux and connected it to a NAT interface so it could connect to the Internet.
Next, I installed benchmark software on the L1 Google VM and the L2 Debian VM and ran the included benchmark tests.
$ sudo apt-get update
$ sudo apt-get install hardinfo
$ hardinfo &
I present my test results below.
Test results
I tested the same benchmarks on a hardware server and on different CPU platforms in Google Cloud. I saw that the differences in performance were mostly due to clock rate. The table below lists my benchmark results. All results are in seconds. Lower is better.
| VM Type | Clock (GHz) | Blowfish | Fibonacci | N-Queens | FPU FFT | FBENCH |
|---|---|---|---|---|---|---|
| Intel XEON Skylake 4-core CPU (HW baseline) | 3.7 | 1.05 | 1.07 | 0.39 | 0.63 | 2.58 |
| L1 VM with 8 vCPU on Intel XEON Skylake 4-core CPU | 3.7 | 1.1 | 1.04 | 0.39 | 0.58 | 2.9 |
| L1 Google n1-standard-8 Skylake VM | 2.0 | 1.34 | 1.66 | 0.58 | 1.22 | 6.91 |
| L2-nested-VM with 8 vCPU on Google n1-standard-8 Skylake VM | 2.0 | 1.55 | 1.57 | 0.61 | 1.04 | 8.02 |
| L1 Google n1-standard-8 Haswell VM | 2.3 | 1.22 | 1.62 | 0.57 | 1.01 | 5.47 |
| L2-nested-VM with 8 vCPU on Google n1-standard-8 Haswell VM | 2.3 | 1.43 | 1.6 | 0.58 | 0.94 | 7.87 |
I then normalized the results for clock rate. I multiplied values from the table above by the ratio of CPU clock speeds, using the 3.7 GHz XEON as the baseline. In the table below. lower values are better.
| Normalized Performance | Clock ratio | Blowfish | Fibonacci | N-Queens | FPU FFT | FBENCH |
|---|---|---|---|---|---|---|
| Intel XEON Skylake 4-core CPU (HW baseline) | 1.00 | 1.05 | 1.07 | 0.39 | 0.63 | 2.58 |
| L1 VM with 8 vCPU on Intel XEON Skylake 4-core CPU | 1.00 | 1.1 | 1.04 | 0.39 | 0.58 | 2.9 |
| L1 Google n1-standard-8 Skylake VM | 0.54 | 0.72 | 0.90 | 0.31 | 0.66 | 3.74 |
| L2-nested-VM with 8 vCPU on Google n1-standard-8 Skylake VM | 0.54 | 0.84 | 0.85 | 0.33 | 0.56 | 4.34 |
| L1 Google n1-standard-8 Haswell VM | 0.62 | 0.76 | 1.01 | 0.35 | 0.63 | 3.40 |
| L2-nested-VM with 8 vCPU on Google n1-standard-8 Haswell VM | 0.62 | 0.89 | 0.99 | 0.36 | 0.58 | 4.89 |
I charted the normalized results. In the figure below, I see that the CPU performance of L2 virtual machines is very close to the performance of the L1 instance on Google Cloud. In fact, when normalized for clock rate, Google Cloud virtual machines outperform the local hardware (except in the FBENCH test).
Conclusion
I have shown how to enable nested virtualization on a Google Cloud VM instance. I am confident that it is possible to run complex network emulation scenarios using popular network emulation tools like GNS3, EVE-NG, and Cloonix — which all use KVM virtual machines to build network nodes in their emulations. And I am confident that the performance of these emulations on cloud virtual machines will be very close to the performance that would be experienced if running them on local hardware.
At this point, users should be able to install their favorite network emulator just like they would if they were using local hardware. Then they can access their tools either using Terminal, X-windows, a web interface, or the client-server user interface models provided by GNS3 and Cloonix.
UPDATE: EVE-NG does not install in Google Cloud. EVE-NG runs scripts that change the interface names and modifies the VM’s Linux kernel. At some point, EVE-NG is incompatible with Google VM startup scripts. I need to spend some time to investigate this. Cloonix works well on Google Cloud. I have not yet tried GNS3 on Google Cloud.