Network Labs Using Nested Virtualization in the Cloud

Many open-source network simulation and emulation tools use full virtualization technologies like VMware, QEMU/KVM, or VirtualBox. These technologies require hardware support for virtualization such as Intel’s VT-x and AMD’s AMD-V. To gain direct access to this hardware support, researchers usually run network emulation test beds on their own PCs or servers but could not take advantage of the inexpensive and flexible computing services offered by cloud providers like Amazon EC2, Google Compute Engine, or Microsoft Azure.

By August 2017, most of the major cloud service providers announced support for nested virtualization. In the cloud context, Nested Virtualization is an advanced feature aimed at enterprises, but it is also very useful for building network emulation test beds. I’ve written about nested virtualization for servers before but, until recently, I was limited to running nested virtual machines on my own PC. Now that the major cloud providers support nested virtualization, I can build more complex network emulation scenarios using cloud servers.

This post will discuss the cloud service providers that support nested virtualization and how this feature supports open source networking simulation and emulation in the cloud.

Cloud service providers support for nested virtualization

The cloud service providers I investigated when writing this post were Amazon EC2, Oracle Cloud IaaS, Google Compute Engine, and Microsoft Azure IaaS. I show the results of my survey in table, below. In every case where a cloud provider supports nested virtualization for Linux virtual machines, I used a free trial account to test how it works. In all supported cases, it worked well.

Cloud provider	Nested virtualization	Level of support for Linux VMs	Free trial period	Free trial limits
Amazon EC2	No	N/A	1 year	8,760 CPU-hours
Oracle Cloud	Yes	Full support	30 days	$300 worth of services. 8 vCPU
Google Compute Engine	Yes	In Beta	1 year	$300 worth of services. 8 vCPU
Microsoft Azure IaaS	Yes	Unofficial, but it works	30 days	$250 worth of services. 4 vCPU

Amazon EC2 does not support nested virtualization in its cloud instances.

Oracle Cloud offers very robust support for nested virtualization. It also offers advanced networking features that make it easier to build complex network emulation scenarios. They even have features that support virtual labs for networking training and testing. Oracle offers a one-month free trial.

Google Cloud offers nested virtualization as a beta feature. You must execute an extra step to get an Google Cloud image that supports nested virtualization. Google Cloud also offers a very generous free trial period that lasts one year.

Microsoft Azure officially supports nested virtualization for cloud instances running Windows and unofficially supports nested virtualization for Linux instances. Azure offers a free trial period. I am using Microsoft Azure for some projects at my workplace so I am building more skills with Azure than with other providers. I’ll probably spend some more time discussing Azure in the future.

Evolution of network emulation

Until a few years ago, most networking was performed by dedicated hardware such as switches and routers. The networking hardware usually contained proprietary silicon that provided differentiating features and its software was tightly integrated with the hardware. It was expensive to build a test lab with this hardware and also to keep it updated.

Over the past decade, users learned how to run the software that comes bundled with networking hardware on low-cost servers or PCs. Skilled users could build test networks using emulation technologies such as Dynamips or QEMU, and virtualization technologies such as VirtualBox or VMware. Network emulation tools like GNS3, EVE-NG, Cloonix, and others simplified the setup and configuration of these virtual test networks. Usually, test lab networks would consist of multiple virtual nodes running on a single, powerful server but multiple servers could be connected together to build larger test networks.

As networks evolved, vendors started providing networking software that could run on standard servers to provide virtual network functions (VNF) and started offering software-defined networking (SDN) solutions. These new products are usually designed to run in virtual machines. Modern networks may consist of a combination of dedicated hardware and also standard servers running network functions.

Researchers are required to emulate larger and more complex networks as they study the operation of new networking technologies.

Network emulation in the Cloud

Researchers often cannot respond fast enough to new and changing technologies because they do not have access to powerful servers or they may have a limited number of servers. Individual researchers, like me, may find it difficult or too expensive to maintain a power-hungry servers, especially if they are not using them all the time.

When faced with problems like these, most organizations consider using cloud providers who can offer large servers on demand, rented per hour or per minute. However, until recently, cloud providers could not support complex network emulation labs.

It is not so simple to run network emulation labs in the cloud. Cloud providers usually require specific drivers to be installed in disk images that run on their hypervisors. We usually cannot modify vendors’ switch or router software images, and we may not be able to modify vendors’ VNF images so we cannot install the required drivers. In other cases, networking software may only support a specific hypervisor which may not be the same hypervisor used by the cloud provider but cloud service providers do not allow users to modify their servers or hypervisors.

Also, the network capabilities provided to cloud virtual machines may not support the types of network traffic that we would need to run between virtual nodes in a network emulation lab. For example, cloud service providers usually block multicast traffic.

Nested virtualization in the Cloud

In 2017, the major cloud service providers announced support for nested virtualization. A cloud instance that supports nested virtualization exposes the hardware acceleration features which are availabe in the base server hardware to the cloud virtual machine we are renting, which allows us to install our own hypervisor on it. We can then run multiple virtual machines “nested” inside the cloud instance.

Nested virtualization enables us to use a cloud instance as if it were a normal virtualization server so we can build virtual network emulation scenarios using a cloud instance instead of a local PC or server. We can run unmodified networking software and VNF images on any hypervisor we install and configure on the cloud instance. We can use any virtual networking technology we wish, such as Open vSwitch or Linux bridging, instead of being forced to use the cloud provider’s virtual networking technology.

We can also take advantage of the benefits of running labs in the cloud. We can build very complex virtual networking scenarios on a single cloud instance and save the entire setup on a single disk image. Then we can start a new lab using that image in only a few minutes. When we are not using the lab, we can stop the cloud instace so we incur costs only when we are using the labs we create. We can run more labs by cloning a lab disk image and starting a new virtual machine from that clone.

What about Bare Metal Cloud?

Many cloud providers also offer bare metal, or dedicated, servers. Innovative companies like Packet.net provide bare metal servers and some of the major cloud providers such as Oracle and Amazon also provide bare metal as a service. These products eliminate the need for nested virtualization because users may install their own hypervisor on a bare metal server that same way they would if that server was located in their own data center.

I don’t recommend bare metal servers for network emulation research because normal cloud instances that support nested virtualization better fit the use-case of the individual researcher who needs maximum flexibility at the lowest possible cost.

Most individual researchers and enthusiasts work on network emulation scenarios in their spare time and need to be able to suspend their work — sometimes for a long time — and then return to it when they are available. If researchers de-allocate bare metal instances to avoid costs, they are then required to rebuild a new instance from scratch when they want to continue their research. Cloud instances based on virtual machines can be shut down when not needed and then started again quickly so researchers can continue to work from the point at which they stopped without having to rebuild the lab from scratch.

Bare metal servers continue to cost the user even if they are shut down, because the hardware resources are still allocated to the user. Normal cloud instances stop costing as soon as they are shut down and, if they support nested virtualization, they can perform the same workloads as bare metal servers.

Conclusion

Now that most major cloud providers support nested virtualization, it is possible to run complex network emulation labs using cloud resources. This can greatly increase the capability of independent open-source networking researchers.