Wistar is an open-source network emulator originally developed by Juniper Networks and released under the Apache license. It simplifies the presentation of Juniper products on its graphical user interface by making the multiple VMs that make up each JunOS virtual router appear as one node in the network topology.
Wistar also supports Linux virtual machines and, interestingly, uses cloud-init to configure Linux routers from the Wistar user interface. Wistar also supports generic virtual appliances, in a basic way. In this post, I will install Wistar and use it to work through two examples using open source routers.
Wistar Documentation
The Wistar installation procedure is documented in the Wistar GitHib page. The Wistar user guide is available at the Read the Docs website and some unpublished chapters are available on GitHub. Juniper published a presentation about using Wistar. In addition, there are a few other other blog posts available about using Wistar and comparing Wistar to other network emulators.
- Wistar GitHub page: https://github.com/Juniper/wistar
- Wistar user guide: https://wistar.readthedocs.io/en/latest/
Wistar documentation is good enough to get started, but seems to be incomplete.
Install Wistar
I installed Wistar on my laptop computer running Ubuntu 18.04 LTS. I modified the Wistar installation procedure a little bit to make Wistar run on my Ubuntu Linux 18.04 laptop. Mainly, I incorporated Python virtual environments into my installation process.
You must first install Python 2 on Ubuntu 18.04. Wistar is written in Python 2. ^[Python 2 will reach the end of life on January 1st, 2020. I asked on the Wistar Slack channel about plans to upgrade the code base to Python 3 but received no response by the time I published this post.]
$ sudo su
# apt update
# apt -y upgrade
# apt -y install python-pip python-dev
Install development tools:
# apt -y install build-essential libz-dev \
libxml2-dev libxslt1-dev libffi-dev \
libssl-dev libxml2-dev libxslt1-dev git
Install KVM and Libvirt:
# apt -y install qemu-kvm libvirt-clients \
libvirt-daemon-system virt-manager \
bridge-utils libguestfs-tools \
libnss-libvirt libvirt-bin socat \
genisoimage dosfstools unzip libvirt-dev
Create the directory structure for Wistar:
# mkdir -p /opt/wistar/user_images/instances
# mkdir -p /opt/wistar/seeds
# mkdir -p /opt/wistar/media
# cd /opt/wistar
Create a Python2 virtual environment and install additional Python modules in it.
# pip install virtualenv
# virtualenv .
# source /opt/wistar/bin/activate
(wistar) #
(wistar) # pip install pexpect libvirt-python netaddr markupsafe mtools
(wistar) # pip install pyvbox junos-eznc pyYAML Django==1.9.9
(wistar) # pip install cryptography websocket-client
Libvirt already creates an external bridge named virbr0 so I skipped the step in the installation guide that asks you to install a new bridge named br0.
Finally, download the Wistar Python program and install it in the virtual environment you previously created.
(wistar) # git clone https://github.com/juniper/wistar.git wistar
(wistar) # cd wistar/
(wistar) # ./manage.py migrate
The migrate command, above, sets up databases and other Wistar infrastructure.
Start Wistar web server
Start a basic webserver, built into the Django framework used by Wistar:
(wistar) # cd /opt/wistar/wistar
(wistar) # ./manage.py runserver 0.0.0.0:8080
Optionally, you may consult the Wistar instalation directions to set up more robust web server, like Apache.
Now, a basic web server is running. This terminal window will run the web server and I like to keep it available so I can see the requests it is serving.
Open a new terminal window for other commands that need to be run on the Wistar host PC.
Wistar GUI
Point your browser to the IP address on which the Wistar web server is running. Use TCP port number 8080. The IP is the IP address of the VM you created to run Wistar or, if you installed Wistar on your local machine, it is http://localhost:8080.
http://localhost:8080
In the browser window, the Wistar GUI starts at the Topologies page, as shown below. If this is your first time using Wistar, it will be blank.
Configure Wistar
Edit the /opt/wistar/wistar/wistar/configuration.py file to include the username, SSH key, and default instance password you wish configured in the Ubuntu Cloud Instances, or any other node type that can be initiaized using cloud-init. All cloud-init-enabled images will have the following configurations automatically installed on boot up.
# vi /opt/wistar/wistar/wistar/configuration.py
Set default userid and password
Note the default userid and password. You can use these or change them to a value you prefer.
default_instance_password = 'Clouds123'
ssh_user = "wistar"
Add your public SSH key
Change the public key to one you will use to login to the nodes you start in Wistar. (Note that the private key needs to be installed in the Wistar system’s /root/.ssh folder, since you need to run Wistar with elevated permissions).
ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6ZPy25MpxGnThisIsFakekeyGbKhClud5BAdrp8mE5aMYzif3g+XNRG1+KhoThisIsFakekeyGTb27mDmur37vS7JKeBThisIsFakekeyoInk1yS9CNb8GeOS2+iB/4w86SBU1IhThisIsFakekeySZ6mLpctzNFWveqcdXrL+E5UBX28hNUDPl+/JgwPB5ai8z root@ubuntu"
Change the default bridge
Change the default external bridge from br0 to virbr0. This will make Wistar automatically connect each new VM to the default Libvirt bridge named virbr0, which is a simple way to ensure every VM can access the Internet to download packages and files.
kvm_external_bridge = "virbr0"
Add images
Get some images that you will use to demonstrate a Wistar simulation scenario. In this example, use the Ubuntu Cloud Image, which supports cloud-init and can be configured directly by Wistar, and Brezular’s TinyCore router image, which comes ready to use as a router and can be used to demonstrate the “other” category of devices in Wistar.
Ubuntu Cloud image
Get the Ubuntu Cloud Image from the Ubuntu web site. Run the following commands in a terminal window on the Wistar host computer:
$ sudo su
# cd /opt/wistar/user_images/
# wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
Brezular’s TinyCore router image
Brezular’s TinyCore Linux router with FRR is stored on Google’s Drive storage service. Download it from Brezular’s Linux Core Appliances page, or download it directly from Google Drive using the Python gdown script, as shown below:
# apt install python3-pip qemu-utils
# pip install gdown
# cd /opt/wistar/user_images/
# gdown https://drive.google.com/uc?id=1VGr3JGwDVGJVxoGZoisBtoANozKVvRnM
After you download Brezular’s TinyCore router image, convert it to QCOW2 image format:
# qemu-img convert -f vmdk -O qcow2 corepure64-9.0-ffr5.0.1.vmdk coreos-9.0.1.qcow2
Brezular userids and passwords
The Brezular router comes with a user already set up. The username is tc, with no password. The system’s root password is tc.
Wistar Image Types
When you import or upload an image into Wistar you must define the image type. Wistar lists many Juniper image types in the drop-down menu but there are only three types that support Linux VMs: “Other”, “Ubuntu 16”, and “Linux”.
Other
The Other type supports QCOW2 disk images. It also tells Wistar to use the traditional interface naming scheme (eth0, eth1) when referring to ports on instances created from the image. It also assumes the image does not support cloud-init.
Instances created from the Other image type can support many interfaces. You must use the Other image type for open-source routers that will need more than two interfaces.
Ubuntu 16
The Ubuntu 16 image type is compatible with Ubuntu 18.04 Cloud images. The Ubuntu 16 type tells Wistar to use the predictable interface naming scheme (ens3, ens4) for instances created from that image, and also enables Wistar to perform basic configuration from the user interface. I suspect the Ubuntu 16 type will work for any Linux distribution that supports configuration via cloud-init.
Instances created from the Ubuntu 16 image type only support two interfaces: one management interface and one interface that connects to a router in the topology. This is an unfortunate restriction, because any Linux node can be configured as a router.
Linux
The Linux type is used for Linux distributions that use the traditional interface naming scheme, and that support cloud-init. You may also define SSH, Netconf, or console scripts that run when an instance is created from a Linux image. See the Templates menu for more options for scripts.
Instances created from the Linux image type only support two interfaces: one management interface and one interface that connects to a router in the topology.
Import images into Wistar
In the Wistar user interface, click on the Images link.
If you are running Wistar on your local Linux PC and you already copied image files to your local disk, as I have done, then click on Define Local Image. If you are working on a VM or with a remote server, you may use the Upload Image menu command to copy image files from your local PC to the VM or server.
Brezular router image (Other type)
Import the Brezular TinyCore router image into Wistar. Click on the Images link, then click on Define Local Image.
In the New Local Image screen, give the image a name, choose the type, and enter the location of the image file. In the example below, you import the Brezular router image as type Other:
Click on Submit Query. When Wistar registers the local image into its database.
Ubuntu 18.04 Cloud Image (Ubuntu16 type)
Repeat the Define Local Image process for the Ubuntu 18.04 Cloud Image. This time, choose the type Ubuntu 16.
In the Images screen, you should see all the images you imported into Wistar:
Create a network topology
Create a network topology. Click on Topologies and then choose Create Topology from the menu.
A blank canvas appears. You may add virtual machine instances or bridges to the network and connect them together. You may also annotate the network topology by adding labels and shapes.
Add a VM
Add the first node to the topology by clicking on the Add VM link.
In the VM configuration panel, name the VM instance and choose its base image — which is one of the two images you previously imported. In the example below, we chose the Ubuntu Cloud Image.
After you click on the Add and Close button, you will see a new server icon on the canvas.
Bridges
Because I configured the default external bridge to be virbr0 in Wistar’s configuration.py file, Wistar will connect each new VM created to the bridge virbr0. So, I do not need to add a bridge to the topology to enable external connectivity.
I might add a private bridge later, to test how they work.
Save and run the topology
Click on the Save link to save the topology. In the dialog box, give the topology a name and description, then click Save.
The next screen that appears is full of options. Frankly, the user interface is a bit of a mess, with buttons all over the place and, if your browser window is too small, you will not see some of them, like the Re-Deploy button on the bottom right of the screen.
To define the VMs in the host’s hypervisor, click on the Deploy to Hypervisor button. This will define the Ubuntu VM you created in Libvirt.
The screen changes to show more information and options. For example, you can now see the KVM status of each element in the topology. You may start individual VMs by clicking on the buttons next to them in the KVM Deployment Status section, or you may start all VMs in the topology by clicking on the Start Topology button, as shown below:
You will be asked to enter the number of seconds between starting domains. Change the number of seconds between starting domains to a low number, like 10 seconds, when working with small numbers of open-source routers.
Next, it asks you if you are sure you want to start all routers at the same time. This warning must be hard-coded into Wistar. I just click OK:
You can now see that the VM has been defined in Libvirt and is running.
You can confirm this by opening a terminal window on the Wistar host PC and running the virsh list command:
# virsh list
Id Name State
----------------------------------------
13 t5_ubuntu1 running
Login to network nodes
Wistar offers many ways to send commands to running instances. In the Wistar user interface, you may click on an icon next to any node to open a VNC console. You may also run commands on nodes directly from the Wistar GUI. In addition, you may access nodes from a terminal on the Wistar host PC using either SSH or the Libvirt console. I’ll descibe all the access methods below.
Access methods differ depending on image type
Instances created from the Ubuntu 16 and Other image types support VNC access, terminal access, and the Linux CLI Automation function. The Linux image type supports two additional tools to run CLI commands from the Wistar GUI: Execute CLI and Instance Tools. I think these additional options are missing from the Ubuntu 16 and Other types due to a bug in Wistar. This is a bit of a shame, since they make it easier to quickly interact with nodes in the topology.
VNC Console
To log into an instance using the VNC console, click on the VNC icon next to the instance name in the Wistar GUI, as shown below:
This will open up a pop-up window in your browser that displays the VNC console of the node:
NOTE: VNC console does not work if you run Wistar in a VM or on a remote server. It seems that Wistar does not allow for firewall or NAT issues.
SSH
Wistar does not support SSH directly from its user interface. You must open a terminal window on the Wistar host PC to use SSH. I prefer to use SSH to connect to the VMs created by Wistar because it allows me to use copy-and-paste.
To use SSH, you must first determine the management IP address assigned to the VM. You should see the IP address of the VM displayed in the Wistar topology canvas. In this example, you should see that the Ip address of the node ubuntu1 is 192.168.122.12.
We already know the ubuntu1 node’s userid is wistar and the password is Clouds123 so, we login with the command:
# ssh [email protected]
Libvirt console
There is no built-in console function in Wistar but, since Wistar uses Libvirt to implement the instances in the topology, you can open a terminal window and use the virsh console command. If SSH will not connect to a node (for example, the Brezular router needs some extra configuration to enable an OpenSSH server), then the virsh console command will work, as long as the network appliance has a serial interface configured.
First, find the VM name created by Wistar:
# virsh list
Id Name State
------------------------------------------
1 t5_ubuntu1 running
We see Wistar named our VM t5_ubuntu1. I suspect Wistar appended the suffix t5_ to the VM name because this is the third topology I created using this installation of Wistar. Then connect to the VM’s console:
# virsh console t5_ubuntu1
Login to ubuntu1 using userid wistar and password Clouds123. Then you should see the instance’s command line prompt:
ubuntu1:~$
Linux CLI Automation – all nodes
Another way to run commands on Linux instances managed by Wistar is to click on the Linux CLI Automation button, as shown below:
This opens up a dialog box that allows you to enter in a command, or string of commands separated by semicolons, that will run on all linux VMs in the topology. I can see this being useful for checking status of VM features or to start a script on every node at the same time, but it would not be suitable for configuration.
To run the command, enter it in the command window at the top of the dialog box, then click the Execute button. See an example, below, where we see output from the one running Instance: ubuntu1.:
Access methods for the Linux image type
The following access methods work only for instances created using the Linux image type. Again, I think that’s a bug. If you create nodes using the linux image type you will see the following access methods available in the Wistar GUI:
Instance Tools
The Instance Tools appear when you click on a Linux node. You may need to scroll down to see them. The Execute Script tool allows you to upload a script from your local PC and execute it on the selected node. This may be useful for pushing and running configuration or test scripts to the selected node.
Execute CLI
You may run Linux commands on instances managed by Wistar using the Execute CLI feature in the Wistar GUI. Click on the node to select it. You should see the option in the GUI change and you will have options related to the node you selected.
Under the Execute cli section, enter the command you wish to run and click on the Go button. Wistar runs this command on the node and returns the result in the window below.
This is useful for simple configurations but we need a real interactive terminal to perform most actions on a Linux node.
Testing the node’s Internet connection
Once logged into the VM instance, test its connectivity to the Internet. I tried pinging the Google DNS server and running the apt update command. Both worked.
ubuntu1:~$ ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=8.64 ms
... output deleted ...
ubuntu1:~$ sudo apt update
[sudo] password for wistar:
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
... output deleted ...
Also, I checked the bridge and virtual connections by running the following commands on the Wistar host PC:
# virsh domiflist t5_ubuntu1
Interface Type Source Model MAC
-------------------------------------------------------
vnet0 bridge virbr0 virtio 52:54:00:05:00:00
# brctl show virbr0
bridge name bridge id STP enabled interfaces
virbr0 8000.52540067f842 yes virbr0-nic
vnet0
From the results above, I see Wistar connected the node to the virbr0 bridge.
You may also verify the IP address assigned to the VM by the DHCP server running on Libvirt’s default network, which manages the bridge virbr0. Use the virsh net-dhcp-leases command:
# virsh net-dhcp-leases default
This will provide output similar to the following:
# virsh net-dhcp-leases default
Expiry Time MAC address Protocol IP address Hostname Client ID or DUID
--------------------------------------------------------------------------------------------------------
2019-04-23 17:55:15 52:54:00:05:00:00 ipv4 192.168.122.12/24 t5_ubuntu1 ff:b5:5e:67:ff:00:02:00:00:ab:11:69:67:57:54:ae:d3:12:a2
You can see the the IP address of VM t5_ubuntu1 is: 192.168.122.12.
Add a node to the topology
One node all by itself is not very interesting. Wistar allows you to add nodes to an existing topology so let’s add a router and connect it to our ubuntu1 VM.
Click on the Add Instance button at the top of the canvas.
In the Add VM dialog box, choose the TinyCore-Router image and name it router1.
The router icon appears on the canvas. Move it beside the ubuntu1 node. Then, create a link between nodes by clicking on the circle in the middle of one node and dragging the circle to the middle of the other node.
To save the new node and link in the topology and to define them in the hypervisor, click on the Re-Deply button in the lower right corner of the Wistar user interface.
You will see a warning that tells you the VMs must be restarted. Wistar will not restart the running VMs for you. The warning box is telling you that you need to restart the VMs, yourself.
It’s a bit confusing, but the warning box is really a call to action. You must destroy and start any running VM that has a link added to it or the interface will not be set up properly on that VM.
We added a link to the running VM ubuntu1. We must stop it, then start it again. Just logging into it and rebooting is not sufficient.
To stop the VM, find it in the KVM Deployment Status and click on the green button. This will power off the VM.
Wait a couple of minutes for the ubuntu1 VM to power down. Then, start the topology by clicking on the Start Topology button. This will start both nodes and, when they start, the link will be enabled at each end.
Next, we need to configure the interfaces on each VM. Click on the link between VMs to get the interface information. See the link details appear on the left side of the Wistar screen.
In this case, we see that ubuntu1 is connected on interface ens4 and router1 is connected on interface eth0. Interface ens4 refers to the second interface on ubuntu1 and eth0 refers to the first interface on router1.
The Wistar GUI option to configure IP addresses on links between source and target devices only works if they were created from the Linux or Ubuntu 16 image type. Since one of the nodes in our example was created from Other image type, we cannot set up IP addresses this way. Also, configurations made from the Wistar GUI are not permanent and will disappear if the nodes are rebooted. It is better to make configurations manually by logging into each node and editing configuration files.
Check the names of each VM. On the Wistar host PC, run the command:
# virsh list
You should see two VMs running:
Id Name State
----------------------------------------------------
15 t5_ubuntu1 running
16 t5_router1 running
Log into ubuntu1.
# virsh console t5_ubuntu1
Login with userid “wistar” and password “Clouds123”. Configure the second interface on ubuntu1, named ens4:
ubuntu1~$ sudo su
ubuntu1~# ip addr add 10.1.1.2/24 dev ens4
ubuntu1~# ip link set dev ens4 up
Exit from the console using the CTRL – ] key combination. Then, log into router1:
# virsh console t5_router1
Login with userid “tc”. No password is required. Then configure the first interface on router1, named eth0
router1~$ sudo su
router1~# ip addr add 10.1.1.3/24 dev eth0
router1~# ip link set dev eth0 up
Then test the connection to ubuntu1 by pinging the far-end IP address:
router1~# ping 10.1.1.2
PING 10.1.1.2 (10.1.1.2): 56 data bytes
64 bytes from 10.1.1.2: seq=0 ttl=64 time=0.684 ms
64 bytes from 10.1.1.2: seq=1 ttl=64 time=0.640 ms
Exit from the console using the CTRL – ] key combination.
You verified that the connection is set up and passes traffic between the two nodes in the Wistar network emulator topology.
Stopping an emulation
To stop the nodes running in a topology, click on the Pause Topology button or shut them down one at a time by clicking on the Status icon next to each instance in the KVM Deployment Status section.
After a while, the nodes will be stopped. The KVM Deployment status will change to the color red.
Stopping Wistar
The only part of Wistar that is running as a daemon is the web server. You can stop the web server by returning to the terminal window in which it is running and pressing the CTRL – C key combination.
To exit the Wistar virtual environment, run the deactivate command in the same terminal window, as shown below:
(wistar) # deactivate
Create more complex topologies
Let’s see what happens if we create a complex topology. Since we observed that adding nodes to a save topology does not work well, we’ll build a new topology that includes six nodes: three routers and three hosts.
Start Wistar
First, start Wistar’s again (assuming you deactivated it in the previous chapter). On the Wistar host PC, open a terminal window and run the following commands:
# source /opt/wistar/bin/activate
(wistar) #
(wistar) # cd /opt/wistar/wistar
(wistar) # ./manage.py runserver 0.0.0.0:8080
Then, open your web browser and enter the URL, http://localhost:8080.
Create new topology
You will see the Topology page and the list of available topologies. Click on Create Topology in the Topologies menu:
Then, click on Add VM. Choose the Ubuntu18 image and name the VM ubuntu1. Next, click on the Add Another Instance button. This will create a VM named ubuntu1 and then increment the form so a new VM named ubuntu2 is ready to be created. Click the Add Another Instance button again and see the form increment to ubuntu3. Finally, click on the Add and Close button to create the third instance and close the Add VM dialog box.
Repeat the process for the router1 VM and choose the TinyCore-Router image. Again, click the Add Another Instance button twice and finish by clicking on the Add and Close button. When you are done, you should see six nodes on the canvas, as shown below.
Reposition the nodes by clicking on them and dragging them around. Connect the nodes together by clicking and dragging the circles in the center of each node to the other nodes. I created a simple network of routers with an ubuntu VM connected to each router, as shown below. Click on the Save link.
Give the topology a name and description in the dialog box that appears.
Record port numbers
At this point, it will be helpful to understand which interfaces on each node are connected to which link. Click on each link in the topology and write down which interface are connected to the links between nodes.
I made a table I will use later when I need to configure the nodes:
| node | interface | to node | interface |
|---|---|---|---|
| ubuntu1 | ens4 | router1 | eth2 |
| ubuntu2 | ens4 | router2 | eth2 |
| ubuntu3 | ens4 | router3 | eth2 |
| router1 | eth0 | router2 | eth0 |
| router2 | eth1 | router3 | eth0 |
| router3 | eth1 | router1 | eth1 |
Start the network emulation
Define the nodes and networks that comprise the network emulation scenario by clicking on the Deploy to Hypervisor button. Then, start the network emulation by clicking on the Start Topology button.
Again, choose 10 seconds as the time to wait between starting nodes.
Configure the VMs
We’ll be using the console command to log into each VM and configure it. First, get a list of the names for each VM, according to Libvirt. On the Wistar host PC, run the virsh list command:
# virsh list
Id Name State
-------------------------------------
18 t6_router1 running
19 t6_router2 running
20 t6_router3 running
21 t6_ubuntu1 running
22 t6_ubuntu2 running
23 t6_ubuntu3 running
Since this is the sixth topology I created, Wistar added the suffix t6_ to each node in the topology.
Network plan
I want to create configuration changes that will persist after a node is rebooted. As I’ve discussed in other posts where I use the same three-router topology, you need to plan your IP address and interfaces before you start configuring nodes.
The network planning table I created for this topology is shown below.
| VM name | Interface name | IP address |
|---|---|---|
| ubuntu1 | ens3 | DHCP |
| ubuntu1 | ens4 | 10.1.1.2/24 |
| ubuntu2 | ens3 | DHCP |
| ubuntu2 | ens4 | 10.2.2.2/24 |
| ubuntu3 | ens3 | DHCP |
| ubuntu3 | ens4 | 10.3.3.2/24 |
| router1 | eth0 | 10.10.10.11/24 |
| router1 | eth1 | 10.30.30.11/24 |
| router1 | eth2 | 10.1.1.1/24 |
| router1 | eth3 | DHCP |
| router2 | eth0 | 10.10.10.12/24 |
| router2 | eth1 | 10.20.20.11/24 |
| router2 | eth2 | 10.2.2.1/24 |
| router2 | eth3 | DHCP |
| router3 | eth0 | 10.20.20.12/24 |
| router3 | eth1 | 10.30.30.12/24 |
| router3 | eth2 | 10.3.3.1/24 |
| router3 | eth3 | DHCP |
Node configurations
From the network plan, configure each node’s interfaces with the appropriate IP addresses. On the router nodes, enable a routing protocol.
Ubuntu1 configuration
On each ubuntu node, we will install test software, configure the interface connected to the router, and modify the SSh configuration so we can log in to the wistar account without using a password.
Login to the ubuntu2 VM using the command:
# virsh console t6_ubuntu1
Login with userid “wistar” and password “Clouds123”.
ubuntu1~$ sudo su
ubuntu1~# apt update
ubuntu1~# apt install -y traceroute tcpdump nmap
Copy and paste the following text into the ubuntu1 VM’s console:
bash <<EOF2
hostname ubuntu1
rm /etc/hostname
echo "ubuntu1" > /etc/hostname
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
systemctl restart sshd
rm /etc/netplan/01-netcfg.yaml
cat >> /etc/netplan/01-netcfg.yaml << EOF
network:
version: 2
renderer: networkd
ethernets:
ens3:
dhcp4: yes
ens4:
addresses:
- 10.1.1.2/24
#gateway4:
routes:
- to: 10.0.0.0/8
via: 10.1.1.1
metric: 100
EOF
chmod 644 /etc/netplan/01-netcfg.yaml
netplan apply
EOF2
Then exit the console with the CTRL – ] key combination.
Ubuntu2 configuration
Login to the ubuntu2 VM using the command:
# virsh console t6_ubuntu2
Login with userid “wistar” and password “Clouds123”.
ubuntu2~$ sudo su
ubuntu2~# apt update
ubuntu2~# apt install -y traceroute tcpdump nmap
Copy and paste the following text into the ubuntu1 VM’s console:
bash <<EOF2
hostname ubuntu2
rm /etc/hostname
echo "ubuntu2" > /etc/hostname
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
systemctl restart sshd
rm /etc/netplan/01-netcfg.yaml
cat >> /etc/netplan/01-netcfg.yaml << EOF
network:
version: 2
renderer: networkd
ethernets:
ens3:
dhcp4: yes
ens4:
addresses:
- 10.2.2.2/24
#gateway4:
routes:
- to: 10.0.0.0/8
via: 10.2.2.1
metric: 100
EOF
chmod 644 /etc/netplan/01-netcfg.yaml
netplan apply
EOF2
Then exit the console with the CTRL – ] key combination.
Ubuntu3 configuration
# virsh console t6_ubuntu3
Login with userid “wistar” and password “Clouds123”.
ubuntu3~$ sudo su
ubuntu3~# apt update
ubuntu3~# apt install -y traceroute tcpdump nmap
Copy and paste the following text into the ubuntu1 VM’s console:
bash <<EOF2
hostname ubuntu3
rm /etc/hostname
echo "ubuntu3" > /etc/hostname
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
systemctl restart sshd
rm /etc/netplan/01-netcfg.yaml
cat >> /etc/netplan/01-netcfg.yaml << EOF
network:
version: 2
renderer: networkd
ethernets:
ens3:
dhcp4: yes
ens4:
addresses:
- 10.3.3.2/24
#gateway4:
routes:
- to: 10.0.0.0/8
via: 10.3.3.1
metric: 100
EOF
chmod 644 /etc/netplan/01-netcfg.yaml
netplan apply
EOF2
Then exit the console with the CTRL – ] key combination.
router1 configuration
The router nodes are bsed on TinyCore so the configuration steps are different than normal Linux. For more information about how configuration on TinyCore works, see my post about persistent configuration changes in TinyCore Linux.
For each router, set up SSH and use FRR to configure networking and routing. The Wistar network emulator did not set up the SSH keys on the router nodes because they are the “Other” node type and are not compatible with cloud-init. In the listing below, replace the SSH public key with your own public key or SSH will not work for you.
Log into router1:
# virsh console t6_router1
Login with userid “tc” and switch to elevated privileges:
router1~$ sudo su
router1~#
Then, paste the following script into the terminal window:
sh <<EOF2
sed -i "2a echo 'router1' > /etc/hostname" /opt/bootlocal.sh
sed -i "3a hostname -F /etc/hostname" /opt/bootlocal.sh
sed -i "4a sed -i 's/box/router1/g' /etc/hosts" /opt/bootlocal.sh
ssh-keygen -A
cp /usr/local/etc/ssh/sshd_config.orig /usr/local/etc/ssh/sshd_config
sed -i 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/g' /usr/local/etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/g' /usr/local/etc/ssh/sshd_config
sed -i 's/#UsePAM no/UsePAM no/g' /usr/local/etc/ssh/sshd_config
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6ZPy25MpxGnThisIsFakekeyGbKhClud5BAdrp8mE5aMYzif3g+XNRG1+KhoThisIsFakekeyGTb27mDmur37vS7JKeBThisIsFakekeyoInk1yS9CNb8GeOS2+iB/4w86SBU1IhThisIsFakekeySZ6mLpctzNFWveqcdXrL+E5UBX28hNUDPl+/JgwPB5ai8z root@ubuntu" > /home/tc/.ssh/authorized_keys
chown tc /home/tc/.ssh/authorized_keys
chmod 600 /home/tc/.ssh/authorized_keys
/usr/local/sbin/sshd
rm /usr/local/etc/frr/babeld.conf
touch /usr/local/etc/frr/babeld.conf
rm /usr/local/etc/frr/bgpd.conf
touch /usr/local/etc/frr/bgpd.conf
rm /usr/local/etc/frr/eigrpd.conf
touch /usr/local/etc/frr/eigrpd.conf
rm /usr/local/etc/frr/isisd.conf
touch /usr/local/etc/frr/isisd.conf
rm /usr/local/etc/frr/ldpd.conf
touch /usr/local/etc/frr/ldpd.conf
rm /usr/local/etc/frr/ospf6d.conf
touch /usr/local/etc/frr/ospf6d.conf
rm /usr/local/etc/frr/ospfd.conf
cat >> /usr/local/etc/frr/ospfd.conf << EOF
frr version 5.0.1
frr defaults traditional
log stdout
router ospf
ospf router-id 1.1.1.1
passive-interface eth2
network 10.1.1.0/24 area 0
network 10.10.10.0/24 area 0
network 10.30.30.0/24 area 0
line vty
EOF
rm /usr/local/etc/frr/pimd.conf
touch /usr/local/etc/frr/pimd.conf
rm /usr/local/etc/frr/ripd.conf
touch /usr/local/etc/frr/ripd.conf
rm /usr/local/etc/frr/ripngd.conf
touch /usr/local/etc/frr/ripngd.conf
rm /usr/local/etc/frr/vtysh.conf
touch /usr/local/etc/frr/vtysh.conf
rm /usr/local/etc/frr/zebra.conf
cat >> /usr/local/etc/frr/zebra.conf << EOF
frr version 5.0.1
frr defaults traditional
hostname router1
password zebra
enable password zebra
interface eth0
ip address 10.10.10.11/24
interface eth1
ip address 10.30.30.11/24
interface eth2
ip address 10.1.1.1/24
line vty
EOF
filetool.sh -b
EOF2
Then reboot the router.
# sudo reboot
Exit the VM using the CTRL – ] key combination.
router2 configuration
Log into router2:
# virsh console t6_router2
Login with userid “tc” and switch to elevated privileges:
router2~$ sudo su
router2~#
Then, paste the following script into the terminal window:
sh <<EOF2
sed -i "2a echo 'router2' > /etc/hostname" /opt/bootlocal.sh
sed -i "3a hostname -F /etc/hostname" /opt/bootlocal.sh
sed -i "4a sed -i 's/box/router2/g' /etc/hosts" /opt/bootlocal.sh
ssh-keygen -A
cp /usr/local/etc/ssh/sshd_config.orig /usr/local/etc/ssh/sshd_config
sed -i 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/g' /usr/local/etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/g' /usr/local/etc/ssh/sshd_config
sed -i 's/#UsePAM no/UsePAM no/g' /usr/local/etc/ssh/sshd_config
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6ZPy25MpxGnThisIsFakekeyGbKhClud5BAdrp8mE5aMYzif3g+XNRG1+KhoThisIsFakekeyGTb27mDmur37vS7JKeBThisIsFakekeyoInk1yS9CNb8GeOS2+iB/4w86SBU1IhThisIsFakekeySZ6mLpctzNFWveqcdXrL+E5UBX28hNUDPl+/JgwPB5ai8z root@ubuntu" > /home/tc/.ssh/authorized_keys
chown tc /home/tc/.ssh/authorized_keys
chmod 600 /home/tc/.ssh/authorized_keys
/usr/local/sbin/sshd
rm /usr/local/etc/frr/babeld.conf
touch /usr/local/etc/frr/babeld.conf
rm /usr/local/etc/frr/bgpd.conf
touch /usr/local/etc/frr/bgpd.conf
rm /usr/local/etc/frr/eigrpd.conf
touch /usr/local/etc/frr/eigrpd.conf
rm /usr/local/etc/frr/isisd.conf
touch /usr/local/etc/frr/isisd.conf
rm /usr/local/etc/frr/ldpd.conf
touch /usr/local/etc/frr/ldpd.conf
rm /usr/local/etc/frr/ospf6d.conf
touch /usr/local/etc/frr/ospf6d.conf
rm /usr/local/etc/frr/ospfd.conf
cat >> /usr/local/etc/frr/ospfd.conf << EOF
frr version 5.0.1
frr defaults traditional
log stdout
router ospf
ospf router-id 2.2.2.2
passive-interface eth2
network 10.2.2.0/24 area 0
network 10.20.20.0/24 area 0
network 10.30.30.0/24 area 0
line vty
EOF
rm /usr/local/etc/frr/pimd.conf
touch /usr/local/etc/frr/pimd.conf
rm /usr/local/etc/frr/ripd.conf
touch /usr/local/etc/frr/ripd.conf
rm /usr/local/etc/frr/ripngd.conf
touch /usr/local/etc/frr/ripngd.conf
rm /usr/local/etc/frr/vtysh.conf
touch /usr/local/etc/frr/vtysh.conf
rm /usr/local/etc/frr/zebra.conf
cat >> /usr/local/etc/frr/zebra.conf << EOF
frr version 5.0.1
frr defaults traditional
hostname router2
password zebra
enable password zebra
interface eth0
ip address 10.10.10.12/24
interface eth1
ip address 10.20.20.11/24
interface eth2
ip address 10.2.2.1/24
line vty
EOF
filetool.sh -b
EOF2
Then reboot the router.
# sudo reboot
Exit the VM using the CTRL – ] key combination.
router3 configuration
Log into router3:
# virsh console t6_router3
Login with userid “tc” and switch to elevated privileges:
router3~$ sudo su
router3~#
Then, paste the following script into the terminal window:
sh <<EOF2
sed -i "2a echo 'router3' > /etc/hostname" /opt/bootlocal.sh
sed -i "3a hostname -F /etc/hostname" /opt/bootlocal.sh
sed -i "4a sed -i 's/box/router3/g' /etc/hosts" /opt/bootlocal.sh
ssh-keygen -A
cp /usr/local/etc/ssh/sshd_config.orig /usr/local/etc/ssh/sshd_config
sed -i 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/g' /usr/local/etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/g' /usr/local/etc/ssh/sshd_config
sed -i 's/#UsePAM no/UsePAM no/g' /usr/local/etc/ssh/sshd_config
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6ZPy25MpxGnThisIsFakekeyGbKhClud5BAdrp8mE5aMYzif3g+XNRG1+KhoThisIsFakekeyGTb27mDmur37vS7JKeBThisIsFakekeyoInk1yS9CNb8GeOS2+iB/4w86SBU1IhThisIsFakekeySZ6mLpctzNFWveqcdXrL+E5UBX28hNUDPl+/JgwPB5ai8z root@ubuntu" > /home/tc/.ssh/authorized_keys
chown tc /home/tc/.ssh/authorized_keys
chmod 600 /home/tc/.ssh/authorized_keys
/usr/local/sbin/sshd
rm /usr/local/etc/frr/babeld.conf
touch /usr/local/etc/frr/babeld.conf
rm /usr/local/etc/frr/bgpd.conf
touch /usr/local/etc/frr/bgpd.conf
rm /usr/local/etc/frr/eigrpd.conf
touch /usr/local/etc/frr/eigrpd.conf
rm /usr/local/etc/frr/isisd.conf
touch /usr/local/etc/frr/isisd.conf
rm /usr/local/etc/frr/ldpd.conf
touch /usr/local/etc/frr/ldpd.conf
rm /usr/local/etc/frr/ospf6d.conf
touch /usr/local/etc/frr/ospf6d.conf
rm /usr/local/etc/frr/ospfd.conf
cat >> /usr/local/etc/frr/ospfd.conf << EOF
frr version 5.0.1
frr defaults traditional
log stdout
router ospf
ospf router-id 3.3.3.3
passive-interface eth2
network 10.3.3.0/24 area 0
network 10.20.20.0/24 area 0
network 10.30.30.0/24 area 0
line vty
EOF
rm /usr/local/etc/frr/pimd.conf
touch /usr/local/etc/frr/pimd.conf
rm /usr/local/etc/frr/ripd.conf
touch /usr/local/etc/frr/ripd.conf
rm /usr/local/etc/frr/ripngd.conf
touch /usr/local/etc/frr/ripngd.conf
rm /usr/local/etc/frr/vtysh.conf
touch /usr/local/etc/frr/vtysh.conf
rm /usr/local/etc/frr/zebra.conf
cat >> /usr/local/etc/frr/zebra.conf << EOF
frr version 5.0.1
frr defaults traditional
hostname router3
password zebra
enable password zebra
interface eth0
ip address 10.20.20.12/24
interface eth1
ip address 10.30.30.12/24
interface eth2
ip address 10.3.3.1/24
line vty
EOF
filetool.sh -b
EOF2
Then reboot the router.
# sudo reboot
Test the network
Now you can run some simple network tests to ensure your setup is correct. List the route table on the routers. Ping from ubuntu1 to ubuntu2 and ubuntu3. Then, you may configure more complex network scenarios.
You may also test that SSH is working by connecting to nodes using ssh.
Test failure scenarios by stopping one of the routers and using the traceroute command to see that the routers converges on a new topology.
You might emulate a broken link by stopping the related bridge but the Wistar GUI does not show you which bridge is related to which link. You would have to do perform some checking at the command line. You can see which bridge is connected to a node using the virsh domiflist command. For example, if you see a bridge is connected to both router2 and router3, then you know it is the bridge related to the link between them.
Once you map out the bridges, you can use the Wistar user interface to stop and restart them.
For example, run the following commands on the Wistar host PC:
# virsh domiflist t6_router2
Interface Type Source Model MAC
---------------------------------------------------
vnet4 bridge t6_br1 virtio 52:54:00:06:00:04
vnet5 bridge t6_br2 virtio 52:54:00:06:00:06
vnet6 bridge t6_br5 virtio 52:54:00:06:00:10
vnet7 bridge virbr0 virtio 52:54:00:06:00:16
# virsh domiflist t6_router3
Interface Type Source Model MAC
---------------------------------------------------
vnet8 bridge t6_br2 virtio 52:54:00:06:00:07
vnet9 bridge t6_br3 virtio 52:54:00:06:00:0a
vnet10 bridge t6_br6 virtio 52:54:00:06:00:13
vnet11 bridge virbr0 virtio 52:54:00:06:00:17
You see that the bridge t6_br2 is the common bridge, so it the link between router2 and router3.
We can verify that with the brctl show command:
# brctl show t6_br2
bridge name bridge id STP enabled interfaces
t6_br2 8000.525400060008 yes t6_br2-nic
vnet5
vnet8
Power off the bridge in Wistar and see that you can no longer send traffic on that link.
Libvirt detaches the bridge’s virtual interfaces when you stop it and Wistar does not re-connect them to the bridge when you start the bridge again.
The brctl show command shows us that the virtual interfaces are no longer attached to the bridge:
# brctl show t6_br2
bridge name bridge id STP enabled interfaces
t6_br2 8000.525400060008 yes t6_br2-nic
You will need to manually re-attach the virtual interfaces. We know, from the data we collected above, that bridge t6_br2 was connected to vnet5 on t6_router2 and to vnet8 on t6_router3. We can reconnect them with the commands:
# brctl addif t6_br2 vnet5
# brctl addif t6_br2 vnet8
Or, you may use Libvirt commands:
# virsh detach-interface t6_router3 bridge --mac 52:54:00:06:00:07
# virsh detach-interface t6_router2 bridge --mac 52:54:00:06:00:06
# virsh attach-interface t6_router3 bridge t6_br2 --mac 52:54:00:06:00:07 --model virtio
# virsh attach-interface t6_router2 bridge t6_br2 --mac 52:54:00:06:00:06 --model virtio
This makes me wonder why Wistar provides any control of Libvirt networks in the user interface. It seems kind of useless to me.
Wistar and switching software
Wistar uses the KVM and the Linux kernel to build virtual machines and emulate network links between them. The Linux kernel blocks some switching protocols, like LLDP. This does not affect most users but, for those who are investigating Layer 2 protocols, Robin Gilijamse, a network architect and blogger working in the Netherlands, wrote a couple of posts that describe how to modify the Linux kernel to allow LLDP and other layer-2 signaling protocols to pass between VMs over a Linux bridge. I list both of his relevant blog posts, below:
Saving node configurations
Wistar creates a separate disk image for each instance in a topolgy and stores them in the /opt/wistar/user-images/intances directory. If you make configuration changes to nodes in your topology, they are saved on the instance disk images.
After you stop a topology and exit Wistar, the topology disk images are preserved. When you start the topology again, it starts the nodes from their saved disk images in the instances directory.
Exporting configurations
Commercial routers like Juniper’s may be configured by importing a text file containing configuration commands. Wistar supports exporting this file from Juniper images in the topology. Click on the Create button under Saved Config Sets in the lower left corner of the Wistar user interface.
However, Wistar does not provide any way to export Linux VM or Linux router configurations in a text format. This makes it difficult to share a topology consisting of open-source routers with colleagues working on other systems.
Conclusion
I showed you how to Wistar to emulate a network of open-source routers. I demonstrated Wistar features and showed how it integrates Libvirt as its virtualization back end.
While using Wistar, I found myself using the Libvirt command line and other terminal commands fairly often. Wistar is probably more impressive when using it with Juniper router images. The Wistar network emulator works well with open-source router images, but it has some rough edges.