netblow¶
Introduction¶
netblow is a vendor agnostic network testing framework to stress network failures.
Why?¶
- You want to stress network failures to validate if the network control plane is converging as expected.
- Maybe you’ve just got a bleeding-edge control plane software update that hasn’t been extensively tested yet :)
- You’d like to make sure that control plane changes are first validated in a CI/CD testing environment for a couple of hours or even days, before pushing to production.
Features¶
- netblow exposes functions to stress the control plane and network failures, such as interfaces_down, interfaces_up, interfaces_flap, reboot, and config_rollback.
- These functions have a common set of arguments, which simplifies the business logic of your tests.
- You can either write your tests directly in Python or in a yml file.
- Tests can be run either asynchronously or synchronously in multiple devices.
- Devices re-connections are handled automatically.
- Data plane validation with salt minions (next release).
- Memory leak detection (next release).
Architecture¶
A general idea of how netblow is designed to perform network failures stress tests in the control plane is illustrated below.
In order to have both control of the control plane, and also to perform validation in the data/forwarding plane, netblow leverages both napalm and salt-pepper. In summary, these are the main components that compose the entire software stack:
- netblow: Testing framework that exposes network failures functions on top of napalm and facilitates the business logic of control plane tests.
- napalm: Enables netblow to manage networking devices in an agnostic manner.
- salt-pepper: Lightweight HTTP client API to interface with salt-master and salt-minions remotely to perform validation of the data/forwarding plane. As a result, this allows netblow to orchestrate tests in the data plane in conjunction with the control plane tests, which is more end-to-end oriented.
- salt-master: Responsible for managing minions.
- salt-minions: Responsible for actually running the forwarding plane tests between other minions in the topology, which is up to you to define. For example, you could simply run ping, fping, nuttcp, iperf3, or use any other testing tool, as long as it’s available in the minions.
Note
You don’t necessarily have to have salt-master and salt-minions running somewhere, they are just needed if you really need.
Installation¶
Pypi¶
pip3 install netblow --user
Upgrading¶
pip3 install netblow --user -U
Note
Legacy python (2.7) is not supported. 2020 is just around the corner, eh?
Docker¶
docker pull quay.io/viniciusarcanjo/netblow
Execution modifiers¶
netblow has the following execution mode modifiers, which are mutually exclusive:
Dry run¶
Dry run is used for validating all network stress tests calls without actually connecting in the device or committing them. You certainly want to experiment with dry run first, before starting to run the actual tests. Plus, dry run is also used to validate if all kwargs are specified correctly.
Connectivity check¶
It’s just used for validating that napalm can in fact connect with all devices in the topology. If any parameters are wrong, or authentication is not allowed you’ll see errors. You probably want to perform some connectivity check when you are first building your topology.
Once¶
Once is really useful when you just want to limit the execution of all specified tests to a single iteration. Plus, when the once mode is on, it will show napalm diffs. So, once is great for the first validating that all specified tests are indeed running as expected and quickly. As soon as you have validated your tests with the once mode, you are good to go for the tests that are supposed to last long.
Warning
Be warned that netblow actually commits the configuration in the network device if you are running in any mode other than dry run or connectivity check. Don’t run in production, unless you know what you’re doing.
Note
napalm diffs are disabled in the normal testing mode by design. netblow assumes you have validated your tests with the once mode first. Also, as a result, the execution is faster, which is important if you are trying to have little delay as possible between iterations.
Writing tests¶
Either you choose to write your tests directly in Python, by instantiating NetBlow, or in yml files and run them in the CLI (command line interface).
Code Snippets¶
Let’s say you want to perform some asynchronous interface flaps and afterwards, you want to keep tearing down the entire BGP configuration for a few seconds. In this case, I have two devices eos1 and junos1:
Note
The devices kwargs (keyword arguments) are exactly the same as documented in napalm RTD http://napalm.readthedocs.io/en/latest/base.html
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from netblow.netblow import NetBlow
def main():
"""Main func."""
topology = {
'devices': {
'eos1': {
'driver': 'eos',
'hostname': 'labhost',
'username': 'vrnetlab',
'password': 'vrnetlab9',
'optional_args': {
'port': 4443
}
},
'junos1': {
'driver': 'junos',
'hostname': 'labhost',
'username': 'vrnetlab',
'password': 'vrnetlab9',
'optional_args': {
'port': 2224
}
}
}
}
nb = NetBlow(topo=topology)
devices = ['eos1', 'junos1']
interfaces = [['Ethernet 2'], ['ge-0/0/2']]
# Interfaces flap stress async on 'eos1' and 'junos1'. Iterates 3 times.
for dut, intfs in zip(devices, interfaces):
nb.interfaces_flap(dut, interfaces=intfs, sync=False, iterations=3)
nb.await_threads() # async await
# Completely tear down current BGP synchronously during 30 secs.
cmds = [['no router bgp'], ['delete protocols bgp']]
for dut, cmds in zip(devices, cmds):
nb.config_rollback(dut, commands=cmds, duration=30)
if __name__ == "__main__":
main()
Note
You can find more snippets in the integration test folder on github.
If you were to run this Python code, you’d see an output similar to this:
~/repos/netblow/docs master*
❯ python examples/flap_rollback.py
2018-05-01 12:49:58 [MainThread] [ INFO] Devices in the topology ['eos1', 'junos1']
2018-05-01 12:49:58 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 12:49:58 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 12:49:58 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 12:50:01 [ junos1] [ INFO] Successfully connected on junos1
2018-05-01 12:50:02 [ eos1] [ INFO] Successfully connected on eos1
2018-05-01 12:50:02 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 12:50:02 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 12:50:02 [ junos1] [ INFO] Test interfaces_flap started on junos1
2018-05-01 12:50:02 [MainThread] [ INFO] Waiting for async tests to finish...
2018-05-01 12:50:02 [ eos1] [ INFO] Iteration #1/3 on eos1
2018-05-01 12:50:02 [ junos1] [ INFO] Iteration #1/3 on junos1
2018-05-01 12:50:02 [ eos1] [ INFO] Shutting interfaces ['Ethernet 2'] down
2018-05-01 12:50:02 [ junos1] [ INFO] Shutting interfaces ['ge-0/0/2'] down
2018-05-01 12:50:04 [ junos1] [ INFO] Bringing interfaces ['ge-0/0/2'] up
2018-05-01 12:50:06 [ junos1] [ INFO] Iteration #2/3 on junos1
2018-05-01 12:50:06 [ junos1] [ INFO] Shutting interfaces ['ge-0/0/2'] down
2018-05-01 12:50:08 [ junos1] [ INFO] Bringing interfaces ['ge-0/0/2'] up
2018-05-01 12:50:09 [ eos1] [ INFO] Bringing interfaces ['Ethernet 2'] up
2018-05-01 12:50:10 [ junos1] [ INFO] Iteration #3/3 on junos1
2018-05-01 12:50:10 [ junos1] [ INFO] Shutting interfaces ['ge-0/0/2'] down
2018-05-01 12:50:12 [ junos1] [ INFO] Bringing interfaces ['ge-0/0/2'] up
2018-05-01 12:50:15 [ eos1] [ INFO] Iteration #2/3 on eos1
2018-05-01 12:50:15 [ eos1] [ INFO] Shutting interfaces ['Ethernet 2'] down
2018-05-01 12:50:21 [ eos1] [ INFO] Bringing interfaces ['Ethernet 2'] up
2018-05-01 12:50:27 [ eos1] [ INFO] Iteration #3/3 on eos1
2018-05-01 12:50:27 [ eos1] [ INFO] Shutting interfaces ['Ethernet 2'] down
2018-05-01 12:50:34 [ eos1] [ INFO] Bringing interfaces ['Ethernet 2'] up
2018-05-01 12:50:41 [ eos1] [ INFO] Test config_rollback started on eos1
2018-05-01 12:50:41 [ eos1] [ INFO] Iteration #1/31536000 on eos1
2018-05-01 12:50:41 [ eos1] [ INFO] Applying commands ['no router bgp'] on eos1
2018-05-01 12:50:48 [ eos1] [ INFO] Performing rollback
2018-05-01 12:50:52 [ eos1] [ INFO] Iteration #2/31536000 on eos1
2018-05-01 12:50:52 [ eos1] [ INFO] Applying commands ['no router bgp'] on eos1
2018-05-01 12:50:59 [ eos1] [ INFO] Performing rollback
2018-05-01 12:51:03 [ eos1] [ INFO] Iteration #3/31536000 on eos1
2018-05-01 12:51:03 [ eos1] [ INFO] Applying commands ['no router bgp'] on eos1
2018-05-01 12:51:10 [ eos1] [ INFO] Performing rollback
2018-05-01 12:51:14 [ eos1] [ INFO] Duration timeout exceeded. Aborting test.
2018-05-01 12:51:14 [ junos1] [ INFO] Test config_rollback started on junos1
2018-05-01 12:51:14 [ junos1] [ INFO] Iteration #1/31536000 on junos1
2018-05-01 12:51:14 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:16 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:17 [ junos1] [ INFO] Iteration #2/31536000 on junos1
2018-05-01 12:51:17 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:19 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:20 [ junos1] [ INFO] Iteration #3/31536000 on junos1
2018-05-01 12:51:20 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:22 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:23 [ junos1] [ INFO] Iteration #4/31536000 on junos1
2018-05-01 12:51:23 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:25 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:26 [ junos1] [ INFO] Iteration #5/31536000 on junos1
2018-05-01 12:51:26 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:28 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:29 [ junos1] [ INFO] Iteration #6/31536000 on junos1
2018-05-01 12:51:29 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:31 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:32 [ junos1] [ INFO] Iteration #7/31536000 on junos1
2018-05-01 12:51:32 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:34 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:35 [ junos1] [ INFO] Iteration #8/31536000 on junos1
2018-05-01 12:51:35 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:37 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:39 [ junos1] [ INFO] Iteration #9/31536000 on junos1
2018-05-01 12:51:39 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:41 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:42 [ junos1] [ INFO] Iteration #10/31536000 on junos1
2018-05-01 12:51:42 [ junos1] [ INFO] Applying commands ['delete protocols bgp'] on junos1
2018-05-01 12:51:44 [ junos1] [ INFO] Performing rollback
2018-05-01 12:51:45 [ junos1] [ INFO] Duration timeout exceeded. Aborting test.
2018-05-01 12:51:45 [MainThread] [ INFO] Closing connections to all devices
CLI¶
netblow also ships with a CLI, which you should probably use if you’d rather write tests in yml file than writing them directly in Python.
Options¶
In addition to the execution modes, in the CLI you also have to specify the topology yml file -f and the tests yml file -t, which describes all the arguments of your tests and how they are supposed to be executed.
❯ netblow -h
usage: netblow [-h] [-d | -c | -1] [-l {info,debug}] [-v] [-f TOPOLOGY]
[-t TESTS]
netblow. Vendor agnostic network testing framework to stress network failures.
required arguments:
-f TOPOLOGY, --topology TOPOLOGY
topology yml file
-t TESTS, --tests TESTS
tests yml file
optional arguments:
-h, --help show this help message and exit
-d, --dryrun show tests calls, won't connect to any devices
-c, --concheck check connectivity with all devices in the topology
-1, --once iterates only once and perfom napalm diffs
-l {info,debug}, --level {info,debug}
logging verbosity level (default: info)
-v, --version show version
XDG-based directories¶
If you intend to also write tests in yml files, you probably want to organize these files somewhere. You could simply use the current working directory, or alternatively, XDG-based directories:
- ~/.config/netblow/topologies: yml files in this folder represent all network devices involved in the tests, which you can target individually in the command line.
- ~/.config/netblow/scenarios_tests: yml files in this folder are the actual tests specification and test execution.
Topology yml files¶
Let’s say you have a topology with two EOS devices, you can create a yml file named, for instance, eos_topo.yml:
---
devices:
eos1:
driver: 'eos'
hostname: 'labhost'
username: 'vrnetlab'
password: 'vrnetlab9'
optional_args:
port: 4443
eos2:
driver: 'eos'
hostname: 'labhost'
username: 'vrnetlab'
password: 'vrnetlab9'
optional_args:
port: 4444
Tests yml file¶
The yml tests file are composed of two main keys:
- tests_specs: specifies all tests, which are nested dictionaries that tell which function on netblow the user wants to run and which kwargs should be used.
- tests_execution: it’s a list of dictionaries that dictates how the tests should be run, and scheduled either synchronously or asynchronously. Essentially, it’s just a cross reference with the definitions in the tests_specs.
Let’s assume I have two EOS networking devices, eos1 and eos2, and I’d like to stress interface flaps. First, I have to specify how exactly I want the interface_flap kwargs for each device and then which order they are supposed to be run. In this case, I created the scenarios_tests/eos_tests.yml, which have two test_specs definitions, and three scheduled tests based on these definitions:
---
tests_specs:
eos1_interfaces_flap:
function: 'interfaces_flap'
dut: 'eos1'
interfaces: ['Ethernet 7', 'Ethernet 8']
eos2_interfaces_flap:
function: 'interfaces_flap'
dut: 'eos2'
interfaces: ['Ethernet 2', 'Ethernet 3']
tests_execution:
- tests: [eos1_interfaces_flap]
kwargs:
iterations: 2
- tests: [eos1_interfaces_flap, eos2_interfaces_flap]
kwargs:
sync: False
- tests: [eos1_interfaces_flap, eos2_interfaces_flap]
kwargs:
duration: 3
CLI Workflow¶
This section gives an example of the recommended workflow to run the eos_tests.yml test file in the eos_topo.yml mentioned in Topology yml files section:
- Dry run mode to verify the yml syntax:
❯ netblow -f topologies/eos_topo.yml -t scenarios_tests/eos_tests.yml -d
2018-05-01 13:56:37 [MainThread] [ INFO] Dry run mode
2018-05-01 13:56:37 [MainThread] [ INFO] Loading topology file /home/arcanjo/repos/netblow/topologies/eos_topo.yml
2018-05-01 13:56:37 [MainThread] [ INFO] Devices in the topology ['eos1', 'eos2']
2018-05-01 13:56:37 [MainThread] [ INFO] Loading test file /home/arcanjo/repos/netblow/scenarios_tests/eos_tests.yml
2018-05-01 13:56:37 [MainThread] [ INFO] Waiting for async tests to finish...
2018-05-01 13:56:37 [MainThread] [ INFO] Mock call trace:
2018-05-01 13:56:37 [MainThread] [ INFO] call.interfaces_flap('eos1', interfaces=['Ethernet 7', 'Ethernet 8'], iterations=2)
2018-05-01 13:56:37 [MainThread] [ INFO] call.interfaces_flap('eos1', interfaces=['Ethernet 7', 'Ethernet 8'], sync=False)
2018-05-01 13:56:37 [MainThread] [ INFO] call.interfaces_flap('eos2', interfaces=['Ethernet 2', 'Ethernet 3'], sync=False)
2018-05-01 13:56:37 [MainThread] [ INFO] call.interfaces_flap('eos1', duration=3, interfaces=['Ethernet 7', 'Ethernet 8'])
2018-05-01 13:56:37 [MainThread] [ INFO] call.interfaces_flap('eos2', duration=3, interfaces=['Ethernet 2', 'Ethernet 3'])
- Connectivity check mode:
❯ netblow -f topologies/eos_topo.yml -t scenarios_tests/eos_tests.yml -c
2018-05-01 13:56:41 [MainThread] [ INFO] Loading topology file /home/arcanjo/repos/netblow/topologies/eos_topo.yml
2018-05-01 13:56:41 [MainThread] [ INFO] Devices in the topology ['eos1', 'eos2']
2018-05-01 13:56:41 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 13:56:41 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 13:56:41 [ eos2] [ INFO] Trying to connect on eos2...
2018-05-01 13:56:44 [ eos2] [ INFO] Successfully connected on eos2
2018-05-01 13:56:45 [ eos1] [ INFO] Successfully connected on eos1
2018-05-01 13:56:45 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 13:56:45 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 13:56:45 [MainThread] [ INFO] Closing connections to all devices
- Once mode:
In this case, I don’t have a long lasting test, but it’s also super useful to see napalm diffs:
❯ netblow -f topologies/eos_topo.yml -t scenarios_tests/eos_tests.yml -1
2018-05-01 13:56:49 [MainThread] [ INFO] Loading topology file /home/arcanjo/repos/netblow/topologies/eos_topo.yml
2018-05-01 13:56:49 [MainThread] [ INFO] Devices in the topology ['eos1', 'eos2']
2018-05-01 13:56:49 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 13:56:49 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 13:56:49 [ eos2] [ INFO] Trying to connect on eos2...
2018-05-01 13:56:51 [ eos1] [ INFO] Successfully connected on eos1
2018-05-01 13:56:52 [ eos2] [ INFO] Successfully connected on eos2
2018-05-01 13:56:52 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 13:56:52 [MainThread] [ INFO] Loading test file /home/arcanjo/repos/netblow/scenarios_tests/eos_tests.yml
2018-05-01 13:56:52 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 13:56:52 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 13:56:52 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:56:59 [ eos1] [ INFO] Diff:
@@ -23,8 +23,10 @@
interface Ethernet6
!
interface Ethernet7
+ shutdown
!
interface Ethernet8
+ shutdown
!
interface Ethernet9
!
2018-05-01 13:57:01 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:57:09 [ eos1] [ INFO] Diff:
@@ -23,10 +23,8 @@
interface Ethernet6
!
interface Ethernet7
- shutdown
!
interface Ethernet8
- shutdown
!
interface Ethernet9
!
2018-05-01 13:57:11 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 13:57:11 [ eos2] [ INFO] Test interfaces_flap started on eos2
2018-05-01 13:57:11 [MainThread] [ INFO] Waiting for async tests to finish...
2018-05-01 13:57:11 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 13:57:11 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 13:57:11 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:57:11 [ eos2] [ INFO] Shutting interfaces ['Ethernet 2', 'Ethernet 3'] down
2018-05-01 13:57:17 [ eos1] [ INFO] Diff:
@@ -23,8 +23,10 @@
interface Ethernet6
!
interface Ethernet7
+ shutdown
!
interface Ethernet8
+ shutdown
!
interface Ethernet9
!
2018-05-01 13:57:17 [ eos2] [ INFO] Diff:
@@ -13,8 +13,10 @@
interface Ethernet1
!
interface Ethernet2
+ shutdown
!
interface Ethernet3
+ shutdown
!
interface Ethernet4
!
2018-05-01 13:57:20 [ eos2] [ INFO] Bringing interfaces ['Ethernet 2', 'Ethernet 3'] up
2018-05-01 13:57:21 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:57:28 [ eos2] [ INFO] Diff:
@@ -13,10 +13,8 @@
interface Ethernet1
!
interface Ethernet2
- shutdown
!
interface Ethernet3
- shutdown
!
interface Ethernet4
!
2018-05-01 13:57:29 [ eos1] [ INFO] Diff:
@@ -23,10 +23,8 @@
interface Ethernet6
!
interface Ethernet7
- shutdown
!
interface Ethernet8
- shutdown
!
interface Ethernet9
!
2018-05-01 13:57:31 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 13:57:31 [ eos1] [ INFO] Iteration #1/31536000 on eos1
2018-05-01 13:57:31 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:57:37 [ eos1] [ INFO] Diff:
@@ -23,8 +23,10 @@
interface Ethernet6
!
interface Ethernet7
+ shutdown
!
interface Ethernet8
+ shutdown
!
interface Ethernet9
!
2018-05-01 13:57:39 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:57:47 [ eos1] [ INFO] Diff:
@@ -23,10 +23,8 @@
interface Ethernet6
!
interface Ethernet7
- shutdown
!
interface Ethernet8
- shutdown
!
interface Ethernet9
!
2018-05-01 13:57:49 [ eos1] [ INFO] Duration timeout exceeded. Aborting test.
2018-05-01 13:57:49 [ eos2] [ INFO] Test interfaces_flap started on eos2
2018-05-01 13:57:49 [ eos2] [ INFO] Iteration #1/31536000 on eos2
2018-05-01 13:57:49 [ eos2] [ INFO] Shutting interfaces ['Ethernet 2', 'Ethernet 3'] down
2018-05-01 13:57:55 [ eos2] [ INFO] Diff:
@@ -13,8 +13,10 @@
interface Ethernet1
!
interface Ethernet2
+ shutdown
!
interface Ethernet3
+ shutdown
!
interface Ethernet4
!
2018-05-01 13:57:58 [ eos2] [ INFO] Bringing interfaces ['Ethernet 2', 'Ethernet 3'] up
2018-05-01 13:58:04 [ eos2] [ INFO] Diff:
@@ -13,10 +13,8 @@
interface Ethernet1
!
interface Ethernet2
- shutdown
!
interface Ethernet3
- shutdown
!
interface Ethernet4
!
2018-05-01 13:58:06 [ eos2] [ INFO] Duration timeout exceeded. Aborting test.
2018-05-01 13:58:06 [MainThread] [ INFO] Closing connections to all devices
- Run the original specified tests without modifiers:
❯ netblow -f topologies/eos_topo.yml -t scenarios_tests/eos_tests.yml
2018-05-01 13:58:18 [MainThread] [ INFO] Loading topology file /home/arcanjo/repos/netblow/topologies/eos_topo.yml
2018-05-01 13:58:18 [MainThread] [ INFO] Devices in the topology ['eos1', 'eos2']
2018-05-01 13:58:18 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 13:58:18 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 13:58:18 [ eos2] [ INFO] Trying to connect on eos2...
2018-05-01 13:58:21 [ eos2] [ INFO] Successfully connected on eos2
2018-05-01 13:58:21 [ eos1] [ INFO] Successfully connected on eos1
2018-05-01 13:58:21 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 13:58:21 [MainThread] [ INFO] Loading test file /home/arcanjo/repos/netblow/scenarios_tests/eos_tests.yml
2018-05-01 13:58:21 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 13:58:21 [ eos1] [ INFO] Iteration #1/2 on eos1
2018-05-01 13:58:21 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:58:29 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:58:35 [ eos1] [ INFO] Iteration #2/2 on eos1
2018-05-01 13:58:35 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:58:41 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:58:49 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 13:58:49 [ eos2] [ INFO] Test interfaces_flap started on eos2
2018-05-01 13:58:49 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 13:58:49 [MainThread] [ INFO] Waiting for async tests to finish...
2018-05-01 13:58:49 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:58:49 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 13:58:49 [ eos2] [ INFO] Shutting interfaces ['Ethernet 2', 'Ethernet 3'] down
2018-05-01 13:58:55 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:58:55 [ eos2] [ INFO] Bringing interfaces ['Ethernet 2', 'Ethernet 3'] up
2018-05-01 13:59:03 [ eos1] [ INFO] Test interfaces_flap started on eos1
2018-05-01 13:59:03 [ eos1] [ INFO] Iteration #1/31536000 on eos1
2018-05-01 13:59:03 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 13:59:09 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 13:59:17 [ eos1] [ INFO] Duration timeout exceeded. Aborting test.
2018-05-01 13:59:17 [ eos2] [ INFO] Test interfaces_flap started on eos2
2018-05-01 13:59:17 [ eos2] [ INFO] Iteration #1/31536000 on eos2
2018-05-01 13:59:17 [ eos2] [ INFO] Shutting interfaces ['Ethernet 2', 'Ethernet 3'] down
2018-05-01 13:59:24 [ eos2] [ INFO] Bringing interfaces ['Ethernet 2', 'Ethernet 3'] up
2018-05-01 13:59:30 [ eos2] [ INFO] Duration timeout exceeded. Aborting test.
2018-05-01 13:59:30 [MainThread] [ INFO] Closing connections to all devices
Development¶
Currently, to test the source code of netblow, the following test stages and test suites are in place:
- linters: flake8, pycodestyle and pydocstyle.
- unit: pytest. All networking I/O are mocked in the CI/CD pipeline.
- integration: pytest. This suite is run outside of the pipeline because it needs actual networking devices (JunOS, EOS, IOS-XR and IOS).
Since I can’t run the integration test suite on travis-ci (CI/CD pipeline), I will use this section to post the results of these tests that I run locally for the record:
Note
Let me know if you can host these instances publicly somewhere, just so I could have full integration with the current CI/CD and increase the test coverage.
Note
I’m running virtual instances of OES, JunOS, IOS-XR and IOS-XE on qemu on Docker engine, so chances are, the performance of the output commands are probably worse that what you would have in a device running the OS natively.
Integration test results¶
❯ pytest tests/integration -s
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/arcanjo/repos/netblow, inifile:
plugins: xdist-1.22.2, forked-0.2, cov-2.5.1
collected 17 items
tests/integration/test_eos.py 2018-05-01 14:21:05 [MainThread] [ INFO] Devices in the topology ['eos1', 'eos2']
2018-05-01 14:21:05 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:21:05 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 14:21:05 [ eos2] [ INFO] Trying to connect on eos2...
2018-05-01 14:21:09 [ eos1] [ INFO] Successfully connected on eos1
2018-05-01 14:21:09 [ eos2] [ INFO] Successfully connected on eos2
2018-05-01 14:21:09 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:21:09 [ eos1] [ INFO] Test interfaces_down started on eos1
2018-05-01 14:21:09 [ eos2] [ INFO] Test interfaces_down started on eos2
2018-05-01 14:21:09 [MainThread] [ INFO] Waiting for async tests to finish...
2018-05-01 14:21:09 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 14:21:09 [ eos2] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 14:21:09 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 14:21:09 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 14:21:17 [ eos1] [ INFO] Test interfaces_up started on eos1
2018-05-01 14:21:17 [ eos2] [ INFO] Test interfaces_up started on eos2
2018-05-01 14:21:17 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 14:21:17 [MainThread] [ INFO] Waiting for async tests to finish...
2018-05-01 14:21:17 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 14:21:17 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 14:21:17 [ eos2] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
.2018-05-01 14:21:24 [ eos1] [ INFO] Test interfaces_down started on eos1
2018-05-01 14:21:24 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 14:21:24 [ eos1] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 14:21:30 [ eos2] [ INFO] Test interfaces_down started on eos2
2018-05-01 14:21:30 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 14:21:30 [ eos2] [ INFO] Shutting interfaces ['Ethernet 7', 'Ethernet 8'] down
2018-05-01 14:21:35 [ eos1] [ INFO] Test interfaces_up started on eos1
2018-05-01 14:21:35 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 14:21:35 [ eos1] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
2018-05-01 14:21:42 [ eos2] [ INFO] Test interfaces_up started on eos2
2018-05-01 14:21:42 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 14:21:42 [ eos2] [ INFO] Bringing interfaces ['Ethernet 7', 'Ethernet 8'] up
.2018-05-01 14:21:49 [MainThread] [ INFO] Devices in the topology ['eos1', 'eos2']
2018-05-01 14:21:49 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:21:49 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 14:21:49 [ eos2] [ INFO] Trying to connect on eos2...
2018-05-01 14:21:53 [ eos1] [ INFO] Successfully connected on eos1
2018-05-01 14:21:53 [ eos2] [ INFO] Successfully connected on eos2
2018-05-01 14:21:53 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:21:53 [ eos1] [ INFO] Test config_rollback started on eos1
2018-05-01 14:21:53 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 14:21:53 [ eos1] [ INFO] Applying commands ['hostname eos1eos1'] on eos1
2018-05-01 14:22:00 [ eos1] [ INFO] Diff:
@@ -3,6 +3,8 @@
! boot system flash:/vEOS-lab.swi
!
transceiver qsfp default-mode 4x10G
+!
+hostname eos1eos1
!
spanning-tree mode mstp
!
2018-05-01 14:22:03 [ eos1] [ INFO] Performing rollback
.2018-05-01 14:22:07 [ eos2] [ INFO] Test show started on eos2
2018-05-01 14:22:07 [ eos2] [ INFO] Iteration #1/1 on eos2
2018-05-01 14:22:07 [ eos2] [ INFO] Applying show commands ['show version'] on eos2
2018-05-01 14:22:09 [ eos2] [ INFO] show version
Arista vEOS
Hardware version:
Serial number:
System MAC address: 5254.005a.ebe5
Software image version: 4.20.1F
Architecture: i386
Internal build version: 4.20.1F-6820520.4201F
Internal build ID: 790a11e8-5aaf-4be7-a11a-e61795d05b91
Uptime: 1 day, 22 hours and 25 minutes
Total memory: 2017260 kB
Free memory: 1038628 kB
.2018-05-01 14:22:09 [ eos1] [ INFO] Test reboot started on eos1
2018-05-01 14:22:09 [ eos1] [ INFO] Iteration #1/1 on eos1
2018-05-01 14:22:09 [ eos1] [ INFO] Rebooting eos1
2018-05-01 14:22:11 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 14:22:41 [ eos1] [ INFO] Retry #1
2018-05-01 14:22:41 [ eos1] [ INFO] Socket error during eAPI connection: _ssl.c:761: The handshake operation timed out
2018-05-01 14:22:41 [ eos1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:22:41 [ eos1] [ INFO] 569 seconds left before timeouting...
2018-05-01 14:23:11 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 14:23:42 [ eos1] [ INFO] Retry #2
2018-05-01 14:23:42 [ eos1] [ INFO] Socket error during eAPI connection: _ssl.c:761: The handshake operation timed out
2018-05-01 14:23:42 [ eos1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:23:42 [ eos1] [ INFO] 509 seconds left before timeouting...
2018-05-01 14:24:12 [ eos1] [ INFO] Trying to connect on eos1...
2018-05-01 14:24:36 [ eos1] [ INFO] Successfully connected on eos1
.
tests/integration/test_ios.py 2018-05-01 14:24:36 [MainThread] [ INFO] Devices in the topology ['ios1']
2018-05-01 14:24:36 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:24:36 [ ios1] [ INFO] Trying to connect on ios1...
2018-05-01 14:24:44 [ ios1] [ INFO] Successfully connected on ios1
2018-05-01 14:24:44 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:24:44 [ ios1] [ INFO] Test interfaces_down started on ios1
2018-05-01 14:24:44 [ ios1] [ INFO] Iteration #1/1 on ios1
2018-05-01 14:24:44 [ ios1] [ INFO] Shutting interfaces ['GigabitEthernet12', 'GigabitEthernet13'] down
2018-05-01 14:26:21 [ ios1] [ INFO] Test interfaces_up started on ios1
2018-05-01 14:26:21 [ ios1] [ INFO] Iteration #1/1 on ios1
2018-05-01 14:26:21 [ ios1] [ INFO] Bringing interfaces ['GigabitEthernet12', 'GigabitEthernet13'] up
2018-05-01 14:26:26 [ ios1] [ERROR] SCP file transfers are not enabled. Configure 'ip scp server enable' on the device.
2018-05-01 14:26:26 [ ios1] [ INFO] Trying to connect on ios1...
2018-05-01 14:26:33 [ ios1] [ INFO] Successfully connected on ios1
.2018-05-01 14:26:33 [MainThread] [ INFO] Devices in the topology ['ios1']
2018-05-01 14:26:33 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:26:33 [ ios1] [ INFO] Trying to connect on ios1...
2018-05-01 14:26:40 [ ios1] [ INFO] Successfully connected on ios1
2018-05-01 14:26:40 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:26:40 [ ios1] [ INFO] Test config_rollback started on ios1
2018-05-01 14:26:40 [ ios1] [ INFO] Iteration #1/1 on ios1
2018-05-01 14:26:40 [ ios1] [ INFO] Applying commands ['router bgp 65000'] on ios1
2018-05-01 14:26:51 [ ios1] [ INFO] Diff:
+router bgp 65000
2018-05-01 14:28:18 [ ios1] [ INFO] Performing rollback
Exception in thread ios1:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/arcanjo/repos/netblow/netblow/netblow.py", line 590, in blow_thread
**kwargs)
File "/home/arcanjo/repos/netblow/netblow/netblow.py", line 785, in config_rollback
dut_driver.rollback()
File "/home/arcanjo/repos/netblow/.direnv/python-3.6.4/lib/python3.6/site-packages/napalm/ios/ios.py", line 471, in rollback
self.device.send_command_expect("write mem")
File "/home/arcanjo/repos/netblow/.direnv/python-3.6.4/lib/python3.6/site-packages/netmiko/base_connection.py", line 1069, in send_command_expect
return self.send_command(*args, **kwargs)
File "/home/arcanjo/repos/netblow/.direnv/python-3.6.4/lib/python3.6/site-packages/netmiko/base_connection.py", line 1051, in send_command
search_pattern))
OSError: Search pattern never detected in send_command_expect: xt\ force
.2018-05-01 14:29:49 [ ios1] [ INFO] Test show started on ios1
2018-05-01 14:29:49 [ ios1] [ INFO] Iteration #1/1 on ios1
2018-05-01 14:29:49 [ ios1] [ INFO] Applying show commands ['show version'] on ios1
2018-05-01 14:29:50 [ ios1] [ INFO] show version
Cisco IOS XE Software, Version 16.06.02
Cisco IOS Software [Everest], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Wed 01-Nov-17 07:24 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2017 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: IOS-XE ROMMON
csr1000v uptime is 1 day, 22 hours, 31 minutes
Uptime for this control processor is 1 day, 22 hours, 32 minutes
System returned to ROM by reload
System image file is "bootflash:packages.conf"
Last reload reason: Reload Command
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
License Level: ax
License Type: Default. No valid license found.
Next reload license Level: ax
cisco CSR1000V (VXE) processor (revision VXE) with 2190795K/3075K bytes of memory.
Processor board ID 9CT15UOLWFI
10 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3984840K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.
Configuration register is 0x2102
.2018-05-01 14:29:50 [ ios1] [ INFO] Test reboot started on ios1
2018-05-01 14:29:50 [ ios1] [ INFO] Iteration #1/1 on ios1
2018-05-01 14:29:50 [ ios1] [ INFO] Rebooting ios1
2018-05-01 14:31:21 [ ios1] [ INFO] Trying to connect on ios1...
2018-05-01 14:31:36 [ ios1] [ INFO] Retry #1
2018-05-01 14:31:36 [ ios1] [ INFO] Error reading SSH protocol banner
2018-05-01 14:31:36 [ ios1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:31:36 [ ios1] [ INFO] 584 seconds left before timeouting...
2018-05-01 14:32:06 [ ios1] [ INFO] Trying to connect on ios1...
2018-05-01 14:32:22 [ ios1] [ INFO] Retry #2
2018-05-01 14:32:22 [ ios1] [ INFO] Error reading SSH protocol banner
2018-05-01 14:32:22 [ ios1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:32:22 [ ios1] [ INFO] 539 seconds left before timeouting...
2018-05-01 14:32:52 [ ios1] [ INFO] Trying to connect on ios1...
2018-05-01 14:32:59 [ ios1] [ INFO] Successfully connected on ios1
.
tests/integration/test_iosxr.py 2018-05-01 14:32:59 [MainThread] [ INFO] Devices in the topology ['iosxr1']
2018-05-01 14:32:59 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:32:59 [ iosxr1] [ INFO] Trying to connect on iosxr1...
2018-05-01 14:33:10 [ iosxr1] [ INFO] Successfully connected on iosxr1
2018-05-01 14:33:10 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:33:10 [ iosxr1] [ INFO] Test interfaces_down started on iosxr1
2018-05-01 14:33:10 [ iosxr1] [ INFO] Iteration #1/1 on iosxr1
2018-05-01 14:33:10 [ iosxr1] [ INFO] Shutting interfaces ['GigabitEthernet0/0/0/1', 'GigabitEthernet0/0/0/2'] down
2018-05-01 14:33:13 [ iosxr1] [ INFO] Test interfaces_up started on iosxr1
2018-05-01 14:33:13 [ iosxr1] [ INFO] Iteration #1/1 on iosxr1
2018-05-01 14:33:13 [ iosxr1] [ INFO] Bringing interfaces ['GigabitEthernet0/0/0/1', 'GigabitEthernet0/0/0/2'] up
.2018-05-01 14:33:15 [MainThread] [ INFO] Devices in the topology ['iosxr1']
2018-05-01 14:33:15 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:33:15 [ iosxr1] [ INFO] Trying to connect on iosxr1...
2018-05-01 14:33:27 [ iosxr1] [ INFO] Successfully connected on iosxr1
2018-05-01 14:33:27 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:33:27 [ iosxr1] [ INFO] Test config_rollback started on iosxr1
2018-05-01 14:33:27 [ iosxr1] [ INFO] Iteration #1/1 on iosxr1
2018-05-01 14:33:27 [ iosxr1] [ INFO] Applying commands ['router bgp 65001'] on iosxr1
2018-05-01 14:33:29 [ iosxr1] [ INFO] Diff:
---
+++
@@ -385,6 +385,8 @@
interface GigabitEthernet0/0/0/127
shutdown
!
+router bgp 65001
+!
xml agent tty
!
netconf-yang agent
2018-05-01 14:33:31 [ iosxr1] [ INFO] Performing rollback
.2018-05-01 14:33:31 [ iosxr1] [ INFO] Test show started on iosxr1
2018-05-01 14:33:31 [ iosxr1] [ INFO] Iteration #1/1 on iosxr1
2018-05-01 14:33:31 [ iosxr1] [ INFO] Applying show commands ['show version'] on iosxr1
2018-05-01 14:33:32 [ iosxr1] [ INFO] show version
Cisco IOS XR Software, Version 6.0.1[Default]
Copyright (c) 2016 by Cisco Systems, Inc.
ROM: GRUB, Version 1.99(0), DEV RELEASE
ios uptime is 1 day, 22 hours, 36 minutes
System image file is "bootflash:disk0/xrvr-os-mbi-6.0.1/mbixrvr-rp.vm"
cisco IOS XRv Series (Pentium Celeron Stepping 3) processor with 3145215K bytes of memory.
Pentium Celeron Stepping 3 processor at 2607MHz, Revision 2.174
IOS XRv Chassis
128 GigabitEthernet
1 Management Ethernet
97070k bytes of non-volatile configuration memory.
866M bytes of hard disk.
2321392k bytes of disk0: (Sector size 512 bytes).
Configuration register on node 0/0/CPU0 is 0x2102
Boot device on node 0/0/CPU0 is disk0:
Package active on node 0/0/CPU0:
iosxr-infra, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-infra-6.0.1
Built on Mon May 9 12:06:47 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-fwding, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-fwding-6.0.1
Built on Mon May 9 12:06:47 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-routing, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-routing-6.0.1
Built on Mon May 9 12:06:47 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-ce, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-ce-6.0.1
Built on Mon May 9 12:06:48 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-os-mbi, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-os-mbi-6.0.1
Built on Mon May 9 12:07:35 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-base, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-base-6.0.1
Built on Mon May 9 12:06:47 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-fwding, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-fwding-6.0.1
Built on Mon May 9 12:06:48 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-mgbl-x, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-mgbl-x-6.0.1
Built on Mon May 9 12:06:55 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-mpls, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-mpls-6.0.1
Built on Mon May 9 12:06:47 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-mgbl, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-mgbl-6.0.1
Built on Mon May 9 12:06:47 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-mcast, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-mcast-6.0.1
Built on Mon May 9 12:06:48 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-mcast-supp, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-mcast-supp-6.0.1
Built on Mon May 9 12:06:48 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-bng, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-bng-6.0.1
Built on Mon May 9 12:06:45 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-bng-supp, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-bng-supp-6.0.1
Built on Mon May 9 12:06:45 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
iosxr-security, V 6.0.1[Default], Cisco Systems, at disk0:iosxr-security-6.0.1
Built on Mon May 9 12:06:39 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
xrvr-fullk9-x, V 6.0.1[Default], Cisco Systems, at disk0:xrvr-fullk9-x-6.0.1
Built on Mon May 9 12:07:39 UTC 2016
By iox-lnx-003 in /auto/srcarchive12/production/6.0.1/xrvr/workspace for pie
.2018-05-01 14:33:32 [ iosxr1] [ INFO] Test reboot started on iosxr1
2018-05-01 14:33:32 [ iosxr1] [ INFO] Iteration #1/1 on iosxr1
2018-05-01 14:33:32 [ iosxr1] [ INFO] Rebooting iosxr1
2018-05-01 14:34:07 [ iosxr1] [ INFO] Trying to connect on iosxr1...
2018-05-01 14:34:19 [ iosxr1] [ INFO] Successfully connected on iosxr1
.
tests/integration/test_junos.py 2018-05-01 14:34:19 [MainThread] [ INFO] Devices in the topology ['junos1']
2018-05-01 14:34:19 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:34:19 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 14:34:21 [ junos1] [ INFO] Successfully connected on junos1
2018-05-01 14:34:21 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:34:21 [ junos1] [ INFO] Test interfaces_down started on junos1
2018-05-01 14:34:21 [ junos1] [ INFO] Iteration #1/1 on junos1
2018-05-01 14:34:21 [ junos1] [ INFO] Shutting interfaces ['ge-0/0/1', 'ge-0/0/2'] down
2018-05-01 14:34:24 [ junos1] [ INFO] Test interfaces_up started on junos1
2018-05-01 14:34:24 [ junos1] [ INFO] Iteration #1/1 on junos1
2018-05-01 14:34:24 [ junos1] [ INFO] Bringing interfaces ['ge-0/0/1', 'ge-0/0/2'] up
2018-05-01 14:34:26 [ junos1] [ INFO] Test interfaces_flap started on junos1
2018-05-01 14:34:26 [ junos1] [ INFO] Iteration #1/1 on junos1
2018-05-01 14:34:26 [ junos1] [ INFO] Shutting interfaces ['ge-0/0/1', 'ge-0/0/2'] down
2018-05-01 14:34:28 [ junos1] [ INFO] Bringing interfaces ['ge-0/0/1', 'ge-0/0/2'] up
.2018-05-01 14:34:30 [MainThread] [ INFO] Devices in the topology ['junos1']
2018-05-01 14:34:30 [MainThread] [ INFO] Trying to open connections to all devices...
2018-05-01 14:34:30 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 14:34:33 [ junos1] [ INFO] Successfully connected on junos1
2018-05-01 14:34:33 [MainThread] [ INFO] All devices are CONNECTED
2018-05-01 14:34:33 [ junos1] [ INFO] Test config_rollback started on junos1
2018-05-01 14:34:33 [ junos1] [ INFO] Iteration #1/1 on junos1
2018-05-01 14:34:33 [ junos1] [ INFO] Applying commands ['set system domain-name lab.com', 'set system ntp peer 10.10.10.10'] on junos1
2018-05-01 14:34:34 [ junos1] [ INFO] Diff:
[edit system]
+ domain-name lab.com;
+ ntp {
+ peer 10.10.10.10;
+ }
2018-05-01 14:34:35 [ junos1] [ INFO] Performing rollback
.2018-05-01 14:34:37 [ junos1] [ INFO] Test show started on junos1
2018-05-01 14:34:37 [ junos1] [ INFO] Iteration #1/1 on junos1
2018-05-01 14:34:37 [ junos1] [ INFO] Applying show commands ['show interfaces terse', 'show version'] on junos1
2018-05-01 14:34:38 [ junos1] [ INFO] show interfaces terse
Interface Admin Link Proto Local Remote
ge-0/0/0 up up
lc-0/0/0 up up
lc-0/0/0.32769 up up vpls
pfe-0/0/0 up up
pfe-0/0/0.16383 up up inet
inet6
pfh-0/0/0 up up
pfh-0/0/0.16383 up up inet
pfh-0/0/0.16384 up up inet
ge-0/0/1 up up
ge-0/0/2 up up
ge-0/0/3 up up
ge-0/0/4 up up
ge-0/0/5 up up
ge-0/0/6 up up
ge-0/0/7 up up
ge-0/0/8 up up
ge-0/0/9 up up
ge-0/0/10 up up
ge-0/0/11 up up
ge-0/0/12 up up
ge-0/0/13 up up
ge-0/0/14 up up
ge-0/0/15 up up
ge-0/0/16 up up
ge-0/0/17 up up
ge-0/0/18 up up
ge-0/0/19 up up
ge-0/0/20 up up
ge-0/0/21 up up
ge-0/0/22 up up
ge-0/0/23 up up
ge-0/0/24 up up
ge-0/0/25 up up
ge-0/0/26 up up
ge-0/0/27 up up
ge-0/0/28 up up
ge-0/0/29 up up
ge-0/0/30 up up
ge-0/0/31 up up
ge-0/0/32 up up
ge-0/0/33 up up
ge-0/0/34 up up
ge-0/0/35 up up
ge-0/0/36 up up
ge-0/0/37 up up
ge-0/0/38 up up
ge-0/0/39 up up
ge-0/0/40 up up
ge-0/0/41 up up
ge-0/0/42 up up
ge-0/0/43 up up
ge-0/0/44 up up
ge-0/0/45 up up
ge-0/0/46 up up
ge-0/0/47 up up
ge-0/0/48 up up
ge-0/0/49 up up
ge-0/0/50 up up
ge-0/0/51 up up
ge-0/0/52 up up
ge-0/0/53 up up
ge-0/0/54 up up
ge-0/0/55 up up
ge-0/0/56 up up
ge-0/0/57 up up
ge-0/0/58 up up
ge-0/0/59 up up
ge-0/0/60 up up
ge-0/0/61 up up
ge-0/0/62 up up
ge-0/0/63 up up
ge-0/0/64 up up
ge-0/0/65 up up
ge-0/0/66 up up
ge-0/0/67 up up
ge-0/0/68 up up
ge-0/0/69 up up
ge-0/0/70 up up
ge-0/0/71 up up
ge-0/0/72 up up
ge-0/0/73 up up
ge-0/0/74 up up
ge-0/0/75 up up
ge-0/0/76 up up
ge-0/0/77 up up
ge-0/0/78 up up
ge-0/0/79 up up
ge-0/0/80 up up
ge-0/0/81 up up
ge-0/0/82 up up
ge-0/0/83 up up
ge-0/0/84 up up
ge-0/0/85 up up
ge-0/0/86 up up
ge-0/0/87 up up
ge-0/0/88 up up
ge-0/0/89 up up
ge-0/0/90 up up
ge-0/0/91 up up
ge-0/0/92 up up
ge-0/0/93 up up
ge-0/0/94 up up
cbp0 up up
demux0 up up
dsc up up
em1 up up
em1.0 up up inet 10.0.0.4/8
128.0.0.1/2
128.0.0.4/2
inet6 fe80::5254:ff:fe55:5801/64
fec0::a:0:0:4/64
tnp 0x4
esi up up
fxp0 up up
fxp0.0 up up inet 10.0.0.15/24
gre up up
ipip up up
irb up up
jsrv up up
jsrv.1 up up inet 128.0.0.127/2
lo0 up up
lo0.16384 up up inet 127.0.0.1 --> 0/0
lo0.16385 up up inet
lsi up up
mtun up up
pimd up up
pime up up
pip0 up up
pp0 up up
rbeb up up
tap up up
vtep up up
2018-05-01 14:34:38 [ junos1] [ INFO] show version
Model: vmx
Junos: 17.2R1.13
JUNOS OS Kernel 64-bit [20170523.350481_builder_stable_10]
JUNOS OS libs [20170523.350481_builder_stable_10]
JUNOS OS runtime [20170523.350481_builder_stable_10]
JUNOS OS time zone information [20170523.350481_builder_stable_10]
JUNOS network stack and utilities [20170601.185252_builder_junos_172_r1]
JUNOS modules [20170601.185252_builder_junos_172_r1]
JUNOS mx modules [20170601.185252_builder_junos_172_r1]
JUNOS libs [20170601.185252_builder_junos_172_r1]
JUNOS OS libs compat32 [20170523.350481_builder_stable_10]
JUNOS OS 32-bit compatibility [20170523.350481_builder_stable_10]
JUNOS libs compat32 [20170601.185252_builder_junos_172_r1]
JUNOS runtime [20170601.185252_builder_junos_172_r1]
JUNOS Packet Forwarding Engine Simulation Package [20170601.185252_builder_junos_172_r1]
JUNOS py extensions [20170601.185252_builder_junos_172_r1]
JUNOS py base [20170601.185252_builder_junos_172_r1]
JUNOS OS vmguest [20170523.350481_builder_stable_10]
JUNOS OS crypto [20170523.350481_builder_stable_10]
JUNOS mx libs compat32 [20170601.185252_builder_junos_172_r1]
JUNOS mx runtime [20170601.185252_builder_junos_172_r1]
JUNOS common platform support [20170601.185252_builder_junos_172_r1]
JUNOS mx libs [20170601.185252_builder_junos_172_r1]
JUNOS mtx Data Plane Crypto Support [20170601.185252_builder_junos_172_r1]
JUNOS daemons [20170601.185252_builder_junos_172_r1]
JUNOS mx daemons [20170601.185252_builder_junos_172_r1]
JUNOS Services URL Filter package [20170601.185252_builder_junos_172_r1]
JUNOS Services TLB Service PIC package [20170601.185252_builder_junos_172_r1]
JUNOS Services SSL [20170601.185252_builder_junos_172_r1]
JUNOS Services Stateful Firewall [20170601.185252_builder_junos_172_r1]
JUNOS Services RPM [20170601.185252_builder_junos_172_r1]
JUNOS Services PTSP Container package [20170601.185252_builder_junos_172_r1]
JUNOS Services PCEF package [20170601.185252_builder_junos_172_r1]
JUNOS Services NAT [20170601.185252_builder_junos_172_r1]
JUNOS Services Mobile Subscriber Service Container package [20170601.185252_builder_junos_172_r1]
JUNOS Services MobileNext Software package [20170601.185252_builder_junos_172_r1]
JUNOS Services Logging Report Framework package [20170601.185252_builder_junos_172_r1]
JUNOS Services LL-PDF Container package [20170601.185252_builder_junos_172_r1]
JUNOS Services Jflow Container package [20170601.185252_builder_junos_172_r1]
JUNOS Services Deep Packet Inspection package [20170601.185252_builder_junos_172_r1]
JUNOS Services IPSec [20170601.185252_builder_junos_172_r1]
JUNOS Services IDS [20170601.185252_builder_junos_172_r1]
JUNOS IDP Services [20170601.185252_builder_junos_172_r1]
JUNOS Services HTTP Content Management package [20170601.185252_builder_junos_172_r1]
JUNOS Services Crypto [20170601.185252_builder_junos_172_r1]
JUNOS Services Captive Portal and Content Delivery Container package [20170601.185252_builder_junos_172_r1]
JUNOS Services COS [20170601.185252_builder_junos_172_r1]
JUNOS AppId Services [20170601.185252_builder_junos_172_r1]
JUNOS Services Application Level Gateways [20170601.185252_builder_junos_172_r1]
JUNOS Services AACL Container package [20170601.185252_builder_junos_172_r1]
JUNOS Extension Toolkit [20170601.185252_builder_junos_172_r1]
JUNOS jfirmware [20170601.185252_builder_junos_172_r1]
JUNOS Online Documentation [20170601.185252_builder_junos_172_r1]
.2018-05-01 14:34:38 [ junos1] [ INFO] Test reboot started on junos1
2018-05-01 14:34:38 [ junos1] [ INFO] Iteration #1/1 on junos1
2018-05-01 14:34:38 [ junos1] [ INFO] Rebooting junos1
2018-05-01 14:34:39 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 14:34:39 [ junos1] [ INFO] Retry #1
2018-05-01 14:34:39 [ junos1] [ INFO] ConnectRefusedError(labhost)
2018-05-01 14:34:39 [ junos1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:34:39 [ junos1] [ INFO] 599 seconds left before timeouting...
2018-05-01 14:35:09 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 14:35:24 [ junos1] [ INFO] Retry #2
2018-05-01 14:35:24 [ junos1] [ INFO] ConnectError(host: labhost, msg: Negotiation failed)
2018-05-01 14:35:24 [ junos1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:35:24 [ junos1] [ INFO] 554 seconds left before timeouting...
2018-05-01 14:35:54 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 14:36:09 [ junos1] [ INFO] Retry #3
2018-05-01 14:36:09 [ junos1] [ INFO] ConnectError(host: labhost, msg: Negotiation failed)
2018-05-01 14:36:09 [ junos1] [ INFO] Waiting for 30 seconds...
2018-05-01 14:36:09 [ junos1] [ INFO] 509 seconds left before timeouting...
2018-05-01 14:36:39 [ junos1] [ INFO] Trying to connect on junos1...
2018-05-01 14:36:42 [ junos1] [ INFO] Successfully connected on junos1
.
========================================================================================= 17 passed in 937.32 seconds =========================================================================================
2018-05-01 14:36:42 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:42 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:42 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:42 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:42 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:42 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:46 [MainThread] [ INFO] Closing connections to all devices
2018-05-01 14:36:46 [MainThread] [ INFO] Closing connections to all devices