So long, for now.

I’m going to move my more popular posts over to and wrap this blog up. I don’t have the time to dedicate to it at the moment and would rather the pushers get the traffic. TTFN!

A Letter to Nick Clegg

Dear Nick,

This is not usually a political blog and is unlikely to be again. However, I wanted to write to you on behalf of the British people you have so faithfully served for the past 5 years. Sir, I salute you. You put people above politics, compassion above career and you gave our Government a heart. The way your party has been punished today is unfair and not a reflection of all you stood for and achieved. You can rightly be proud of your record and my hope is that history remembers you in the light you deserve.

So thank you Nick Clegg. We wish you well for the future and, personally, I wish you were still in Government fighting for those values which make such a difference to ordinary people.

Yours Sincerely,

Guy Morrell

Quick Ubuntu Package Update Script

I don’t want to have to remember to login to my syslog server and check for required updates. Here is the script I wrote and added to cron so that I don’t have to:


use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Sender::Transport::Sendmail qw();
use Try::Tiny;

my $to = "noc\";
my $from = "root\";
my $subject = "update script";
my $body =  `/usr/bin/apt-get update && /usr/bin/apt-get upgrade -s`;

my $email = Email::Simple->create(
    header=>[To=>$to, From=>$from,

try {
} catch {
    print "Can't send mail: $_";

To get this to work I needed to install some packages:

apt-get install sendmail
apt-get install libemail-sender-perl 
apt-get install libemail-simple-perl


I then added my script to cron:

root@itssys01:~# crontab -l
# Edit this file to introduce tasks to be run by cron.
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# For more information see the manual pages of crontab(5) and cron(8)
# m h  dom mon dow   command
0 0 * * * /usr/local/bin/ > /dev/null

Now I get an email with a list of packages which would be updated were I to run the following:

/usr/bin/apt-get upgrade

I don’t want my system blindly updating itself so this way I can still check things over before committing.

Northbound and Southbound APIs

I regularly hear the packet pushers and others discussing Northbound and Southbound APIs and keep forgetting which does what. Here is a summary, comments and corrections welcome.

First a few definitions for context:

  • SDN – Somewhat Defined Networking ūüėČ Or Software..
  • SDN Management Plane – here is the realm of the applications, where network orchestration takes place
  • SDN Control Plane – interacts with the SDN Management Plane and Data Plane. The realm of APIs. In classic SDN this lives on the SDN Contoller
  • SDN Data Plane – where traffic is forwarded, the realm of hardware
  • SDN Controller – Platform on which Applications run. That platform may be a single server, collection of distributed devices in your network or hosted in the cloud

Southbound API

Communication between the SDN Controller (or possibly simply an administrators workstation running C, Java, Phython,… scripts) and network switches, routers and other devices happens via the Southbound API.

Open Standards

  • Openflow¬†– maintained by the Open Network Foundation (ONF)
  • NETCONF¬†– IETF RFC 6241. Standard for managing Network Device Configuration
    • Often used with the YANG modelling language defined in RFC 6020

As an example I might write a Python Script ‘StandardSNMP’ which takes a switch IP as an argument and uses an appropriate southbound API to retrieve and correct its SNMP config based on my companies best practice.

Vendor Proprietary

Northbound / External API

Communication between applications (possibly running on the SDN Controller) and the controller itself. It is a network abstraction presented to high level applications or management systems. This enables developers to write apps for the network without needing to call the southbound API directly. This could mean the same code could be re-used across multiple network devices, from a variety of vendors.

Disclaimer: This is only an example of how I *think* things may work. It doesn’t reflect operational experience.

Say I am writing a configuration application ‘NetConfPLS'[1] which includes a standardisation feature. In my network are a few switches and routers from multiple vendors, all of which support OpenFlow. Once again, I want to verify whether my companies SNMP¬†standard is adhered to across my network and correct it where it is not.

NetConfPLS would make a Northbound API call to the controller such as ‘give me the SNMP state of all devices’. The Controller then makes a Southbound API call to each network device to pull that config and presents this data to NetConfPLS in a standardised way. NetConfPLS then presents this data to the application user.¬†Perhaps the actual config of each device would be shown together with the proposed replacement if running in debug mode.

The user then selects which devices to standardise. NetConfPLS would issue a second Northbound call to the controller like ‘standardise the following devices’. The controller would then issue a southbound call to each device containing the API version of¬†the appropriate configuration commands.

[1] Potential LawSuit



Screen Shot 2015-02-09 at 12.11.31


Hopefully¬†you can start to see the appeal for developers. Just as we used to write Perl libraries (e.g. Net::Appliance::Session) to communicate with devices, now we can use the device manufacturer’s (or the Open Standard) API to do this. Whereas before the script and library were tightly coupled, now the developer can make use of their programming language of choice to communicate with a network abstraction provided by the controller. Scripts or Applications written in one environment could be more easily ported, given to the community or commercialised.

Of course, as ever, each vendor has their own story to sell. I suspect¬†the road to an Open Source controller playing nicely across all platforms is long and perilous. For example, I can’t imagine Cisco would allow a third party controller access to its Northbound APIs. More likely you’ll be forced to buy a Cisco¬†eXtensible Network Controller (XNC) to program your Openflow switches and onePK switches from one place, protecting that revenue stream for Cisco.


Cisco have confused me a little in the way they have brought their simulator to market. First it was VIRL and free, then CML was announced and the name change blamed on marketing. Now we have both. Here is a birds eye view of each for reference. I’ve also included GNS3 as it is worth considering if you’re in the market for an emulation tool.


  • Part of the /dev/innovate program at Cisco
  • In Beta at the time of writing
  • Community supported
  • Runs on Linux KVM Hypervisor / OpenStack frontend
    • This can be installed bare metal or run via VMWare Fusion / Workstation / Player etc
  • Local hardware model (laptop / desktop)
  • Supports IOS, IOS-XE, IOS-XR and NX-OS
  • No Layer 2
  • Annual Licence either Academic or Personal ($80 or $200)
  • Capped at 15 nodes


  • Corporate Edition
  • TAC Supported
  • Client / Server Model
  • IOSv only today
  • NXOS support roadmapped
  • Heavy server requirements
    • Min: 4 Core CPU / 16 GB RAM for basic 15 node limit
    • 16 Core CPU / 128 GB RAM for full 50 nodes
  • Server runs Ubuntu image
  • Java Client (oh Cisco, why do you try my patience so?)
  • $13,000 Base Install (15 node limit)
  • 5% discount for 50 nodes, 10% for 100
  • More pricing detail and other useful information here

All-in-One Virtual Machine

  • Free
  • Limited version of VIRL
  • 3 Node Development Environment for the Open Network Environment Platform Kit (onePK)
  • Designed to allow would-be app writers to develop and test their Apps


  • Free
  • Community Supported via the GNS3 Jungle
  • Multi-Vendor – Cisco, Juniper, HP, Arista, Citrix and Brocade
  • L2 Supported using Cisco L2IOU Images (native on Linux / Solaris only, VMs on Windows / Mac)
    • Although features relying on L3 Hardware such as¬†¬†L3 Etherchannel, ISL trunks, DHCP snooping, Private VLAN, SPAN/RSPAN/ERSPAN, Port-security, Voice VLANs, MLS QoS and QinQ won’t work
  • Supports any Cisco platforms available as a virtual machine using VitualBox or Qemu



Network Cable and Optic Distances

Here is a list of common network cable and optic types, together with their distances. The costs are very rough and a little on the low side, I’ll try and update them.

Speed / Base Media Distance Cisco Part Rough Cost
10/100/1000BaseT Cat 5e 100m GLC-T SFP £150
10GBaseT Twinax Passive 1 to 5m SFP-H10GB-CU<N>M £50-£100
10GBaseT Twinax Active 7 or 10m SFP-H10GB-ACU<N>M
10GBaseT Cat 6 55m N/A N/A
10GBaseT Cat 6a (may be too thick for flood wiring) 100m N/A N/A
1000BaseSX OM1 (62.5/125) 220m GLC-SX-MMD £200
1000BaseSX OM3 (50/125) 550m GLC-SX-MMD £200
1000BaseSX OM4 (50/125) 1000m GLC-SX-MMD £200
10GBaseSR OM1 (63.5/125) 26m SFP-10G-SR £400
10GBaseSR OM3 (50/125) 300m SFP-10G-SR £400
10GBaseSR OM4 (50/125) 400m SFP-10G-SR £400
10GBaseLRM OM1 with mode conditioning patch cord (62.5/125) 220m SFP-10G-LRM £400
10GBaseLRM OM3 (50/125) 220m SFP-10G-LRM £400
10GBaseLRM SM 300m SFP-10G-LRM £400
10GBaseLR SM 10km SFP-10G-LR £1500
X2 to SFP+ Converter OneX N/A CVR-X2-SFP10G £80
40GBaseCR4 -> 4x10GBaseT Twinax Passive Breakout 1-5m QSFP-4SFP10G-CU<N>M
40GBaseCR4 -> 4x10GBaseT Twinax Active Breakout 7 or 10m QSFP-4SFP10G-AC<N>M
40GBaseCR4 Twinax Passive 1-5m QSFP-H40G-CU<N>M
40GBaseCR4 Twinax Active 7 or 10m QSFP-H40G-ACU<N>M
40GBaseAOC -> 4x10G Optical Active Breakout 1-10m QSFP-4X10G-AOC<N>M
40GBaseAOC Optical Active 1-15m QSFP-H40G-AOC<N>M
40GBaseSR4 OM3 (50/125) 100m QSFP-40G-SR4
40GBaseSR4 OM4 (50/125) 150m QSFP-40G-SR4
40GBaseCSR4 OM3 (50/125) 300m QSFP-40G-CSR4 £800
40GBaseCSR4 OM4 (50/125) 550m QSFP-40G-CSR4 £800
40GBaseSR BiDi (Bi-Directional over one fibre pair) OM3 (50/125) 100m QSFP-40G-SR-BD £400
40GBaseSR BiDi OM4 (50/125) 150m QSFP-40G-SR-BD £400
40GBaseLR4 SM 10km QSFP-40GE-LR4

Cisco Nexus TACACS config

According to the team where I work, our standard Catalyst TACACS+ config didn’t work properly. Here is the snippet they used to get going again:

user = DEFAULT {
default service = permit
service = exec {
shell:roles*"network-admin vdc-admin"

User based role authorisation didn’t work well either apparently.

The * rather than the = means it’s an optional attribute so should be ignored by switches which aren’t compatible.

Thanks to Emma Cardinal-Richards for the snippet.

IPv6 in a Global Company – a response: NPTv6?

This is a response this post by Ivan. In summary he is talking about the issues companies will face if they wish to use IPv6 and multi-home. Leaving aside the fact the most Small/Medium Enterprises (SMEs) aren’t going to want to run BGP, the questions of NAT in IPv6 (a terrible idea) and Internet Routing table explosion remain.

I was at the IPv6 Council meeting last week and Eric Vyncke drew our attention to the experimental RFC, RFC6296, IPv6-to-IPv6 Network Prefix Translation (NPTv6). The clue is in the name; rather than translating the entire address and breaking the end-to-end communication model, translate the Prefix section of the IPv6 address and leave the Host part unchanged. Unlike NAT, NPTv6 is stateless which considerably reduces the burden on network equipment. Also ports at L4 can be left alone and asymmetric flows won’t cause an issue.

By way of a simple example, a business is assigned 2001:10::/48 Provider Independent Address (PIR) space. They are a small business with no BGP skills but would like resilience. They take out an Internet connection with two different providers and set up outbound routing however they like. Provider A gives them 2001:A::/48 and provider B allocates 2001:B::/48.

Now, any flows which egress Provider A get prefix translated to 2001:A::/48 and any going out via provider B get 2001:B::/48. For client access to the Internet this is great, hosts can use SLAAC or manually assigned addresses, troubleshooting is easy as any issues can quickly be narrowed down to a host and ISP.

For example, let’s assume the SME uses /64s for its clients. A client with the address 2001:10::baad:food:cafe:d00d is part of the first subnet of the PIR 2001:10::/48 (2001:10:0:0/64) and has a host address of baad:food:cafe:d00d. If that user accesses the Internet via provider A their source address will appear as 2001:A::baad:food:cafe:d00d, within the 2001:A::/64 subnet of provider A’s 2001:A::/48. If they egress via provider B they’ll appear as 2001:B:baad:food:cafe:d00d in B’s 2001:B::/64 subnet of 2001:B/48.

If the company wishes to run Internet facing services, then further design work would be needed with a careful consideration of the impact of different failures, but for the small business with blah blah cloud hosting, or the remote sites Ivan mentions, this RFC looks promising. Head office may choose to use NTP for client subnets and PIR for any onsite data centres, thus reducing global routing table explosion and keeping the design simple.

Admittedly I’ve cheated a bit on requirement 1 (No NAT) by replacing it with NPTv6, but requirements 2 (Ubiquitous redundancy) and 3 (Local Internet Exit) are met by this in my opinion.

Arrival of the Camel (Cisco Modelling Labs)

A horse designed by a committee?

A horse designed by a committee?

My Account Manager kindly sent me this link: this morning. So VIRL CML is almost here. Most of what the site said is positive, Cisco has finally listened to its customers and produced an official modelling tool. Sure, it would be better if it could run standalone on my laptop for study on the move. The client server model should offer rock solid performance provided I have 2GB of RAM and 1GB of storage on my laptop, which I do and most modern machines will too. The server requirements aren’t to be sniffed at – 128GB of RAM! I assume that is for those who have purchased the L-CML-CE-100N= licence (increments of 100 licences on top of the base 10+5). The docs do say to visit the page I linked above to determine your memory requirements although I couldn’t find where to do that.

Since it is used in a virtual environment any hardware-related functionality, including Layer 2 functions such as Spanning Tree Protocol, are not currently supported.

This sentence doesn’t make a lot of sense. I could imagine them saying that UDLD isn’t supported for example, but surely STP is just a software feature that happens to run at layer 2? At any rate, I’d rather a rock solid release which is missing features than a feature and bug rich release. Hopefully switches are coming to a camel near you soon!

DMVPN with Crypto

Following on from my previous post, I’ll record how to add encryption to an established DMVPN setup.

A word of warning

This will hammer your throughput. For example, the ASR1001 will do 5Gbps of throughput in ideal conditions [1] but only 1.8Gbps of crypto. This is because traffic which needs to be encrypted needs to be recursively fed back into the Embedded Services Processor (ESP, the card used to manipulate and forward most traffic). Also, unless the size of the packets in your flow is approximately a multiple of the size of the internal cell used to shift traffic around the bus in the ASR, you’ll get padding and achievable throughput will be considerably smaller.

For example, if the cells of data used on the ASRs internal bus were 30 bytes, and your packets were 60 bytes each, then you’d get two cells per packet and 100% efficiency.

| Packet size 60 Bytes | ¬†—> ASR Internal BUS ¬†—> | 30 Byte Cell |¬†30 Byte Cell |

61 byte packets would result in three cells per packet and 29 bytes of padding and a waste of something like half the bandwidth.

| Packet size 61 Bytes | ¬†—> ASR Internal BUS ¬†—> | 30 Byte Cell |¬†30 Byte Cell | 1 Byte Cell + padding |

This is all rather contrived and the statistical nature of network traffic means that real world performance is likely to be between the two extremes.¬†Cell sizes don’t tend to be published as other vendors could use this data to make equipment seem rather worse than it really is.


We create a crypto map and apply it to our tunnel interface. We’ll use a pre-shared-key and tie it to the physical addresses of the routers in our topology. This introduces a snag: in the lab it is easy as we have a nice summarisable range for our fake public IPs. In real life this won’t be the case as you probably won’t know the real IPs of all your spoke routers at initial deployment. To simplify things and in¬†order that the dynamic nature of DMVPN is possible, you’ll probably need to use in the pre-shared key command. For the lab, we just need to cover all the routers (or the spoke-to-spoke tunnels would fail).

Getting to the Config

The following config should be applied to R1, R2, R3 and R4:

crypto isakmp policy 10
encr aes 256
authentication pre-share
group 2
crypto isakmp key 0 DMVPN_LAB address
crypto ipsec transform-set ESP_AES256_SHA_TRANSPORT esp-aes 256 esp-sha-hmac
mode transport
crypto ipsec profile DMVPN_IPSEC_PROFILE
set transform-set ESP_AES256_SHA_TRANSPORT
interface Tunnel0
tunnel protection ipsec profile DMVPN_IPSEC_PROFILE

You’ll need to shut / no shut the tunnel interfaces to re-establish the DMVPN tunnel.


The usual gotcha applies with IPSec tunnels, you need traffic to traverse it to bring the tunnel up. However, we have a nice IGP running so the tunnels to the hub come up:

R4#show crypto isakmp sa
dst src state conn-id slot status QM_IDLE 1001 0 ACTIVE
R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 UP 00:00:17 S

We do a ping to bring up a spoke-to-spoke tunnel:

R4#ping source fa 0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Packet sent with a source address of
Success rate is 100 percent (5/5), round-trip min/avg/max = 36/95/164 ms

R4#show crypto isakmp sa
dst src state conn-id slot status QM_IDLE 1001 0 ACTIVE QM_IDLE 1002 0 ACTIVE

R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:2,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 UP 00:00:49 S
1 UP never D

A word of thanks

This post was helped hugely by reading an old Jeremy Stretch article.