A Letter to Nick Clegg

Dear Nick,

This is not usually a political blog and is unlikely to be again. However, I wanted to write to you on behalf of the British people you have so faithfully served for the past 5 years. Sir, I salute you. You put people above politics, compassion above career and you gave our Government a heart. The way your party has been punished today is unfair and not a reflection of all you stood for and achieved. You can rightly be proud of your record and my hope is that history remembers you in the light you deserve.

So thank you Nick Clegg. We wish you well for the future and, personally, I wish you were still in Government fighting for those values which make such a difference to ordinary people.

Yours Sincerely,

Guy Morrell

Quick Ubuntu Package Update Script

I don’t want to have to remember to login to my syslog server and check for required updates. Here is the script I wrote and added to cron so that I don’t have to:


use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Sender::Transport::Sendmail qw();
use Try::Tiny;

my $to = "noc\@howfantastic.net";
my $from = "root\@syslog.howfantastic.net";
my $subject = "update script";
my $body =  `/usr/bin/apt-get update && /usr/bin/apt-get upgrade -s`;

my $email = Email::Simple->create(
    header=>[To=>$to, From=>$from,

try {
} catch {
    print "Can't send mail: $_";

To get this to work I needed to install some packages:

apt-get install sendmail
apt-get install libemail-sender-perl 
apt-get install libemail-simple-perl


I then added my script to cron:

root@itssys01:~# crontab -l
# Edit this file to introduce tasks to be run by cron.
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# For more information see the manual pages of crontab(5) and cron(8)
# m h  dom mon dow   command
0 0 * * * /usr/local/bin/update-check.pl > /dev/null

Now I get an email with a list of packages which would be updated were I to run the following:

/usr/bin/apt-get upgrade

I don’t want my system blindly updating itself so this way I can still check things over before committing.

Northbound and Southbound APIs

I regularly hear the packet pushers and others discussing Northbound and Southbound APIs and keep forgetting which does what. Here is a summary, comments and corrections welcome.

First a few definitions for context:

  • SDN – Somewhat Defined Networking ūüėČ Or Software..
  • SDN Management Plane – here is the realm of the applications, where network orchestration takes place
  • SDN Control Plane – interacts with the SDN Management Plane and Data Plane. The realm of APIs. In classic SDN this lives on the SDN Contoller
  • SDN Data Plane – where traffic is forwarded, the realm of hardware
  • SDN Controller – Platform on which Applications run. That platform may be a single server, collection of distributed devices in your network or hosted in the cloud

Southbound API

Communication between the SDN Controller (or possibly simply an administrators workstation running C, Java, Phython,… scripts) and network switches, routers and other devices happens via the Southbound API.

Open Standards

  • Openflow¬†– maintained by the Open Network Foundation (ONF)
  • NETCONF¬†– IETF RFC 6241. Standard for managing Network Device Configuration
    • Often used with the YANG modelling language defined in RFC 6020

As an example I might write a Python Script ‘StandardSNMP’ which takes a switch IP as an argument and uses an appropriate southbound API to retrieve and correct its SNMP config based on my companies best practice.

Vendor Proprietary

Northbound / External API

Communication between applications (possibly running on the SDN Controller) and the controller itself. It is a network abstraction presented to high level applications or management systems. This enables developers to write apps for the network without needing to call the southbound API directly. This could mean the same code could be re-used across multiple network devices, from a variety of vendors.

Disclaimer: This is only an example of how I *think* things may work. It doesn’t reflect operational experience.

Say I am writing a configuration application ‘NetConfPLS'[1] which includes a standardisation feature. In my network are a few switches and routers from multiple vendors, all of which support OpenFlow. Once again, I want to verify whether my companies SNMP¬†standard is adhered to across my network and correct it where it is not.

NetConfPLS would make a Northbound API call to the controller such as ‘give me the SNMP state of all devices’. The Controller then makes a Southbound API call to each network device to pull that config and presents this data to NetConfPLS in a standardised way. NetConfPLS then presents this data to the application user.¬†Perhaps the actual config of each device would be shown together with the proposed replacement if running in debug mode.

The user then selects which devices to standardise. NetConfPLS would issue a second Northbound call to the controller like ‘standardise the following devices’. The controller would then issue a southbound call to each device containing the API version of¬†the appropriate configuration commands.

[1] Potential LawSuit



Screen Shot 2015-02-09 at 12.11.31


Hopefully¬†you can start to see the appeal for developers. Just as we used to write Perl libraries (e.g. Net::Appliance::Session) to communicate with devices, now we can use the device manufacturer’s (or the Open Standard) API to do this. Whereas before the script and library were tightly coupled, now the developer can make use of their programming language of choice to communicate with a network abstraction provided by the controller. Scripts or Applications written in one environment could be more easily ported, given to the community or commercialised.

Of course, as ever, each vendor has their own story to sell. I suspect¬†the road to an Open Source controller playing nicely across all platforms is long and perilous. For example, I can’t imagine Cisco would allow a third party controller access to its Northbound APIs. More likely you’ll be forced to buy a Cisco¬†eXtensible Network Controller (XNC) to program your Openflow switches and onePK switches from one place, protecting that revenue stream for Cisco.


Cisco have confused me a little in the way they have brought their simulator to market. First it was VIRL and free, then CML was announced and the name change blamed on marketing. Now we have both. Here is a birds eye view of each for reference. I’ve also included GNS3 as it is worth considering if you’re in the market for an emulation tool.


  • Part of the /dev/innovate program at Cisco
  • In Beta at the time of writing
  • Community supported
  • Runs on Linux KVM Hypervisor / OpenStack frontend
    • This can be installed bare metal or run via VMWare Fusion / Workstation / Player etc
  • Local hardware model (laptop / desktop)
  • Supports IOS, IOS-XE, IOS-XR and NX-OS
  • No Layer 2
  • Annual Licence either Academic or Personal ($80 or $200)
  • Capped at 15 nodes


  • Corporate Edition
  • TAC Supported
  • Client / Server Model
  • IOSv only today
  • NXOS support roadmapped
  • Heavy server requirements
    • Min: 4 Core CPU / 16 GB RAM for basic 15 node limit
    • 16 Core CPU / 128 GB RAM for full 50 nodes
  • Server runs Ubuntu image
  • Java Client (oh Cisco, why do you try my patience so?)
  • $13,000 Base Install (15 node limit)
  • 5% discount for 50 nodes, 10% for 100
  • More pricing detail and other useful information here

All-in-One Virtual Machine

  • Free
  • Limited version of VIRL
  • 3 Node Development Environment for the Open Network Environment Platform Kit (onePK)
  • Designed to allow would-be app writers to develop and test their Apps


  • Free
  • Community Supported via the GNS3 Jungle
  • Multi-Vendor – Cisco, Juniper, HP, Arista, Citrix and Brocade
  • L2 Supported using Cisco L2IOU Images (native on Linux / Solaris only, VMs on Windows / Mac)
    • Although features relying on L3 Hardware such as¬†¬†L3 Etherchannel, ISL trunks, DHCP snooping, Private VLAN, SPAN/RSPAN/ERSPAN, Port-security, Voice VLANs, MLS QoS and QinQ won’t work
  • Supports any Cisco platforms available as a virtual machine using VitualBox or Qemu



Network Cable and Optic Distances

Here is a list of common network cable and optic types, together with their distances. The costs are very rough and a little on the low side, I’ll try and update them.

Speed / Base Media Distance Cisco Part Rough Cost
10/100/1000BaseT Cat 5e 100m GLC-T SFP £150
10GBaseT Twinax Passive 1 to 5m SFP-H10GB-CU<N>M £50-£100
10GBaseT Twinax Active 7 or 10m SFP-H10GB-ACU<N>M
10GBaseT Cat 6 55m N/A N/A
10GBaseT Cat 6a (may be too thick for flood wiring) 100m N/A N/A
1000BaseSX OM1 (62.5/125) 220m GLC-SX-MMD £200
1000BaseSX OM3 (50/125) 550m GLC-SX-MMD £200
1000BaseSX OM4 (50/125) 1000m GLC-SX-MMD £200
10GBaseSR OM1 (63.5/125) 26m SFP-10G-SR £400
10GBaseSR OM3 (50/125) 300m SFP-10G-SR £400
10GBaseSR OM4 (50/125) 400m SFP-10G-SR £400
10GBaseLRM OM1 with mode conditioning patch cord (62.5/125) 220m SFP-10G-LRM £400
10GBaseLRM OM3 (50/125) 220m SFP-10G-LRM £400
10GBaseLRM SM 300m SFP-10G-LRM £400
10GBaseLR SM 10km SFP-10G-LR £1500
X2 to SFP+ Converter OneX N/A CVR-X2-SFP10G £80
40GBaseCR4 -> 4x10GBaseT Twinax Passive Breakout 1-5m QSFP-4SFP10G-CU<N>M
40GBaseCR4 -> 4x10GBaseT Twinax Active Breakout 7 or 10m QSFP-4SFP10G-AC<N>M
40GBaseCR4 Twinax Passive 1-5m QSFP-H40G-CU<N>M
40GBaseCR4 Twinax Active 7 or 10m QSFP-H40G-ACU<N>M
40GBaseAOC -> 4x10G Optical Active Breakout 1-10m QSFP-4X10G-AOC<N>M
40GBaseAOC Optical Active 1-15m QSFP-H40G-AOC<N>M
40GBaseSR4 OM3 (50/125) 100m QSFP-40G-SR4
40GBaseSR4 OM4 (50/125) 150m QSFP-40G-SR4
40GBaseCSR4 OM3 (50/125) 300m QSFP-40G-CSR4 £800
40GBaseCSR4 OM4 (50/125) 550m QSFP-40G-CSR4 £800
40GBaseSR BiDi (Bi-Directional over one fibre pair) OM3 (50/125) 100m QSFP-40G-SR-BD £400
40GBaseSR BiDi OM4 (50/125) 150m QSFP-40G-SR-BD £400
40GBaseLR4 SM 10km QSFP-40GE-LR4

Cisco Nexus TACACS config

According to the team where I work, our standard Catalyst TACACS+ config didn’t work properly. Here is the snippet they used to get going again:

user = DEFAULT {
default service = permit
service = exec {
shell:roles*"network-admin vdc-admin"

User based role authorisation didn’t work well either apparently.

The * rather than the = means it’s an optional attribute so should be ignored by switches which aren’t compatible.

Thanks to Emma Cardinal-Richards for the snippet.

IPv6 in a Global Company – a response: NPTv6?

This is a response this post by Ivan. In summary he is talking about the issues companies will face if they wish to use IPv6 and multi-home. Leaving aside the fact the most Small/Medium Enterprises (SMEs) aren’t going to want to run BGP, the questions of NAT in IPv6 (a terrible idea) and Internet Routing table explosion remain.

I was at the IPv6 Council meeting last week and Eric Vyncke drew our attention to the experimental RFC, RFC6296, IPv6-to-IPv6 Network Prefix Translation (NPTv6). The clue is in the name; rather than translating the entire address and breaking the end-to-end communication model, translate the Prefix section of the IPv6 address and leave the Host part unchanged. Unlike NAT, NPTv6 is stateless which considerably reduces the burden on network equipment. Also ports at L4 can be left alone and asymmetric flows won’t cause an issue.

By way of a simple example, a business is assigned 2001:10::/48 Provider Independent Address (PIR) space. They are a small business with no BGP skills but would like resilience. They take out an Internet connection with two different providers and set up outbound routing however they like. Provider A gives them 2001:A::/48 and provider B allocates 2001:B::/48.

Now, any flows which egress Provider A get prefix translated to 2001:A::/48 and any going out via provider B get 2001:B::/48. For client access to the Internet this is great, hosts can use SLAAC or manually assigned addresses, troubleshooting is easy as any issues can quickly be narrowed down to a host and ISP.

For example, let’s assume the SME uses /64s for its clients. A client with the address 2001:10::baad:food:cafe:d00d is part of the first subnet of the PIR 2001:10::/48 (2001:10:0:0/64) and has a host address of baad:food:cafe:d00d. If that user accesses the Internet via provider A their source address will appear as 2001:A::baad:food:cafe:d00d, within the 2001:A::/64 subnet of provider A’s 2001:A::/48. If they egress via provider B they’ll appear as 2001:B:baad:food:cafe:d00d in B’s 2001:B::/64 subnet of 2001:B/48.

If the company wishes to run Internet facing services, then further design work would be needed with a careful consideration of the impact of different failures, but for the small business with blah blah cloud hosting, or the remote sites Ivan mentions, this RFC looks promising. Head office may choose to use NTP for client subnets and PIR for any onsite data centres, thus reducing global routing table explosion and keeping the design simple.

Admittedly I’ve cheated a bit on requirement 1 (No NAT) by replacing it with NPTv6, but requirements 2 (Ubiquitous redundancy) and 3 (Local Internet Exit) are met by this in my opinion.

Arrival of the Camel (Cisco Modelling Labs)

A horse designed by a committee?

A horse designed by a committee?

My Account Manager kindly sent me this link: www.cisco.com/go/cml this morning. So VIRL CML is almost here. Most of what the site said is positive, Cisco has finally listened to its customers and produced an official modelling tool. Sure, it would be better if it could run standalone on my laptop for study on the move. The client server model should offer rock solid performance provided I have 2GB of RAM and 1GB of storage on my laptop, which I do and most modern machines will too. The server requirements aren’t to be sniffed at – 128GB of RAM! I assume that is for those who have purchased the L-CML-CE-100N= licence (increments of 100 licences on top of the base 10+5). The docs do say to visit the page I linked above to determine your memory requirements although I couldn’t find where to do that.

Since it is used in a virtual environment any hardware-related functionality, including Layer 2 functions such as Spanning Tree Protocol, are not currently supported.

This sentence doesn’t make a lot of sense. I could imagine them saying that UDLD isn’t supported for example, but surely STP is just a software feature that happens to run at layer 2? At any rate, I’d rather a rock solid release which is missing features than a feature and bug rich release. Hopefully switches are coming to a camel near you soon!

DMVPN with Crypto

Following on from my previous post, I’ll record how to add encryption to an established DMVPN setup.

A word of warning

This will hammer your throughput. For example, the ASR1001 will do 5Gbps of throughput in ideal conditions [1] but only 1.8Gbps of crypto. This is because traffic which needs to be encrypted needs to be recursively fed back into the Embedded Services Processor (ESP, the card used to manipulate and forward most traffic). Also, unless the size of the packets in your flow is approximately a multiple of the size of the internal cell used to shift traffic around the bus in the ASR, you’ll get padding and achievable throughput will be considerably smaller.

For example, if the cells of data used on the ASRs internal bus were 30 bytes, and your packets were 60 bytes each, then you’d get two cells per packet and 100% efficiency.

| Packet size 60 Bytes | ¬†—> ASR Internal BUS ¬†—> | 30 Byte Cell |¬†30 Byte Cell |

61 byte packets would result in three cells per packet and 29 bytes of padding and a waste of something like half the bandwidth.

| Packet size 61 Bytes | ¬†—> ASR Internal BUS ¬†—> | 30 Byte Cell |¬†30 Byte Cell | 1 Byte Cell + padding |

This is all rather contrived and the statistical nature of network traffic means that real world performance is likely to be between the two extremes.¬†Cell sizes don’t tend to be published as other vendors could use this data to make equipment seem rather worse than it really is.


We create a crypto map and apply it to our tunnel interface. We’ll use a pre-shared-key and tie it to the physical addresses of the routers in our topology. This introduces a snag: in the lab it is easy as we have a nice summarisable range for our fake public IPs. In real life this won’t be the case as you probably won’t know the real IPs of all your spoke routers at initial deployment. To simplify things and in¬†order that the dynamic nature of DMVPN is possible, you’ll probably need to use in the pre-shared key command. For the lab, we just need to cover all the routers (or the spoke-to-spoke tunnels would fail).

Getting to the Config

The following config should be applied to R1, R2, R3 and R4:

crypto isakmp policy 10
encr aes 256
authentication pre-share
group 2
crypto isakmp key 0 DMVPN_LAB address
crypto ipsec transform-set ESP_AES256_SHA_TRANSPORT esp-aes 256 esp-sha-hmac
mode transport
crypto ipsec profile DMVPN_IPSEC_PROFILE
set transform-set ESP_AES256_SHA_TRANSPORT
interface Tunnel0
tunnel protection ipsec profile DMVPN_IPSEC_PROFILE

You’ll need to shut / no shut the tunnel interfaces to re-establish the DMVPN tunnel.


The usual gotcha applies with IPSec tunnels, you need traffic to traverse it to bring the tunnel up. However, we have a nice IGP running so the tunnels to the hub come up:

R4#show crypto isakmp sa
dst src state conn-id slot status QM_IDLE 1001 0 ACTIVE
R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 UP 00:00:17 S

We do a ping to bring up a spoke-to-spoke tunnel:

R4#ping source fa 0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Packet sent with a source address of
Success rate is 100 percent (5/5), round-trip min/avg/max = 36/95/164 ms

R4#show crypto isakmp sa
dst src state conn-id slot status QM_IDLE 1001 0 ACTIVE QM_IDLE 1002 0 ACTIVE

R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:2,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 UP 00:00:49 S
1 UP never D

A word of thanks

This post was helped hugely by reading an old Jeremy Stretch article.


Intro to DMVPN

Although this has been well blogged before, as we’ll be rolling DMVPN out where I work I’d like to record the steps I took to configure and verify a working example in GNS3.


The first hurdle I came across is that the image I’d upgraded my virtual Cisco 3725s to (
c3725-adventerprisek9-mz124-25.bin, in order to support EIGRP for IPv6) doesn’t have everything you need for DMVPN. Although it seems to be possible to get things working, the ‘show dmvpn’ command in missing. I rolled back to c3725-adventerprisek9-mz.124-15.T14.bin and all was well.


Here is a screenshot of my GNS3 topology. Linknets between routers are shown in green, client networks in dark blue. I’ve included a box showing the multipoint GRE tunnel peers in light blue.

Screen Shot 2014-02-24 at 11.33.29

Base Config

I’ve attached full configs to the post. Prior to configuring DMVPN I set up OSPF on all routers. The intention here is to mimic each sites’ Internet connection. I probably should have used BGP or a static route to via R5 on each router but at this stage I simply wanted all routers to be aware of all green subnets so that the mGRE tunnel would come up.

interface FastEthernet0/0
 description R1
 ip address
interface FastEthernet0/1
 description R2
 ip address
interface FastEthernet1/0
 description R3
 ip address
interface FastEthernet2/0
 description R4
 ip address
router ospf 100
 network area 0
 default-information originate
ip route Null0

A sister config was added to each router (without the default-information originate), lighting up the protocol on fa 0/0 only, that is matching the local green subnet. A quick look at the OSPF entires in R1s routing table shows that we’re ready:

R1#show ip route ospf is subnetted, 1 subnets
O [110/11] via, 01:47:25, FastEthernet0/0 is subnetted, 1 subnets
O [110/20] via, 01:47:25, FastEthernet0/0 is subnetted, 1 subnets
O [110/11] via, 01:47:25, FastEthernet0/0
O*E2 [110/1] via, 01:47:25, FastEthernet0/0

Multipoint GRE Tunnel

I’ve selected for my tunnel interfaces. The hub and all spokes will need a logical interface in this subnet to act as the local tunnel ingress/egress. Each router gets a config like this, the example here being R1. Note that there is no tunnel destination on any router.

interface Tunnel0
 ip address
 tunnel source
 tunnel mode gre multipoint

R2 would have a tunnel source of etc.

The multipoint GRE tunnel is treated like a Non Broadcast Multi Access (NBMA) network, much like point-to-multipoint frame-relay.


The point of DMVPN is that any spoke should be able to bring up a tunnel to any other spoke on demand. We need a mechanism to define our hub router, and for spoke routers to be able to resolve the physical IP address in use by another spoke, enter Next Hop Resolution Protocol. Hub routers – NHRP clients – query the spoke router or Next Hop Server (NHS) to learn the address of another hub router with which they need to bring up a tunnel. It is notable that although each router in the topology has an interface in, this network is not known by R5 (my ‘Internet’ router). Instead, NHRP maps interfaces in this subnets to real interfaces in ‘globally routable’ subnets (inverted commas as I’ve used RFC1918 address in this example).

There a four commands needed to configure NHRP, although only two are used on the hub.

The ‘ip nhrp network-id’ must be the same across the DMVPN instance; it is quite possible and sometimes desirable to run multiple concurrent DMVPN overlays. For this lab, one is enough.

The ‘ip nhrp map multicast’ command sets up a mapping so that our NBMA network can process multicast and unknown unicast packets. We then have two options:

R1(config-if)#ip nhrp map multicast ?
  A.B.C.D  IP NBMA address
  dynamic  Dynamically learn destinations from client registrations on hub

On a spoke, we enter the IP NBMA address of our hub so that broadcast and unknown unicast traffic is sent there. We need to do that so that we can run an IGP over our mGRE network later. In my example that would be On our hub we use the dynamic keyword.

Spokes need an additional map entry to link the hub’s tunnel address to its physical address, ‘ip nhrp map’ in our example.

Finally, for each spoke we need to define its NHS using the ‘ip nhrp nhs’ commend

Bringing it all together, to define our hub we add two commands to Tun0 on R1:

interface Tunnel0
 ip nhrp map multicast dynamic
 ip nhrp network-id 1

To define a spoke, we add four commends to Tun0 on R2, R3 and R4:

interface Tunnel0
 ip nhrp map
 ip nhrp map multicast
 ip nhrp network-id 1
 ip nhrp nhs

Dynamic MVPN

Like many things, DMVPN relies on a sensible routing table on each router. Here is the logical topology:



We need each router to have routes to the other routers’ client networks. EIGRP lends itself well to this. We configure it to match on each router, as well as the local client networks. We’ll make the client interfaces passive to stop our routers peering with anything downstream. Finally we’ll need to tweak the defaults on the hub so that:

* split-horizon doesn’t prevent us advertising subnets back out to the other spokes
* we advertise the next hop IP as that of the originating router

Here is R1’s eigrp config, the only change on the other routers is to match their local client networks as they all use the same local physical interfaces in my lab.

router eigrp 1
 passive-interface FastEthernet0/1
 no auto-summary

On R1 we also add:

interface Tunnel0
! ip address
 no ip next-hop-self eigrp 1
 no ip split-horizon eigrp 1


First, let’s check the routing – each hub and spoke should know about all the other client networks via the mGRE tunnel:

R1#show ip route eigrp is subnetted, 1 subnets
D [90/297270016] via, 02:58:14, Tunnel0 is subnetted, 1 subnets
D [90/297270016] via, 02:58:16, Tunnel0 is subnetted, 1 subnets
D [90/297270016] via, 02:58:16, Tunnel0

R4#show ip route eigrp is subnetted, 1 subnets
D [90/310070016] via, 02:58:06, Tunnel0 is subnetted, 1 subnets
D [90/297270016] via, 02:58:06, Tunnel0 is subnetted, 1 subnets
D [90/310070016] via, 02:58:06, Tunnel0

On the hub, we should see our three peers:

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
	N - NATed, L - Local, X - No Socket
	# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Hub, NHRP Peers:3, 
 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1    UP    never D    
     1    UP    never D    
     1    UP    never D

On each spoke, we’ll just see the hub:

R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
	N - NATed, L - Local, X - No Socket
	# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:1, 
 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1    UP 02:56:54 S

Now, let’s try and ping (R3’s client interface) from (R4’s client interface):

R4#ping source fa 0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Packet sent with a source address of 
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/34/64 ms

Great, it worked. You can see that the dynamic tunnel has come up:

R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
	N - NATed, L - Local, X - No Socket
	# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:2, 
 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1    UP 03:00:22 S    
     1    UP    never D

I’ve uploaded a packet capture taken during the above test (on the R4-R5 link) to cloudshark.

Click on that link and you’ll see the capture filtered to NHRP traffic only. Note that packets 16 and 19 flow via the hub, but 20 and 21 are spoke-to-spoke. If you open up the NHRP Mandatory Part in packet 16 you’ll see that the source and destination protocol addresses are the mGRE tunnel endpoints and the source NBMA address is the physical interface of the router.

Screen Shot 2014-02-24 at 13.17.08

For packet 19 the protocol addresses are reversed (as expected for a reply) but now the source NBMA address is that of R3. Looking at the IP header confirms that this packet came from R1.Screen Shot 2014-02-24 at 13.32.05

In packet 20, NHRP provides R4 with R3’s physical address:


Screen Shot 2014-02-24 at 13.35.39

In packet 21, R4 provides R3 with its physical address.Screen Shot 2014-02-24 at 13.36.12

In packets 20 and 21, open the client information entry section to see that NHRP has returned, the physical address of R3 and endpoint for our dynamic, spoke-to-spoke tunnel.

Finally, let’s have a look at the pings. The first thing I should point out is that we are tunnelling the traffic, so there are two IP headers. In real life I would drop the IP MTU on each Tu interface to allow for the GRE encapsulation and extra IP header, plus any crypto.

Packet 15’s outer IP headers have source and destination physical addresses of R4 and R1. The inner headers are the local client subnets foreach router.¬†Screen Shot 2014-02-24 at 13.47.35This continues until packet 22 (right after NHRP resolved the true address). From this point, the traffic is direct. Note that in this case, the request went via R1, but the reply came direct as the spoke-to-spoke tunnel had come up.

Screen Shot 2014-02-24 at 13.48.41


In this post I’ve described and implemented a simple (crypto-free) DMVPN topology. I may expand on this in future to add encryption.

Update: I’ve covered how to add encryption here.