Category Archives: Service Provider

Quick Ubuntu Package Update Script

I don’t want to have to remember to login to my syslog server and check for required updates. Here is the script I wrote and added to cron so that I don’t have to:

#!/usr/bin/perl

use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Sender::Transport::Sendmail qw();
use Try::Tiny;

my $to = "noc\@howfantastic.net";
my $from = "root\@syslog.howfantastic.net";
my $subject = "update script";
my $body =  `/usr/bin/apt-get update && /usr/bin/apt-get upgrade -s`;

my $email = Email::Simple->create(
    header=>[To=>$to, From=>$from,
             Subject=>$subject],
    body=>$body,
);

try {
    sendmail($email,
             {from=>$from,
             transport=>Email::Sender::Transport::Sendmail->new});
} catch {
    print "Can't send mail: $_";
}

To get this to work I needed to install some packages:

apt-get install sendmail
apt-get install libemail-sender-perl 
apt-get install libemail-simple-perl

 

I then added my script to cron:

root@itssys01:~# crontab -l
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
0 0 * * * /usr/local/bin/update-check.pl > /dev/null

Now I get an email with a list of packages which would be updated were I to run the following:

/usr/bin/apt-get upgrade

I don’t want my system blindly updating itself so this way I can still check things over before committing.

Intro to DMVPN

Although this has been well blogged before, as we’ll be rolling DMVPN out where I work I’d like to record the steps I took to configure and verify a working example in GNS3.

Software

The first hurdle I came across is that the image I’d upgraded my virtual Cisco 3725s to (
c3725-adventerprisek9-mz124-25.bin, in order to support EIGRP for IPv6) doesn’t have everything you need for DMVPN. Although it seems to be possible to get things working, the ‘show dmvpn’ command in missing. I rolled back to c3725-adventerprisek9-mz.124-15.T14.bin and all was well.

Lab

Here is a screenshot of my GNS3 topology. Linknets between routers are shown in green, client networks in dark blue. I’ve included a box showing the multipoint GRE tunnel peers in light blue.

Screen Shot 2014-02-24 at 11.33.29

Base Config

I’ve attached full configs to the post. Prior to configuring DMVPN I set up OSPF on all routers. The intention here is to mimic each sites’ Internet connection. I probably should have used BGP or a static route to 0.0.0.0/0 via R5 on each router but at this stage I simply wanted all routers to be aware of all green subnets so that the mGRE tunnel would come up.

interface FastEthernet0/0
 description R1
 ip address 192.168.15.2 255.255.255.252
!
interface FastEthernet0/1
 description R2
 ip address 192.168.25.2 255.255.255.252
!
interface FastEthernet1/0
 description R3
 ip address 192.168.35.2 255.255.255.252
!
interface FastEthernet2/0
 description R4
 ip address 192.168.45.2 255.255.255.252
!
router ospf 100
 router-id 5.5.5.5
 network 0.0.0.0 255.255.255.255 area 0
 default-information originate
!
ip route 0.0.0.0 0.0.0.0 Null0

A sister config was added to each router (without the default-information originate), lighting up the protocol on fa 0/0 only, that is matching the local green subnet. A quick look at the OSPF entires in R1s routing table shows that we’re ready:

R1#show ip route ospf
192.168.45.0/30 is subnetted, 1 subnets
O 192.168.45.0 [110/11] via 192.168.15.2, 01:47:25, FastEthernet0/0
192.168.25.0/30 is subnetted, 1 subnets
O 192.168.25.0 [110/20] via 192.168.15.2, 01:47:25, FastEthernet0/0
192.168.35.0/30 is subnetted, 1 subnets
O 192.168.35.0 [110/11] via 192.168.15.2, 01:47:25, FastEthernet0/0
O*E2 0.0.0.0/0 [110/1] via 192.168.15.2, 01:47:25, FastEthernet0/0

Multipoint GRE Tunnel

I’ve selected 172.16.0.0/24 for my tunnel interfaces. The hub and all spokes will need a logical interface in this subnet to act as the local tunnel ingress/egress. Each router gets a config like this, the example here being R1. Note that there is no tunnel destination on any router.

interface Tunnel0
 ip address 172.16.0.1 255.255.255.0
 tunnel source 192.168.15.1
 tunnel mode gre multipoint

R2 would have a tunnel source of 192.168.25.1 etc.

The multipoint GRE tunnel is treated like a Non Broadcast Multi Access (NBMA) network, much like point-to-multipoint frame-relay.

NHRP

The point of DMVPN is that any spoke should be able to bring up a tunnel to any other spoke on demand. We need a mechanism to define our hub router, and for spoke routers to be able to resolve the physical IP address in use by another spoke, enter Next Hop Resolution Protocol. Hub routers – NHRP clients – query the spoke router or Next Hop Server (NHS) to learn the address of another hub router with which they need to bring up a tunnel. It is notable that although each router in the topology has an interface in 172.16.0.0/24, this network is not known by R5 (my ‘Internet’ router). Instead, NHRP maps interfaces in this subnets to real interfaces in ‘globally routable’ subnets (inverted commas as I’ve used RFC1918 address in this example).

There a four commands needed to configure NHRP, although only two are used on the hub.

The ‘ip nhrp network-id’ must be the same across the DMVPN instance; it is quite possible and sometimes desirable to run multiple concurrent DMVPN overlays. For this lab, one is enough.

The ‘ip nhrp map multicast’ command sets up a mapping so that our NBMA network can process multicast and unknown unicast packets. We then have two options:

R1(config-if)#ip nhrp map multicast ?
  A.B.C.D  IP NBMA address
  dynamic  Dynamically learn destinations from client registrations on hub

On a spoke, we enter the IP NBMA address of our hub so that broadcast and unknown unicast traffic is sent there. We need to do that so that we can run an IGP over our mGRE network later. In my example that would be 192.168.15.1. On our hub we use the dynamic keyword.

Spokes need an additional map entry to link the hub’s tunnel address to its physical address, ‘ip nhrp map 172.16.0.1 192.168.15.1’ in our example.

Finally, for each spoke we need to define its NHS using the ‘ip nhrp nhs 172.16.0.1’ commend

Bringing it all together, to define our hub we add two commands to Tun0 on R1:

interface Tunnel0
 ip nhrp map multicast dynamic
 ip nhrp network-id 1

To define a spoke, we add four commends to Tun0 on R2, R3 and R4:

interface Tunnel0
 ip nhrp map 172.16.0.1 192.168.15.1
 ip nhrp map multicast 192.168.15.1
 ip nhrp network-id 1
 ip nhrp nhs 172.16.0.1

Dynamic MVPN

Like many things, DMVPN relies on a sensible routing table on each router. Here is the logical topology:

DMVPN

 

We need each router to have routes to the other routers’ client networks. EIGRP lends itself well to this. We configure it to match 172.16.0.0/24 on each router, as well as the local client networks. We’ll make the client interfaces passive to stop our routers peering with anything downstream. Finally we’ll need to tweak the defaults on the hub so that:

* split-horizon doesn’t prevent us advertising subnets back out to the other spokes
* we advertise the next hop IP as that of the originating router

Here is R1’s eigrp config, the only change on the other routers is to match their local client networks as they all use the same local physical interfaces in my lab.

router eigrp 1
 passive-interface FastEthernet0/1
 network 10.10.10.0 0.0.0.255
 network 172.16.0.0 0.0.0.255
 no auto-summary

On R1 we also add:

interface Tunnel0
! ip address 172.16.0.1 255.255.255.0
 no ip next-hop-self eigrp 1
 no ip split-horizon eigrp 1

Verification

First, let’s check the routing – each hub and spoke should know about all the other client networks via the mGRE tunnel:

R1#show ip route eigrp 
     20.0.0.0/24 is subnetted, 1 subnets
D       20.20.20.0 [90/297270016] via 172.16.0.2, 02:58:14, Tunnel0
     40.0.0.0/30 is subnetted, 1 subnets
D       40.40.40.252 [90/297270016] via 172.16.0.4, 02:58:16, Tunnel0
     30.0.0.0/24 is subnetted, 1 subnets
D       30.30.30.0 [90/297270016] via 172.16.0.3, 02:58:16, Tunnel0

R4#show ip route eigrp 
     20.0.0.0/24 is subnetted, 1 subnets
D       20.20.20.0 [90/310070016] via 172.16.0.2, 02:58:06, Tunnel0
     10.0.0.0/24 is subnetted, 1 subnets
D       10.10.10.0 [90/297270016] via 172.16.0.1, 02:58:06, Tunnel0
     30.0.0.0/24 is subnetted, 1 subnets
D       30.30.30.0 [90/310070016] via 172.16.0.3, 02:58:06, Tunnel0

On the hub, we should see our three peers:

R1#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
	N - NATed, L - Local, X - No Socket
	# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Hub, NHRP Peers:3, 
 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1    192.168.25.1      172.16.0.2    UP    never D    
     1    192.168.35.1      172.16.0.3    UP    never D    
     1    192.168.45.1      172.16.0.4    UP    never D

On each spoke, we’ll just see the hub:

R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
	N - NATed, L - Local, X - No Socket
	# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:1, 
 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1    192.168.15.1      172.16.0.1    UP 02:56:54 S

Now, let’s try and ping 30.30.30.254 (R3’s client interface) from 40.40.40.254 (R4’s client interface):

R4#ping 30.30.30.254 source fa 0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 30.30.30.254, timeout is 2 seconds:
Packet sent with a source address of 40.40.40.254 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/34/64 ms

Great, it worked. You can see that the dynamic tunnel has come up:

R4#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incompletea
	N - NATed, L - Local, X - No Socket
	# Ent --> Number of NHRP entries with same NBMA peer

Tunnel0, Type:Spoke, NHRP Peers:2, 
 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1    192.168.15.1      172.16.0.1    UP 03:00:22 S    
     1    192.168.35.1      172.16.0.3    UP    never D

I’ve uploaded a packet capture taken during the above test (on the R4-R5 link) to cloudshark.

Click on that link and you’ll see the capture filtered to NHRP traffic only. Note that packets 16 and 19 flow via the hub, but 20 and 21 are spoke-to-spoke. If you open up the NHRP Mandatory Part in packet 16 you’ll see that the source and destination protocol addresses are the mGRE tunnel endpoints and the source NBMA address is the physical interface of the router.

Screen Shot 2014-02-24 at 13.17.08

For packet 19 the protocol addresses are reversed (as expected for a reply) but now the source NBMA address is that of R3. Looking at the IP header confirms that this packet came from R1.Screen Shot 2014-02-24 at 13.32.05

In packet 20, NHRP provides R4 with R3’s physical address:

 

Screen Shot 2014-02-24 at 13.35.39

In packet 21, R4 provides R3 with its physical address.Screen Shot 2014-02-24 at 13.36.12

In packets 20 and 21, open the client information entry section to see that NHRP has returned 192.168.35.1, the physical address of R3 and endpoint for our dynamic, spoke-to-spoke tunnel.

Finally, let’s have a look at the pings. The first thing I should point out is that we are tunnelling the traffic, so there are two IP headers. In real life I would drop the IP MTU on each Tu interface to allow for the GRE encapsulation and extra IP header, plus any crypto.

Packet 15’s outer IP headers have source and destination physical addresses of R4 and R1. The inner headers are the local client subnets foreach router. Screen Shot 2014-02-24 at 13.47.35This continues until packet 22 (right after NHRP resolved the true address). From this point, the traffic is direct. Note that in this case, the request went via R1, but the reply came direct as the spoke-to-spoke tunnel had come up.

Screen Shot 2014-02-24 at 13.48.41

 Summary

In this post I’ve described and implemented a simple (crypto-free) DMVPN topology. I may expand on this in future to add encryption.

Update: I’ve covered how to add encryption here.

Auto QoS and ASIC / Port mappings on a 6500

Today I came across an interesting qwerk which I felt worth sharing. If you enable auto-qos on a 6500 port, the QoS features will be applied to every port which shares the relevant ASIC. Exactly how this plays out will depend on the architecture of the switch, we need to take a closer look. In this excellent post James Ventre says:

“In the 6500 platform, [the ASIC port mapping is] easily displayed with ‘show interface capabilities’… The portion we’re interested in is labeled ‘Ports-in-ASIC’.”

In my case, I have a WS-X6748-SFP in slot 3 of my 6500 and I’m interested in port 1:

my-6500# show interfaces gig 3/1 capabilities | i ASIC
 Ports-in-ASIC (Sub-port ASIC) : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 (1,3,5,7,9,11,13,15,17,19,21,23)

According to this post, there are three levels of ASIC on a 6748 line card:

  1. Two JANUS Bus ASICs
  2. Two SSA Fabric ASICs
  3. Four ROHINI Port ASICs
The section in parenthesis refers to the ASIC we are interested in. The backplane on a sup-720 offers 40G per line card. My 6748 has 48 ports, managed by four ROHINI or Sub-port ASICs, each connected at 10Gbps (as an aside, this gives an oversubscription ratio of 1.2:1 as 12 Gigabit port has access a 10G connection to the backplane).
When I enabled auto-qos on interface Gig 3/1, I saw the following on the remaining (default config) ports sharing its ROHINI ASIC:
interface GigabitEthernet3/n ! where n is odd and < 24
 no ip address
 shutdown
 wrr-queue cos-map 2 1 1 2
 wrr-queue cos-map 3 5 3 4
 wrr-queue cos-map 3 7 6 7
 rcv-queue cos-map 1 2 1
 rcv-queue cos-map 1 3 2
 rcv-queue cos-map 1 4 3
 rcv-queue cos-map 1 5 4
 rcv-queue cos-map 1 6 5
 rcv-queue cos-map 1 7 6
 rcv-queue cos-map 1 8 7

So there you have it – no need to panic if a load of config appears on your box from a single auto-qos command. However, do be aware of your switch architecture before trying to squeeze everything you can out of an old line card.

6500 Transit ACLs

A while ago we found that our FWSMs were no longer up to the job. As it happens we were doing nothing more than stateless packet filtering with them so could replicate their functionality with access-lists. I noticed the hit count on the ACLs was rather low and came across this blog post, which explained what was going on rather well.

In summary:

show ip access-list NAME

will only show packets destined for the router, not those passing through it. To see transit traffic you need this command:

show tcam interface <INT> acl [in | out] ip

This makes sense given that ACLs are implemented in TCAM on the 6500 platform.

Verification

We have an interface which originates a default route into our campus, with an ACL in each direction applied to it. We permit ip any any at the end after filtering out the cruft we don’t want to see. Here is a comparison of the two commands for that ACL.

ROUTER#show ip access-lists TO_CAMPUS | i permit ip any
    230 permit ip any any (4475 matches)

Given that this ACL has around 30,000 users behind it and the counters had been cleared earlier on the day the command was run, I would expect number to be larger.

Now the tcam command:

ROUTER#show tcam interface vlan 80 acl out ip | i ip any any
    permit       ip any any (4141914 matches)

That is more realistic.

Janet6

I watched this presentation on Janet6 today. Since I work in ac.uk I’m always interested in their developments. Some key points:

  1. Exponential growth in day to day inbound traffic from external peers since records began in 2005. Peaked at 120Gbps last summer. Forecasted to reach 240Gbps in 2014, 0.5 Tbps by 2016.
  2. More and more hungry ‘big Science’ users are also driving the requirement for further bandwidth (LHC, Square Kilometer Array, Bioinformatics Genome Transfers, ITER Fusion Reactor, Radio Astronomy, Climate Data etc).
  3. To cope with this Janet6 will have a brand new network, to be deployed in parallel to SuperJanet5.
  4. Janet6 will use different PoP to Janet5 with all sites to be migrated by October 2013.
  5. For the first time Janet will operate the optical layer directly using DWDM and dedicated hardware to manage this.
  6. A new optical network is being procured with the SSE Telecoms to be the provider.
  7. The Juniper routers will be upgraded to T4000s which provide 240Gbps per slot or 192 10Gbps ports per chassis.
  8. The lightpath service will now be run over the existing EoMPLS network with upgraded 100GE interconnects.
  9. A new non-CIR L2 service will be offered using the MPLS backbone in addition to this.
  10. Requirement for a second connection to Janet becoming more common for institutions. Details around how these are financed are still being worked out.

This guy’s opinion

It is always good to see proactive capacity management in action. Higher Education in the UK is often seen as being at the bleeding edge of networking. The bandwidth figures here would seem to corroborate that point of view – it is one of the reasons ac.uk is such a fun environment to work in. Certainly science departments at my place of work are demanding more and more bandwidth, as are our 30,000 users. The new backbone looks very beefy both at Layer 1 and above and that is a good thing.

Resilience is a word I’m hearing more and more at work and like many others it would seem, we are in the process of introducing a second connection to Janet (maybe we’re not bleeding edge there). I’ll be interested in how this develops. For us it was affordable because we are a PoP for several other institutions so we are paid to host the router. In tight economic times others may find the expense hard to justify if they have to go it alone, especially given the incredible reliability of Janet5. The only outages I’ve experienced have been as a result of our FWSMs flaking out or IOS upgrades on our border 6500.

Finally, I’m pleased that the lightpath service is getting some love. I wonder if Universities would start co-locating their datacentres and use lightpath as the DCI if the service could be fast and reliable enough?

Diagram

Using BGP communities and local preference

At my current place of work we have a single ISP. We have two routed connections spread across two line cards, on a dual-sup Cisco 6500. However, with 30,000 Internet hungry users and the upcoming 6500 chassis replacement this no longer offers enough resilience. We’re going to move one of our links to a second router and do some simple load balancing using a combination of BGP communities and local preference.

I’ve mimicked our IPv4 unicast setup as far as I am able using Dynamips and RFC1918 addresses. I will document it here. We also run IPv6 and IPv4 multicast but the setup is the same.

Requirements

  1. We want both links to be active. If one site were to fail, we want to know the other site is ready to go.
  2. We don’t want routing loops.We do want to stay online if the uplink interface fails on either of our routers. Therefore we’ll run iBGP between our two Internet routers.
  3. iBGP peers use loopbacks. Our two Internet routers have multiple paths between them so we’ll peer on loopbacks. For this lab they only have one link. I’ll use OSPF so that all my lab routers know each others loopback addresses.
  4. eBGP peers use interface addresses. We don’t run an IGP with our ISP so we need static routes to peer on loopbacks. Now that each router will only have one link, this isn’t needed.

Network Diagram

We want to draw traffic to 10.1.0.0/16 via 3.3.3.3, and traffic to 10.2.0.0/16 via 4.4.4.4. Both routers will offer a default route to our IGP if they have one.

Diagram

Outbound traffic

Our ISP will offer us a default route. We will configure each Internet router so that if it has a valid route to 0/0, it will advertise it into our IGP. Therefore outbound traffic will flow via both Internet routers in normal operation.

Inbound traffic

We have two /16s and will configure each Internet router to draw traffic for just one of them in normal operation. Our ISP will accept two BGP communities from its customers. Any routes including 64511:1 will get a better (higher) local pref than routes with 64511:2. Although I don’t know exactly how they achieve this, I will mimic it by setting the local pref to 150 and 50 respectively.

Asymmetric routing then?

This configuration does not guarantee that the traffic will flow in and out the same way. However, this shouldn’t matter. My own experience is that turning on HSRP causes a similar issue locally [1] and that hasn’t broken any applications yet.

Configuration

I’ve attached full Router Configs to this post.

A basic BGP setup with no funny stuff

The four routers are peering with one another as per the diagram.

  • 3.3.3.3 and 4.4.4.4 are both offering the /16s with no tweaks.
  • 1.1.1.1 and 2.2.2.2 are offering 0.0.0.0/0 with no tweaks.

So we have resiliency in our Internet connection. However, we have no influence over which router our ISP will use as the next hop for our prefixes. Since BGP will choose one next hop for a given prefix by default, only one of our routers will get used for inbound traffic until we take further action.

Community spirit

Here is what we need to do on each enterprise router (3.3.3.3 & 4.4.4.4):

  1. Create an ACL or prefix list to match the subnets we wish to influence
  2. Create a route-map to add the BGP community we chose to each prefix
  3. Configure BGP so that BGP communities are offered and the route-map is applied to our eBGP peer.

Our ISP routers (1.1.1.1 & 2.2.2.2) will need the following:

  1. Create a community list for each BGP community we will accept
  2. Create a route-map to match on the appropriate list and the adjust the local pref
  3. Configure BGP to apply the route-map to the appropriate neighbor.
Here is the relevant config:
! R3
ip prefix-list 10_1 seq 5 permit 10.1.0.0/16
ip prefix-list 10_2 seq 5 permit 10.2.0.0/16
!
route-map TO_ISP permit 10
 match ip address prefix-list 10_1
 set community 64496:1
route-map TO_ISP permit 20
 match ip address prefix-list 10_2
 set community 64496:2
!
router bgp 64496
 bgp log-neighbor-changes
 neighbor 4.4.4.4 remote-as 64496
 neighbor 4.4.4.4 update-source Loopback3
 neighbor 192.168.13.1 remote-as 64511
 !
 address-family ipv4
  neighbor 4.4.4.4 activate
  neighbor 192.168.13.1 activate
  neighbor 192.168.13.1 send-community
  neighbor 192.168.13.1 soft-reconfiguration inbound
  neighbor 192.168.13.1 route-map TO_ISP out
  no auto-summary
  no synchronization
  network 10.1.0.0 mask 255.255.0.0
  network 10.2.0.0 mask 255.255.0.0
 exit-address-family

R4 is pretty well the same, apart from the route-map which reverse the community choice:

route-map TO_ISP permit 10
 match ip address prefix-list 10_1
 set community 64496:2
route-map TO_ISP permit 20
 match ip address prefix-list 10_2
 set community 64496:1 

R1 and R2  have virtually the same config:

! R1
ip community-list 1 permit 64496:1
ip community-list 2 permit 64496:2
!
route-map FROM_CUSTOMER permit 10
 match community 1
 set local-preference 150
route-map FROM_CUSTOMER permit 20
 match community 2
 set local-preference 50
!
router bgp 64511
 bgp log-neighbor-changes
 neighbor 2.2.2.2 remote-as 64511
 neighbor 2.2.2.2 update-source Loopback1
 neighbor 192.168.13.2 remote-as 64496
 !
 address-family ipv4
  neighbor 2.2.2.2 activate
  neighbor 192.168.13.2 activate
  neighbor 192.168.13.2 soft-reconfiguration inbound
  neighbor 192.168.13.2 route-map FROM_CUSTOMER in
  no auto-summary
  no synchronization
  network 0.0.0.0
 exit-address-family

Verification

First we verify on R1. Notice that it only has one route for 10.1/16 because R2’s best route for 10.1/16 is learned from R1 via iBGP – iBGP rules mean it won’t re-advertise the prefix.

R1#show ip bgp 10.0.0.0/8 longer-prefixes 
BGP table version is 5, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.0.0/16      192.168.13.2             0    150      0 64496 i
*>i10.2.0.0/16      192.168.24.2             0    150      0 64496 i
*                   192.168.13.2             0     50      0 64496 i

Then on R2 we notice a similar situation, although now we only have one route for 10.2/16:

R2#show ip bgp 10.0.0.0/8 longer-prefixes 
BGP table version is 7, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i10.1.0.0/16      192.168.13.2             0    150      0 64496 i
*                   192.168.24.2             0     50      0 64496 i
*> 10.2.0.0/16      192.168.24.2             0    150      0 64496 i

Thoughts

This method words well in our environment as our IPv4 allocation gives us a natural way of roughly dividing our traffic between our two Internet routers. This lab has also shown me one of the reasons ISPs like BGP communities. They are simple, extensible, scalable and flexible. In this case we can select our primary router – the ISP could easily overwrite our choice. Clearly in a production network the ISP would be a little more careful about which routes it accepts from its customers. Even so, the config is simple and powerful.


Appendix – the basic BGP setup

R1 and R2 each have a static route as follows:

 ip route 0.0.0.0 0.0.0.0 Serial1/5 192.168.[1|2]5.2

R1 learns both /16s from both of its R3 and R2. It also learns another router to 0/0 from R2:

R1#show ip bgp summary | b Neighbor
Neighbor        V          AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4      64511      21      20        4    0    0 00:15:52        3
192.168.13.2    4      64496      39      40        4    0    0 00:37:06        2

Since it has a local route for 0/0 with a local preference of 1, the RTM prefers that to the iBGP route so we only see two BGP routes.

R1#show ip route bgp
     10.0.0.0/16 is subnetted, 2 subnets
B       10.2.0.0 [20/0] via 192.168.13.2, 00:38:08
B       10.1.0.0 [20/0] via 192.168.13.2, 00:38:08

R2 is the same as R1.

R3 and R4 learn a default route only from their eBGP neighbor. They learn the two /16s from each other, as well as the default route.

R4#show ip bgp summary | b Neighbor
Neighbor        V          AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
3.3.3.3         4      64496      25      24       11    0    0 00:21:03        3
192.168.24.1    4      64511      30      32       11    0    0 00:26:37        1 

Again, I’m only showing one router’s output.

R4#show ip route bgp
B*   0.0.0.0/0 [20/0] via 192.168.24.1, 00:22:36

Before I start mucking about with metrics, I’ll just shut s1/5 on R2. That should drop the static route from the RIB. This should cause 4.4.4.4 to start using 3.3.3.3 as its next hop for the Internets:

R4#show ip route bgp
B*   0.0.0.0/0 [20/0] via 192.168.24.1, 00:41:27
R4#
*Jul 16 11:37:02.595: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/2, changed state to down
*Jul 16 11:37:02.631: %BGP-5-ADJCHANGE: neighbor 192.168.24.1 Down Interface flap
R4#show ip route bgp
B*   0.0.0.0/0 [200/0] via 192.168.13.1, 00:00:06
Which it does.

[1] The difference is that with HSRP we find that both routers advertise the prefixes for incoming traffic but only one receives outgoing traffic.

Network Diagram

Adding a second Internet route to our MPLS VPNs.

At my current place of work we use MPLS VPNs for our wireless service. The design wasn’t mine, but it was done well and has enabled us to remove a couple of nasty trunked VLANs from our core. The only flaw was that it relied on a single 6500 chassis for Internet connectivity. We’re busy forklifting out our 6500 chassis to replace them with 6500Es as they are going end of support in November.  Since we were allowing 4 hours for the upgrade I wanted to add a second link to the Internets.

Network Diagram

The configuration is typical of an MPLS VPN with some subtleties around CE PE routing which isn’t important here. A requirement was that clients connected to different PE routers and clients on different WLANs should not have visibility of each other. This was achieved by using a different VRF for each direction of traffic. The outbound VRFs only import an RT carrying the default route. The inbound VRF imports routes from all outbound VRFs so the traffic can return.

Here are the important bits of config for the Internet VRF. Clearly you need iBGP, MPLS and LDP configured for this to work. We redistribute connected routes so that the client WLANs have a route for the gateway of last resort in their routing tables.

ip vrf WIRELESS-EGRESS
 rd 5.5.5.5:99
 route-target export 5.5.5.5:99
 route-target import 5.5.5.5:10
 route-target import 3.3.3.3:10
! and the rest
router bgp 65501
 ! <snip>
 address-family ipv4
  vrf WIRELESS-EGRESS
  redistribute connected
  no synchronization
  network 0.0.0.0
 exit-address-family
!
ip route vrf WIRELESS-EGRESS 0.0.0.0 0.0.0.0 ser 1/2 192.168.25.1
!
int s 1/2
 ip address 192.168.25.2 255.255.255.252
 ip vrf forwarding WIRELESS-EGRESS
!

So on my second 6500 I added similar config. Although the VRF name and RD are locally significant, I’ve used different ones to keep things clear for myself:

ip vrf WIRELESS-EGRESS2
 rd 6.6.6.6:99
 route-target export 6.6.6.6:99
 route-target import 3.3.3.3:10
!
router bgp 65501
 ! <snip>
 address-family ipv4 vrf WIRELESS-EGRESS2
  redistribute connected
  redistribute static
  no synchronization
  network 0.0.0.0 route-map WIRELESS-EGRESS2-MAP
 exit-address-family
!
ip route vrf WIRELESS-EGRESS2 0.0.0.0 0.0.0.0 ser 1/2 192.168.26.1 tag 10
!
int s 1/2
 ip address 192.168.26.2 255.255.255.252
 ip vrf forwarding WIRELESS-EGRESS2

The route-map simply drops the local-pref to 50 on our second default route, so that the main route is used where available (we have only one application firewall). It also drops the weight to 0 so that hosts using the second 6500 as their gateway don’t default to the local route).

route-map WIRELESS-EGRESS2-RM
 permit 10 match tag 10
 set local-preference 50
 set weight 0

The Internet router, 2.2.2.2 on the diagram, has a couple of static routes to permit return traffic, with the route via our firewall preferred.

ip route 0.0.0.0 0.0.0.0 Serial1/6 192.168.26.1 210
ip route 0.0.0.0 0.0.0.0 Serial1/5 192.168.25.1 220

Finally we just need to import both Internet RTs at each client VRF. We redistribute connected routes into BGP too or else traffic wouldn’t get back. Here is a config sample:

route bgp 65501
 ! <SNIP>
 address-family ipv4 vrf WLAN1
  redistribute connected
  no synchronization
 exit-address-family
 !
ip vrf WLAN1
 rd 3.3.3.3:10
 route-target export 3.3.3.3:10
 route-target import 5.5.5.5:99
 route-target import 6.6.6.6:99

Let’s see if this has worked. First we look at the WLAN RIB before we add our second router:

R3#show ip route vrf WLAN1

Gateway of last resort is 5.5.5.5 to network 0.0.0.0
     192.168.25.0/30 is subnetted, 1 subnets
B       192.168.25.0 [200/0] via 5.5.5.5, 01:05:24
     10.0.0.0/24 is subnetted, 2 subnets
C       10.10.30.0 is directly connected, Loopback10
B*   0.0.0.0/0 [200/0] via 5.5.5.5, 01:05:24

Now we add the second router and wait for the neighborships to come up:

*Jul  3 17:51:46.079: %BGP-5-ADJCHANGE: neighbor 6.6.6.6 Up 
R3#show ip route vrf WLAN1
<SNIP>
Gateway of last resort is 6.6.6.6 to network 0.0.0.0

     192.168.25.0/30 is subnetted, 1 subnets
B       192.168.25.0 [200/0] via 5.5.5.5, 01:15:23
     192.168.26.0/30 is subnetted, 1 subnets
B       192.168.26.0 [200/0] via 6.6.6.6, 00:00:04
     10.0.0.0/24 is subnetted, 2 subnets
C       10.10.30.0 is directly connected, Loopback10
B*   0.0.0.0/0 [200/0] via 6.6.6.6, 00:00:04

If we shut any relevant interfaces on or reboot 6.6.6.6, then the previous state is restored.

*Jul  3 17:57:41.399: %BGP-5-ADJCHANGE: neighbor 6.6.6.6 Down Peer closed the session
R3#show ip route vrf WLAN1

Gateway of last resort is 5.5.5.5 to network 0.0.0.0

     192.168.25.0/30 is subnetted, 1 subnets
B       192.168.25.0 [200/0] via 5.5.5.5, 01:21:05
     10.0.0.0/24 is subnetted, 2 subnets
C       10.10.30.0 is directly connected, Loopback10
B*   0.0.0.0/0 [200/0] via 5.5.5.5, 00:00:02

Technical detail about our environment is deliberately very light as this post is only about adding the second route to the Internet.