Category Archives: Enterprise

ASA source port

I was troubleshooting an issue today where an ASA 5505 had been configured as a replacement for an old PIX, but authentication with an external RADIUS server was failing.

A number of stateful firewalls sit between the outside interface of the ASA in our enterprise network and the public Internet. However the outbound rules permitted the appropriate traffic.

First I set up a capture to verify things from the perspective of the ASA:

! Version 8.X
!
access-list CAPTURE permit ip any host 192.0.2.10
access-list CAPTURE permit ip host 192.0.2.10 any
!
! Version 9.X has separate ACEs for ipv4 and ipv6, if you enter the above you'll get:
! ERROR: Capture doesn't support access-list containing mixed policies
! so, change the ACL to look like this:
!
access-list CAPTURE permit ip any4 host 192.0.2.10
access-list CAPTURE permit ip host 192.0.2.10 any4
!
! Now enable the capture
capture RADIUS access-list CAPTURE interface outside

 

Here is the output. The ASA actually has a public IP.

ASA# show capture RADIUS

54 packets captured

1: 12:16:06.883545 802.1Q vlan#8 P0 172.16.1.1.1025 > 192.0.2.10.1645: udp 158
2: 12:16:07.975794 802.1Q vlan#8 P0 172.16.1.1.1025 > 192.0.2.10.1645: udp 158
3: 12:16:09.075771 802.1Q vlan#8 P0 172.16.1.1.1025 > 192.0.2.10.1645: udp 158
…..

You can also view the capture in a web browser like this: https:///capture/RADIUS. Or you can download the file for wireshark analysis by going to an URL like this: https:///capture/RADIUS/pcap. Clearly you’ll need to substitute ‘RADIUS’ for whatever you called your capture.

Also, if you’re captures are big, you can used the circular-buffer and buffer command to make the buffer eat its own tail and increase the size (up to around 32M).

Finally, use the headers-only keyword if you aren’t interested in the content of the packet.

Okay, so we weren’t getting a response. It was at this point that I checked the external interfaces on our Internet facing routers and found an inbound ACL which did not permit traffic to UDP 1025. One minor, conservative adjustment to the ACLs later and we were in business.

Summary

I’ve come across the ASA source port issue before. Blocking TCP/UDP 1024-5 is a common policy as there were well known trojans which used them. Something to look out for when migrating to an ASA, or any system which uses those ports as a source. I see that the question has already been asked, but if anyone knows how to change the source port an ASA uses for radius requests, do let me know!

Migrating to flexible netflow

Background

We deployed some 6500s running IOS 15.0(1)SY and when I came to configure netflow I found the usual commands didn’t work. The configuration guide didn’t contain anything helpful but I came across this whitepaper which did. What follows is the config I deployed. It doesn’t quite match the whitepaper as the syntax is slightly different on our code, but it does work.

Configuration

Here is how you declare a simple exporter. This is the server collecting the netflow data.

flow exporter MY_EXP
 destination 192.0.2.14
 source Loopback0
 transport udp 19221
 export-protocol netflow-v5
!

Then you define a monitor. This is just a container for the exporter, as well as a place you can select what to export (it is a bit more flexible you see):

flow monitor MY_MON
 record platform-original ipv4 full
 exporter MY_EXP
!

Then you enable netflow on the interface(s) you wish to monitor:

int vlan 31
 ip flow monitor MY_MON input
!

In case you were wondering, our standard netflow config looks like this:

ip flow-export source Loopback0
ip flow-export version 5
ip flow-export destination 192.0.2.14 19211
int vlan 31
 ip flow ingress
!

Verification

Check the exporter:

ROUTER#show flow exporter MY_EXP            
Flow Exporter MY_EXP:
  Description:              User defined
  Export protocol:          NetFlow Version 5
  Transport Configuration:
    Destination IP address: 192.0.2.14
    Source IP address:      192.168.1.1
    Source Interface:       Loopback0
    Transport Protocol:     UDP
    Destination Port:       19211
    Source Port:            55248
    DSCP:                   0x0
    TTL:                    255
    Output Features:        Not Used

Check the monitor:

ROUTER#show flow monitor MY_MON
Flow Monitor MY_MON:
  Description:       User defined
  Flow Record:       platform-original ipv4 full
  Flow Exporter:     MY_EXP
  Cache:
    Type:              normal
    Status:            allocated
    Size:              4096 entries / 278544 bytes
  Cache:
    Type:              normal (Platform cache)
    Status:            allocated
    Size:              Unknown
  Timers:
                       Local        Global
    Inactive Timeout:  15 secs      300 secs
    Active Timeout:    1800 secs    1920 secs
    Update Timeout:    1800 secs
    Fast Timeout:                   Disabled

Check UDP datagrams are being sent:

ROUTER#show flow exporter MY_EXP statistics 
Flow Exporter MY_EXP:
  Packet send statistics (last cleared 00:14:09 ago):
Successfully sent:         1                     (100 bytes)
Client send statistics:
Client: Flow Monitor CERT_MON
Records added: 1
- sent: 1
Bytes added: 48
- sent: 48

This guy’s opinion

I had to dig around and do a bit of guessing to get this working, but it only took about five minutes. At this stage there isn’t much to say, other than I’m looking forward to digging into the finer grained goodness that can be extracted from flexible netflow, and that I’m glad to have a basic config which works on my new boxes.

 

 

Diagram

Using BGP communities and local preference

At my current place of work we have a single ISP. We have two routed connections spread across two line cards, on a dual-sup Cisco 6500. However, with 30,000 Internet hungry users and the upcoming 6500 chassis replacement this no longer offers enough resilience. We’re going to move one of our links to a second router and do some simple load balancing using a combination of BGP communities and local preference.

I’ve mimicked our IPv4 unicast setup as far as I am able using Dynamips and RFC1918 addresses. I will document it here. We also run IPv6 and IPv4 multicast but the setup is the same.

Requirements

  1. We want both links to be active. If one site were to fail, we want to know the other site is ready to go.
  2. We don’t want routing loops.We do want to stay online if the uplink interface fails on either of our routers. Therefore we’ll run iBGP between our two Internet routers.
  3. iBGP peers use loopbacks. Our two Internet routers have multiple paths between them so we’ll peer on loopbacks. For this lab they only have one link. I’ll use OSPF so that all my lab routers know each others loopback addresses.
  4. eBGP peers use interface addresses. We don’t run an IGP with our ISP so we need static routes to peer on loopbacks. Now that each router will only have one link, this isn’t needed.

Network Diagram

We want to draw traffic to 10.1.0.0/16 via 3.3.3.3, and traffic to 10.2.0.0/16 via 4.4.4.4. Both routers will offer a default route to our IGP if they have one.

Diagram

Outbound traffic

Our ISP will offer us a default route. We will configure each Internet router so that if it has a valid route to 0/0, it will advertise it into our IGP. Therefore outbound traffic will flow via both Internet routers in normal operation.

Inbound traffic

We have two /16s and will configure each Internet router to draw traffic for just one of them in normal operation. Our ISP will accept two BGP communities from its customers. Any routes including 64511:1 will get a better (higher) local pref than routes with 64511:2. Although I don’t know exactly how they achieve this, I will mimic it by setting the local pref to 150 and 50 respectively.

Asymmetric routing then?

This configuration does not guarantee that the traffic will flow in and out the same way. However, this shouldn’t matter. My own experience is that turning on HSRP causes a similar issue locally [1] and that hasn’t broken any applications yet.

Configuration

I’ve attached full Router Configs to this post.

A basic BGP setup with no funny stuff

The four routers are peering with one another as per the diagram.

  • 3.3.3.3 and 4.4.4.4 are both offering the /16s with no tweaks.
  • 1.1.1.1 and 2.2.2.2 are offering 0.0.0.0/0 with no tweaks.

So we have resiliency in our Internet connection. However, we have no influence over which router our ISP will use as the next hop for our prefixes. Since BGP will choose one next hop for a given prefix by default, only one of our routers will get used for inbound traffic until we take further action.

Community spirit

Here is what we need to do on each enterprise router (3.3.3.3 & 4.4.4.4):

  1. Create an ACL or prefix list to match the subnets we wish to influence
  2. Create a route-map to add the BGP community we chose to each prefix
  3. Configure BGP so that BGP communities are offered and the route-map is applied to our eBGP peer.

Our ISP routers (1.1.1.1 & 2.2.2.2) will need the following:

  1. Create a community list for each BGP community we will accept
  2. Create a route-map to match on the appropriate list and the adjust the local pref
  3. Configure BGP to apply the route-map to the appropriate neighbor.
Here is the relevant config:
! R3
ip prefix-list 10_1 seq 5 permit 10.1.0.0/16
ip prefix-list 10_2 seq 5 permit 10.2.0.0/16
!
route-map TO_ISP permit 10
 match ip address prefix-list 10_1
 set community 64496:1
route-map TO_ISP permit 20
 match ip address prefix-list 10_2
 set community 64496:2
!
router bgp 64496
 bgp log-neighbor-changes
 neighbor 4.4.4.4 remote-as 64496
 neighbor 4.4.4.4 update-source Loopback3
 neighbor 192.168.13.1 remote-as 64511
 !
 address-family ipv4
  neighbor 4.4.4.4 activate
  neighbor 192.168.13.1 activate
  neighbor 192.168.13.1 send-community
  neighbor 192.168.13.1 soft-reconfiguration inbound
  neighbor 192.168.13.1 route-map TO_ISP out
  no auto-summary
  no synchronization
  network 10.1.0.0 mask 255.255.0.0
  network 10.2.0.0 mask 255.255.0.0
 exit-address-family

R4 is pretty well the same, apart from the route-map which reverse the community choice:

route-map TO_ISP permit 10
 match ip address prefix-list 10_1
 set community 64496:2
route-map TO_ISP permit 20
 match ip address prefix-list 10_2
 set community 64496:1 

R1 and R2  have virtually the same config:

! R1
ip community-list 1 permit 64496:1
ip community-list 2 permit 64496:2
!
route-map FROM_CUSTOMER permit 10
 match community 1
 set local-preference 150
route-map FROM_CUSTOMER permit 20
 match community 2
 set local-preference 50
!
router bgp 64511
 bgp log-neighbor-changes
 neighbor 2.2.2.2 remote-as 64511
 neighbor 2.2.2.2 update-source Loopback1
 neighbor 192.168.13.2 remote-as 64496
 !
 address-family ipv4
  neighbor 2.2.2.2 activate
  neighbor 192.168.13.2 activate
  neighbor 192.168.13.2 soft-reconfiguration inbound
  neighbor 192.168.13.2 route-map FROM_CUSTOMER in
  no auto-summary
  no synchronization
  network 0.0.0.0
 exit-address-family

Verification

First we verify on R1. Notice that it only has one route for 10.1/16 because R2’s best route for 10.1/16 is learned from R1 via iBGP – iBGP rules mean it won’t re-advertise the prefix.

R1#show ip bgp 10.0.0.0/8 longer-prefixes 
BGP table version is 5, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.0.0/16      192.168.13.2             0    150      0 64496 i
*>i10.2.0.0/16      192.168.24.2             0    150      0 64496 i
*                   192.168.13.2             0     50      0 64496 i

Then on R2 we notice a similar situation, although now we only have one route for 10.2/16:

R2#show ip bgp 10.0.0.0/8 longer-prefixes 
BGP table version is 7, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i10.1.0.0/16      192.168.13.2             0    150      0 64496 i
*                   192.168.24.2             0     50      0 64496 i
*> 10.2.0.0/16      192.168.24.2             0    150      0 64496 i

Thoughts

This method words well in our environment as our IPv4 allocation gives us a natural way of roughly dividing our traffic between our two Internet routers. This lab has also shown me one of the reasons ISPs like BGP communities. They are simple, extensible, scalable and flexible. In this case we can select our primary router – the ISP could easily overwrite our choice. Clearly in a production network the ISP would be a little more careful about which routes it accepts from its customers. Even so, the config is simple and powerful.


Appendix – the basic BGP setup

R1 and R2 each have a static route as follows:

 ip route 0.0.0.0 0.0.0.0 Serial1/5 192.168.[1|2]5.2

R1 learns both /16s from both of its R3 and R2. It also learns another router to 0/0 from R2:

R1#show ip bgp summary | b Neighbor
Neighbor        V          AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4      64511      21      20        4    0    0 00:15:52        3
192.168.13.2    4      64496      39      40        4    0    0 00:37:06        2

Since it has a local route for 0/0 with a local preference of 1, the RTM prefers that to the iBGP route so we only see two BGP routes.

R1#show ip route bgp
     10.0.0.0/16 is subnetted, 2 subnets
B       10.2.0.0 [20/0] via 192.168.13.2, 00:38:08
B       10.1.0.0 [20/0] via 192.168.13.2, 00:38:08

R2 is the same as R1.

R3 and R4 learn a default route only from their eBGP neighbor. They learn the two /16s from each other, as well as the default route.

R4#show ip bgp summary | b Neighbor
Neighbor        V          AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
3.3.3.3         4      64496      25      24       11    0    0 00:21:03        3
192.168.24.1    4      64511      30      32       11    0    0 00:26:37        1 

Again, I’m only showing one router’s output.

R4#show ip route bgp
B*   0.0.0.0/0 [20/0] via 192.168.24.1, 00:22:36

Before I start mucking about with metrics, I’ll just shut s1/5 on R2. That should drop the static route from the RIB. This should cause 4.4.4.4 to start using 3.3.3.3 as its next hop for the Internets:

R4#show ip route bgp
B*   0.0.0.0/0 [20/0] via 192.168.24.1, 00:41:27
R4#
*Jul 16 11:37:02.595: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/2, changed state to down
*Jul 16 11:37:02.631: %BGP-5-ADJCHANGE: neighbor 192.168.24.1 Down Interface flap
R4#show ip route bgp
B*   0.0.0.0/0 [200/0] via 192.168.13.1, 00:00:06
Which it does.

[1] The difference is that with HSRP we find that both routers advertise the prefixes for incoming traffic but only one receives outgoing traffic.