Monthly Archives: September 2012

Troubleshooting high CPU on a Catalyst 6500

Thanks to Nicola Pezzi for this tip.

If the CPU is pegged on a 6500 due to IP packets being punted up to it [1], there is a nice trick to see what packets are doing the damage in the form of the debug netdr command. To prevent us from overwhelming our box, we can dump 4096 packets (configurable) from the debug to a buffer like this:

6500# debug netdr capture

Here are the options:

6500#debug netdr capture ?
  and-filter              (3) Apply filters in an and function: all must match
  continuous              (1) Capture packets continuously: cyclic overwrite
  destination-ip-address  Capture all packets matching ip dst address
  dmac                    Capture packets matching destination mac
  dstindex                (7) Capture all packets matching destination index
  ethertype               (8) Capture all packets matching ethertype
  interface               (4) Capture packets related to this interface
  or-filter               (3) Apply filters in an or function: only one must match
  rx                      (2) Capture incoming packets only
  smac                    Capture packets matching source mac
  source-ip-address       (9) Capture all packets matching ip src address
  srcindex                (6) Capture all packets matching source index
  tx                      (2) Capture outgoing packets only
  vlan                    (5) Capture packets matching this vlan number

Let’s grab packets transmitted from VLAN 20 which hit the CPU:

 debug netdr capture tx vlan 20

The buffer can be seen with:

show netdr captured-packets

Here is an example capture:

A total of 4096 packets have been captured
The capture buffer wrapped 0 times
Total capture capacity: 4096 packets

------- dump of incoming inband packet -------

interface Vl20, routine mistral_process_rx_packet_inlin, timestamp 15:38:33
dbus info: src_vlan 0x14(20), src_indx 0x342(834), len 0x47E(1150)
bpdu 0, index_dir 1, flood 0, dont_lrn 0, dest_indx 0x380(896)
08020000 00143800 03460004 7E000000 00F70438 10000008 00000000 03804A64
mistral hdr: req_token 0x0(0), src_index 0x342(834), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x14(20)
destmac 00.00.0C.07.AC.00, srcmac, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1132, identifier 7234
df 0, mf 1, fo 0, ttl 254, src, dst, proto 103

You can customise the capture as you wish. In this example host is sending PIM (IP Protocol 103) traffic to router, which is processing the traffic in software. In this case CGMP had been enabled on an on 3500 switch causing us to fall foul of the issues documented here.

This is a simple, safe and powerful tool from troubleshooting when a chassis starts doing things in software.

[1] You’ll see a high percentage under ‘IP input’ when you run ‘show process cpu sorted’.

ASA source port

I was troubleshooting an issue today where an ASA 5505 had been configured as a replacement for an old PIX, but authentication with an external RADIUS server was failing.

A number of stateful firewalls sit between the outside interface of the ASA in our enterprise network and the public Internet. However the outbound rules permitted the appropriate traffic.

First I set up a capture to verify things from the perspective of the ASA:

! Version 8.X
access-list CAPTURE permit ip any host
access-list CAPTURE permit ip host any
! Version 9.X has separate ACEs for ipv4 and ipv6, if you enter the above you'll get:
! ERROR: Capture doesn't support access-list containing mixed policies
! so, change the ACL to look like this:
access-list CAPTURE permit ip any4 host
access-list CAPTURE permit ip host any4
! Now enable the capture
capture RADIUS access-list CAPTURE interface outside


Here is the output. The ASA actually has a public IP.

ASA# show capture RADIUS

54 packets captured

1: 12:16:06.883545 802.1Q vlan#8 P0 > udp 158
2: 12:16:07.975794 802.1Q vlan#8 P0 > udp 158
3: 12:16:09.075771 802.1Q vlan#8 P0 > udp 158

You can also view the capture in a web browser like this: https:///capture/RADIUS. Or you can download the file for wireshark analysis by going to an URL like this: https:///capture/RADIUS/pcap. Clearly you’ll need to substitute ‘RADIUS’ for whatever you called your capture.

Also, if you’re captures are big, you can used the circular-buffer and buffer command to make the buffer eat its own tail and increase the size (up to around 32M).

Finally, use the headers-only keyword if you aren’t interested in the content of the packet.

Okay, so we weren’t getting a response. It was at this point that I checked the external interfaces on our Internet facing routers and found an inbound ACL which did not permit traffic to UDP 1025. One minor, conservative adjustment to the ACLs later and we were in business.


I’ve come across the ASA source port issue before. Blocking TCP/UDP 1024-5 is a common policy as there were well known trojans which used them. Something to look out for when migrating to an ASA, or any system which uses those ports as a source. I see that the question has already been asked, but if anyone knows how to change the source port an ASA uses for radius requests, do let me know!

IPSec Part 1: Glossary

This first post is a brief summary of the concepts, protocols and relationships involved in IPSec. I’ll look at how to make use of it in a future post.

IPSec is an IETF open standard which was designed as part of IPv6 and backported to IPv4. It can be used for both remote access and site-to-site VPNs and operates at L3 of the OSI model.


  • IKE – Internet Key Exchange: used to negotiate and establish VPN connections
  • ISAKMP – Internet Security Association & Key Management Protocol: Framework which provides IKE used to establish SAs.
    • ISAKMP Phase 1 (IKE Negotiation)
      • Negotiate ISAKMP SA
      • Creates secure two-way comms between VPN peers
      • Uses UDP 500 (sometimes blocked by service providers)
    • ISAKMP Phase 2
      • Protected by ISAKMP SA
      • Negotiate IPSec SA
      • Enables payload traffic between VPN peers to be encrypted
      • ISAKMP header is the only non-encrypted part, hence phase 1.
      • One per subnet, per direction. Summarisation can help!
  • IPSec Protocol Suite
    • Two protocols used to encapsulate tunnel data:
      • AH – Authentication Headers
        • L3, IP protocol number 51
        • Protects payload and immutable header fields.
      • ESP – Encapsulating Security Payloads
        • L3, IP Protocol number 50
        • Provides origin authenticity, integrity and confidentiality
    • SA – Security Association
      • the bundle of algorithms and parameters (such as keys) being used to encrypt and authenticate a particular flow in one direction.
  • Phase 1 Policy components:
  • Encryption Algorithms (to encrypt the traffic)
    • DES – Data Encryption Standard (64 bit)
    • 3DES – Triple DES (164 bit)
    • AES – Advanced Encryption Services (128 bit)
    • AES192 (192 bit)
    • AES256 (256 bit) (recommended)
  • Hashing Algorithms (for data integrity)
    • SHA – Secure Hash Algorithm (recommended)
    • MD5 – Message Digest Algorithm 5
  • IPSec Peer Authentication
    • PSK – Pre Shared Keys (small – medium enterprise)
    • PKI – Public Key Infrastructure (medium – large enterprise)
  • Phase 1 SA Establishment modes
    • Main – protects identity if PSKs are used (typically site-to-site)
    • Agressive – protects identity if PKI is used (typically remote-access). Has known vulnerabilities so don’t use this.
  • Phase 2
    • AKA Quick mode
    • Creates one IPSec SA per direction, each with a unique SPI
  • SPI – Security Parameter Index
    • Used with destination IP address to select protection applied to outgoing packet
    • Used by IPSec pass-thru to work around PAT issues
  • Initiator – outbound SA
  • Responder – inbound SA
  • Diffie-Hellman – protocol enabling hosts to authenticate each other’s PSKs without transmitting them.
  • IPSec Modes
    • Transport
      • Protects L4
    • Tunnel
      • Cisco default
      • Protects whole packet
      • Increases the packet size as additional IP Header is needed
      • Watch for MTU issues

A note on NAT

Since the IPSec protocol suite operates at L3, there is no L4 port information for NAT to munge leading to AH and ESP packets being dropped. IPSec pass-thru builds L4 information from the SPI. Another option is NAT Traversal (NAT-T), where VPN peers dynamically discover that NAT is happening between them and encapsulate the traffic in UDP 4500 if so.