Thanks to Nicola Pezzi for this tip.
If the CPU is pegged on a 6500 due to IP packets being punted up to it , there is a nice trick to see what packets are doing the damage in the form of the debug netdr command. To prevent us from overwhelming our box, we can dump 4096 packets (configurable) from the debug to a buffer like this:
6500# debug netdr capture
Here are the options:
6500#debug netdr capture ? and-filter (3) Apply filters in an and function: all must match continuous (1) Capture packets continuously: cyclic overwrite destination-ip-address Capture all packets matching ip dst address dmac Capture packets matching destination mac dstindex (7) Capture all packets matching destination index ethertype (8) Capture all packets matching ethertype interface (4) Capture packets related to this interface or-filter (3) Apply filters in an or function: only one must match rx (2) Capture incoming packets only smac Capture packets matching source mac source-ip-address (9) Capture all packets matching ip src address srcindex (6) Capture all packets matching source index tx (2) Capture outgoing packets only vlan (5) Capture packets matching this vlan number
Let’s grab packets transmitted from VLAN 20 which hit the CPU:
debug netdr capture tx vlan 20
The buffer can be seen with:
show netdr captured-packets
Here is an example capture:
A total of 4096 packets have been captured The capture buffer wrapped 0 times Total capture capacity: 4096 packets ------- dump of incoming inband packet ------- interface Vl20, routine mistral_process_rx_packet_inlin, timestamp 15:38:33 dbus info: src_vlan 0x14(20), src_indx 0x342(834), len 0x47E(1150) bpdu 0, index_dir 1, flood 0, dont_lrn 0, dest_indx 0x380(896) 08020000 00143800 03460004 7E000000 00F70438 10000008 00000000 03804A64 mistral hdr: req_token 0x0(0), src_index 0x342(834), rx_offset 0x76(118) requeue 0, obl_pkt 0, vlan 0x14(20) destmac 00.00.0C.07.AC.00, srcmac 10.11.14.16.15.10, protocol 0800 protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1132, identifier 7234 df 0, mf 1, fo 0, ttl 254, src 192.0.2.1, dst 172.16.0.31, proto 103
You can customise the capture as you wish. In this example host 192.0.2.1 is sending PIM (IP Protocol 103) traffic to router 172.16.0.31, which is processing the traffic in software. In this case CGMP had been enabled on an on 3500 switch causing us to fall foul of the issues documented here.
This is a simple, safe and powerful tool from troubleshooting when a chassis starts doing things in software.
 You’ll see a high percentage under ‘IP input’ when you run ‘show process cpu sorted’.