Category Archives: Documentation

Quick Ubuntu Package Update Script

I don’t want to have to remember to login to my syslog server and check for required updates. Here is the script I wrote and added to cron so that I don’t have to:

#!/usr/bin/perl

use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Sender::Transport::Sendmail qw();
use Try::Tiny;

my $to = "noc\@howfantastic.net";
my $from = "root\@syslog.howfantastic.net";
my $subject = "update script";
my $body =  `/usr/bin/apt-get update && /usr/bin/apt-get upgrade -s`;

my $email = Email::Simple->create(
    header=>[To=>$to, From=>$from,
             Subject=>$subject],
    body=>$body,
);

try {
    sendmail($email,
             {from=>$from,
             transport=>Email::Sender::Transport::Sendmail->new});
} catch {
    print "Can't send mail: $_";
}

To get this to work I needed to install some packages:

apt-get install sendmail
apt-get install libemail-sender-perl 
apt-get install libemail-simple-perl

 

I then added my script to cron:

root@itssys01:~# crontab -l
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
0 0 * * * /usr/local/bin/update-check.pl > /dev/null

Now I get an email with a list of packages which would be updated were I to run the following:

/usr/bin/apt-get upgrade

I don’t want my system blindly updating itself so this way I can still check things over before committing.

Cisco Nexus TACACS config

According to the team where I work, our standard Catalyst TACACS+ config didn’t work properly. Here is the snippet they used to get going again:

user = DEFAULT {
default service = permit
service = exec {
shell:roles*"network-admin vdc-admin"
}
}

User based role authorisation didn’t work well either apparently.

The * rather than the = means it’s an optional attribute so should be ignored by switches which aren’t compatible.

Thanks to Emma Cardinal-Richards for the snippet.

IPSec Part 1: Glossary

This first post is a brief summary of the concepts, protocols and relationships involved in IPSec. I’ll look at how to make use of it in a future post.

IPSec is an IETF open standard which was designed as part of IPv6 and backported to IPv4. It can be used for both remote access and site-to-site VPNs and operates at L3 of the OSI model.

Glossary

  • IKE – Internet Key Exchange: used to negotiate and establish VPN connections
  • ISAKMP – Internet Security Association & Key Management Protocol: Framework which provides IKE used to establish SAs.
    • ISAKMP Phase 1 (IKE Negotiation)
      • Negotiate ISAKMP SA
      • Creates secure two-way comms between VPN peers
      • Uses UDP 500 (sometimes blocked by service providers)
    • ISAKMP Phase 2
      • Protected by ISAKMP SA
      • Negotiate IPSec SA
      • Enables payload traffic between VPN peers to be encrypted
      • ISAKMP header is the only non-encrypted part, hence phase 1.
      • One per subnet, per direction. Summarisation can help!
  • IPSec Protocol Suite
    • Two protocols used to encapsulate tunnel data:
      • AH – Authentication Headers
        • L3, IP protocol number 51
        • Protects payload and immutable header fields.
      • ESP – Encapsulating Security Payloads
        • L3, IP Protocol number 50
        • Provides origin authenticity, integrity and confidentiality
    • SA – Security Association
      • the bundle of algorithms and parameters (such as keys) being used to encrypt and authenticate a particular flow in one direction.
  • Phase 1 Policy components:
  • Encryption Algorithms (to encrypt the traffic)
    • DES – Data Encryption Standard (64 bit)
    • 3DES – Triple DES (164 bit)
    • AES – Advanced Encryption Services (128 bit)
    • AES192 (192 bit)
    • AES256 (256 bit) (recommended)
  • Hashing Algorithms (for data integrity)
    • SHA – Secure Hash Algorithm (recommended)
    • MD5 – Message Digest Algorithm 5
  • IPSec Peer Authentication
    • PSK – Pre Shared Keys (small – medium enterprise)
    • PKI – Public Key Infrastructure (medium – large enterprise)
  • Phase 1 SA Establishment modes
    • Main – protects identity if PSKs are used (typically site-to-site)
    • Agressive – protects identity if PKI is used (typically remote-access). Has known vulnerabilities so don’t use this.
  • Phase 2
    • AKA Quick mode
    • Creates one IPSec SA per direction, each with a unique SPI
  • SPI – Security Parameter Index
    • Used with destination IP address to select protection applied to outgoing packet
    • Used by IPSec pass-thru to work around PAT issues
  • Initiator – outbound SA
  • Responder – inbound SA
  • Diffie-Hellman – protocol enabling hosts to authenticate each other’s PSKs without transmitting them.
  • IPSec Modes
    • Transport
      • Protects L4
    • Tunnel
      • Cisco default
      • Protects whole packet
      • Increases the packet size as additional IP Header is needed
      • Watch for MTU issues

A note on NAT

Since the IPSec protocol suite operates at L3, there is no L4 port information for NAT to munge leading to AH and ESP packets being dropped. IPSec pass-thru builds L4 information from the SPI. Another option is NAT Traversal (NAT-T), where VPN peers dynamically discover that NAT is happening between them and encapsulate the traffic in UDP 4500 if so.

 

6500 Transit ACLs

A while ago we found that our FWSMs were no longer up to the job. As it happens we were doing nothing more than stateless packet filtering with them so could replicate their functionality with access-lists. I noticed the hit count on the ACLs was rather low and came across this blog post, which explained what was going on rather well.

In summary:

show ip access-list NAME

will only show packets destined for the router, not those passing through it. To see transit traffic you need this command:

show tcam interface <INT> acl [in | out] ip

This makes sense given that ACLs are implemented in TCAM on the 6500 platform.

Verification

We have an interface which originates a default route into our campus, with an ACL in each direction applied to it. We permit ip any any at the end after filtering out the cruft we don’t want to see. Here is a comparison of the two commands for that ACL.

ROUTER#show ip access-lists TO_CAMPUS | i permit ip any
    230 permit ip any any (4475 matches)

Given that this ACL has around 30,000 users behind it and the counters had been cleared earlier on the day the command was run, I would expect number to be larger.

Now the tcam command:

ROUTER#show tcam interface vlan 80 acl out ip | i ip any any
    permit       ip any any (4141914 matches)

That is more realistic.

TACACS+ on MRV LX4000T Console Servers

In this forth post in the TACACS+ series, I’ll look at using TACACS+ for access to the console port of an IOS device via an MRV console server.

MRV LX4000T Console Servers

Configure TACACS+ for authentication and accounting.  The “local subscriber” means that if a username is defined locally, it can be authenticated by TACACS+ and use the properties defined locally.

TOUCS:0 >>config
Config:0 >>aaa
AAA:0 >>tacacs+ primary authentication server address <IP>
AAA:0 >>tacacs+ primary authentication server secret ...    
AAA:0 >>tacacs+ secondary authentication server address <IP2>
AAA:0 >>tacacs+ secondary authentication server secret ...   
AAA:0 >>tacacs+ primary accounting server address <IP>
AAA:0 >>tacacs+ primary accounting server secret ...   
AAA:0 >>tacacs+ secondary accounting server address <IP2>  
AAA:0 >>tacacs+ secondary accounting server secret ...   
AAA:0 >>tacacs+ local subscriber enable

Enable authentication and accounting on the ethernet interfaces.  The fallback statement allows the local authentication database to be used if the TACACS+ servers are unreachable.

TOUCS:0 >>config
Config:0 >>interface 1
Warning Interface active
Intf 1-1:0 >>authentication tacacs+ enable
Intf 1-1:0 >>tacacs+ accounting enable
Intf 1-1:0 >>authentication fallback attempts 3 
Intf 1-1:0 >>exit 
Config:0 >>interface 2
Warning Interface active
Intf 2-2:0 >>authentication tacacs+ enable
Intf 2-2:0 >>tacacs+ accounting enable
Intf 2-2:0 >>authentication fallback attempts 3 
Intf 2-2:0 >>end

Confirm our configuration

TOUCS:0 >>show tacacs+ characteristics
 Time:                                          Mon, 21 Jun 2010 14:22:58 UTC
 Primary TACACS+ Authentication Server:
 IP Address:              <IP>  TACACS+ Auth. TCP Port:            49
 Secret:                    Configured  Timeout:                            5
 Retry:                              3
 Secondary TACACS+ Authentication Server:
 IP Address:               <IP2>  TACACS+ Auth. TCP Port:            49
 Secret:                    Configured  Timeout:                            5
 Retry:                              3
 Primary TACACS+ Authorization Server:
 IP Address:                   0.0.0.0  TACACS+ Author. TCP Port:          49
 Secret:                Not configured  Timeout:                            5
 Retry:                              3
 Secondary TACACS+ Authorization Server:
 IP Address:                   0.0.0.0  TACACS+ Author. TCP Port:          49
 Secret:                Not configured  Timeout:                            5
 Retry:                              3
 Primary TACACS+ Accounting Server:   
 IP Address:              <IP>  TACACS+ Acct. TCP Port:            49
 Secret:                    Configured  Timeout:                            5
 Retry:                              3
 Secondary TACACS+ Accounting Server: 
 IP Address:               <IP2>  TACACS+ Acct. TCP Port:            49
 Secret:                    Configured  Timeout:                            5
 Retry:                              3
 Superuser Request:           Disabled  Accounting Server Period:           5
 Local Subscriber:             Enabled  Source Interface:                   1
 Command Authorization:       Disabled  Command Logging:             Disabled
 Command Authorization Fallback:                                     Disabled
 TACACS+ Authentication Serial Ports:
 TACACS+ Authentication Interfaces: 1
 TACACS+ Accounting Serial Ports:
 TACACS+ Accounting Interfaces: 1

TACACs+ on Cisco WLCs

In this third post in the TACACS+ series, I’ll cover using TACACS+ for administering a Cisco WLC device.

Cisco WLC

Server Config

			
group = wlc {
  service = ciscowlc {
    role1 = ALL
  } 
}

group = wlc-read-only {
  cmd = show {
    permit .*
  }
  cmd = ping {
    permit .*
  }
  cmd = traceroute {
    permit .*
  }
  service = exec {
    priv-lvl = 15
  }   
  service = ciscowlc {
    role1 = ALL
  }   
}

Client Config

This is fairly trivial and best done through the GUI. Just go to security->tacacs+ and add the servers and keys for Authentication and Authorization. I didn’t find the Accounting data very useful so left that off. To work out the server settings I ran the daemon in debugging mode and looked at what the WCS was sending. Something like:

# tac_plus -C /path/to/tac_plus.conf -g -d <level>

TACACS+ on Cisco ASAs

In this second post in the TACACS+ series, I’ll cover using TACACS+ for administering an ASA via SSH and ASDM, as well as for SSL VPN access.

Cisco ASA 5500 Series

  • After you ssh in, you’ll need to enable.
  • You can use your TACACS+ password to do this
  • Users with privilege level 5 are read only

Server Config

# Groups
group = asa {
  default service = permit
  service = exec {
    priv-lvl = 15
  }
}
group = asa-read-only {
  default service = permit
  service = exec {
    priv-lvl = 5
  }
}
# Users
user = admin {
  member = all
  login = des <snip>
  enable = des <snip>
}
user = read-only {
  member = asa-read-only
  login = des <snip>
}

Client Config

# To generate an RSA key pair, which is required for SSH, enter the following command:
crypto key generate rsa modulus 2048
# Give the device a hostname / domain name
!
hostname foo
domain-name bar.domain
!
# Add local AAA users
username user1 password <snip>
enable password <snip>
!
# Set up the management interface
interface Management0/0
 nameif manage
 security-level 50
 ip address 192.168.1.254 255.255.255.0
!
# ACL which selects who should use tacacs for AAA
access-list LOGIN extended permit tcp 192.168.1.0 255.255.255.0 interface manage eq ssh
access-list LOGIN extended permit tcp 192.168.1.0 255.255.255.0 interface manage eq https
!
# Set a default route for management access
route manage 0.0.0.0 0.0.0.0 192.168.1.254 1
!
# Set up tacacs
aaa-server data-tacacs protocol tacacs+
aaa-server data-tacacs (manage) host [ip] key <snip>
aaa authentication match LOGIN manage data-tacacs
aaa authentication ssh console data-tacacs LOCAL
# Console access local auth - optional
# aaa authentication enable console LOCAL
aaa authentication http console data-tacacs LOCAL
aaa authentication enable console data-tacacs
aaa authorization command data-tacacs LOCAL
aaa accounting command data-tacacs
aaa accounting enable console data-tacacs
aaa accounting ssh console data-tacacs
!
# Enable ASDM
http server enable
# ACL for ASDM
http 192.168.1.0 255.255.255.0 manage
!
# Allow ssh in for management subnet
ssh 192.168.1.0 255.255.255.0 manage
!
# You'll need NTP for TACACS to work - best have > 1
ntp server ntp0.domain source outside prefer
ntp server ntp1.domain source outside
ntp server ntp2.domain source outside
!

If your want to use tacacs+ as the auth mechanism for an SSL VPN running on an ASA:

tunnel-group DefaultRAGroup general-attributes
 authentication-server-group data-tacacs

Be careful if you run command authorisation on an ASA and have two TACACS+ servers. The default reactivation-mode is timed so if the networking on your device fails you can lock yourself out of it.

WARNING: 
If fallback authentication is configured with this server and reactivation mode is set to timed. 
Multiple aaa servers may prevent the appliance from ever invoking the fallback auth mechanism.
*** Output from config line 126, " reactivation-mode timed"
WARNING: 
Fallback authentication is configured, but reactivation mode is set to timed. Multiple aaa servers 
may prevent the appliance from ever invoking the fallback auth mechanism.

If you do have two servers available the answer is to do this:

asa(config-aaa-server-group)# reactivation-mode ?
aaa-server-group mode commands/options:
    depletion Failed servers will remain inactive until all other servers in this group are inactive
    timed Failed servers will be reactivated after 30 seconds of down time
asa(config-aaa-server-group)# reactivation-mode depletion

TACACS+ on IOS devices

In this first post in the TACACS+ series, I’ll look at some general server stuff and then configuring TACACS+ on IOS devices. I’ll cover ASAs, WLCs and MRV LX-4000T console servers in later posts.

Packaging

These days I used the stock Debian package. A few years ago I hand rolled an RPM based on the Shrubbery Networks code. You can download that here:tac_plus-F4.0.4-15.i386.rpm.

Generating a password for the config file

htpasswd -nd <username>

A script to parse and email the logs

I use a script I wrote called tac_logmail to dump a summary of the logs in our RT queue every day. It assumes you use syslog-ng to dump your tacacs logs to a central syslog server.

On tacacs+ servers:

source s_tacacs {
file("/var/log/tacacs" follow_freq(1) flags(no-parse));
};
destination d_remote {
tcp("syslog.domain", localip([% hostname %]), destport());
};

On central server:

template t_isotemplate { 
    template("$S_ISODATE $HOST_FROM $MSGHDR$MSG\n");    
    template_escape(no); 
};
# === Tacacs+ 
filter f_tacacs_plus_host {
    host ("foo") or host("bar");
};
destination d_tacacs_plus {
    file("/var/log/remote/tacacs+/tacacs+-$YEAR-$MONTH-$DAY" template(t_isotemplate));
};
log {
    source(s_tcp);
    source(s_udp);
    filter(f_tacacs_plus_host);
    destination(d_tacacs_plus);
};

Cisco IOS Switch / Router Configuration

Server Config

		
group = ios {
  default service = permit
  service = exec {
    priv-lvl = 15
  }
}

Client Config

aaa new-model
!
aaa authentication login default local group tacacs+
aaa authentication enable default enable group tacacs+
aaa authorization console
aaa authorization exec default local group tacacs+ if-authenticated 
aaa authorization commands 15 default local group tacacs+ if-authenticated 
aaa accounting commands 1 default stop-only group tacacs+
aaa accounting commands 15 default stop-only group tacacs+
!
ip tacacs source-interface [int]
!
tacacs-server host [ip] key [foo]
tacacs-server directed-request
!
line con 0
authorization exec default
line vty 0 [n]
login authentication default
!
# You'll need NTP for TACACS to work - best have > 1
ntp server [ip] key 0

 

Migrating to flexible netflow

Background

We deployed some 6500s running IOS 15.0(1)SY and when I came to configure netflow I found the usual commands didn’t work. The configuration guide didn’t contain anything helpful but I came across this whitepaper which did. What follows is the config I deployed. It doesn’t quite match the whitepaper as the syntax is slightly different on our code, but it does work.

Configuration

Here is how you declare a simple exporter. This is the server collecting the netflow data.

flow exporter MY_EXP
 destination 192.0.2.14
 source Loopback0
 transport udp 19221
 export-protocol netflow-v5
!

Then you define a monitor. This is just a container for the exporter, as well as a place you can select what to export (it is a bit more flexible you see):

flow monitor MY_MON
 record platform-original ipv4 full
 exporter MY_EXP
!

Then you enable netflow on the interface(s) you wish to monitor:

int vlan 31
 ip flow monitor MY_MON input
!

In case you were wondering, our standard netflow config looks like this:

ip flow-export source Loopback0
ip flow-export version 5
ip flow-export destination 192.0.2.14 19211
int vlan 31
 ip flow ingress
!

Verification

Check the exporter:

ROUTER#show flow exporter MY_EXP            
Flow Exporter MY_EXP:
  Description:              User defined
  Export protocol:          NetFlow Version 5
  Transport Configuration:
    Destination IP address: 192.0.2.14
    Source IP address:      192.168.1.1
    Source Interface:       Loopback0
    Transport Protocol:     UDP
    Destination Port:       19211
    Source Port:            55248
    DSCP:                   0x0
    TTL:                    255
    Output Features:        Not Used

Check the monitor:

ROUTER#show flow monitor MY_MON
Flow Monitor MY_MON:
  Description:       User defined
  Flow Record:       platform-original ipv4 full
  Flow Exporter:     MY_EXP
  Cache:
    Type:              normal
    Status:            allocated
    Size:              4096 entries / 278544 bytes
  Cache:
    Type:              normal (Platform cache)
    Status:            allocated
    Size:              Unknown
  Timers:
                       Local        Global
    Inactive Timeout:  15 secs      300 secs
    Active Timeout:    1800 secs    1920 secs
    Update Timeout:    1800 secs
    Fast Timeout:                   Disabled

Check UDP datagrams are being sent:

ROUTER#show flow exporter MY_EXP statistics 
Flow Exporter MY_EXP:
  Packet send statistics (last cleared 00:14:09 ago):
Successfully sent:         1                     (100 bytes)
Client send statistics:
Client: Flow Monitor CERT_MON
Records added: 1
- sent: 1
Bytes added: 48
- sent: 48

This guy’s opinion

I had to dig around and do a bit of guessing to get this working, but it only took about five minutes. At this stage there isn’t much to say, other than I’m looking forward to digging into the finer grained goodness that can be extracted from flexible netflow, and that I’m glad to have a basic config which works on my new boxes.