Configuring RSPAN on Cisco Catalyst Switches

I recently wrote a post on configuring port mirroring (SPAN) on Cisco Catalyst switches.  SPAN (switched port analyzer) allows you to mirror traffic from a source or multiple sources on a switch to a destination interface or interfaces on the same switch.  RSPAN (remote SPAN) takes this a step further and allows you to mirror traffic to an interface on a remote switch or switches.

RSPAN

RSPAN configuration is relatively simple and builds upon existing SPAN functionality and configuration syntax.

• Create an RSPAN VLAN on the source switch, destination switch, and all switches in the transit path.
• Take traffic from a specified source on switch A, and mirror it to an RSPAN VLAN.  
• Then, on switch B, use traffic from this VLAN as the source and mirror it to a physical interface

As shown below, traffic mirrored from the switch on the right to the switch on the left can traverse other switches as long as there is end to end L2 connectivity between them (ie. the RSPAN VLAN exists on all switches).

Basic RSPAN configuration is as follows:

Configuring Port Mirroring (SPAN) on Cisco Catalyst Switches

So you have a network issue.  Or perhaps you don't, but you need to help find the root cause of a performance issue and conclusively show that it's not network related.  In either case, packet analysis is your friend.

At times, it can be convenient (and effective) to capture directly on an affected server or host.  However, you may not always be able to access the affected device.  Even you can, capturing from the affected device is not always the best option due to TCP segmentation offload, checksum offload, and a number of other factors.  (These are outside of the scope of this post, but Kary over at packetbomb.com has a ton of great content on packet analysis including why you shouldn't capture on a host.  See here.)

A network tap is the best solution when absolute precision is required.  However, this can be impractical and is often overkill.  This is where port mirroring comes into play.  Cisco gear provides a number of ways to mirror traffic from a specified source (or sources) and get frames from point A to point B for analysis. 

vMotion Fails - Failed to connect to remote host. Network unreachable.

When migrating a VM using vMotion, the migration may stall at 14% and eventually fail with the following error:

Migration [xxxx:xxxx] failed to connect to remote host <x.x.x.x> from host <y.y.y.y>: Network unreachable.

Usually this is a pretty straightforward fix: correct whatever network issue is preventing communication between the vmkernel ports.  However, I recently encountered an issue where the network was configured properly, traffic was flowing, and vMotion still failed.

Everything with the multi-NIC vMotion config checked out:

• Two separate VMkernel ports on the relevant vSwitch with IPs on the same subnet.
• One vmnic active and one standby for each VMkernel port.
• Active/standby adapters on the second VMkernel port were the inverse of the first.
• vMotion enabled on the vMotion VMkernel ports.
• 9000 MTU on each vSwitch and VMkernel port
• 9000 MTU on the relevant switch and switchports.
• Relevant switchports tagged for the appropriate VLAN.

(The configuration is pretty straightforward, as outlined in the VMware KB: kb.vmware.com/kb/2007467)

ASA Remote Access User Prevent SSH Access

When configuring a remote access VPN on an ASA, there are times when an external authentication server (RADIUS, TACACS+, etc) is not available.  In this case, the local AAA database can be used:

asa01(config)# username vpnuser password p@ssw0rd privilege 0
asa01(config)# username vpnuser attributes
asa01(config-username)# service-type remote-access
asa01(config-username)# exit
asa01(config)#

You might think that specifying privilege 0 and service-type remote access as shown above would be enough to prevent this user from logging in through SSH.  However, this may not be the case.  Let's look at the following example:

asa01# sh run user vpnuser
username vpnuser password jpCK6VfivhvBp0Pn encrypted privilege 0
username vpnuser attributes
 service-type remote-access
asa01# sh run aaa
aaa authentication ssh console LOCAL
aaa authorization exec LOCAL
asa01#

With this configuration, it is still possible for "vpnuser" to log in through SSH:

vpnuser@asa01's password:
login as: vpnuser
vpnuser@asa01's password:
Type help or '?' for a list of available commands.
asa01>

This is possible because the above configuration only specifies AAA authentication, not authorization.  Therefore the local user account's password is checked against the local database, but no check is performed to determine whether or not this local user is authorized for EXEC shell access.  This behavior can be changed by enabling management authorization with the following command:

asa01(config)# aaa authorization exec LOCAL
asa01(config)#

Now if we attempt to log in with this same account, the login will fail:

login as: vpnuser
vpnuser@asa01's password:
Access denied
vpnuser@asa01's password:

The ASA configuration guide goes into more detail about this feature here:
http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/general/asa_91_general_config/admin_management.html#86134

Choosing an ExtremeXOS Software Release

When updating ExtremeXOS, it is important to choose your software release carefully.  The newest builds may contain new features that are not yet stable.  I learned this the hard way.  The Extreme support portal currently does not label their software builds with anything that indicates what is a stable build and what is may contain new unstable features.

Extreme Networks EXOS Cheat Sheet

After working in primarily Cisco or Cisco-esque CLIs, ExtremeXOS can have a bit of a learning curve.  At the time of this post, Extreme Networks' documentation is almost entirely in PDFs.  In my opinion, these are a bit of a pain to navigate and are not well indexed by search engines.

This post is meant to serve as a basic EXOS cheat sheet.  It is by no means meant to be a configuration guide or thorough command reference.  I will update this post periodically to add additional commands.

update: I reached out to Extreme regarding their documentation (I work for a partner), and it turns out they already have an EXOS cheat sheet, so my channel SE sent it over.  I was not able to find this document anywhere on Extreme's site.  However, an Extreme SE confirmed this document is public, so here it is:  EXOS Quick Guide - Cheat Sheet.  I may still update this page on occasion, but Extreme's document is more complete.

HOWTO: Basic Cisco ASA AnyConnect VPN 8.2(5)

A while back I posted a how-to for configuring AnyConnect in ASA version 8.3+.  I recently received a request to post the 8.2(5) configuration, so here it is.  The example below uses split tunneling and local authentication.  For RADIUS authentication, see this post.

Before beginning, verify you have the AnyConnect essentials license (without this license, only two simultaneous sessions are permitted).

asa# sh ver | inc AnyConnect
AnyConnect Premium Peers          : 25             perpetual
AnyConnect Essentials             : 25             perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Enabled        perpetual
asa#