Monday, November 14, 2016

Rethinking Micro-segmentation

Traditional Security Architectures

Traditional security architectures enforce security policy at rigidly defined trust boundaries. At the most basic level, this is the perimeter of the network. A firewall sits between the untrusted public Internet and the trusted private network. If inbound access from the Internet is required, a DMZ is often created to segment Internet exposed resources from the trusted internal network. A network can be further segmented using additional zones on the perimeter firewall, access-lists on distribution switches, and additional layers of security at various points in the network.

In this traditional model, as security increases, so does configuration complexity, management overhead, and margin for human error. In addition, implicit trust between devices on a network segment is inherent to traditional security architectures. If one device is breached, an attacker can use the compromised device to launch an attack against other devices on the same network segment. Therefore, traditional security architectures are often ill equipped to secure east-west traffic in a modern data center.

What is micro-segmentation?

In two words: Trust nothing. The goal is to eliminate implicit trust and apply security policy between all devices within the purview of the micro-segmentation solution. By using this zero-trust model, micro-segmentation solutions aim to prevent attackers from moving laterally through a network after breaching an initial target.

There are a few fundamentally different approaches to micro-segmentation in the data center. Several current micro-segmentation solutions are built into larger data center orchestration and automation platforms. I'll avoid mentioning specific products, because comparisons often end up like those of vi vs. Emacs or which is the best Linux distribution.

That said, the solutions I am most familiar with enforce security policy in one of two ways:
    • Enforce policy in the network device and/or vSwitch
    • Enforce policy in the hypervisor kernel

Despite where the actual enforcement occurs, at a high level the micro-segmentation functionality itself is comparable. An engineer logs into a controller, defines a security policy, and centrally pushes this security policy to a number of devices in order to restrict traffic between endpoints. These endpoints can be baremetal servers, VMs, containers, or other resources supported by the micro-segmentation platform. The fundamental difference is the point of policy enforcement - hypervisor, vSwitch, or network device.

Thursday, October 6, 2016

Big Data Analytics for Your Network

The help desk just called. Users are reporting the wireless is down in your office, and nobody can get on the network. The wireless seems fine to you. You're connected. You ask a few people nearby, and they're connected too. You log into the WLC and don't see any problems. works fine. Maybe you should just turn the controller off and then back on again. That worked last time. No, that's a bad idea. It's the middle of the day and you actually need to troubleshoot it.

After a bit of troubleshooting, you determine the cause of the issue is not the wireless. The DHCP scope is exhausted. Users could connect, but they couldn't obtain an IP address. You shorten the lease time, expand the scope, and call it a day. While you're at it, you wonder if DHCP is the reason connecting has been taking longer than usual, so you fire up Wireshark.

Discover, offer, request, acknowledge. You remember that from a CCNA class half a lifetime ago. Looks good. Well, you think it looks good. It takes about 227 milliseconds from discover to offer. That's normal, right? You realize you're not sure what normal is. You don't know your baseline, and you have no idea how long DHCP should take from discover to offer or request to acknowledge. What about dot1x? Is the RADIUS server slowing things down? You really have no idea. It works. It's lunch time. Nobody is complaining - right now.

Ok, hopefully the way you run your network is nothing like this. However, let's face it: this is an exaggerated version of the reality that many deal with on a day to day basis. There is often little insight into the individual operations that contribute to network performance as a whole. "The wireless is down" could mean any number of things, many of which may be out of the purview of the team managing the wireless network. Troubleshooting is often a reactive process. Even when there is visibility into network operations and baselines are known, it can be difficult to determine if your "normal" is actually optimal.

I recently attended a presentation by Nyansa at Networking Field Day 12. Nyansa is a startup focusing on what they call Cloudsourced Network Analytics. Their goal is to go beyond providing visibility in the form of pretty graphs and actually provide actionable insight about how to improve the end user experience.

Friday, September 9, 2016

Mnemonic: Syslog Severity Levels

Ever have trouble remembering syslog severity levels?

I was organizing some old study notes and came across this mnemonic. It's easy to remember, and I'm sure many network engineers can relate.

Everyone always complains even when nothing is different. 

[E]veryone  [A]lways [C]omplains [E]ven  [W]hen    [N]othing      [I]s            [D]ifferent
[E]mergency [A]lert  [C]ritical  [E]rror [W]arning [N]otification [I]nformational [D]ebugging

Level              Description
0 - emergency      System is unusable
1 - alert          Immediate action needed
2 - critical       Critical condition
3 - error          Error condition
4 - warning        Warning condition
5 - notification   Normal but significant condition
6 - informational  Informational message only
7 - debugging      Appears during debugging only

More information about syslog (system message logging):

Saturday, September 3, 2016

Spanning Tree Protocol Visualization - Initial Convergence

Spanning tree: everyone's favorite protocol! Thousands of pages have already been written about spanning tree, so I've decided to take a different approach.

I find it helpful to visualize protocol elections and traffic flow in order to better understand protocol behavior, so I created a visualization illustrating the initial spanning convergence process. This visualization only addresses the initial root bridge election and STP convergence process, for example, if all switches were to boot at the same time. This does not address converging a new STP topology after a topology change.

The visualization below is basically an embedded slideshow that can be advanced by clicking on the image. There are notes for each slide that briefly explain each step of the convergence process. The numbers on each side of the links between switches represent the port number of each uplink.

Here are the basic steps of the initial STP convergence process:
  1. Elect the root bridge.
  2. Determine the root ports.
  3. Determine the designated ports.
  4. Remaining ports are blocking ports.

Click the image to advance the visualization. 

Tap here if you are on a mobile device.

As you can see, the resulting topology (the logical forwarding topology that is created after STP blocks redundant links) looks something like an upside-down tree.

Here is an example of of what this tree might look like when redundant links are blocked by STP in a larger L2 topology (although hopefully your network looks nothing like this). The links shown in black can be used, because each port is in the forwarding state. The links shown in red cannot be used, because one of the ports is in the blocking state. Reducing the topology to a tree of forwarding links is how spanning tree maintains a loop free L2 topology.

Once again, this is not meant to be a complete description of STP, but rather a visualization and basic description of the initial convergence process.

Feel free to leave questions or comments below.

Tuesday, August 9, 2016

Opengear and the Evolution and Consolidation of Network Devices

Opengear at Tech Field Day Extra 2016

I recently attended Cisco Live 2016 in Las Vegas and was invited to attend Tech Field Day Extra as a delegate. The first presenter was Opengear, a maker of console access servers and remote management gateways. They describe their products as "next generation Smart Solutions for managing and protecting critical IT and communications infrastructure."

While the term "next generation" is frequently overused, I can't argue with Opengear. Opengear extends the functionality of a console access server into a more complete out-of-band management solution. First, the Opengear presentation made me reevaluate what I should look for in an a console access server. What should it do? What shouldn't it do, and what roles should be held by separate devices?

Friday, February 5, 2016

Configuring ERSPAN on Cisco Routers and Switches

In two recent posts, I covered SPAN, for mirroring traffic to a port on a local switch, and RSPAN, for mirroring traffic across a VLAN to a port on a remote switch.  What if we want to mirror traffic traffic to a destination across a L3 link?  Cisco provides the ability to do this natively with a feature called ERSPAN, or encapsulated RSPAN.  However, this feature is only available on higher end platforms such as Catalyst 6500 and 6800 series switches, 7600 series routers, ASR1000, and CSR1000v (this is not a complete list).


Like SPAN and RSPAN, configuring ERSPAN is pretty straightforward.  ERSPAN simply requires L3 connectivity between source and destination devices.  The ERSPAN monitor session then builds a GRE tunnel that transports mirrored frames from the source port to the destination port.

Basic ERSPAN configuration is as follows:
! Source switch
monitor session SESSION-NUMBER type erspan-source 
 source-interface INTERFACE(S)|VLAN(S) {TX|RX|BOTH}
 no shutdown
  erspan-id ERSPAN-ID
  ip address DESTINATION-IP
  origin ip address ORIGIN-IP

! Destination switch
monitor session SESSION-NUMBER type erspan-destination
 destination-interface INTERFACE(S)
 no shutdown
  erspan-id ERSPAN-ID
  ip address SOURCE-IP
    It is important to note that when configuring the destination switch "source IP," you should select the source IP on the destination switch itself - the GRE tunnel endpoint.  Source IP does not refer to the GRE tunnel origin IP address.  Therefore, the "ip address" command should match on the source and destination.

    Below is a basic ERSPAN config to mirror data from R1 interface g3 to R3 interface g3.  I created this topology using VIRL using CSR1000V routers for R1 and R3.

    Saturday, January 30, 2016

    Configuring RSPAN on Cisco Catalyst Switches

    I recently wrote a post on configuring port mirroring (SPAN) on Cisco Catalyst switches.  SPAN (switched port analyzer) allows you to mirror traffic from a source or multiple sources on a switch to a destination interface or interfaces on the same switch.  RSPAN (remote SPAN) takes this a step further and allows you to mirror traffic to an interface on a remote switch or switches.


    RSPAN configuration is relatively simple and builds upon existing SPAN functionality and configuration syntax.
    • Create an RSPAN VLAN on the source switch, destination switch, and all switches in the transit path.
    • Take traffic from a specified source on switch A, and mirror it to an RSPAN VLAN.  
    • Then, on switch B, use traffic from this VLAN as the source and mirror it to a physical interface

    As shown below, traffic mirrored from the switch on the right to the switch on the left can traverse other switches as long as there is end to end L2 connectivity between them (ie. the RSPAN VLAN exists on all switches).

    Basic RSPAN configuration is as follows:

    Thursday, January 28, 2016

    Configuring Port Mirroring (SPAN) on Cisco Catalyst Switches

    So you have a network issue.  Or perhaps you don't, but you need to help find the root cause of a performance issue and conclusively show that it's not network related.  In either case, packet analysis is your friend.

    At times, it can be convenient (and effective) to capture directly on an affected server or host.  However, you may not always be able to access the affected device.  Even you can, capturing from the affected device is not always the best option due to TCP segmentation offload, checksum offload, and a number of other factors.  (These are outside of the scope of this post, but Kary over at has a ton of great content on packet analysis including why you shouldn't capture on a host.  See here.)

    A network tap is the best solution when absolute precision is required.  However, this can be impractical and is often overkill.  This is where port mirroring comes into play.  Cisco gear provides a number of ways to mirror traffic from a specified source (or sources) and get frames from point A to point B for analysis. 

    Saturday, January 16, 2016

    CCIE "GAP Analysis"

    I am in the beginning stages of CCIE preparation and recently had the opportunity to work on a lab scenario with three senior network architects - all current CCIEs.  I'll spare you the details and just say my performance (and troubleshooting approach) in this particular lab fell far short of what I would have liked.

    So why is that?  On a daily basis, I design networks.  I perform network assessments and network remediations.  I scale existing networks.  I troubleshoot persistent network issues, at times down to the packet level.  I do all of this quite well.  I work with junior members of my team and explain various networking technologies.  Yet when my feet were held to the fire in a lab scenario, I faltered.

    I've put in a tremendous amount of effort over the past few years to master networking fundamentals, and throughout this process, I have earned a number of relevant certifications.  I have read well over a dozen Cisco Press books front to back, I and took the time to process what I read.  I studied using video training materials from multiple sources (including INE whenever possible - I highly recommend Brian McGahan's content).  I built an extensive lab environment (see here).  I configured the relevant technologies, broke them, and troubleshot them.  I implement the bulk of these technologies in production, many of them on a regular basis.

    So, again, why did I fall short in a lab scenario?  In an effort to strengthen my weak points and prepare for my CCIE, I've done a high very high level "gap analysis."

    Configuration vs. Troubleshooting

    There are many technologies I configure on a regular basis but don't often have to troubleshoot outside of a lab scenario.  And quite frankly, I haven't done many troubleshooting labs in the past year since I earned my CCNP.  My responsibilities shifted more to the design side of things, and troubleshooting why an OSPF adjacency won't form isn't something I often do.  In hindsight, I should have kept the momentum going after earning my CCNP and continued with CCIE prep.  At this point, I'm going to have to circle back to solidify some CCNP level topics.

    Understanding vs. Execution

    I have a solid understanding of a wide variety of technologies.  I could explain many of them at a low level during a quick trip to the whiteboard.  This is a useful skill to have.  However, this doesn't mean I can configure or troubleshoot these technologies on demand.  When it comes to recalling the relevant configuration, show, or debug commands for some of these technologies, I come up up short and have to use reference material.  Conceptual understanding does not equal the ability to configure or troubleshoot a technology on the fly.

    Reliance on References

    I have developed a lot of my own work notes and reference materials, some of which are posted on this blog.  When writing new configurations, I tend to rely upon known good configurations and references that I wrote.  I have always seen this as being efficient and scalable; why do the same work twice?  However, I now realize this is also a crutch that is going to hold me back as I prepare for the CCIE.

    Spread too thin

    I tend to spread myself a bit too thin.  I started writing about this here, and then I decided this topic warrants a blog post of its own:
    Habits vs. Goals.

    Next steps...

    To be successful in my CCIE prep, I need to get back to basics and hit the CLI.  In an effort to stay on track and solidify my own understanding of various technologies, I am going to be writing a series of blog posts that corresponds to the CCIE R&S v5 blueprint.  Stay tuned for more.

    Sunday, January 10, 2016

    Habits vs. Goals

    So it's that time of year again.  People set goals.  People chase goals.  Many ultimately fail.  I recently read an article on habits vs. goals over here, and it got me thinking about goals, habits, and balance in general.

    I have big goals, and I want to achieve them all at once.   I tend to spread myself a bit too thin.  For example, in 2014 I set the following goals:
    • Study for and pass CCNP route, switch, and t-shoot
    • Run a marathon
    • Help plan my own wedding (I wasn't tremendously helpful), and get married
    • Go on my honeymoon, and while on my honeymoon, study for the CCNP and train for the marathon.
    • Work enough after-hours to cover the flex time for the wedding/honeymoon
    • Achieve a laundry list of professional goals for a major promotion

    Generally just a CCNP, marathon, wedding, or major promotion would be a pretty intimidating goal.  Attempting to achieve all of these in one year was definitely quite the challenge.  I ultimately succeeded, but the road to get there was unstructured.  It was a very hectic year. Last year (2015) was more of the same.

    Right now I'm working toward even more time consuming goals:
    • Earn CCIE R&S
    • Finish B.S. Information Technology-Security degree at WGU
    • Learn Python and become proficient using it to manage network devices and manipulate configs
    • Develop greater understanding of TCP/IP and packet analysis
    • Train for another marathon, and begin cycling in preparation for an Ironman
    • Start weight training (again)
    • Blog consistently
    • Achieve a laundry list of other professional goals and certifications

    As my goals get larger and larger, holding myself accountable and managing my time becomes even more important.  This is where building solid habits comes into play.  I've seen great results using fitness tracking apps such as Garmin Connect, MyFitnessPal, and Withings.  The datafication and gamification these apps provide helps me stay on track.  Tracking progress over time helps me hold myself accountable.

    This year I've decided to take a similar approach and track all of the other goals I'm working toward.  There is no shortage of apps for tracking and forming habits.  I found a useful list here.

    I ultimately chose because it is cross-platform and easy to use.  It appears to be full featured, and has a nice pretty UI.  As with all of these apps, you get out what you put in.  However, I'm definitely looking forward to focusing on habits rather than goals.  If you develop the habits, the goals will fall into place.

    UPDATE 1/29/16:  While I like the premise and basic functionality, the app leaves a lot to be desired.  There is currently no reporting functionality or any way to view historical data.  It seems like a wasted opportunity to collect data on a daily basis and do absolutely nothing with it.  My ideal app would provide a visual representation of habit consistency, trends, and how habits/goals change over time.