Rant: Licensing - It never gets better, it just gets more awkward

I have worked in the VAR/reseller space for over 8 years, and in that time I can’t recall a major shift in licensing that left me with the opportunity to go have enjoyable conversations with my customers.  Never did licensing get simpler, easier or seem to make more sense.

Let’s take a few examples from my career working with Cisco and see if you can spot the awkward parts of these conversations with customers?

Ancient History: We sold 4400 series controllers and they were sold with a fixed amount of licenses.  Need to go from a 4402-12 to a 4402-25? The conversation was to “buy a new controller.”

The Middle Ages:  Then we had 5508s and 2504s.  This was confusing as licensing costs varied with the volume and platform you were buying for.  Upgrading from a 4400 series?  Yeah, you’re going to have to rebuy those licenses along with the new controller.  But the bonus is that you can always add more up to the platform limit.

The Modern Age:  Now we have the 3504 and the 5520 series controllers.  You can now buy per-AP licenses, so no more blocks of licenses that you only needed 1 of.  Also, it’s Right To Use, so no awkward pak files.  Things would pretty good.  But you still need to rebuy licenses from your 5508 to get to this platform.

The Postmodern Age:  Everything in the modern age is still alive and well, but now we have this thing called CiscoOne.  It’s a licensing bundle and it decouples hardware from software licensing.  You buy it per-AP, and it includes a WLC, Prime, CMX Base and 25  ISE base licenses.  This enables portability so in the future you don’t have to rebuy licenses AND access to ongoing innovation.  YAY.  But you have to rebuy into this program from your existing 5508 or 5520 license, and the entry price is steeper.  But it includes most everything you could want.  There are a two different tiers, with different features that you can choose from. 

Today:  We still have the Modern Age, and Postmodern Age licensing. But now we have DNA licensing.  Which is also a bundling of multiple licenses, but now it’s term based. So when the term is up, you have to renew or you’re super-awesome WLAN hardware is just postmodern artwork.   Also the bundles don’t align exactly with the previous generation of C1.  ISE is missing from all but the top bundle, and now it’s 15 base licenses vs 25 before.  Also, if you bought C1, you get ongoing innovation.  At least for a while, and then you gotta rebuy into that innovation.  Now there are 3 more tiers of this licensing, all term based depending on how much of the new DNA architecture you want to go into.

Does your head hurt?  Mine has been hurting for years. What was a hard conversation is now a number of hard conversations with a ton of complexity and nuance that I am now expected to help my customer navigate through.  Every new licensing tier has areas where customers get lost, feel like they misspent money and ultimately get a call from Jake to talk about how the next generation of licensing is different and try to navigate a path forward.

Now, if you think I’m sticking it to Cisco, you are mistaken. My experience has been that the Cisco account teams have done a good job on a case by case basis to help customers with some of this pain.  I can’t say enough good things about my local account teams and they continue to help us navigate these waters.  But Cisco isn’t alone in this licensing evolution.

Aruba recently revamped their Clearpass licensing and on the surface it looks like an improvement.  But they took out the 25 enterprise licenses that you could run Onboard, Onguard, or Guest to test with.  And if you bought into the idea that enterprise licenses gave you the flexibility to a generic type of licenses that could be used for any feature (Guest, Onboard or Onguard), in 6.7 you now have to decide how you want to split them up.  And before they used to market it as it was an average of the last 7 days, so the weekends being low kept the average down for you so you really didn’t have to buy “peak” licensing, Not any longer, it’s a 24 hour window.  So while Aruba listened to their customers about their frustrations about how the product was licensed, they did very little to get existing customers over the gap between where they were and the new licensing scheme.

I’m sure the Aruba account teams will step up and help their customers, but it sure doesn’t give me warm fuzzy feelings.  Nor to I look forward to those conversations with customers who have bought into a product only to have their plans altered by new monetization models.

I hate licensing, and feel like efforts to “Make Licensing Great Again” are just ways to flip revenue from old revenue models to newer ones and don't benefit the customers of these products.

Cisco SD-Access and the Complexity Monster

A few weeks back, I attended Cisco Live 2017.  The headline announcement for the week was the Catalyst 9k series and DNA Center making up Cisco's SD-Access solution.  This represents a major shift in direction for Cisco’s networking business.  And while it looks very exciting, I am approaching DNAC and the 9k with some skepticism.

Why skepticism?  Because I’m a little nervous about the complexity of technology that this solution brings to bear.  Most of the components have been available in the Cisco portfolio for a while, but few have seen tremendous adoption.

CMX Cloud with Mobility Express is here

For some, Mobility Field Day 1 may be a distant memory.  The first Mobility Field Day Cisco brought Cisco to the delegates to talk about CMX Cloud.  Simply taking presence analytics to the cloud wasn't all that interesting to me.  Prior to 8.3 code you needed a proxy server to convert the NMSP data from the controller to an HTTPS connection to the CMX server in the cloud.  This was a bummer for me because now I have to deploy a server to run the proxy on, as well as a WLC to connect to CMX.   And if I'm deploying a server infrastructure at a site, why not just run CMX locally?

But not all was lost, Cisco told us the plan was for 8.3 to support a native connection to CMX cloud.  So that removes the server requirement, but what about the WLC?  They also announced that Mobility Express (the WLC software running on an AP) would support this direct connection to CMX Cloud in 8.3, removing the requirement to have a physical WLC and controller licensing for this deployment.  This is great news for distributed organizations like retail that want to gain presence analytics, but are shy about either deploying gear locally, or building/leveraging their WAN circuits for this.

---

Fast forward to today and 8.3 code for Mobility Express is published on Cisco.com.  I was asked to build a CMX demo for the upcoming Interface Boise conference in a few weeks, so I decided what a great opportunity to try out CMX Cloud with Mobility Express.  In just a few minutes I had reset my 1832 AP running Mobility Express and run through the initial setup.  A reboot later and the Mobility Express AP was up and operational.

Now for the CMX Cloud component.  I signed up for a CMX Cloud trial at cmxcisco.com.  It warns you that it can take up to a day to receive your account info.  Less than an hour later, I had my login credentials, server url and authentication token.   Going to the Mobility Express GUI, I was able to configure this with a minimal effort.

*Note: I ran into an issue where during the Mobility Express initial setup, DNS was listed as optional.  When you setup CMX, it NEEDS the DNS server functional to resolve the CMX server URL and to build the needed ACLs for the CMX Connect functionality.  You can configure DNS on the CLI of the Mobility Express if you don't want reset the AP and rerun the setup.  Once DNS was setup, the CMX configuration on the AP worked flawlessly.

If you're familiar with CMX Presence, the setup of CMX Cloud for presence is quite straight forward.  First you create a site and add your AP to CMX.  Remember it's the base radio MAC, not the ethernet MAC.  

Once you have the site created and the AP added to the site, I verified that I saw the Mobility Express controller connected to CMX and that I started to receive data.

Now that the AP is talking to CMX Cloud, for the next part all I had to do was wait (for data to populate).

The AP was running at my desk in the CompuNet Meridian office.  Pretty quickly i started to see the Insights panes starting to get populated.

I found the insights pretty interesting.  Our local engineering team is still edged out on Intel PCs vs Macs, but it's close (51 to 46 in the last 7 days).  I did find it interesting that 12-1 was he peak hour, considering the number of people who go out for lunch.

All things considered, this solution looks really well thought out, easy to deploy and simple from the fact that the only hardware required on-site is the Mobility Express AP.  I’m also really interested to see what this data looks like for the day of the conference (which is the whole point of the demo).

For the future, I think having location as part of this offering would be cool.  However, I think that the need to have maps in Prime Infrastructure and sync that to CMX might be more cumbersome and really take away from the minimal hardware needed.  Maybe this is an opportunity for Prime Infrastructure in the cloud (Or a “Prime Lite” in the cloud).  I also like the fact that I don’t have to manage the box.  It has been exceptionally stable and not having to update it periodically exemplifies why this belongs in the cloud.  This is really attractive when you realize that Cisco markets this to non-IT folks.  IT Doesn’t have to support it aside from 3 lines of config in the Mobility Express AP, and the rest can be managed by the marketing or BI folks.

It should be noted that Cisco provided me with the 1832 access point as part of participating in previous Wireless Field Day activities.  The CMX Cloud was provided by Cisco as part of a 60 day trial.   

Hallway Design Nightmares Part2: TXPower

One of the effects of the hallway design is that Radio Resource Management (RRM) frequently doesn't work as expected.  It's not that it doesn't work, it's just that hallway designs significantly limit the perspective of how APs see each other, which is primarily how RRM determines what it should be doing.  Given that all the APs in the hall generally see each other at or about the same level, they all have a very similar view of the network.  For illustration purposes, I've mocked up an imaginary residence hall.  A specific hall that I visited recently was build with concrete walls.  We will talk about some of the things I saw, and how they apply to this generic sample floor.

For this example, I've place 3 APs where the black dots are.  Let's also assume that this is the 2nd floor in a 3 story building.

Hallway Design Nightmares Part 1: Introduction

A lot of people know that Hallways designs are a bit of a pet peeve of mine.  My opinion is that they do not work, and beyond moving the APs out of the hall, you cannot fix them.  But the reality is that moving APs and existing infrastructure can be really expensive, time consuming and difficult to get management to buy off on.  This means that you probably will have to live with some of your existing hallway designs for the foreseeable future.

In this series, my aim is to explore the options available to improving performance of a hallway design.  As an engineer working for a Cisco Gold Partner, I'm going to talk about some of the tools available in the Cisco Unified Wireless Network (CUWN) to help deal with these designs.

I'll leave this short intro post with the following advise:

Don't put APs in the hallway.

Put APs as close to clients as you can get them.

Think about how RRM works, and incorporate that into your designs

Don't put APs in the hallway, PLEASE!

Airtight Networks: A look at WIPS Part 1: Over the Wire

When it comes to security, I have a personal motto: Think maliciously, act responsibly.  I really enjoy trying to manipulate clients, exploiting behavior and finding ways to prevent it in the "real world".

For protecting wireless networks in the "real world" one of the best tools is is WIPS.  Word on the street is that the Airtight solution is pretty good.  The presentation by Rick Farina at Wireless Field Day 6 was fun for me.  Rick is a great presenter and brought a lot of fun and energy to a security presentation.  For more on the WFD6 presentation, check out what fellow delegate Lee Badman wrote on his personal blog: http://wirednot.wordpress.com/2014/02/02/airtight-networks-rising/

Also watch his presentation here:

This is the first of a few posts on Airtight Networks WIPS solution.  Part 1 will cover how Airtight does rogue detection, Part 2 will cover containment and OTA communication and Part 3 will cover common Rogue scenarios and look at how

*I will note that Airtight gave me a C55 AP during my visit during WFD5.  While I'm grateful to them for this, these posts are my own opinion and not influenced by their generosity.

---

Coming from the Cisco world, I see that Airtight takes a very different approach to identifying on-network rogues. Instead of trying to correlate Wi-Fi traffic to wired traffic by listening on the wire (Rogue Detector), scanning CAM tables on switches, or trying to connect to open access points and sending traffic towards the controller (RLDP),  Airtight sends broadcast (or potentially unicast) frames on a vlans connected to the Sensor/AP and then listens to see if those frames are ever sent over the air.

Let's look at the wired side of my Airtight C55 AP.  For example, here the mac addresses from my WIPS mode AP on my network:

You'll notice that there's the management mac and the rest are mac addresses created by the WIPS-mode AP, one for each vlan that we are doing WIPS on.  Here is what I have my AP configured for:

*Note, VLAN125 is present, just not in the picture.

Spanning the port on the Airtight AP, I'm able to capture some of the packets coming from the WIPS-AP.

For simplicity, i wrote a display filter to clean up the data only from the Airtight AP:

(eth.src > f2:91:4a:7f:00:00 and eth.src < f2:91:4a:7f:ff:ff)

You can see it does a lot of GARP for addresses on the VLANs I am monitoring.  I also observed it sending DHCP requests on the VLANs configured as well:

So what does this give us?  Well, by sending these L2 broadcast messages out, the hope is that a rogue (on network) AP will hear the L2 broadcast and forward this traffic out to clients over the air.  Once this happens, the WIPS wireless radio will hear the packet and be able to see that it is from itself.  This allows the Airtight system to tell that an AP is connected to the network.  From here we can Airtight take action against this rogue network/clients connecting to this network.

What's next?  In part 2 of this series, I'm going to look at how the Airtight WIPS prevents clients from connecting to a Rogue AP, as well as looking at some of the options settings.  For Part 3, I'm going to try my hand at some common scenarios to see where the Airtight system works and where it does.

Begun, the Chipset Wars Have

As you might expect, 802.11ac was a hot topic at Wireless Field Day 5.  While there wasn't a lot of talk about it onscreen, there was a lot of talk about chipset vendors between the delegates and sponsors.  Surprisingly, Meru  and Xirrus were some of the most open about talking about their chipset vendors of choice.  Props to Xirrus for being the only vendor to admit they were using Qualcomm Atheros on-air for a yet-to-ship product.

The general consensus was that Broadcom surprised a lot of people with their focus on the enterprise and their speed to market.  At the time of writing this, all of the shipping enterprise 802.11ac access points are Broadcom chipsets in the enterprise space (Cisco, Aruba and Meru).

Atheros was the king of the enterprise AP market in 802.11n.  With the exception of Cisco who is known for their Marvel chipsets, almost every enterprise class AP vendor used Atheros for their 11n chipsets.  Now Broadcom is the first to market and a number of the enterprise manufacturers have made the jump to Broadcom due to a significant lead time of getting their gear to market.

But being early to market didn't come without a price.  There are some restrictions around the number of encrypted clients, number of beam-formed clients and whether they support promiscuous  packet capture.

I would expect to see vendors bringing their Atheros based chipsets to market soon. Xirrus passed around their 802.11ac module based on Qualcomm Atheros.  I would expect to see a number of product announcement in the coming months and for the vendors to promote how they are different from their competitors products.

The real question is whether vendors who have jumped on the Broadcom bandwagon will continue to stay there, or if they will make the jump back to Atheros when those products become available.  This will also help AP vendors help differentiate themselves from these early generation products.  One thing you can bet on, it will be an exciting ride.

My thoughts:

Competition in this space will be good for everyone.  It will cause chipset manufactures to innovate with features and functionality, drive down price and generate product differentiation for enterprise customers.

Cisco Prime Infrastructure: Bug ID CSCud39395 Logins not Processed

A couple weeks ago I ran into an issue with Prime Infrastructure 1.2 where it was not responding to a login request.  It just sat there working on the login request but would never respond with a success or failure.

After stopping the NCS software, it would not start again.  I was presented with a failure and told to check the launchout.log

So I performed the Backup-logs and untar'd the file.  Here is what I was receiving in the file.

Starting Health Monitor as a primary
Checking for Port 8082 availability... OK
truststore used is /opt/CSCOlumos/conf/truststore
truststore used is /opt/CSCOlumos/conf/truststore
Starting Health Montior Web Server...
Health Monitor Web Server Started.
Starting Health Monitor Server...
Health Monitor Server Started.
Starting Remoting Service: Reporting Server
Checking for running servers.
00:00 Check complete. No servers running.
Starting Server ... 
 Start failed. Initiating shutdown. Please check logs/Startup.log.

Powershell Module to Backup configs

One of my FAVORITE tools is powershell.  It's quick, easy to write and easy to reuse.
The esteemed Mr @BlakeKrone was asking for a script to backup configs off of a Cisco Switch.

To use this module, you have to run over and grab SharpSSH, a .Net implementation of JSCH.  I've used this in projects as far back as 2008.  It's a great library and very easy to implement in C#, VB.net or Powershell.  Just throw it in the same folder with the module and update the path in the script.

Howto convert a Cisco Sniffer AP capture for Metageek's EyePA

One of the limitations of EyePA is that it only supports a direct WLAN Packet capture through AirPCap or through another type of raw wireless packet capture.  Specifically, it needs to have the radiotap headers or the regular 802.11 headers.

Cisco uses the Airopeek encapsulation and to further complicate matters, the packet is encapsulated inside a UDP packet.  If you want to know how to configure a Lightweight AP as a Sniffer, here is a great guide:  https://supportforums.cisco.com/docs/DOC-19214

So now that we have our Cisco.pcap file, we can see that EyePA won't open the file:

After a few hours of research I stumbled onto a tool called AiroXtractor.   http://micky.ibh.net/~liske/airoxtractor/

Since this is a linux program, power up your BackTrack linux or whatever your favorite distro is.
You can download AiroXtractor with the following command:
wget http://micky.ibh.net/debian/pool/stable/main/airoxtractor/airoxtractor_0.1.tar.gz

Extract the files with:

tar xzvf ./airoxtractor_0.1.tar.gz

Run airoxtractor

./airoxtractor/airoxtractor --in=<pathtocapture>/Cisco.pcap --out=<pathtodestination>/EyePA.pcap

Let's look at our capture now:

Once the program finishes, you should now have a capable packet for EyePA. 

Just to be clear, I did not write this software.  I credit the original owner over at http://micky.ibh.net/~liske/airoxtractor/

I'd also like to throw a shout-out to the team at Metageek and specifically Trent.  It's their software that makes this all happen.

How to remove backup repositories in NCS

Ran into this during an NCS install today, and just wanted to pass along the request.  The request was to delete backup repositories, both from testing and from the original WCS to NCS migration.

The basic structure follows this:

configure
no repository <repository name>

This is case sensitive

Example:

test-NCS/testadmin# conf
Enter configuration commands, one per line.  End with CNTL/Z.
test-NCS/testadmin(config)# no repository wcs-ftp-repo
test-NCS/testadmin(config)# exit
test-NCS/testadmin#

While this is not in the configuration guide, Cisco did do a good job of making an intuitive CLI for the linux appliance.

The art of GRE troubleshooting with keepalives

Today I was presented with a couple interesting problems surrounding GRE tunnels.  Because I am no expert with GRE, it took a while in order for me to figure out what was wrong and how to fix it.

I've heard a lot of engineers say that keepalives will break a tunnel and that you shouldn't use them.  This is a half truth. Since GRE Tunnels are supposed to be stateless, they should never go down and from a routers perspective they are up unless you explicitly shut the interface for the tunnel.  But the tunnel can be down without the router knowing about it.  Keepalives are the router testing the tunnel to make sure that the tunnel is functioning both ways.

Here's how this works.  Router A takes a data packet and wraps it in a GRE header for Router B to send back.  Then it wraps that packet in another GRE header destined for Router B to look something like this:

Router A now sends this to Router B:

When Router B receives this packet, it removes the outer encapsulation and is left with a packet destined to Router A.  So it returns the packet to Router A:

When Router A receives the packet and removes the GRE header, it's left with the original keepalive packet and knows the tunnel is functional.  If the packet does not return, it knows the tunnel is down and will eventually take the state of the tunnel to DOWN after enough keepalives fail.

On a Cisco router, the keepalive command is pretty simple to implement.

RouterA# conf t
RouterA(config) interface tunnel 0
RouterA(config) keepalive

To see the status of the keepalives, you can do a "debug tunnel keepalive" and see the following output:

Dec 15 23:16:29.413: Tunnel19: sending keepalive, 10.0.0.2-> 10.0.0.3 (len=24 ttl=255), counter=1
Dec 15 23:16:39.413: Tunnel19: sending keepalive, 10.0.0.2-> 10.0.0.3 (len=24 ttl=255), counter=2
Dec 15 23:16:49.413: Tunnel19: sending keepalive, 10.0.0.2->10.0.0.3- (len=24 ttl=255), counter=3

As you can see, we're not seeing the return traffic.  In this case, the issue was a NAT problem where the GRE packets were being NAT'd to the outside instead of being put into the IPSec tunnel.  Running a packet capture on the firewall indicated that the packets were hitting the PAT Overload rule and not going into the VPN crypto map.  With this fixed we see the normal behavior.

Now we are seeing happy keepalives and the tunnel is back up.

Dec 15 23:21:52.425: Tunnel19: sending keepalive, 10.19.2.1->10.4.11.1 (len=24 ttl=255), counter=1
Dec 15 23:21:52.497: Tunnel19: keepalive received, 10.19.2.1->10.4.11.1 (len=24 ttl=252), resetting counter
Dec 15 23:22:02.425: Tunnel19: sending keepalive, 10.19.2.1->10.4.11.1 (len=24 ttl=255), counter=1
Dec 15 23:22:02.497: Tunnel19: keepalive received, 10.19.2.1->10.4.11.1 (len=24 ttl=252), resetting counter
Dec 15 23:22:12.425: Tunnel19: sending keepalive, 10.19.2.1->10.4.11.1 (len=24 ttl=255), counter=1
Dec 15 23:22:12.497: Tunnel19: keepalive received, 10.19.2.1->10.4.11.1 (len=24 ttl=252), resetting counteru all